Product News from the Core Team

Release Notes - 7.12 Development Release

Tetyana Loskutova — Tue, 20 Oct 2009 14:36:00 GMT

Hi! A new db4o version is out - 7.12 Development.

Here is what is in it:

OMN-178 - Reviewing OMN code. Changes made in code so that OMN works for java db

OMJ-149 - Include OMJ in 7.4 Production and Stable downloadable db4o distribution

COR-1793 - BlockingQueue may throw BlockingQueueStoppedException in case of spurious wakeups

COR-1789 - Introduce PerformanceCounters/JMX statistics for activation/deactivation, objects stored/deleted

COR-1787 - [.Net] Make sure PerformanceCounters instance are removed at ObjectContainer.Close()

COR-1781 - Blog about Silverlight assembly loading issue fixed

COR-1780 - Blog about Guid typehandler

COR-1778 - Introduce optional, user-specified name/label for databases

COR-1775 - Separate sorting/ordering step from SODA processing

COR-1773 - Enable defragment before actual migration in format migration tests.

COR-1768 - Base class classmetadata should not be take into account while storing derived ones

COR-1765 - JMX/perfmon: WeakReferences and FreespaceManager

COR-1761 - LINQ-Ordering on multiple properties throws a Db4oException: Unexpected exception: this..size()=1, other.size()=0

COR-1760 - ArrayList4 Generic IEnumerableGetEnumerator() should be implicity declared and the IEnumerable.GetEnumerator() explicitly declared

COR-1759 - [LINQ] Calls to String.Contains() produce wrong results

COR-1755 - Refactor deep prefetching client slot cache to use Cache4

COR-1752 - [.Net] Expose performance counter instances for each container

COR-1749 - JMX/perfmon: Networking statistics

COR-1748 - Investigate system assembly not being resolved in OMN

COR-1730 - create proper bundles with OSGi headers

COR-1678 - [Silverlight] Profile application to spot bottlenecks

COR-1646 - File lock must be released after emergency shutdown

COR-1637 - Prepare Silverlight tests for running in CC build

COR-1519 - Typehandler for System.Guid

COR-1062 - Sort result incorrect after OR

COR-377 - InMemoryObjectContainer creates a lot of short lived byte[]

Watching over db4o's shoulders the .NET way

Adriano Verona — Tue, 13 Oct 2009 11:40:00 GMT

Hi.

This post presents a .Net centric view of the newly introduced support for better runtime monitoring into db4o (as described in this post), so, before you continue reading please, go read the aforementioned post! I'll assume you've read it.

Have you read it yet?

No? Go there, I'll wait for you.

Ok. let's continue...

As you may already know for now, in the last few weeks, we have been busy adding more sophisticated monitoring support to db4o in the form of "Runtime Statistics" a term that we coined because we needed a common way to refer to "these things" in a platform agnostic way (you know, Java developers would not be happy if we just used "Performance Counters" and .Net developers would not be happy either had we choose MBean :); in this post I'm going to use the more natural name in the platform: Performance Counters. If, for some reason, you are not familiar with this concept or want to refresh your knowledge on the subject we recommend to read the MSDN documentation here and here and also this article on codeproject.

After reading the companion post you should already have a good idea about what monitoring data db4o allows you to collect and how it can be useful but we intentionally omitted a description of how to configure and consume this information which is the goal of this post.

The first step to be able consume monitoring information is to let windows know about the counters which means that you either run Db4oTool with the install-performance-counters command line parameter or call the method Db4oPerformanceCounters.Install() from your code. Note that this step is required only once per machine running db4o and also that it requires administrative rights so you probably will perform it at application install time (as opposed to runtime).

With the counters in place, the next step is to configure db4o to start collecting the data. As we explained in the companion post, the process of collecting such information does pose some overhead so we adopted an opt in model grouping counters in the following categories (each category in bold configures db4o to collect statistics for the respective set of counters):

Queries
- General
- Native Queries / LINQ
FreeSpaceManager
Reference System
IO
Networking (x Connected Clients)

In order to select the category/categories you are interested in, add an instance of the respective xxxxxMonitoringSupport() class to db4o configuration as in the following sample:

IEmbeddedConfiguration config = Db4oEmbedded.NewConfiguration();

// Query
config.Common.Add(new QueryMonitoringSupport());
config.Common.Add(new NativeQueryMonitoringSupport());

// Networking
config.Common.Add(new NetworkingMonitoringSupport());

// IO
config.Common.Add(new IOMonitoringSupport());

// Reference System:
config.Common.Add(new ReferenceSystemMonitoringSupport());

// FreeSpaceManager
config.Common.Add(new FreespaceMonitoringSupport());

Thats it!

Once configured no other action is required to start collecting the information (other than exercising the pieces of code that triggers the interesting counters :)

Next step is to consume the information being collected which can be done either through PerfMon (the main tool in Windows designed to view such counters) or programmatically. We are going to show you how to see the counters using PerfMon first, so start PerfMon.exe and you should see its main window (since I'm running Windows 7, your actual PerfMon window may look a little bit different than mine).

Next step, press + button in the toolbar to select which counters you want to monitor;

In the window that shows up (in Available Counters) locate and expand Db4o and then select the desired counters. Next, select the instances you want to monitor.

Each opened db4o database (or client) will be represented as an instance in the list "Instances of selected object". Local and server databases will display the database file name (in the sample above mydb.db4o) and clients (in C/S mode) will be shown as local port => server ip:port). You may monitor as many instances as you want simultaneously.

That's it again ;)

Now that you already know how to monitor db4o performance counters it is time to learn how to consume them programmatically since this can be useful in some scenarios. Accessing db4o performance counters programmatically is just a matter of calling the Db4objects.Db4o.Monitoring.Db4oPerformanceCounters.CounterFor() method passing the counter and the object container you want to monitor as in the following sample:

PerformanceCounter bytesReadPerSec;
bytesReadPerSec = Db4oPerformanceCounters.CounterFor(PerformanceCounterSpec.QueriesPerSec, db);

Now you can query PerformanceCounter's reference for any of it's properties / methods.

Have fun (and let us know what you think could be improved in this subject!)

Db4o Team!

Watching over db4o's shoulders the Java way

Carl Rosenberger — Tue, 13 Oct 2009 11:37:00 GMT

As described in this posting, db4o 7.12 comes with new monitoring support to collect runtime statistics about performance relevant behaviour and state.

On the Java platform, statistics are published through Java Management Extensions (JMX). A variety of JMX clients is available. Many of them are linked from here.

In this posting we will describe how to use the rather basic jconsole tool that comes with the Java JDK.

For the purpose of providing sample readings we have written a simple program that exercises various db4o tasks in a long running process.

It can be found in the 7.12 download in
/src/db4oj.tests/src/com/db4o/db4ounit/optional/monitoring/samples/MonitoringDemo.java

You also need
/src/db4oj.tests/src/com/db4o/db4ounit/optional/monitoring/samples/AllMonitoringSupport.java
that bulk registers all the available monitoring ConfigurationItems.

To view the generated statistics, run the MonitoringDemo application, then start jconsole from the command line.

A general tutorial on how to use jconsole is provided here.

First you need to connect to the MonitoringDemo process. Then you will find the db4o related readings on the "MBeans" tab. In the treeview on the left, you will see "com.db4o.monitoring".

Below this tree node you will find all open ObjectContainers that have JMX monitoring enabled. For each of them the respective runtime statistics categories will be listed. If you select the "Attributes" node for any of them, the corresponding set of attribute values will be shown along with their most recent reading.

By doubleclicking on the number value a graph will be displayed. (Admittedly not a very intuitive user interface, but that's jconsole.)

Here is a screenshot how results for IO readings could look like:

For some "exceptional" events, we also provide the option to subscribe to notifications.

In the categories where the "Notifications" node is visible, you can select it and click "Subscribe" on the bottom right. For example we provide notifications about unoptimized native queries and about class index scans.

If you consider looking into other tools than jconsole to monitor JMX statistics, we recommend to take a look at jManage. Here is a sample screenshot of how jManage nicely allows to combine multiple readings into one graph:

If you want to test whether your mail client gracefully handles full mailboxes: jManage also allows to send an email for every JMX notification received.

Enjoy the new functionality!

Your db4o team.

db4o runtime statistics

Adriano Verona — Mon, 12 Oct 2009 17:31:00 GMT

Once a project hits production status and gets deployed it is in the wild (at least from the point of view of the development team) and it requires constant monitoring from IT. The more important the process, the more important becomes the ability to monitor the application behavior so IT members can take proactive actions and avoid potential problems.

Also, monitoring support can be a life/time saver during development, allowing developers to spot performance issues, wrong assumptions and the like before the application reaches the magical production status. Traditional enterprise applications (such RDBMS, application servers, EJB containers, etc) has been delivering such monitoring capabilities either through well established platform mechanisms (MBeans, Performance Counters) or custom approaches.

While db4o provides API hooks to add custom logging for performance relevant figures, until now there's been no out-of-the-box way to have an ad hoc look under the hood before getting into real coding, or a convenient, standardized mechanism to monitor and display these values for a long-running process over a period of time.

We are proud to announce that starting with version 7.12 various "Runtime Statistics" will be exposed by db4o; we do believe that such statistics can provide useful information for developers and/or IT administrators who can consume this information either through the platform infrastructure (you can learn more details about .Net and Java specifics) or an API.

As of version 7.12 the following statistics will be made available:

Queries
- # class scans / sec
  A class scan means that db4o will need to read each object of a given class in order to figure out whether the object should be included in the
  result set of a query. In this case indexing the parent referencing field would avoid this costly operation and improve query times.
- # queries / sec
  Number of SODA queries executed per second.
- Average query execution time
  The time to retrieve the list of ids for the matching objects (this doesn't take the time to activate the objects into account)
- # unoptimized native queries / sec
  Number of native queries (per second) that were not optimized. Developers should strive to run only optimized native queries in order to
  get the best possible performance.
- # native queries / sec
  Total number of native queries (either optimized or not) executed per second.
FreeSpaceManager
- average slot size
  The average size of free storage slots available.
  Smaller slots are less likely to be reused.
- # free slots
  A high number of free slots indicates a high level of fragmentation.
- # reused slots / sec
  Number of reused free slots per second.
- # bytes in free space management
  A high value (with a high value of #free slots) is a good indication that defragmenting the database may bring performance improvements.
Reference System
- # objects in reference system
  Total number of objects in the db4o reference system.
IO
- # bytes read / sec
- # bytes written / sec
- # reads / sec
- # writes / sec
- # syncs / sec
Networking
- Bytes Sent/Received
  Bytes sent/received from/to clients.
- Connected Clients
  Number of currently connected clients.
- Messages sent
  Number of messages sent from/to db4o clients and servers. Developers can use this statistic as an indicator whenever their object models are
  driving db4o to exchange too many messages between clients / server. This statistic is also useful to help developers to tweak
  prefetchDepth/ prefetchObjectCount configuration in order to minimize messages exchange between client / servers.
.Net Specific (For more details about .Net specifics follow the link in the end of this post)

Note that since gathering these statistics does introduce some performance overhead (for most of then this overhead is so small that it is immeasurable) each set of runtime statistics (Queries, FreeSpaceManager, ReferenceSystem, etc.) must be configured independently (so you can decide which ones you want to track and pay the performance hit only for that ones).

One selects the category(ies) he is interested in publishing statistics for by adding an object of the respective xxxxxMonitoringSupport() class to db4o configuration as in the following sample:

IEmbeddedConfiguration config = Db4oEmbedded.NewConfiguration();

// Query
config.Common.Add(new QueryMonitoringSupport());
config.Common.Add(new NativeQueryMonitoringSupport());

// Networking
config.Common.Add(new NetworkingMonitoringSupport());

// IO
config.Common.Add(new IOMonitoringSupport());

// Reference System:
config.Common.Add(new ReferenceSystemMonitoringSupport());

// FreeSpaceManager
config.Common.Add(new FreespaceMonitoringSupport());

Last, but not least, these statistics are per object container (database file) so you can monitor as many open databases as you want.

If you can't wait for this functionality we invite you to go and download db4o continuous build version and give it a try, but please remember that it's a work in progress for while, so you may expect to see changes in the API and / or list of actual statistics supported.

For more details on how to consume these statistics, samples and more details, please, read the following blog posts:

For .Net see Watching over db4o's shoulders the .Net way.
For Java see Watching over db4o's shoulders the Java way.

Have fun! (and provide feedback ;)

Db4o team!

New typehandler on the block: System.Guid

Adriano Verona — Tue, 06 Oct 2009 18:51:00 GMT

Hi!

If db4o cannot find a more specific typehandler for a type it will fall back to the StandardReferenceTypeHandler. That means the type will be handled as a complex object.

Types handled by the StandardReferenceTypeHandler carry some interesting properties:

Db4o will pretend that objects of this type have an identity

Indexing happens at id level

A class index entry will be created for every stored object

These three points have some interesting implications for value types on the .Net platform. By definition, Value Types have no identity so it makes no sense to db4o to keep an id for such types. Also, db4o handles types with no identity by embedding marshalled instances into their parent's slots, which in most cases will induce less overhead (no need to keep track of ids), potentially resulting in better performance.

Since we didn't have a specific typehandler for System.Guid this is the status quo for this type in db4o in versions prior to 7.12. As we continue to improve db4o, we strive to make its usage as natural and its performance as high as possible on each platform it's designed to run on. We are proud to announce that based on the contribution from Judah Himango, a community member, we are introducing a new typehandler for the System.Guid, so starting with db4o 7.12, System.Guid has got it's own (deserved) type handler with better indexing support, no more artificial ids and less overhead.

You may be wondering what will happen to old databases when they get updated to version 7.12. As always the database format update should be transparent and require no effort from developers; as objects holding guids get updated they will use the new typehandler.

You can use Db4oTestRunner with GuidTypeHandlerTester to see it with your own eyes (you can even try with different db4o versions ;)

Alternativally you can grab Db4oTestRunner / GuidTypeHandlerTest sources from our SVN.

Have fun!

Your db4o team.

Rocks are still rolling on db4o Silverlight port

Adriano Verona — Tue, 06 Oct 2009 17:09:00 GMT

Some time ago we discussed the status of the db4o Silverlight port effort at that point in time.

Well, we are happy to let you know that since then we have been working on the issues presented and that we have made slow but steady progress towards a better developer experience.

Basically we have improvements in three areas:

- Tests ported to Silverlight

- continuous build integration

- assembly loading issues

When we announced that we were working on a db4o Silverlight port we had just a few tests of our test suite running and, worst, the Silverlight build was not integrated into our continuous build loop.

As of today we have fixed, to some extent, both issues; the major bulk of our test suite is already running under Silverlight (with the ones that don't make sense disabled or removed) and better, we are currently building and running the tests in our continuous build process! This ensures that we are not going to introduce regressions on the Silverlight port! The goal is to continue adding tests and eventually start to write specific tests for Silverlight.

One of the most frustrating issues I've found while working on the db4o Silverlight port was related to assembly loading; for some reason we were misled to believe that we could not load assemblies in a Silverlight application and that led us to introduce a workaround that required users to write more code in order to be able to use db4o under Silverlight. Thanks to a Silverlight program manager and some extra research we figured out that actually it is possible to load assemblies and so we fixed our code rendering the proposed workaround unnecessary (the test application has been updated to reflect this change). For those who read the previous post, that means that you don't need to worry about that scary, red warning message anymore :).

In other words, there's no more need for the following code:

   #region Assembly.Load() issue workaround

        private static readonly IDictionary<string, Assembly> _assemblyCache = new Dictionary<string, Assembly>();
        static App()
        {
            _assemblyCache[AssemblyNameFor(typeof(Person))] = typeof(Person).Assembly;
            _assemblyCache[AssemblyNameFor(typeof(Db4oFactory))] = typeof(Db4oFactory).Assembly;

            TypeReference.AssemblyResolve += (sender, args) => args.Assembly = _assemblyCache[args.Name];
        }

        #endregion

Actually if you try to run a Silverlight application that was written with this workaround you'll notice that it won't compile because the Event AssemblyResolve on the TypeReference class is no more; if you find yourself with such an error, just remove the offending line and related code!

Cool!

The message? We are still committed to provide a solid solution for object persistence on the Silverlight platform!

As always, feel free to comment.

Best

Your db4o team.

Networking C/S preloaded

Patrick Roemer — Wed, 09 Sep 2009 14:19:00 GMT

In a networking client/server environment, most of the cost is incurred by lots of small messages being interchanged between client and server. The classic cure is to apply some more or less sophisticated heuristics to batch or preload data, thus reducing the number of round trips in exchange for a bigger message size. Previous db4o versions already implemented some variants of this general approach, most notably:

ID prefetching: Clients have to request IDs for newly created objects from the server. Instead of requesting a single ID on demand, clients will rather acquire a full range of IDs to be used at their own discretion.
Message batching: Whenever it is not required that a message reaches the server asap, it will be queued on the client side and be marshalled together with other accumulated messages when either the queue threshold is exceeded or an "urgent" messages (like a commit) enters the queue.
Object prefetching: When the client receives a query result, it will only get the collection of IDs. Instead of requesting each single object on access, the client would prefetch a configured number of root objects from the result set upon first access and cache it internally.

As a next step, we have extended this tool box with deep prefetching, introducing general changes to object prefetching. "Root" objects in the above description means: Only the actual object (and its primitive members) is retrieved. However, it will be very likely in most scenarios that "child" objects will also be accessed. In order to improve round trip times in these scenarios, we introduced a prefetch depth strategy for query results that will push the object graph below the prefetched "root" objects up to a preconfigured depth to the client along with the ID set. You can think of object prefetching as "horizontal" pre-loading, while deep prefetching covers the "vertical" axis.

Deviating from the existing object prefetching implementation that actually would transfer and cache "live", activated objects on the client (that may never be needed there), we transfer and cache the file level representation (i.e. byte arrays) only and let the client handle them just as it would actual file slots. Object prefetching has been merged into this deep prefetching mechanism, it uses the same approach now.

The following graph illustrates the effect of prefetch count and prefetch depth on a hypothetical result set. Prefetched objects/slots are marked green.

Configuring deep prefetching is as simple as invoking

ClientConfiguration clientConfig = Db4oClientServer.newClientConfiguration();
// ...
clientConfig.prefetchDepth(prefetchDepth);

The default setting is 0, i.e. the feature is switched off. It can be re-configured arbitrarily at runtime without reopening the client.

Release Notes - 7.11 Development Release

Tetyana Loskutova — Tue, 18 Aug 2009 14:58:00 GMT

New release is out and here is the list of bug fixes and improvements that you will find in it:

OMN-177 - OMNAddin.addin file location detection fails on non english language

OMN-176 - ObjectManagerEnterprise fails to run on non english boxes

OMN-175 - In Query result, Empty any cell displaying some primitive value. It throws NullReference Exception

OMN-174 - exception occurs on dragging some fileds to attribute list and then running a query or click on view all objects.

OMN-173 - Queries not returning correct results for datetime field.

OMN-172 - For Bool and datetime fields, query builder does not show dropdown for bool values and calender for datetime

OMN-171 - If such a search string is added which has no matching class and we switch to assembly view then the db4o broweser has treeview with just assembly name

OMN-169 - Recent queries remains for a db, if the db is deleted and created again with same name and location

OMJ-144 - Error by running a query

OMJ-142 - Query string doesn't appear in text field of Build query view

OMJ-128 - Unhandled event loop exception

DRS-103 - Collection not replicating properly if configured with TransparentActivationSupport

COR-1746 - HybridDictionary can not be stored

COR-1744 - Native Query optimizer accepts multi-statement predicates

COR-1743 - ObjectContainer in member field causes NullPointerException during activation

COR-1741 - Enable ObjectContainer#Store to be used inside Deleting callback (and event handler)

COR-1739 - Db4oTool should support db4o performance counters installation

COR-1736 - JMX/PerfMon: Query statistics

COR-1735 - TransportObjectContainer should not create a WeakReferenceCollector

COR-1731 - Skip nested activation calls in NQ optimization

COR-1729 - update svnkit to 1.3

COR-1728 - PerformanceCounter/JMX bean for IO Read and Write for Java and .NET

COR-1725 - BLOAT cache fills up the heap

COR-1723 - ClassHasNoFields diagnistic is not called on a class with no fields

COR-1721 - After refactoring a field to transient/NonSerialized, field contents are still retrieved

COR-1718 - DB4OTool.exe crashes on some class names

COR-1715 - When #setSemaphore() waits for the semaphore to become available it blocks the server.

COR-1714 - Batched messages hold on to the server container lock while they are processed

COR-1712 - Reduce TransportObjectContainer code overhead

COR-1707 - Consistent client/server exception (and shutdown) handling

COR-1706 - Per class UUID config is overridden by global config

COR-1703 - UniqueFieldValueConstraint should refuse to be configured on a client

COR-1701 - com.db4o.foundation.ArgumentNullException about ActivatableHashMap

COR-1700 - Allocate memory in the MemoryBin in byte[] pages

COR-1699 - Consolidate type handler querying requirements into QueryableTypeHandler

COR-1687 - Removing super class from hierarchy leads to ClassCastException

COR-1685 - Deleting updated objects with TP configured has no effect

COR-1683 - Updating objects that contain enums causes exceptions

COR-1682 - DatabaseFileLockedException is no longer thrown when the file is locked by another process

COR-1681 - [Silverlight] Introduce helper class for assembly resolution.

COR-1677 - Simple boolean fields not supported

COR-1676 - [Silverlight] Add CLI1 tests (adapting the ones not supported) to Silverlight test suite

COR-1675 - review and update astoria demo

COR-1672 - integrate Data.Services.Db4oDataContext tests into the continous build

COR-1671 - test cases for Data.Services.Db4oDataContext in server mode with concurrent clients

COR-1670 - Activation doesn't work properly when UniqueFieldValueConstraint is set

COR-1669 - TA enhancement plugin for eclipse fails when the super class of a class that's being enhanced resides in a referenced project

COR-1667 - Support simple method calls on LINQ queries

COR-1666 - Support method/property chaining on LINQ queries

COR-1665 - support ordering for value types that implement IComparable

COR-1664 - NullPointerException encountered in client server mode when multiple client connect to server simultaneously using Configuration along with openclient

COR-1662 - On upgrading database to 7.10 NullReferenceException is thrown

COR-1661 - Defrag fails with NullReferenceException

COR-1656 - Support non generic types derived from generic collection types

COR-1652 - DateTimeOffset field in ORDER By throws exception

COR-1651 - LINQ 'Where Not' clause against boolean field throws exception when native query optimization is enabled

COR-1649 - split embedded C/S from networking C/S and introduce API to open embedded clients against local ObjectContainer

COR-1648 - During emergency shutdown, db4o should not write cached data to disk if cache could possibly be corrupted through causing exception

COR-1647 - Exceptions during emergency shutdown should not shadow the causing exception

COR-1640 - EventRegistry cleanup

COR-1635 - Run db4o common tests on Silverlight

COR-1632 - Deep Prefetching

COR-1134 - Poleposition alerts when the performance decreases or increases by a specific threshold

COR-1054 - Ensure Enums are handled correctly by Java TA instrumentation

COR-1046 - updateDepth sets on configuration overrides updateDepth on ObjectClass

COR-656 - SODA throws exception on identity constraint against "unknown" object

COR-391 - DateTime in dotnet looses "kind" member

COR-192 - Pluggable ClientServer Communication

Improved Exception Handling

Carl Rosenberger — Fri, 14 Aug 2009 15:33:00 GMT

In the latest build we have introduced an even stricter Exception handling. The new strategy: If the exception occured is a Db4oRecoverableException we rethrow this exception to the user application. In all other cases we do an emergency shutdown of the ObjectContainer/ObjectServer without commit, without even writing FreeSpace information. This way we try to do our best to keep the database in the last committed state.

The new behaviour is quite different to the previous behaviour where we did not catch VM errors. In Client/Server mode an OutOfMemoryError for a specific client would have only terminated the handling thread for the client that it serviced. The server would have continued to work. For this case there was a small risk that BTrees or cache pages ended up in an inconsistent state.

The change of the Exception handling is also relevant for single user mode: If db4o would have been accessed by multiple threads, the one thread that experienced an OutOfMemoryException would have died. Other threads with references to the same ObjectContainer may have worked on, also against a possibly inconsistent state.

Now we make sure the ObjectContainer becomes unusable as soon as any unexpected Exception occurs.

We have also tried to improve against the possible risk of shadowed exceptions: In case exceptions occur during the emergency shutdown, we wrap all occurred exceptions in a CompositeDb4oException to make sure no exception information gets lost.

For callbacks and events that call into user code we wrap possible exceptions into Db4oRecoverableException. That way we try to ensure that exceptions that occur outside of the db4o core do not shut down ObjectContainers. For these cases it is up to your application to decide what to do.

We plan to continue improving local exception handling to catch even more cases where it is safe to keep an ObjectContainer alive.

As always we rely on your feedback. Please tell us how you like the new Exception handling and how it works for you.

Memories - Running db4o in-memory revisited

Patrick Roemer — Thu, 13 Aug 2009 12:23:00 GMT

Running a db4o database completely in main memory certainly is not the most ubiquitous usage. Nevertheless there are quite some users out there who are using db4o that way - caching is one possible use case where this option may be helpful. db4o has supported in-memory use from the very beginning, but it has rather been considered a step child feature. For historical reasons, there have been two redundant, although semantically slightly different ways to start db4o in-memory: As an InMemoryObjectContainer (the legacy variant), or as a vanilla container configured to use MemoryStorage (the "proper" variant). Both options would not fully support all the features the file-based scenario would offer. Here's a brief summary of the current status of db4o in-memory usage.

There can only be one

We eventually decided to deprecate the InMemoryObjectContainer variant. Although code using InMemoryObjectContainer will remain functional, we strongly encourage users to switch to PagingMemoryStorage.

Reduced memory consumption

Our original MemoryStorage (as well as the MemoryFile used by the InMemoryObjectContainer) would keep the raw database content in one single byte array that would grow when its capacity was exceeded - "growing" here means that a new, larger array would be allocated and the old content would be copied over, temporarily doubling memory consumption. Furthermore, huge objects are generally problematic with regards to heap compaction and specifically troublesome for the .net runtime which allocates objects requiring 85000 bytes or more in what's called the Large Object Heap that is never defragmented. We recommend to use PagingMemoryStorage instead, which keeps the "file" in a list of small byte arrays, eliminating the need for copying and playing nicer with the heap overall. Configuration is as simple as:

config.file().storage(new PagingMemoryStorage(pageSize));

To better visualize the difference we have monitored a simple application that keeps on adding simple objects with a 60K byte array payload to an in-memory database. In the following graph we can see the behavior of both implementations with regards to overall memory consumption and Large Object Heap saturation in particular:

For MemoryStorage, Large Object Heap accounts for the bulk of total memory usage and as a result the application quickly crashes with an OutOfMemoryException around 80K objects with half of the physical memory still available. With PagingMemoryStorage on the other hand, available memory can be fully utilized and the application just gets slower as garbage collection takes more and more time.

Defragment and Backup

Earlier this year we already made sure that defragmenting and online backup blends in well with in-memory usage. In order to provide more flexible support for these features, we extended the Storage interface to become kind of a minimalist file system rather than a mere Bin factory. As a result, you can specify the backup storage as well as the temporary storage used during defragmentation - e.g. to create a file based backup of your in-memory db:

db.backup(new FileStorage(), "backup.db4o");

Wrapping Up

In memory usage of db4o is now more closely integrated with the storage API which removes redundancies while bringing better support for administrative features such as defragment and backup. The new PagingMemoryStorage implementation significantly reduces memory footprint.

LINQ improvements.

Adriano Verona — Thu, 18 Jun 2009 12:27:00 GMT

Hi,

I want to let you know that during the last few days we have made some enhancements and fixed some bugs / issues related to our LINQ implementation.

Some of them were just annoyances that forced developers to use "not so idiomatic" styles as workarounds, some of them either resulted in crashes or wrong results being returned or even bad performance. Just in case you are curious, here are the tickets to the related issues.

COR-1651(bug): LINQ 'Where Not' clause against boolean field throws exception when native query optimization is enabled

COR-1652 (bug): DateTimeOffset field in ORDER By throws exception

COR-1666 (enhancement) : Support method/property chaining on LINQ queries

COR-1667 (enhancement) : Support simple method calls on LINQ queries

I'll use the class definition bellow to discuss these issues (Assume we have a bunch of objects of this class stored in the database):

class Person
{
public string Name { get; set; }
public bool IsMale { get; set; }
public Guid Id { get; set; }
public Person Parent { get; set; }

      public int GetName()
      {
          return Name;
      }

public Person()
     {
          Id = Guid.NewGuid();
     }
}

The first bug (not operator on simple boolean expressions) only manifests itself in its most basic form:

from Person p in db
where !p.IsMale
select p;

The result was a crash with a InvalidOperationException inside Stack<T> class. The next issue was also related to simple boolean fields without a explicit constraint:

from Person p in db
where p.IsMale // no == true here.
select p;

but instead of throwing an exception this version would silently ignore the where clause effectively querying for all Person class objects as if the following query had been executed:

from Person p in db
select p;

(note the where clause absence)

The good news is that in latest db4o version they work as expected (revision > 13278). If you still needs to use this kind of expressions on previous versions, the workaround is to use the more verbose form:

from Person p in db
where p.IsMale == false // instead of where !p.IsMale
selec p;

and

from Person p in db
where p.IsMale == true // instead of where p.IsMale
select p;

The next two issues were related to our Linq -> Soda translator. In both cases we (wrongly) concluded that the expressions were not convertible to Soda falling back to LINQ for Objects. You would still get the right results but the performance could be seriously affected depending on the actual expression. The following sample shows the failing constructs:

from Person p in db
where p.GetName() == "John Doe"
selet p;

from Perosn p in db
where p.Parent.Name == "John Doe"
select p;

In the first one, GetName() just returns the contents of a property (or a field). The second example just happened to have two indirections (p.Parent.Name). In both cases we used to fail to recognize these patterns as "SODA compliant" and felt back to LINQ for Objects. So, for instance, in the second sample, instead of getting something like:

IQuery q = db.Query();
q.Constrain(typeof(Person));
q.Descend("_parentBackingField").Descend("_nameBackingField").Constrain("John Doe");

foreach(var p in q.Execute())
{
}

we would get something like:

IQuery q = db.Query();
q.Constrain(typeof(Person));

foreach(var p in db.Execute().Where(candidate => candidate.Parent.Name == "John Doe"))
{
}

which is less efficient since all objects of type Person will be retrieved.

The last issue was not directly related to LINQ; instead it was a Soda bug regarding ordering based on value types that implements IComparable / IComparable<T>.

So queries like (note that Id is a Guid, which implements these interfaces):

from Person p in db
orderby p.Id
select p;

would result in an exception like:

Db4objects.Db4o.Internal.IllegalComparisonException: Exception of type 'Db4objects.Db4o.Internal.IllegalComparisonException' was thrown.
at Db4objects.Db4o.Internal.Handlers.StandardReferenceTypeHandler.PrepareComparison(IContext context, Object source)

It was fixed by making StandardReferenceTypeHandler aware of these interfaces.

Well, the good notice (as you already know :) is that all of them were fixed in revision 13278, so if you faced any of these issues you may wish to check it out again :)

As usual, you are invited to download a console application (attached to this post) that accepts a base folder as its argument and runs some tests against all db4o versions found as subdirectories under this base folder.

We can identify all the issues discussed before in the following screen shot (run against version 7.10.96)

And confirm that they were gone on version 7.10.99!

I could not finish this post without saying that three of these issues were reported by community members. To you, our sincere thanks you!

Best.

Adriano

Give your objects a REST (interface)

Rodrigo B. de Oliveira — Wed, 17 Jun 2009 16:35:00 GMT

ADO.NET Data Services Framework [1], codename Astoria, a framework released with the Service Pack 1 of the .Net framework 3.5 makes it very simple to create data services that are accessible by HTTP.

Data services can be created from any data source that implements the IQueryable<> interface (such as db4o). For updates to work however the new IUpdatable interface must also be implemented. Db4o 7.10 ships with the new Db4objects.Db4o.Data.Services assembly to provide just such an implementation.

Let's go for a step by step example on using it.

Here's a simple object model that we want to make accessible by HTTP:

        public class Hacker
        {
            public string Name { get; set; }
            public Team Team { get; set; }
        }
        
        public class Team
        {
            public string Name { get; set; }
            public List<Hacker> Hackers { get; set; }
        }

The first step is to decorate the public types with the DataServiceKey attribute as required by Astoria:

        [DataServiceKey("Name")]
        public class Hacker
        {
            public string Name { get; set; }
            public Team Team { get; set; }
        }
        
        [DataServiceKey("Name")]
        public class Team
        {
            public string Name { get; set; }
            public List<Hacker> Hackers { get; set; }
        }

The next step is to create a «Data Context». A «Data Context» is a type that binds Astoria to a specific data source and exposes IQueryable<> entry points into the data model.

For a db4o data source we need to subclass Db4objects.Db4o.Data.Services.Db4oDataContext:

        public class TeamDataContext : Db4oDataContext
        {
            public IQueryable<Hacker> Hackers
            {
                get { return Container.AsQueryable<Hacker>(); }
            }
    
            public IQueryable<Team> Teams
            {
                get { return Container.AsQueryable<Team>(); }
            }
    
            protected override IObjectContainer OpenSession()
            {
                // ...
            }
            
            // ...
        }

TeamDataContext exposes the two entry points to the object model as IQueryable<> properties: Hackers and Teams.

It also needs to provide an IObjectContainer to service each request, a good strategy is to keep an IEmbeddedObjectContainer open throughout and just delegate the OpenSession invocation:

            protected override IObjectContainer OpenSession()
            {
                return EmbeddedObjectContainer.OpenSession();
            }
    
            public static IEmbeddedObjectContainer EmbeddedObjectContainer;

As a last step we have to write an ADO.NET Data Service and bind it to our custom data context:

            public class TeamDataService : DataService<TeamDataContext>
            {
                public static void InitializeService(IDataServiceConfiguration config)
                {
                    config.SetEntitySetAccessRule("*", EntitySetRights.All);
                }
            }

And that's all that is to the server part. After we have the service running (either as part of a web application or standalone as in the provided sample) Visual Studio can be used to create a service proxy which makes remotely consuming the data as convenient as:

        var context = new TeamDataContext(new Uri("http://uri.to/service.svc"));
        var hacker = (from hacker in context.Hackers where hacker.Name == "Rodrigo").Single();

You can find the complete sample code in the attachments.

Have fun!

[1] http://en.wikipedia.org/wiki/ADO.NET_Data_Services

db4o goes Queryable

Rodrigo B. de Oliveira — Wed, 17 Jun 2009 11:55:00 GMT

The LINQ framework design guidelines [1] describes the "... three means by which a type can be designed to participate in LINQ queries: implementing IEnumerable<> ..., implementing IQueryable<>, or by defining the Query Pattern on the type".

For db4o we chose the Query Pattern implementation because only a subset of the available LINQ operators could be reasonably supported. By implementing the pattern for the operators we support any code that references an unsupported operator automatically falls back to LINQ to Objects.

Yes, implementing IQueryable<> [2] used to be that slightly more complicated approach recommended when most LINQ operators were supported by the provider. Things have changed and IQueryable<> is now being used by APIs everywhere as the lowest common denominator for optimizing LINQ providers.

Which is why we decided to make LINQ to db4o support the IQueryable interface as well.

Just as with LINQ to db4o, it's very easy to use and completely integrated:

        using Db4objects.Db4o.Linq;
        
        // ...
        
        IObjectContainer container = Db4oEmbedded.OpenFile("HR.db4o");
        IQueryable<Person> query = container.AsQueryable<Person>();

Et voila, you can pass `query` around as an IQueryable/IQueryable<Person> reference to every API which is dynamically manipulating a LINQ query using the IQueryable interface or the Queryable extension methods.

Shall a scalar method be called on this IQueryable or shall it be iterated over, the underlying implementation will rewrite the query expression tree into an optimized LINQ to db4o query. For the unsupported operators the query would still fallback to LINQ to Objects, meaning that all Queryable methods can be used without distinction.

We'll see in a next post how we've leveraged our IQueryable implementation to support an API which actually demands it.

[1] http://blogs.msdn.com/mirceat/archive/2008/03/13/linq-framework-design-guidelines.aspx
[2] http://msdn.microsoft.com/en-us/library/system.linq.iqueryable.aspx

Note: if you want to check out a demos that use these new features see this page

        using Db4objects.Db4o.Linq;
        
        // ...
        
        IObjectContainer container = Db4oEmbedded.OpenFile("HR.db4o");
        IQueryable<Person> query = container.AsQueryable<Person>();

In the search of silver

Adriano Verona — Wed, 10 Jun 2009 12:00:00 GMT

Hi!

If you develop for .Net (and even if you don't :) the odds are high that you've already heard about Microsoft Silverlight, a platform for developing RIA style applications.

In this post I'd like to share some of our experiences while working on adding support for this platform to db4o, but before we move on, I'd like to stress some points:

The goal of this post is to describe the way we plan to add Silverlight support but we cant guarantee that we'll be able to stick to these plans. Also, we assume that readers have a good understanding of the related technologies (db4o, .Net, Silverlight)
Even though we're committed to this goal, as of today, Silverlight is not an officially supported platform.
All decisions are being taken based on the assumption that db4o most common usage scenario in this platform will be to store objects locally
(RIA) hence we're focusing on embedded mode. Of course, we are a community driven project so this can change in the future.
Last but not least, current code assumes Silverlight Beta 3 (it might work on Silverlight 2 but that hasn't been checked).

That said, let's take a look on how the porting process looks like (strikethrough activities have been completed already):

~~Feature set / limitations and possible fixes~~
~~Understand the required changes / Port the selected projects~~
~~Update/extend development tools like Sharpen~~
Port the tests
Update documentation
Write samples
Integrate into the build
Publish Silverlight artifacts to web site

Feature set / Limitations

The first step was relatively simple: given the tight constrains imposed by Silverlight we just picked a doable set of features. For the initial support we decided

to concentrate our efforts in embedded mode, leaving some features out.

Also, while working on porting db4o to run on Silverlight we assembled a list of limitations and possible improvements:

By default Silverlight apps have a quota of 1.0 Mb of disk space. Using the right APIs it's possible to ask for more (through a user consent dialog). It would be nice if db4o could use this API to increase its quota once it detects that it is running out of space.
Only public fields are allowed (AFAIK, Silverlight doesn't support reflection over private fields). That means that fields need to be declared as public to be "storable".
db4o requires some additional configuration (this is the most severe limitation IMHO).
Client / Server mode: Silverlight doesn't allow direct socket connections so, at least with db4o's current implementation, it's not possible to support CS mode. We could make the communication channel pluggable and, maybe, use "REST" instead of TCP/IP but this would require more investigation.
[edited on Jun / 23] Some days ago, Ashish Shetty, a kind program manager at Silverlight team just alerted me that this sentence may be misleading and that actually Silverlight support client sockets (with some port restrictions). The main point is that, at least for now, we'll not pursue adding client / server support on Silverlight platform [/edited]
Native queries / Linq: Since both technologies rely on Mono.Cecil and friends and these assemblies were not ported to Silverlight it's not possible to support them for now (anyway, we saw some good news regarding Mono.Cecil in this forum thread).

We are planing to investigate whether the Db4oTool can be used to instrument Silverlight assemblies and optimize native queries at compile time. If we manage to instrument Silverlight assemblies Db4oTool could be used to turn storable types into "self reflectors" through interface injection.

Understanding required changes

Understanding the changes required to successfully compile for Silverlight consisted mainly in getting the list of unsupported interfaces/classes/methods and adapting the code accordingly. The biggest changes were due to the lack of some I/O classes as well non generic collections, restrictions on assembly loading and a tighter security model. These constraints were the cause of most of the compilation errors we found.

For the I/O classes db4o is using an implementation from Michael Sync (which, by the way, was really helpful in the porting process and is the kind developer willing to spend time and help; hey guy, thank you very much!). Technically speaking, Michael wrote a db4o storage based on .Net IsolatedStorage model.

The lack of non generic classes was trickier to solve than we expected; since most of db4o C# code is converted from java (and from some historical reasons we convert from java 1.2 codebase which by the way is converted from java 1.5 code :) most of the collections used were the non-generic ones. This forced us to review the code and, when possible, use the generic version. In some cases the simplest solution was to introduce a non-generic collection (based on its generic counter-part) mimicking the native, non-generic, missing collection.

The last restriction in this list was the most frustrating one. db4o relies heavily on being able to load assemblies referenced by persisted objects and Silverlight security model doesn't allow arbitrary assemblies to be loaded explicitly. Actually (AFAIK) it is possible to load assemblies only if you specify a fully qualified assembly name, including its version. The problem with this approach is that db4o doesn't know about the assembly version (to be more precise db4o strips assembly versions away when storing objects). One possible solution would be to try to iterate over the already loaded assemblies and grab the reference from this list instead of trying to reload "already loaded" assemblies, but this brought us another frustration: this same security model disallows iterating over this list also (the decision to always load assemblies instead of applying the process described above was taken based on the fact that the performance impact of calling Assembly.Load() on "already loaded" assemblies is barely perceptible).

The solution we found (at least for now) relies on the fact that most applications know, at compile time, the assemblies declaring storable types. On Silverlight, every time db4o needs to get an assembly reference it'll raise a TypeReference.AssemblyResolve (static) event and will allow the application to resolve and return the required reference. The included sample shows a possible implementation for this. We are thinking on having a new class encapsulate the details of this pattern but we decided to postpone the introduction of this class until we better understand the restrictions and get some feedback on the design.

[edited on Jun / 23] Ashish Shetty, just pointed me to this page and explained that using this API we could workaround the assembly loading issue. We are gonne to investigate it soon.[/edited]

Current Status

As of today the following projects compile successfully under the Silverlight platform (see Db4o-Silverlight-2008.sln):

Core (Db4objects.Db4o)
Tests (Db4objects.Db4o.Tests)

No CS tests
No Migration tests
No Interoperability tests
No Linq / NQ tests

Optional (Db4objects.Db4o.Optional)
Db4oUnit
Db4oUnit.Extensions

Unfortunately successful compilation has very little to do with a software being usable / stable. Right now we have a subset of a full db4o test suite running against this platform. As we manage to run more of these tests we are sure that new issues will be found (for instance this one was caught when we started to run the first tests).

Required changes to Applications

In order to make the discussion more practical I've written a very simple CRUD Silverlight application. Keep in mind that this is my first Silverlight application :), so please give me some feedback if you see that something smells bad :)

Before you continue reading I suggest you download the application source code and take a look at it (bellow you can see a screenshot)

Through this really innovative user interface (seriously, it's a break through :) it's possible to add a new person by entering its first name/last name and then pressing the "add" button. To query, fill in the fields you want to query on (first name and/or last name) and press "query" button. To delete an object, select it in the listbox and press "delete". Thats it :)

The most important aspect of this sample lies in the following lines:

        #region Assembly.Load() issue workaround

        private static readonly IDictionary<string, Assembly> _assemblyCache = new Dictionary<string, Assembly>();
        static App()
        {
            //_assemblyCache[AssemblyNameFor(typeof(Queue<>))] = typeof(Queue<>).Assembly;
            //_assemblyCache[AssemblyNameFor(typeof(List<>))] = typeof(List<>).Assembly;
            _assemblyCache[AssemblyNameFor(typeof(Person))] = typeof(Person).Assembly;
            _assemblyCache[AssemblyNameFor(typeof(Db4oFactory))] = typeof(Db4oFactory).Assembly;

            TypeReference.AssemblyResolve += (sender, args) => args.Assembly = _assemblyCache[args.Name];
        }

        #endregion

and

        private static IEmbeddedConfiguration Config()
        {
            IEmbeddedConfiguration config = Db4oEmbedded.NewConfiguration();
            config.File.Storage = new IsolatedStorageStorage();
            return config;
        }

in app.xaml.cs file.

Basically it just populates a dictionary mapping from an assembly name to the respective assembly reference and registers for TypeReference.AssemblyResolve events. Once this event gets raised we just look for the assembly name in our map. This is required as a workaround on assembly loading constraints.

Note that any type referenced by your model (stored objects) should have its assembly listed in this map otherwise bad things will happen to you :) Jokes apart, if you fail to list an assembly in this map db4o will handle types defined in this assembly as generic classes, i.e, classes that don't have the original class definition, and this will not work as expected in most cases. Since this is hardly the desired behavior, a possible improvement would be to throw an exception (inside AssemblyResolve event if the requested assembly could not be found in the map) instead of returning null.

Also note that we declared the Person's fields as public otherwise we'd get exceptions at runtime. We discussed these limitations on the "limitations" topic above.

Wrapping it up

In order to support Silverlight we're required to introduce changes into db4o (including some deprecated classes/methods that were wiped out). For developers, the most important requirements are to register for TypeReference.AssemblyResolve event in order to return assembly references and to use IsolatedStorageStorage, as shown in the sample.

Silverlight support is on the way, but even though we had a steady progress (it can be tracked through COR-1611 issue) we do have some homework to do before we can declare it officially supported.

Based on our initial list of tasks we can see that currently we are working on porting / adapting the tests. As the next steps we're considering the following:

Integrate Silverlight into the build (mainly run the tests in our continuous build)
Finish porting tests
Investigate performance issues
Update documentation
Write samples
Publish Silverlight artifacts to the website

Of course, community involvement is a strong force that'll help us decide how much effort we shall put into this. So, if you are learning or already develop for the Silverlight platform we invite you to grab the db4o sources from our svn repository, try it with your projects and provide as much feedback as possible (reporting any issues, questions, suggestions, ideas, etc).

Finally, fell free to drop me a line at adriano at db4o dot com if you need any help to get started.

[edited on Jun / 23] As I said, some days after publishing this post, Ashish Shetty got in touch to alert me about some issues; I'd like to thank you for taking the time :)[/edited]

Best.

Adriano

Release Notes - 7.10 Development Release

Tetyana Loskutova — Fri, 29 May 2009 13:10:00 GMT

db4o version 7.10 is out and here is the list of modifications, improvements and bug fixes that you will find in it:

COR-1642 - Ensure proper exception handling in Eclipse plugin

COR-1633 - Blog about Silverlight

COR-1631 - schema evolution not working correctly for client server mode.

COR-1630 - Attributes are not being respected by the configuration

COR-1629 - Class filter configuration UI for Eclipse instrumentation plugin

COR-1628 - Get Db4objects.Db4o.Tests projects compiling under Silverlight

COR-1627 - Provide instrumentation step in Eclipse plugin

COR-1626 - Adding a new indexed value typed field to an existing persisted class triggers NPE

COR-1621 - merge type handlers branch

COR-1620 - PropertyChangedEventHandler causes InvalidSlotException when used with ArrayList4

COR-1617 - Add Db4o core Silverlight project to build

COR-1616 - Extend Java HashSet for TA

COR-1615 - Extend Java TreeSet for TA

COR-1613 - remove ExtObjectContainer#collections and all related code

COR-1612 - Bring trunk as close as possible to silverlight version

COR-1608 - Review MsgD implementations to make sure exceptions are transmitted to the client

COR-1605 - [Java] running TA instrumented AllTestsDb4ounitJdk5

COR-1604 - Activation decision is being taken upon field type instead of actual type

COR-1603 - Extend Java Stack for TA

COR-1602 - Extend Java HashTable for TA

COR-1599 - Synchronize KnownClassesRepository access in GenericReflector against database lock

COR-1598 - Diagnostics warns about missing field indexes on class-only queries

COR-1597 - db4o_osgi with rcp app problem

COR-1596 - Allow to specify dedicated target storage for backup

COR-1591 - migration support for TreeSet - from the old translator based format to the new type handler based one

COR-1590 - define expected behavior for multiple TypeHandlers (including InstantiatingTypeHandlers) in a class hierarchy

COR-1588 - Migration support for .net enums (broken after type handler api changes)

COR-1587 - .net enums should be indexable

COR-1584 - Remove dependency on Thread.SetData() / GetData() to enable progress on Silverlight support.

COR-1579 - move from FirstClass/SecondClass terminology to ReferenceType/ValueType terminology

COR-1578 - Detach FieldHandler from TypeHandler and ClassMetadata hierarchies and assign responsibilities

COR-1577 - PlainObjectHandler should not be embedded and should not have to manage identity explicitly

COR-1576 - Defragmenting an encrypted file throws OutOfMemoryException

COR-1573 - Separate ClassMetadata from TypeHandler

COR-1572 - Extend Java HashMap for TA

COR-1560 - Db4oTool doesn't instrument constructors for transparent persistence

COR-1539 - Readding a deleted object from a different client changes database ID in embedded mode

COR-1531 - Enum upgrade from 7.7 to 7.8 causes loss of information

COR-1527 - Type Handlers - Design discussion

COR-1514 - Implement/register TypeHandler for TreeSet

COR-1481 - TypeReference class fails with 'open' generic types

COR-1357 - Db4o-2008.sln does not compile in the distribution

COR-1336 - Emmit a warning when instrumenting classes with non private fields.

COR-1136 - Defrag an in-memory database

COR-864 - When lockDatabaseFile() is configured to be false, database files should not be locked.

COR-715 - DatabaseFileLockedException when database file folder doesn't exist