Implementing System Catalog or Data Dictionary

It is the classic chicken and egg problem.

I would like to implement the system catalog in database tables. However, the system catalog will contain information that is a pre-requisite for working with tables. I do not know how other database implementations handle this cyclic relationship. It is made more difficult for SimpleDBM because the core database engine has no knowledge of the data dictionary or the type-system. These are implemented by add-on modules.

I considered following options for implementing the system catalog:

1. Simplest option would be to pre-define the schema for a database using a configuration file, which can be loaded at database startup. This may be adequate for many use cases, but implies that the database schema is statically defined, and tables cannot be created or dropped dynamically.

2. Second option is to implement the catalog as dynamic tables. The catalog tables would be pre-created when the database is created, and their structure would be known to the database manager. When a regular table is created, a row would be added to the system catalog table, to record the structure of the new table. The table creation would be handled as a standalone transaction (covering both the container creation and updating of the system catalog table), to ensure that it is completed before any attempt is made to access the new table. Index creation would be handled similarly.

When a regular table is accessed, its type structure will have to be retrieved from the catalog tables. This means some recursive locking, which I am worried about because so far, SimpleDBM has not needed to carry out locking during system restarts. Perhaps, we could optimize away the need for locking because SimpleDBM locks containers anyway when they are accessed, which means that there is already a lock that protects the table definition. Unfortunately, at present the SimpleDBM API does not support a read uncommitted isolation mode - which would be necessary to avoid obtaining locks on catalog tables when they are being read.

To avoid accessing the catalog tables every time a container is accessed, the first time a table is accessed, its definition would be retrieved and cached in memory for subsequent use.

3. A third option is to maintain the table definitions in special containers, and manage updates to these containers without going through the buffer manager. Here is a tentative design:

The structure of a table or index will be stored in a special container. This may be a single shared container or one container for each table or index. The multiple container option has the advantage that a simple naming convention could be used to quickly locate the appropriate container.

The database module will perform special reads and writes to these containers, bypassing the Buffer Manager, and Lock Manager.

The problem is how to guarantee that the data in the special container is updated atomically as part of the transaction that created the table or index. One way of achieving this would be to generate PostCommitActions for updating the metadata when a table or index is created or dropped. The PostCommitAction would be executed even during restart recovery, and would therefore ensure that the table or index definition is atomically updated as part of the transaction that created the table.

When a table or index is accessed, its definition can be retrieved and cached. Subsequent access will refer to the cache.

Java on Mac OS X

I posted elsewhere that I now own an iMac and am doing most of my Java development on it. The big problem with Java on Mac is that the JDK is pretty out of date. The official version that comes with Mac OS X 10.4 Tiger is 1.5.0_07. This version seems buggy, as the JVM seems to hang when running test cases for one of my projects. I was getting quite frustrated, and in the end decided to run my test cases under OpenSolaris, which I installed as a VM using VMWare Fusion.

Fortunately, Apple has recently made available a pre-release version 1.5.0_13 from its Developer Connection web site. You need to be registered to be able to download this release. The good news is that it fixes the bug that was causing my tests to hang the JVM. I think it has something to do with the java.util.concurrency classes. Ofcourse, if you don't use these, you would not have faced the problems I was facing.

The long gap between the Java releases on Mac seems unacceptable, especially since the JDK is so buggy. The other problem is the lack of support from major vendors for the Mac platform. I would have liked to install and use IBM WebSphere and Rational products on my Mac, but it isn't amongst the supported platforms. Linux has much better support now-a-days.

Fast forward

It has been sometime since I last blogged about general programming topics. My day job has kept me from the joys of exploring the JPA. But many other things have happened to me in the meantime, life just goes on, and there is never enough time to catch up on everything.

For a start, I want to retitle this blog, by removing the Java from it. Let this just be a Programmer's Blog.

New release 1.0.6

A new SimpleDBM RSS release (1.0.6 BETA) is out. This release contains an important bug fix to the Buffer Manager. The release can be downloaded from www.simpledbm.org.

Along with this release, there is also updated documentation. The User Manual is for developers who wish to use the SimpleDBM RSS component within their own projects. The Developer's Guide is for hackers who want to understand how SimpleDBM works and help me enhance it.

I recently installed Solaris 10 Developer edition on my home server. I was pleasantly surprised by how fast Java is on this platform. I am using a Mac as well, but the Java support on Mac is disappointing. The JVM just hangs sometimes when I am running SimpleDBM tests. Not because of locking or any other issue - it literally hangs - and if I connect to it using JConsole, there is no response.

I am working on a new sample which will demonstrate a higher level API than currently available. I hope to get a working version out by the middle of the month.

How portable is Java code?

In my last blog entry, I mentioned the differences in file locking behaviour I found between Sun's Windows JDK and Mac JDK. Well, I recently acquired a Dell server, and am running SLES 10.1 64-bit edition on it. The default Java 5 SDK that comes with SLES 10.1 is IBM's JDK 5 SR3. SimpleDBM fails to pass all its test cases under this JVM. All tests pass successfully when I run using Sun's Java 6 SE Update 2 64-bit JVM.

I haven't tried any other JDK's yet, but it would be interesting to see the results.

The point is that when I code in Java, I expect my code to run correctly on all JDKs. I do not want to have to worry about portability issues. But reality is that the JDK and its libraries are so large, that it must be very hard to make independently developed JVM's to be completely compatible.

Update: One of the test cases started failing because the HashMap.values().toArray() on IBM's JVM returns values in reverse order of the Sun JVM. This changed the results of the test to be different from what was expected, and therefore the test failed.

Java on Mac

Recently bought an iMac - with 24inch screen, and this is now my development platform for simpledbm. I am using JDK 5 and Eclipse 3.3.

So far one test case has failed. Upon investigation, found that the behaviour of FileChannel locking is different on the Mac. It seems that Mac JVM allows multiple threads within the same JVM to acquire exclusive locks on a file concurrently - whereas on Windows, we get an OverlappingFileLockException exception. It also seems that within the same thread, it is okay to acquire a lock more than once on the same file on Mac but not on Windows.

I was relying on the locking behaviour I saw on Windows, to allow simpledbm to detect multiple instances of simpledbm running concurrently against the same database.

Interestingly, the specification of locking behaviour seems contradictory:

File locks are held on behalf of the entire Java virtual machine. They are not suitable for controlling access to a file by multiple threads within the same virtual machine.

But on the other hand:

The locks held on a particular file by a single Java virtual machine do not overlap. The overlaps method may be used to test whether a candidate lock range overlaps an existing lock.

Which is right?

SimpleDBM Beta release is out

At long last, an early Beta release of SimpleDBM is out. Downloads are available from SimpleDBM project website. A User Manual is also available.

Due to work pressures, progress has been very slow during the last one year. I am now hoping to build a few sample applications. Testing is still a priority as there are still not enough test cases in some areas, such as the Tuple Manager module, and in particular, coverage of crash/recovery scenarios.