IndexedDB Polyfill - Under the covers


I have been working on an IndexedDB polyfill that enables the API for browsers like Opera and Safari that still do not support it. The polyfill enables the IndexedDB API to be available on browsers that support WebSql including devices like iPad and iPhone. This post is about the interesting issues that I encountered while developing the plugin. For more context, you should check out the IndexedDB Specification, WebSQL Specification and the polyfill source code.

Housekeeping data
IndexedDB is a non-relational database that saves Keys and Javascript objects. Certain meta data like Object Store Key Paths, index names, auto increment flags, database versions, etc. need to be store. Like WebSql using the sqlite_sequence to store autoincrement sequences, a database called __sysdb__ is used to save database versions. Additionally, every database created has a __sys__ table that stores the names of all stores with their meta-data (indexes, auto-increment flags, etc). This is very similar to the way Firefox IndexedDB Implementation persists meta data.

Keys and collation
The keys for data store can be of any Javascript type with rules specified for collation. WebSql however requires column types to be specified. To be able to save multi-type data, a blob column is used to keys and values. A key encoding scheme like the one by Firefox needs to be implemented to ensure collations.

Indexes
Indexes are created by simply creating a new column in the same table, extracting the index-key-path from the value and then and indexing that column. This is different from the way Firefox or Chrome implement indexes by using a different table, but this was a lot easier to write. I am still trying to find the reason why indexes were implemented as separate tables (multi value indices ? )

Synchronous Create Operations
The IndexedDB specification indicates that createObjectStore and createIndex operations are synchronous. However, WebSql does not allow synchronous operations. To overcome this limitation, all object stores have an internal __ready flag that is set to false for createObjectStore or createIndex when object stores or indexes are created. Operations are added to the transaction queue once these are reset. This has to be replaced with a mechanism that ensures that operations are only queued during a version transaction. Note that this needs to be persisted such that the flag is available across tabs - possibly saving this in the database, or simply using a cookie. 

Aborting Transactions
WebSQL does not provide a direct way to abort transaction. To mock an abort operation, an exception is thrown when inside the transaction.

Events and event.target
I noticed that the older specification and some libraries use event.target.result instead of request.result to get the return value of IDBRequest. Unfortunately, an Event object cannot have a custom target set on it. Hence, events for now are simply Javascript objects.

Note that some of the points discussed above are still marked as open issues - any help would be welcome!
You can watch out this space for more updates.


at

IndexedDB polyfill


In my last post, I had introduced the polyfill for IndexedDB that I was working on. IndexedDB is not yet supported on all browsers. Firefox, Chrome and Internet Explorer have implementations of IndexedDB while Safari and Opera still support WebSql. The IndexedDB specifications is also at its final stages and most browsers should have a native implementation of IndexedDB soon.
However, a client side database technology can make web apps very powerful and I thought that it would be a good idea to bridge the gap between IndexedDB and WebSql.
The polyfill exposes IndexedDB like API and uses the underlying WebSql to actually store data. This enables web applications to start using IndexedDB APIs across all the major browsers. The examples using the shim work on browsers and even devices like iPad or iPhone.

Including the Polyfill
<script type = "text/javascript" src = "http://nparashuram.com/IndexedDBShim/indexeddb.shim.js"></script>

To use the polyfill, simply link or include the indexeddb.shim.js file. The polyfill assigns window.indexedDB to be window.mozIndexedDB, window.webkitIndexedDB or window.msIndexedDB is any of those implementations are available. If no implementation is available, the polyfill kicks in and assigns window.shimIndexedDB to window.indexedDB. Hence, web applications can start using window.indexedDB as the starting point for all database operations.

To force the shim to be used, add the following statement.

window.shimIndexedDB  && window.shimIndexedDB.__useShim();

The source is on github at http://github.com/axemclion/IndexedDBShim.

Libraries Tested
The polyfill is based on the IndexedDB specification and hence should work with any library that requires IndexedDB. Some of the libraries that were tested with the polyfill include the following


Specification compliance
Almost all features in the IndexedDB specifications are supported by the polyfill except the following.
  • Issue 1: The keys are not encoded properly and numbers are currently sorted like strings. Hence, 1 and 100 come before 2. 
  • Issue 2: Version Transactions are currently non-blocking. Hence, if you try to read a database when a version transaction is in progress, a race condition occurs and the results are non-deterministic.
  • Issue 3: Transaction abort is not currently supported.
I am working on resolving these issues as soon as possible, but would appreciate any help.You can either send me a pull request with the fix or contact me.

I am planning to follow up this post with an article that explains how the polyfill works under the covers. You can follow my work on IndexedDB here.
at