HTML5 Dev Conf

I was speaking at the HTML5 Dev Conf on May 21, 2012 on Client Storage.

The talk covered the state of client storage, touching on cookies and local/session storage. It looked at the WebSQL specification, the reasons it was deprecated and how IndexedDB is the next big thing for a database in the browsers.
It also spoke about the limitations on IndexedDB and the list of libraries that could help using IndexedDB now. Some libraries covered include gazeljs, db.js, queryIndexedDB and backbone's IndexedDB Adapter. I also spoke about the JQuery IndexedDB library that I wrote, in addition to Linq2IndexedDB that Kristof and I are working on. I also introduced the IndexedDB shim on WebSql that would let people use IndexedDB API on Safari, Opera - enabling it on iPad and the iPhone.

Client storage

Introducing - IndexedDB Shim over WebSql

IndexedDB Shim over Web on Github

Run the test cases

IndexedDB is a client side database technology supported by Firefox, Internet Explorer and Chrome. It is a replacement to the now deprecated WebSql standard. Unfortunately, IndexedDB is not yet supported in Opera or Safari, make adoption of this standard slower.
The specification is also evolving and its implementation has differences even in the browsers that implement it. In the previous post, I wrote about how Chrome implements setVersion, while Firefox has been supporting the onupgradeneeded method as per the newer version of the specification.
This post is about a project that I am currently working on - bringing the IndexedDB technology to browsers that only have WebSql as the database. WebSql iteself has shims implemented and this should work well over those shims too.
So far, the simpler operations like object stores, data get and put, and cursors have been implemented. The shim is based on the implementation of IndexedDB in Firefox. Since WebSql provides a transactional SQLite like platform, translating IndexedDB concepts to the WebSql world was easy.


IDBDatabase
A database in IndexedDB corresponds directly to a database in WebSql. An additional, top level database called __sysdb__ is added to keep track of the database versions etc. WebSql database versions were not used in this implementation.Note that there is no implementation of db.close()

IDBTransaction
The transactions for IndexedDB directly correspond to transactions in WebSql. The lack of the abort method in WebSQL Transactions is overcome by throwing an exception to cancel the transaction.

IDBObjectStore
An IndexedDB Object Store is corresponds to a table in WebSql. Every database also has a __sys__ table that tracks meta data like autoIncrement, keyPath, etc about object stores. Adding data to the object store is added using a record in the table that looks like {keyId, value}. I hope to use the Firefox Key implementation to allow collations, but for now, all keys are simply JSON strings. Values are also not obtained using the Structured Cloning algorithm, but using JSON.stringify for now. KeyPaths are evaluated when required using eval. Since SQLite allows only integer keys for autoIncrement, an extra inc column is added to such tables to keep track of the auto increment values.

IDBCursors
Cursors are implemented using offset and limit queries in WebSql. Continue simply selects the next  element and a variable saves the offset value that should be applied to the query.

IDBIndexes
Indexes are still not implemented yet and I am working on getting Indexes working. I am looking at the way Firefox implements indexes.

Testing
You can run the tests on Chrome, Opera and Safari using the page at http://nparashuram.com/IndexedDB/polyfill/. In chome, you should be able to open the Resources tab to see the various tables created by the implementation and how they correspond to IndexedDB constructs.

You can watch out this space for updates. Please feel free to fork the project on github and help me confirm to the standards better. I am doing test driven development, and you can run the tests at http://nparashuram.com/IndexedDB/polyfill/

I hope that this plugin makes the IndexedDB API available in more places, enabling faster adoption of the specification. 

IndexedDB setVersion vs onupgradeneeded

Try out the latest version of the plugin - http://nparashuram.com/jquery-indexeddb/test/

The latest specification of IndexedDB has removed the setVersion method and uses the newupgradeneeded method instead. In an older post, I had written about this change to the jquery-indexeddb plugin. However, Chrome still does not support the onupgradeneeded method and hence, the new version of jquery indexeddb plugin had to have a different name.
Over the past few weeks, I was able to work on some code that would wrap the indexeddb.open() method to allow the onupgradeneeded method. Instead of deeply coupling the setVersion-onupgradeneeded shim with the jquery plugin, I was able to get it to work as an independent shim that could also be used with standard IndexedDB applications.

Using the shim
To use the shim, simply include this javascript, and instead of calling window.indexedDB.open, call openReqShim. The events fired would be onsuccess, onerror, onblocked, and onupgradeneeded, as in the specification. The onupgradeneeded would have a versionTransaction that can be used to create/delete objectStores or Indexes.

Inside the shim

A database is opened when the openReqShim() call is made, very similar to indexedDB.open() call. The database name, and an optional database version are passed to this call, similar to the original call. The call returns the IDBRequest object.

For Firefox
In case of firefox, when the database is opened with the specified version greater than the database version, the onupgradeneeded method is called. This call is passed to the callback specified by the user. This is followed by a call to onsuccess, that is also passed to the user's onsuccess function.

For Chrome
In case of chrome, the onupgradeneeded function is not called. The database's onsuccess function is called. Here, the existence of the setVersion method is checked. If the method exists, and the specified version is greater than the database version, a the setVersion method is called. The onsuccess of the setVersion's request call invokes the user's onupgradeneeded method with the version transaction. Once the method completes, the versionTrasnaction is committed by closing the database. The database is opened again with the latest version and this is passed to the onsuccess defined by the user.

Future
The above shim tries to make the programming interface for the IndexedDB API confirm to the specification for Chrome. The latest version of the specification has also deprecated constants like IDBTransaction.READ_WRITE, replacing them with string constants like "readwrite". This change is currently available in Firefox Aurora and Chrome Canary, and I am working on it now, so you can watch out this space for any updates.

Running QUnit tests sequentially in order

Download QueuedUnit.js

Wikipedia defines Unit Tests as a method to test individual pieces of code independently. This basically implies that the order in which the QUnit tests run should not really matter. However, there could be cases where such tests may have to run in order and it is not very easy to do so on QUnit.

Use Case

There could be cases where you would like tests to run in order for better coverage, or simply because you have to write lesser setup/tear down code for each case. I have been working on a couple of API-centric projects and I found that I could be lazy and write lesser code if I could get the tests to run in order.   The projects, IndexedDB WebSql Shim and a jquery plugin for IndexedDB have tests where a database is first created, and sample data entered, and the subsequent tests use this setup to iterate over or search for data.

Alternatively, I could have written a common set up method that creates the mock database for every test. 

Experiment

Looking at this code snippet the QUnit tests are written such that they are started long after (100 milliseconds) the script tag is encountered. This causes the first test to run, but then, the order of the tests becomes 1,4,3,2. 

If the time interval is reduced to something like 10 milliseconds, the test execution order now changes to 1,2,3,4. 

I tried playing with QUnit.config.autostart, QUnit.config.autorun, but none of them seem to get the tests to execute in order. 

Solution

To get it working quickly, I had to brew my own solution, and the result was just a tiny framework. The gist simply loads the tests, one after another. The only changes from traditional QUnit async tests would be 

• Replace asyncTest with queuedAsyncTest().
• Replace module() with queuedModule(). 
• At all points that the test is over, the nextTest() function should be called to add the next test to the queue. 
• Once the tests are all ready to run, call nextText(), to start running test 1. 

A timeout method checks if the current test has finished in time, and if it has not, the current test simply fails and it moves on to the next text.
Here is the modified code snippet working with the QueuedUnit function calls.

Compare the test results - Before and After.

Issues

The requirement to call nextTest() to start the next test seems annoying. Also, if nextTest() is not specified, QUnit simply stops, unaware if the test passed of failed, but these are errors similar to a missing start() that can be avoided.

You can grab QueuedUnit.js on github, and leave a comment letting us know if it helped you become lazier:). After all, lazy testing is better than no testing!!