LINQ on IndexedDB

LINQ API on IndexedDB : http://nparashuram.com/IndexedDB/LINQ/index.html

In my previous post, I had written about a jquery plugin for IndexedDB. The objective of the plugin was to make the IndexedDB API easier to use. Treading on the same direction, I started exploring the use of LINQ like syntax that can be built on IndexedDB.

Background
IndexedDB API is a part of the HTML5 standards that allows web pages to persist data in the browser. Basically, it is a database embedded into the browser. The standard is currently supported by Internet Explorer (prototype), Firefox 4 and Google Chrome 12 (Canary build) . More details about IndexedDB API here.

Summary
The current API specification is still in editor’s draft (and in my humble opinion, very verbose).
Would it not be great if I could use the familiar LINQ like syntax data operations using IndexedDB on the browser also ?

… from("BookList").in("LibraryDB").select().forEach(showRecords)…

Read without the parenthesis and it would sound more like LINQ.. The parenthesis are a due to the Javascript language.

Problem

• The IndexedDB APIs are very verbose and even simple get/put operations require huge amounts of boiler plate code. 
• The asynchronous function call model of getting a request is something that users have not really used. When dealing with data, people love LINQ.

API Today
To perform a database operation, a developer would have to write boilerplate code,  as large as

 
    var request = window.indexedDB.open("BookShop-1");
    request.onsuccess = function(event){
        var db = request.result;
        var transaction = db.transaction(["BookList"], IDBTransaction.READ_WRITE);
        var objectStore = transaction.objectStore("BookList");
        var request = DAO.objectStore.openCursor();
        request.onsuccess = function(event){
            var cursor = request.result;
            if (cursor) {
              write(cursor.key + "" + cursor.value);
              cursor["continue"]();                
            }
        };
    };

The above code would increase if objects have to be created with version transaction, and error conditions are to be added.

Solution

Would it not be simpler if a user would just need to write something like

From ObjectStore in Database, Select and ForEach, print.

The syntax for this could be as simple as

linq4idb().from("BookList").in("LibraryDB")select().forEach(showRecords);

for iterating through the records or the following for filtering records.

linq4idb().from("BookList").in("LibraryDB").where(“rating”,{“equals”: “good”}).select().forEach(showRecords);

Note that since this is Javascript, we cannot do away with the parenthesis. Read the above like without the parenthesis and it would sound like LINQ.

Code & Demo

The demo is available at http://nparashuram.com/IndexedDB/LINQ/index.html. It requires JQuery 1.5 and is built on the concept of "promises". The demo illustrates the various LINQ
operators that are currently supported. It requires Firefox 4 or Google Chrome 12 Canary build.
The code is available here on Github.

Advantages

• This is better than passing the entire LINQ query as a string with auto-completion, syntax checking, etc.
• Better than SQL Statement string in Javascript 
• This library is not built at on the native implementation but build on IndexedDB API that can run across all browsers.
• Server Side JS could use similar syntax with LINQ on NOSQL databases whose APIs are similar to IndexedDB.

Design
The library tries to use ranged cursors for the where() operator and indexes for the orderby(propertyName) operator.
A limitation of the IndexedDB API is that it does not allow for filtering and sorting over multiple properties. In cases where sorting and filter are user over more than one property, the sorting is does first using Indexes and then the results are filtered with the assumption that native, index-based sorting is faster than filtering using cursors and then sorting in Javascript.

Next Steps

• LINQ is a query language and there did not seem any obvious way to "put" data. Still exploring the ways to have a good syntax to "put" data.
• LINQ expands to "Language" Integrated Query. The syntax above has parenthesis which do not make it look integrated with the language. Still looking at way to make it look more like LINQ. 
• Find ways to optimize the combinations of the where() and the orderby() clause so that all operations are performed natively.

Fiddler as a CDN substitute for web development

Problem
As the performance rules suggest, common Javascript libraries like JQuery or YUI should be served from CDNs. Though the advice is great for production systems, it makes web development without connectivity hard.

Alternatives

• To circumvent the fact that a developer may not always have access to the internet and hence the CDN, some web pages include a second script that check for the absence of the global object and then include it from the local filesystem. In my humble opinion, this method is a little ugly and adds unnecessary extra logic. 
• Another common alternative is to include the final paths of the when the pages are built using ant or make. Web development is usually about making a change and hitting F5, building every time to see a change is hard. 
• The most common approach I have seen is to alter the /etc/hosts file (or a dedicated proxy server in the middle) that intercepts requests to redirect them to the local file.

Solution
I noticed that I usually have Tamper Data or Fiddler open when I do web development to inspect traffic to my server. Instead of modifying the hosts file or spinning up a special proxy server, it is easier to add rules to Fiddler that redirect CDN requests to a local file.
All that we need to do is start fiddler and filter all requests to internet resources and then save the session to import it as auto-responder.

An easier approach that would work for all applications would be to save the entire CDN archive as a .saz file. Unfortunately, I could not find the entire list of Google or Microsoft CDN files.
I tried setting up a crawler that picks up all versions and files/sub-files for all libraries from Google, but then, I just worry about just 2 or 3 libraries. Hence, it is easier to set up mapping rules just for them.

Next Steps
I am planning to write a Fiddler or a Firefox plugin that automatically looks at CDN requests, caches them, and makes them available in case of absence of connectivity.

IndexedDB Jquery Plugin

Summary

IndexedDB Jquery plugin demo - http://nparashuram.com/trialtool/index.html#example=/IndexedDB/jquery/trialtool.html

Background

The IndexedDB API is in its draft state but is already available in Firefox 4 and Chrome 12 Canary Builds. Internet Explorer also has a version of it on their HTML5 prototypes site. In my previous posts, I have also written about IndexedDB API examples using TrialTool.

Problem

I wrote a couple of "non-production" applications and noticed that I was frequently copying non-application-logic, IndexedDB-related code across applications. Should that code be a part of a library? In my opinion, the amount of boiler plate code written to perform simple tasks like persisting or fetching data is not little. 

Code

A typical operation using the IndexedDB API would involve using the request model, creating transactions, checking for existence of object store using error responses and exceptions and then finally getting to the part where the data is actually iterated over.  

    var request = window.indexedDB.open("BookShop-1");
    request.onsuccess = function(event){
        var db = request.result;
        var transaction = db.transaction(["BookList"], IDBTransaction.READ_WRITE);
        var objectStore = transaction.objectStore("BookList");
        var request = DAO.objectStore.openCursor();
        request.onsuccess = function(event){
            var cursor = request.result;
            if (cursor) {
              write(cursor.key + "" + cursor.value);
              cursor["continue"]();                
            }
        };
    };

The above code would increase if objects have to be created with version transaction, and error conditions are to be added.

Solution

The IndexedDB Jquery plugin is an attempt to reduce this boiler plate code in addition to bring back concepts like method chaining and Deferred calls.
The philosophy followed in the library is the same as the $.ajax() call that is more than just a wrapper over the XMLHTTPRequest object. 
Some Jquery goodness that I wanted while I designed the API include

• Method chaining, access to most probable next operation in a chain. For example, once an objectStore is referenced, the next operations usually are CRUD, cursors or indexes. These operations should be available in the chain. 
• APIs for most common functions. For example, most people would not want to care about transactions. They would simply like transactions to be implicit and hence, the API opens a READ_WRITE transaction. Alternatively, the user can also explicitly open transactions and use them.
• Since the IndexedDB API is an asynchronous API, use of Deferreds() for completion. This model is much more familiar than the request and request.result model exposed in IndexedDB API.
• Default error handling, taking advantage of error event propagation in IndexedDB. Error cases are not common and should not be required at every step. The promises in Deferreds() are easier to handle errors.
• Falling back on smart defaults. For example, when a user accesses an object store and it does no exist, they would probably want to create on a version change transaction. This is done in the object store call by default. However, the user can explicitly specify that the call should fail if the objectStore does not exist.

The equivalent code for the snippet above looks something like this

$.indexeddb("BookShop-1").objectStore("BookList").openCursor().each(write); 



Code and Demo
The library is currently hosted here on GitHub and the examples illustrating the API are available at http://nparashuram.com/trialtool/index.html#example=/IndexedDB/jquery/trialtool.html.

Watch out this space for more updates on my experiments with IndexedDB.