NoSQL, No Peace

After several months of research, review and revision, a white paper I wrote for the SQL Azure team, "NoSQL and the Windows Azure Platform", has been published by Microsoft. If you go to http://www.microsoft.com/windowsazure/whitepapers and do a find within the page for "NoSQL" you'll see a link for it. If you'd rather download the PDF directly, you can do so by clicking here. The 25-page (not including cover and TOC) paper provides an introduction to NoSQL database technology, and its major subcategories, for those new to the subject; an examination of NoSQL technologies available in the cloud using Windows Azure and SQL Azure; and a critical discussion of the NoSQL and relational database approaches, including the suitability of each to line-of-business application development.

As I conducted my research for the paper, and read material written by members of the NoSQL community, I found a consistent sentiment toward, and desire for, cleaning the database technology slate. NoSQL involves returning to a basic data storage and retrieval approach. Many NoSQL databases, including even Microsoft's Azure Table Storage, are premised on basic key-value storage technology -- in effect, the same data structure used in collections, associative arrays and cache products.

I couldn't help thinking that the recent popularity of NoSQL is symptomatic of a generational and cyclical phenomenon in computing. As product categories (relational databases in this case) mature, products within them load up on features and create a barrier to entry for new, younger developers. The latter group may prefer to start over with a fresh approach to the category, rather than learn the wise old ways of products whose market presence predates their careers -- sometimes by a couple of decades.

The new generation may do this even at the risk of regression in functionality. In the case of NoSQL databases, that regression may include loss of "ACID" (transactional) guarantees; declarative query (as opposed to imperative inspection of collections of rows); comprehensive tooling; and wide availability of trained and experienced professionals. Existing technologies have evolved in response to the requirements, stress tests, bug reports, and user suggestions accumulated over time. And sometimes old technologies can even be used in ways equivalent to the new ones. Two cases in point: the old SQL Server Data Services was a NoSQL store, and its underlying implementation used SQL Server. Even the developer fabric version of Azure Table Storage is implemented using SQL Server Express Edition's XML columns.

So if older technologies are proven technologies, and if they can be repurposed to function like some of the newer ones, what causes such discomfort with them? Is it mere folly of younger developers? Are older developers building up barriers of vocabulary, APIs and accumulated, sometimes seldom used, features in their products, to keep their club an exclusive one?

In other engineering disciplines, evolution in technology is embraced, built upon, made beneficial to consumers, and contributory to overall progress. But the computing disciplines maintain a certain folk heroism in rejecting prior progress as misguided. For some reason, we see new implementations of established solutions as elegant and laudable. And virtually every player in the industry is guilty of this. I haven't figured out why this phenomenon exists, but I think it's bad for the industry. It allows indulgence to masquerade as enlightenment, and it holds the whole field back.

Programming has an artistic element to it; it's not mere rote science. That's why many talented practitioners are attracted to the field, and removing that creative aspect of software work would therefore be counter-productive. But we owe it to our colleagues, and to our customers, to conquer fundamentally new problems, rather than provide so many alternative solutions to the old ones. There's plenty of creativity involved in breaking new ground too, and I dare say it brings more benefit to the industry, even to society.

NoSQL is interesting technology and its challenge to established ways of thinking about data does have merit and benefit. Nevertheless, I hope the next disruptive technology to come along says yes to conquering new territory. At the very least, I hope it doesn't start with "No."

Posted by Andrew J. Brust on 04/25/2011