One that won’t go away (under load), one that doesn’t keep me up all night (worrying about scaling), one that won’t make me sleep all day (when I should be adding features). If you have tried to build an Internet site or application recently that needs to work for thousands or tens of thousands of concurrent users you may share this desire.
Why is this still so hard? Why do we find ourselves worrying about locating experts in the dark art of database performance tuning? Why are we spending time haggling with Rackspace over the price for another set of servers or racking our own servers at Equinix? Why are we writing our own user management (from scratch)?
Because we are using tools that simply were not made for the job. Relational databases have gotten faster and better, but their fundamental construct of data stored in rows and neatly parceled out across tables (which frequently need to be joined) does not really match up with either rapid development using some notion of objects or with scaling horizontally using commodity hardware. Web servers started out by simply providing static content and were then forced into running applications with the result of not doing either particularly well. Yes, people have been writing new ones to address that, but even those are fundamentally designed to work on a single machine. The second you go beyond one machine you need separate load balancers, reverse proxies, caches and all sorts of other paraphernalia just to make stuff work together. Worse yet, the web server, the application and the database are connected to each other via thin straws that were bolted on after the fact with important information either not passed at all or only painfully (e.g., information about which user is requesting a particular set of data).
What about the growing cast of frameworks, like Ruby on Rails, CakePHP, Django, etc? Don’t they provide a solution to these issues? Not really. These frameworks are for the most part about enabling rapid development. They do great on the add features rapidly front but often make matters worse on the scaling front. The reason is that they layer more stuff on the very tools that are wrong for the job to with.
What then is needed? A platform that is created from the ground up to enable modern web sites and applications. What would such a platform look like? It would be hosted and (nearly) infinitely scaleable. It would provide object storage that’s as simple as saying “here’s an object, store it” (you get back a handle, ideally, if you want a human readable, search engine optimized one). Later on, retrieval should be something like – “here’s a handle, give me back the object” (with full user level access control baked in). Stored stuff should be easily indexed so that one could say “give me back all the handles for objects that match this pattern” (and to which the user has access). The same should work for media: “Here’s a picture, store it for me with this metadata” and “Find all the pictures for me tagged x.”
Aside from storage there are other useful services the new platform should come with, since essentially every modern web site / application needs them, such user authentication, authorization and access control. Flexible processing of pretty URLs. Easy creation and maintenance of page templates. Ability to send emails and process bounces. Handling of RSS feeds (inbound and outbound). Support for mobile access and possibly even voice capabilities.
Code would run inside the platform (this is what Marc Andreessen calls a Level 3 platform) but it would not be cut off from the outside world. It would have full access to other services that live on the net via web services. So, for instance, there would be no need for the platform to have its own payment service.
Let me try to head off some potential objections.
Can such a platform be built at all? Won’t there be problems with distributed s (or some other technical objection)? Nobody said it would be easy (besides, if it were it wouldn’t be half the fun or represent a big opportunity). But there is no fundamental impossibility here, especially in a hosted environment and keeping in mind that the goal is not to create a platform on which anything can scale automagically, just a large set of web sites and applications that are currently hand crafted. For instance, the object storage magic does not have to work for arbitrarily complex objects. My guess is that the vast majority of sites and applications can live with (fairly) shallow objects based on just a few underlying types.
Isn’t this just a fancy way of saying you want a hosted object-oriented database and didn’t those fail miserably as software products? Just because stuff failed in the past doesn’t mean it won’t work now, especially when circumstances change (the current ugly picture in the credit markets is a reminder that the opposite is equally true).
Won’t folks be completely locked into this platform? We are not really worried about being locked into MySQL, because we feel that if push came to shove, if we really had to, we could switch to say Postgres or DB2 or Oracle. This of course is a lot harder to do in practice because in the quest for performance we often resort to using custom SQL extensions that are specific to our current DB. But it does point to standards as an important way to mitigate lock in and anybody providing a platform like this would do well by working with existing standards where available and promoting new ones where not (e.g. an object storage standard). A more dramatic way to mitigate lock in might be to offer an open source “community” version of the platform and compete based on hosting the most reliable version.
Hasn’t Amazon has already built this? Isn’t this S3 and EC2? S3 and EC2 and the upcoming database service are steps in the right direction. But they are not nearly as easy to program as the “new” platform (in particular EC2 is really just high speed provisioning of traditional servers). Amazon could, however, use their pieces to offer something like the new platform.
I am sure there are other objections (look forward to the comments), but those aside, where might this new platform come from? Marc Andreessen lists companies who are working on it now, including his own Ning, Salesforce and Amazon. I would not be surprised if folks at google were also thinking about offering something. Based on their recent announcements, Salesforce seems furthest along in offering a new platform but they have made many idiosyncratic choices which result in high lock in (not to mention that they run an applications business which might compete). Despite the big guns already working on this, I believe there is still an opportunity for someone new to build the new platform. It’s the kind of disruptive change that requires throwing overboard a lot of existing assumptions and conventional wisdom and trying something radically different. It will also require maniacal focus and independence from the needs and influences of existing operations. Sounds like a startup to me.
UPDATE (December 16, 2007): Amazon has taken the wraps off SimpleDB and it looks promising. It's schema free and appears very reasonably priced. If the performance is good this will bring Amazon a big step closer.