June 11, 2009

Featureserver on AppEngine

AppEngine is awesome. The more we use it, the more we like it.

Recently, someone contacted us who needed a site up, in a hurry, to serve up some points on a google map. The catch was there were about 50k points (so it seemed server side clustering might be nice). Also they wanted to be able to serve up at _least_ tens of millions of requests a day. And maybe quite a lot more.

Given the scaling requirements, it seemed like AppEngine might be a nice fit, since then we wouldn't have to worry so much about tons of caching, or ensuring clients made similar bounding box requests, and so forth. And as for the posting/getting of points to/from appengine, we decided to go for using FeatureServer as a base.

If you're not familiar with featureserver, a quick overview: It makes it easy to (amongst other things) post/update your features to some datastore, and pull them out with bounding box and/or attribute queries in a variety of vector formats (kml, json, wfs, etc). Also it not only supports a bunch of different backend datastores (shapefiles, twitter, postgis, flickr, etc.), but it makes creating new ones simple. And, thanks to crschmidt's usual paving-the-way, setting up FeatureServer on AppEngine was trivial.

So there I am with a nice little featureserver running on AppEngine. We set up some cron jobs to do the clustering, and with 50,000 points I run some tests at about 75 queries/second. Everything seems great.

But on further examination, the FeatureServer datastore that currently exists for AppEngine has a couple problems:
* Because it is based on geohash , it uses up your only inequality query on your location (bounding box) search, which means you can't filter on other stuff.
* The geohash implementation it uses has some quirks (but that's for another post)

Fortunately, WhereCamp was on during the time I was thinking about how to solve this, so I was able to ask all kinds of smart people their advice. One of them immediately pointed out to me that a colleague had implemented a clever method for storing points on AppEngine that might just do the trick: GeoModel

And so it was that I gave GeoModel a try, and it did indeed solve the problems I was having with the geohash implementation. On the downside GeoModel currently only works with points, but as that is all this particular project needs, it's not a problem at all. Long story short, I simplified our custom datastore this morning, and committed it to the featureserver codebase. So if you want to very quickly put up a scalable, reasonably robust geo-point datastore, with a restful (sorry, sean) interface, GeoModel on AppEngine might be a good way to go.

1 comment :

Jamie Vleeshouwer said...

Do you have any installation instructions available which helped you to set up an instance of featureserver with AppEngine. We are also trying to do the same thing.