TileMill

I've played briefly with TileMill before, but after learning more about the advances that Development Seed is putting into their MapBox stack (such as Node.js integration with Mapnik, utfgrid and more) I realized it was time to sit down and play with it for real.

My main interest right now is in getting a feel for the CSS-like syntax used in Carto, but as long as I needed to set up a full instance of TileMill to play with I figured I might as well make it into an Amazon EC2 AMI, so anyone can easily boot up an instance and get started.

Setup was extremely simple (thanks to Dane telling me how to cut and paste their very straightforward install instructions; not that he noticed I was installing everything to /tmp). After some much needed sleep and attempt #2 here on the plane, you can now go to the Amazon AWS console and load up ami-56ae563f, or search for tilemill, and you're done. Just wait for the instance to start up, and TileMill should be running on port 80 (if you want to get to the tilemill console, ssh in with the keypair you associated with the instance, and type in:

sudo su
screen -r tilemill

Shape Escape

Well it's been a while, so just a quick note: Since the last post, I started working full time for Google. And with that out of the way, here's a post on how and why I made shpescape.com, which lets you upload shapefiles to Google Fusion Tables.

Why shpescape?

Google Fusion tables makes it easy to import and visualize data from spreadsheets and KML, and while it has increasingly robust spatial support it does not currently let you upload shapefiles directly. And since shapefiles are still incredibly common in the wild, I thought I'd make a quick tool to let people upload shapefiles to Fusion Tables.

Which platform?

I thought I'd try Google App Engine to avoid any hosting costs (given this will likely not be an extremely popular website), but while there's a decent shapefile reader or two for python, there's not a lot of support for things like reprojecting and other geometry manipulation without additional c++ libraries that App Engine won't run. So I just went for a simple GeoDjango app.

Authentication

I used my colleague Kathryn's Fusion Tables python client to handle the authentication (OAuth). And I decided against having OpenID in the mix as well for actually associating an account with various uploads. The downside is that you can't log in and view your previous uploads. But you can always go to the main Fusion Tables page to see all your tables, and the upside was one less thing for me to consider (for example, if you are logged in with multiple accounts in the same browser, OAuth does not return which account gave permissions for). [Edit: It turns out you can actually request the email address of an authorized user using the scope noted at http://sites.google.com/site/oauthgoog/Home/emaildisplayscope]

Handling a Shapefile Upload

I used a simple fork of Dane Springmeyer's django-shapes app to handle the shapefile import. The customizations let users upload a zipfile that has a shapefile in a subfolder, and/or multiple shapefiles in a single zip. I had never really noticed shapefiles being zipped up this way, and it really surprised me how common these scenarios are with shapefiles from various US counties and other agencies -- my first 3 test users all had their uploads fail until I added this. After the upload is verified as valid, it creates a shapeUpload object, which is processed separately so the end user can view a reloading page with updated status.

Processing the Upload

My initial attempt was pretty straightforward:

• Attempt to get the projection of the shapefile (from the .prj)
• For each feature, get it's KML and attributes
• Upload 'em to Fusion Tables, ensuring a max of a few hundred, or <1MB, at a time (the API can handle at most 500 rows and 1MB per POST)

Additional Features

Next up, I started adding a few extra bits, which led to an import method begging for a refactor.

• Simplification for KML over 1M characters long (which is the max characters allowed by Fusion Tables per cell)
• Process/Upload 10k rows at a time (so we don't use too much memory for very large shapefiles)
• Added numeric styling columns for string fields that don't have too many unique values (Fusion Tables only allows robust styling like gradients and buckets on numeric fields)
• Allow users to specify some additional geometry columns:
  • Simplified Geometry
  • Centroid (only works for polygon shapefiles)
  • Buffered Centroid (so you can apply the more robust polygon styling rules on the 'centroid')

Finishing up

This whole project was a pretty quick attempt at what I hope is a useful solution to a common problem, so any comments on how to make it better are appreciated. And if you want to see how it all works in more detail, I also open sourced the code. Enjoy!

Oceans Showcase

Last night at the San Francisco Ocean Film Festival Google launched the Oceans Showcase, which is the second contract I've had the opportunity to work on with them. The showcase is a set of Google Earth based Tours for playing in a webpage (plugin required) or via download.

The Ocean Film Festival is going on until Sunday, and has a really interesting lineup - check it out if you're in the Bay Area. Either way, take a peek at some of the Tours: There's some really amazing content available for the Oceans layer in Google Earth that I was totally unaware of before looking more closely.

Fun with Layars

Last night I installed Layar on my phone, and had some fun checking out the twitter and wikipedia layers. So I signed up for an API key, and 30 seconds later saw a tweet mentioning the California Data Camp. Perfect! After a rare and blissful sleep-in, I wandered over to see what was going on at Citizen Space, thinking I'd try get a proof of concept demo showing some City of SF data in Layar.

Turns out, despite a number of interesting conversations taking precedence over my coding, I managed to get a simple demo working, and even win Honorary Mention (and an iPod touch) for my efforts. And a couple of Layars (crime data and handicapped parking spaces) are just waiting for publishing approval from Layar, and will hopefully be available in a few hours. Just search for "datasf" in your Layar app.

Since GeoDjango was the reason I was able to get a mockup going so quickly, I thought I'd just write a few short notes on the steps I took to build the Layar compatible API, and make the code available. Note that the code here is not particularly pretty - it's the result of a partial afernoon of work (including finding/downloading the layers, going over the layar api docs, and dealing with incredibly spotty internet connectivity). Nonetheless it may be interesting to some geodjango newbies as an example of a quick proof of concept.

The homepage is at http://code.google.com/p/geodjango-layar/, and the wiki includes some play-by-play instructions if you're just getting started with this stuff. Enjoy!

[EDIT]: I now see that someone else (sunlight foundation) had already built a more robust generic view for django, called django-layar. You should definitely use theirs instead .. but if you're curious for more ways to load spatial data into geodjango you might find my notes on google code interesting anyway.