Porcupine Alley

Last week I updated Shape Escape to convert shapefiles to GeoJSON and TopoJSON , in the hopes of making it easier for developers to quickly get that web-unfriendly shapefile into some clientside useful vector format. First, it's worth noting that TopoJSON has got a bunch of webmap folks excited. And rightfully so. Many people seem excited because topology (although I haven't seen many non-demo sites taking advantage of this yet). And also because it advertises a more compact representation of data. Clearly if the topology it makes available will help your visualization, TopoJSON is the way to go. But what's this about a smaller representation? As noted on the wiki page , 'TopoJSON can also be more efficient to render since shared control points need only be projected once. To further reduce file size, TopoJSON uses fixed-precision delta-encoding for integer coordinates rather than floats. This eliminates the need to round the precision of coordinate values...

I've played briefly with TileMill before, but after learning more about the advances that Development Seed is putting into their MapBox stack (such as Node.js integration with Mapnik, utfgrid and more) I realized it was time to sit down and play with it for real. My main interest right now is in getting a feel for the CSS-like syntax used in Carto , but as long as I needed to set up a full instance of TileMill to play with I figured I might as well make it into an Amazon EC2 AMI, so anyone can easily boot up an instance and get started. Setup was extremely simple (thanks to Dane telling me how to cut and paste their very straightforward install instructions; not that he noticed I was installing everything to /tmp). After some much needed sleep and attempt #2 here on the plane, you can now go to the Amazon AWS console and load up ami-56ae563f, or search for tilemill, and you're done. Just wait for the instance to start up, and TileMill should be running on port 80...

Well it's been a while, so just a quick note: Since the last post, I started working full time for Google. And with that out of the way, here's a post on how and why I made shpescape.com , which lets you upload shapefiles to Google Fusion Tables . Why shpescape? Google Fusion tables makes it easy to import and visualize data from spreadsheets and KML, and while it has increasingly robust spatial support it does not currently let you upload shapefiles directly. And since shapefiles are still incredibly common in the wild, I thought I'd make a quick tool to let people upload shapefiles to Fusion Tables. Which platform? I thought I'd try Google App Engine to avoid any hosting costs (given this will likely not be an extremely popular website), but while there's a decent shapefile reader or two for python, there's not a lot of support for things like reprojecting and other geometry manipulation without additional c++ libraries that App Engine won't...

Last night at the San Francisco Ocean Film Festival Google launched the Oceans Showcase , which is the second contract I've had the opportunity to work on with them. The showcase is a set of Google Earth based Tours for playing in a webpage (plugin required) or via download. The Ocean Film Festival is going on until Sunday, and has a really interesting lineup - check it out if you're in the Bay Area. Either way, take a peek at some of the Tours: There's some really amazing content available for the Oceans layer in Google Earth that I was totally unaware of before looking more closely.

Last night I installed Layar on my phone, and had some fun checking out the twitter and wikipedia layers. So I signed up for an API key, and 30 seconds later saw a tweet mentioning the California Data Camp . Perfect! After a rare and blissful sleep-in, I wandered over to see what was going on at Citizen Space, thinking I'd try get a proof of concept demo showing some City of SF data in Layar. Turns out, despite a number of interesting conversations taking precedence over my coding, I managed to get a simple demo working, and even win Honorary Mention (and an iPod touch) for my efforts. And a couple of Layars (crime data and handicapped parking spaces) are just waiting for publishing approval from Layar, and will hopefully be available in a few hours. Just search for "datasf" in your Layar app. Since GeoDjango was the reason I was able to get a mockup going so quickly, I thought I'd just write a few short notes on the steps I took ...

The upcoming Map Kibera project acquired some imagery recently, and I got ahold of it yesterday to set up a quick tilecache preview. There's actually been quite a few requests here recently for getting some tiles up quickly from various sets of source imagery, so I thought I'd write a few blog posts on some different ways to go about it. First, I'm assuming the end user will be requesting tiles, and that these tiles will be projected in Spherical Mercator for viewing on the web in a browser like OpenLayers or Google Maps (so I'm skipping over the bits for creating tiles that might be used in a browser like Google Earth). With that in mind, there are a few ways to get your tiles.  Note that the Kibera imagery is a nice simple example, because the area of the imagery is not that large (about 25 square km), and the source file is only a couple hundred megs as an uncompressed TIF. Option A: Pre-generate all your tiles in advance The easiest way to generate all y...

AppEngine is awesome. The more we use it, the more we like it. Recently, someone contacted us who needed a site up, in a hurry, to serve up some points on a google map. The catch was there were about 50k points (so it seemed server side clustering might be nice). Also they wanted to be able to serve up at _least_ tens of millions of requests a day. And maybe quite a lot more. Given the scaling requirements, it seemed like AppEngine might be a nice fit, since then we wouldn't have to worry so much about tons of caching, or ensuring clients made similar bounding box requests, and so forth. And as for the posting/getting of points to/from appengine, we decided to go for using FeatureServer as a base. If you're not familiar with featureserver, a quick overview: It makes it easy to (amongst other things) post/update your features to some datastore, and pull them out with bounding box and/or attribute queries in a variety of vector formats (kml, json, wfs, etc). Al...

A few months ago, Walkscore.com launched their new API , aimed a Real Estate sites, academic and other largescale studies, and anyone else who might want programmatic access to the walkability of a set of locations. Recently, a number of large sites ( zillow.com , Estately.com , BaseEstate.com , and ColoProperty.com amongst many others) have started to incorporate walkscore in their listings, which has led to the happy situation of ensuring the site can handle the popularity. When the good folks at Walkscore first contacted me about designing the API, we talked about the likelyhood we'd be serving many, many millions of daily requests shortly after launch. This quickly led into discussions about what framework we wanted to build it on, and how much IT we were interested in taking on. Eventually, we decided to use Google's AppEngine . Given the current, and quickly growing, popularity of the API, I think this ended up being a great choice - no worries about having to even...

Earlier this year, I read this post on local foodsheds, and got to thinking that there must be a good way to automate and generalize this type of analysis.  So I put the idea into the big bucket of ideas that I don't have time to implement, until it came up in a conversation with Brent Pedersen .   He mentioned a great  way to optimize the raster summarization that I hadn't thought of, so we got excited, and put it into the bucket of ideas that we'd like to implement (but probably wouldn't).  Until finally, a few weeks ago, we decided to get together for a Sunday hackfest. A few hours later, with a good percentage of my time wasted learning the strange ways of git , we had a functional demo that summarized NLCD data by class for any given buffer or polygon, in a simple Django framework. To get things vaguely useable, we added a pie chart to show the classified summary, and integrated some census data to get the population.  Our initial demo is now at lan...

At the same time I was taking a great nature hike just before the start of FOSS4G, there was a mapping party in Houte Bay .  The party clearly came out a success, and thanks to Mikel for passing on the contact info of the gracious hosts at Tobi, I was invited out to the area today to continue the work that was started the other week. This morning, Sebastian, Grant, Belinda, and I started out in Imizamu Yethu (sometimes called IY, or more recently Mandela Park), the local township.  With the excellent guidance of Afrika Mone, we got the main road and a few side streets, stopping by a few local spots, including a tavern , preschool , museum, community center , and a surprisingly nicely set up computer lab .  With a little luck and communication, I'm optimistic some of the kids taking courses there will continue our efforts and get the rest of the township mapped.  As usual, the few folks I chatted with in the township were friendly, and did not seem to mind some to...

The last couple of days have yet again highlighted for me the importance of trying to take advantage of locals and local knowledge.  A few days ago I met Chris -- a local geology/botany expert, and friend of a friend that I haven't seen in years -- for a hike up Lions Head.  It was a perfect evening, and the 360 views of Table Mountain, the Apostles, the Atlantic, the city and suburbs was almost overwhelming.  The perfect sunset didn't hurt either.  Chris mentioned the local mountaineering club was hosting a slideshow/video, so last night we checked it out.  A couple guys who were part of a group that did a mission to climb the three towers in Southern Patagonia recounted a humerous, personable, and mindblowing story of their trip, bringing back for me good memories of (much easier) trips I've done in the past, and getting me quite excited to get back in shape and outdoors more in the future.  After chatting with a few of the local mountaineer members, we h...

The presentations for FOSS4G are over, and just a few workshops and the code sprint are left in the official conference.  This year I ran a workshop on GeoDjango with Chris Schmidt, and gave a presentation on the work I did recently with walkscore.com.  The workshop had a few technical glitches (no power, internet, or projector for the first bit), and as Chris mentioned, we probably should have practiced together in a fully offline mode beforehand.  On that note, thanks a lot to Chris for agreeing to help out as a replacement at the last minute -- it would have been quite a struggle without him there.  Still, we got through most of the material, and participants ended up with working admin editing for points and polygons, and some public facing pages that showed a good hint of GeoDjango's possibilities.  I learned quite a lot from this workshop, and am looking forward to repeating it in the future.  After fixing up a few  bugs in docs and vmware image,...

Yesterday Karen and Hanlie and I went on a nice walk down near Cape Point, with a group of local botany experts who meet once a month to wander around, rip out some invasive plants, and talk endlessly about the indigenous and endemic species.  It was a beautiful area, with views of the Atlantic and the bay, and I learned a lot about fynbos and local history.  I also saw a lot of tortoises, and these really fast caterpillers.  One of the guys brought along his grandkids, Gabriel and Ruben.  These kids were amazing at finding animals or animal parts (numerous porcupine quills, tortoise shell segments, owl eggs, etc), and it was a lot of fun to jump around on some large rocks again with people who truly enjoy that activity.  After the hike, I went got dropped off at the University of Cape Town campus, and headed to the Geomatics lab to help set up some of the workshops with the conference organizers.  A couple machines had mysterious issues installing vmware, ...

I arrived in Cape Town about 5am (dark outside), and stepped carefully back into Africa (stepped/stumbled:  they were unable to locate lights for the path from the plane to the terminal).  There wasn't much of a line for citizens in the passport-checking area, but that didn't stop at least a few people from wandering right to the front of the line, as if to cut, and then just continuing on right past the passport-checkers, who were unable to really notice due to the poor design of the booths they sit in.  Hopefully those guys have no trouble with lack of passport stamps on their next leg, or perhaps it's just a known thing here that there's no need to bother with the hassle of official entrance/exit.  Next up was a quick check on pricing for transportation:  For two people, it was R170 for the shuttle, or $240 for a cab.  The cab people were clearly not pleased with being undercut, but the shuttle was a great choice.  Despite my misdirections, Chris Ho...

  I had a great backpacking trip in the Southern Sierras (near Mt. Whitney in the John Muir Wilderness) a few weeks ago, and finally decided to see about getting some geocoded pics online. Since I've been so spoiled with my eye-fi card, I'm not used to needing to manually geocode my pics.  Still, I had my trusty old GPS with me for the trip, and I figured it shouldn't be too hard.  Here's what I did: Used GPS Babel to quickly export a .gpx of my tracks from the trip Ran the gpsPhoto perl script to geocode my photos:  ./gphoto.pl  --gpsfile hike08.gpx--timeoffset 28800 --maxtimediff 3600 --dir ./hike_photos/ Note:  the time offset (in seconds) is because my camera and I are set to -8 hours from GMT, and the time diff allows my photos to be up to an hour off from the closest gps point.  This is because I sometimes hiked with with the gps off.  Now that the photos were geocoded, I uploaded them to Flickr, and noticed th...

I finally had a chance to get my first flight in at Mussel Rock earlier today -- soaring over the coast was everything I had hoped for.  No need for a vario at this place; lift is everywhere.  So much lift, that when I was ready to land, it took judicious use of big-ears (which is very common to see out here, and clearly helpful to ensure you avoid blow-back ) to get past the incredible buoyancy I was getting.  I have no idea why some folks call it cliff-boring .. soaring is a ton of fun, and it's great to be a little more familiar with this site.

A few weeks ago, I purchased Eye-Fi's fancy geolocating and wireless-uploading SD Card , which I'd seen a demo of at WhereCamp . I love gadgets, perhaps too much, but this thing is indeed a great implementation of a cool idea. And, as yet another harbinger of how almost everything will soon be location aware, it has all kinds of potential. For example, devices like this could be used as the poor man's (no iphone) location updater.  Since they haven't opened up their API (allegedly it's on their to-do list), and because I've been looking for any excuse to play with FireEagle , I just finished up a little python script that queries my flickr photos every few minutes and (if there's a new one that eye-fi has uploaded), it grabs the WOEID and updates my FireEagle profile.  Working with the Fllickr and FireEagle API's was fun. Obviously a bit of a hack, but once they do open their API (so I don't have to pull for data), or otherwise allow us to ...

Ate lunch at Eric's today. Despite their scary restaurant inspection report ( EveryBlock reports are yet another nice feature of San Francisco), it was one of the better meals, and deals, I've had in a while. $6.50 total bill for a huge plate of tasty kung-pao goodness, soups, and tea. I'll be eating lunch here again.

I spent the morning at Mussel Rock Park , kiting and getting back up to speed on some ground handling basics with some good tips, and a great site overview from Jeff of Airtime San Francisco . I've never gone cliff soaring before, and I'm certainly not used to the fog! Starting around noon, there were a few gliders in the air, and I understand the weekends can be packed. If the weather looks good, I'm excited to get some flying time in early next week.

When I first saw walkscore.com I was impressed with their simple but clever approach to creating a walk index. Ranking walkability is an interesting topic for me: I've had the opportunity to work on a few different sprawl and walkability models in the past, but these often tended to be complex. And I don't mean they used tricky or especially unintuitive algorithms (although do believe this type of analysis is often ruined by unintuitive modeling). I mean complex in terms of the data sources. In the world of ranking walkability, many sources (assessor, zoning, sidewalk attributes, etc) that the more academic models tend to rely on are just not comparable from city to city. To me, one of the key features of the new version of walkscore is that it keeps true to it's initial approach by keeping it simple (business listings and census data), and using a straightforward methodology . By focusing on amenities and where people live, can a model like this be use...