Shape Escape

Well it's been a while, so just a quick note: Since the last post, I started working full time for Google. And with that out of the way, here's a post on how and why I made shpescape.com, which lets you upload shapefiles to Google Fusion Tables.

Why shpescape?

Google Fusion tables makes it easy to import and visualize data from spreadsheets and KML, and while it has increasingly robust spatial support it does not currently let you upload shapefiles directly. And since shapefiles are still incredibly common in the wild, I thought I'd make a quick tool to let people upload shapefiles to Fusion Tables.

Which platform?

I thought I'd try Google App Engine to avoid any hosting costs (given this will likely not be an extremely popular website), but while there's a decent shapefile reader or two for python, there's not a lot of support for things like reprojecting and other geometry manipulation without additional c++ libraries that App Engine won't run. So I just went for a simple GeoDjango app.

Authentication

I used my colleague Kathryn's Fusion Tables python client to handle the authentication (OAuth). And I decided against having OpenID in the mix as well for actually associating an account with various uploads. The downside is that you can't log in and view your previous uploads. But you can always go to the main Fusion Tables page to see all your tables, and the upside was one less thing for me to consider (for example, if you are logged in with multiple accounts in the same browser, OAuth does not return which account gave permissions for). [Edit: It turns out you can actually request the email address of an authorized user using the scope noted at http://sites.google.com/site/oauthgoog/Home/emaildisplayscope]

Handling a Shapefile Upload

I used a simple fork of Dane Springmeyer's django-shapes app to handle the shapefile import. The customizations let users upload a zipfile that has a shapefile in a subfolder, and/or multiple shapefiles in a single zip. I had never really noticed shapefiles being zipped up this way, and it really surprised me how common these scenarios are with shapefiles from various US counties and other agencies -- my first 3 test users all had their uploads fail until I added this. After the upload is verified as valid, it creates a shapeUpload object, which is processed separately so the end user can view a reloading page with updated status.

Processing the Upload

My initial attempt was pretty straightforward:
  • Attempt to get the projection of the shapefile (from the .prj)
  • For each feature, get it's KML and attributes
  • Upload 'em to Fusion Tables, ensuring a max of a few hundred, or <1MB, at a time (the API can handle at most 500 rows and 1MB per POST)


Additional Features

Next up, I started adding a few extra bits, which led to an import method begging for a refactor.
  • Simplification for KML over 1M characters long (which is the max characters allowed by Fusion Tables per cell)
  • Process/Upload 10k rows at a time (so we don't use too much memory for very large shapefiles)
  • Added numeric styling columns for string fields that don't have too many unique values (Fusion Tables only allows robust styling like gradients and buckets on numeric fields)
  • Allow users to specify some additional geometry columns:
    • Simplified Geometry
    • Centroid (only works for polygon shapefiles)
    • Buffered Centroid (so you can apply the more robust polygon styling rules on the 'centroid')


Finishing up

This whole project was a pretty quick attempt at what I hope is a useful solution to a common problem, so any comments on how to make it better are appreciated. And if you want to see how it all works in more detail, I also open sourced the code. Enjoy!

Comments

Unknown said…
Hello. This is great. Though, it doesn't work well with non latin languages. Do you have any idea how to fix non latin characters?
priya sharma said…
Such A nice post... thanks For Sharing !!Great information for new guy like Happy New year 2020
anita said…
Such A nice post... thanks For Sharing !!Great information for new guy like Hanuman Chalisa Lyrics
archna said…
Good effort and This is a very valuable concept.Such A nice post... thanks For Sharing !!Great information for new guy like showbox for android
gadgetsdora said…
Your style is very unique compared to other folks I've read stuff from. Many thanks for posting when you have the opportunity, Guess I will just bookmark this site.
tech
centre99 said…
it is really a great and helpful piece of info. I am glad that you shared this helpful information with us. Please keep us informed like this. Thank you for sharing.
china embassy singapore
travel5212 said…
Amazing Article,Really useful information to all So, I hope you will share more information to be check and share here.thanks for sharing .
Website: Vietnam Cycling Tours

riya singh said…
I enjoyed over read your blog post. This was actually what i was looking for and i am glad to came here!
Website: artificial jewellery online
jenifer027 said…
MyAssignmentHelpNow provides assignment help service to the students of Australia...We are well known and most demanding writing service provider over there. If you are looking for the best writing service provider over there, then we are the one-stop-shop for you.

MyAssignmentHelp
anthonyhall said…
an interesting article, it was pleasant to read, well written, I myself sometimes write articles, and it helps me https://soclikes.com/
Banumadhu said…
Useful article which was very helpful. also interesting and contains good information.
to know about python training course , use the below link.

Python Training in chennai

Python Course in chennai
pkcartoon said…
Post is very good its amzazing post I love them thanks for sharing.
visit here - Desi Call Recordings in Hindi
rajmohan1140 said…
Always good to find good articles on the net. Thank you. This is very valuable thing shared, I thanked the writer.

Java Training in Chennai

Java Course in Chennai
it is really a great and helpful piece of info. I am glad that you shared this helpful information with us. Please keep us informed like this. Thank you for sharing.
Website: Online punjab state lottery
Ankit Ron said…
Very genuine post i love to share like this kind of article, eventually thanks for sharing. ball wala game
buyonline123 said…
Thanks for another great post. Where else could anyone get that kind of information in such a perfect way of writing? I've a presentation next week, and I'm on the look for such information.
buy electronics online
marketshare said…
I am new here. I like your post very much. It is very usefull post for me.
website: Global 5G Infrastructure Market
Anonymous said…
Great post and I got extremely information from your article. I appreciate your work and requesting you to please keep posting other articles.
Speedify Crack
Crack Software said…
I’m impressed, I must say. Genuinely rarely can i encounter a blog that’s both educative and entertaining, and without a doubt, you have hit the nail on the head. Your concept is outstanding; the issue is something that too little individuals are speaking intelligently about

Crack Software

PC License Key

World PC Crack
Gaurish said…
I always like to read a quality content having accurate information regarding the subject and the same thing I found in this post.

Website : Website Development Company |
Gaurish said…
Great information, i was searching of this kind of information, thank you very much for sharing with us.

Website : Craigslist Posting Service for Car Dealers |
Gaurish said…
This is really amazing website that I have been found on google regarding website Blog Commenting sites. and I would like to thank admin who also given us to post the link on his side.

Website : Lubbock moving company |
Gaurish said…
This was something I was looking for, really helpful, and great work is done. Thank you so much for sharing such valuable information.

Website : Car Auction Software |
Gaurish said…
It’s really a cool and helpful piece of information. I am glad that you shared this useful information with us. Please keep us up to date like this. Thanks for sharing.

Website : Best CRM for Small Businesses |
Gaurish said…
I am really like it very much for the interesting info in this blog that to this website is providing the wonderful info in this blog that to utilize the great technology in this blog.

Website : Web Development Company in Gwalior |
sattamatka25 said…

Thank you so much for sharing this article. I really love it. This is a very nice one and gives in-depth information. I am really happy with the quality and presentation of the article. I’d really like to appreciate the efforts you get with writing this post. Thanks for sharing.
website: Matka Result
Evo Syah said…
Evo Syah adalah seorang Freelancer Website Developer Bali, dia sangat menyukai hobbynya dalam menelusuri setiap blok yang Anda di internet. Dia memiliki banyak pengetahuan tentang Digital Marketing dan bagaimana penerapan dalam bisnis.
Evo Syah said…
Satya Puri Studio is Bali architects have worked on historically-inspired homes & villa around the world — including “Lux Villa in Bali; Sukh Sagar Haveli in Jodhpur, Rajasthan; Kahani Paradise in Gokarna. Their work shows inspiration across history and styles, tied together with the goal of enhancing life through good design. Above all, Satya Puri Studio Architects aims to produce personalized living environments.
Evo Syah said…
SEO adalah proses meningkatkan visibilitas dan peringkat website bisnis Anda di laman hasil penelusuran mesin telusur, seperti Google, Bing, dan Yahoo, sehingga memudahkan pelanggan menemukan produk atau layanan Anda secara online.

Kami menawarkan layanan Jasa SEO terbaik di Bali dengan Strategi Optimasi Situs Website secara tepat dan akurat sesuai dengan prinsip kerja mesin pencari.
Evo Syah said…
When you think Indonesia, you probably picture its Labuan Bajo, and sunny beaches in Canggu. Yet, many people don’t realize that Indonesia is also a great place to start a business. According to International Labour Organization, there are about 700 thousand small businesses in Indonesia in 2018, which 57 million are small and medium sized enterprises Start Business in Indonesia

But what makes Indonesia so great? Major cities like Jakarta, Bandung, Yogyakarta, and Surabaya, an impressive university system, and low taxes means the Sunshine country is ripe with opportunity for savvy business owners. Keep reading to learn more about why Indonesia could be the perfect spot to start your new business!
crack software said…
Remarkable! Its in fact remarkable paragraph, I have got much clear idea concerning from this article.

explorecrack.com

brave-browser-crack

bandicam-crack-product-key

wondershare-filmora-crack
Marty Sockolov said…
CBD isolates. CBD (cannabidiol) is an active ingredient found in th CBD Cannabis plant that does not cause psychoactive "high". Instead, the active ingredient provides soothing energy, such as a stronger Ginkgo Biloba tea
Rani tyagi said…
I just would like to give a huge thumbs up for the great info you have here on this post. Thanks for sharing with us. Are you looking for Escort Services in Ahmedabad and Call Girls in Ahmedabad
Click For More Information Hire Call Girls Ahmedabad
Including fast withdrawals With a system that has high precision, meets international standards,can start playing for free. You will find the rules and the hand rank. Find other poker games that help hone your skills withPlay Money Texas Hold'em, Omaha and or join real money games. There is no better place to learn and play poker from us.

Marty Sockolov said…
it is not the most popular on this list, but fans do love its benefits ดูบอลฟรี . One of these is that no sign up is required, but you can sign up for free to get some convenient options. For instance, it provides intelligent content recommendations and a calendar for you to set up reminders. That way, you never miss your favorite live content. If you do miss it, though, you can always pull it up on Duball24hrs.com recordings.

Popular posts from this blog

TopoJSON

Featureserver on AppEngine