Shape Escape

Well it's been a while, so just a quick note: Since the last post, I started working full time for Google. And with that out of the way, here's a post on how and why I made shpescape.com, which lets you upload shapefiles to Google Fusion Tables.

Why shpescape?

Google Fusion tables makes it easy to import and visualize data from spreadsheets and KML, and while it has increasingly robust spatial support it does not currently let you upload shapefiles directly. And since shapefiles are still incredibly common in the wild, I thought I'd make a quick tool to let people upload shapefiles to Fusion Tables.

Which platform?

I thought I'd try Google App Engine to avoid any hosting costs (given this will likely not be an extremely popular website), but while there's a decent shapefile reader or two for python, there's not a lot of support for things like reprojecting and other geometry manipulation without additional c++ libraries that App Engine won't run. So I just went for a simple GeoDjango app.

Authentication

I used my colleague Kathryn's Fusion Tables python client to handle the authentication (OAuth). And I decided against having OpenID in the mix as well for actually associating an account with various uploads. The downside is that you can't log in and view your previous uploads. But you can always go to the main Fusion Tables page to see all your tables, and the upside was one less thing for me to consider (for example, if you are logged in with multiple accounts in the same browser, OAuth does not return which account gave permissions for). [Edit: It turns out you can actually request the email address of an authorized user using the scope noted at http://sites.google.com/site/oauthgoog/Home/emaildisplayscope]

Handling a Shapefile Upload

I used a simple fork of Dane Springmeyer's django-shapes app to handle the shapefile import. The customizations let users upload a zipfile that has a shapefile in a subfolder, and/or multiple shapefiles in a single zip. I had never really noticed shapefiles being zipped up this way, and it really surprised me how common these scenarios are with shapefiles from various US counties and other agencies -- my first 3 test users all had their uploads fail until I added this. After the upload is verified as valid, it creates a shapeUpload object, which is processed separately so the end user can view a reloading page with updated status.

Processing the Upload

My initial attempt was pretty straightforward:
  • Attempt to get the projection of the shapefile (from the .prj)
  • For each feature, get it's KML and attributes
  • Upload 'em to Fusion Tables, ensuring a max of a few hundred, or <1MB, at a time (the API can handle at most 500 rows and 1MB per POST)


Additional Features

Next up, I started adding a few extra bits, which led to an import method begging for a refactor.
  • Simplification for KML over 1M characters long (which is the max characters allowed by Fusion Tables per cell)
  • Process/Upload 10k rows at a time (so we don't use too much memory for very large shapefiles)
  • Added numeric styling columns for string fields that don't have too many unique values (Fusion Tables only allows robust styling like gradients and buckets on numeric fields)
  • Allow users to specify some additional geometry columns:
    • Simplified Geometry
    • Centroid (only works for polygon shapefiles)
    • Buffered Centroid (so you can apply the more robust polygon styling rules on the 'centroid')


Finishing up

This whole project was a pretty quick attempt at what I hope is a useful solution to a common problem, so any comments on how to make it better are appreciated. And if you want to see how it all works in more detail, I also open sourced the code. Enjoy!

Comments

Unknown said…
Hello. This is great. Though, it doesn't work well with non latin languages. Do you have any idea how to fix non latin characters?
archna said…
Good effort and This is a very valuable concept.Such A nice post... thanks For Sharing !!Great information for new guy like showbox for android
gadgetsdora said…
Your style is very unique compared to other folks I've read stuff from. Many thanks for posting when you have the opportunity, Guess I will just bookmark this site.
tech
jenifer027 said…
MyAssignmentHelpNow provides assignment help service to the students of Australia...We are well known and most demanding writing service provider over there. If you are looking for the best writing service provider over there, then we are the one-stop-shop for you.

MyAssignmentHelp
pkcartoon said…
Post is very good its amzazing post I love them thanks for sharing.
visit here - Desi Call Recordings in Hindi
rajmohan1140 said…
Always good to find good articles on the net. Thank you. This is very valuable thing shared, I thanked the writer.

Java Training in Chennai

Java Course in Chennai
Ankit Ron said…
Very genuine post i love to share like this kind of article, eventually thanks for sharing. ball wala game
marketshare said…
I am new here. I like your post very much. It is very usefull post for me.
website: Global 5G Infrastructure Market
Anonymous said…
Great post and I got extremely information from your article. I appreciate your work and requesting you to please keep posting other articles.
Speedify Crack
Gaurish said…
It’s really a cool and helpful piece of information. I am glad that you shared this useful information with us. Please keep us up to date like this. Thanks for sharing.

Website : Best CRM for Small Businesses |
Including fast withdrawals With a system that has high precision, meets international standards,can start playing for free. You will find the rules and the hand rank. Find other poker games that help hone your skills withPlay Money Texas Hold'em, Omaha and or join real money games. There is no better place to learn and play poker from us.

Marty Sockolov said…
it is not the most popular on this list, but fans do love its benefits ดูบอลฟรี . One of these is that no sign up is required, but you can sign up for free to get some convenient options. For instance, it provides intelligent content recommendations and a calendar for you to set up reminders. That way, you never miss your favorite live content. If you do miss it, though, you can always pull it up on Duball24hrs.com recordings.

Along with it also has a live casino 24 hours a day. สมัคร ufa Bet online gambling website is available to play with all casinos, baccarat, slots, roulette, dice and many other card games. We have gathered it here. With many promotions Apply for UEFA Bet now, we are happy to serve you all. With a professional team ready to solve problems And serve all of you There is a team to support the service. And answer your questions 24 hours a day.
Scottishkilt said…
Great article! This is the type of information that are meant to
be shared across the internet. Thank you for sharing such a useful post. Very Interesting Post! I regularly follow this kind of Blog.
visit this page
Cracked Apps said…



I’m amazed, I have to admit. Rarely do I come across a blog that’s both equally educative and entertaining, and let me tell you, you have hit the nail on the head.Thanks a lot for sharing.
Freemake Video Converter Crack
Spotify Crack
EaseUS Todo Backup Crack
Smart Game Booster Crack
Malwarebytes Crack
Guitar Pro Crack
Recover My Files Crack
Device Doctor Pro Crack
thomas said…
Here's an example: Let's say you use your CareCredit card to pay for a $3,000 dental bill. You qualified for a 24-month 0% annual percentage rate promotional period. You do your best to pay it off, but there's still $1,000 left after two years.

fishyfacts4u.com
fishyfacts4u.com
fishyfacts4u.com
fishyfacts4u.com
fishyfacts4u.com
fishyfacts4u.com
fishyfacts4u.com
fishyfacts4u.com
fishyfacts4u.com
fishyfacts4u.com
thomas said…
Best Airline Credit Cards: U.S. News’ overall issuer satisfaction rating, ease and flexibility of rewards redemption, airline rewards earning rate, rewards redemption value, annual fee, APR, sign-up bonus value, cardholder benefits, foreign transaction fee, and balance transfer fee.

inewshunter.com
inewshunter.com
inewshunter.com
inewshunter.com
inewshunter.com
inewshunter.com
inewshunter.com
inewshunter.com
inewshunter.com
inewshunter.com
thomas said…
"Everything about the way we operated (this year) was a little bit different," Weston says. "I would say to my clients: Anytime you are anticipating or coming into a major life change, call me. Say, 'Hey Cari, we're moving, we want to sell a house, we want to buy a new car.' Those are the kinds of things that your client only has every five to 10 years. But then, think of how many of those things people were going through last year."

juicyfactor.com
juicyfactor.com
juicyfactor.com
juicyfactor.com
juicyfactor.com
juicyfactor.com
juicyfactor.com
juicyfactor.com
juicyfactor.com
juicyfactor.com
thomas said…
Joshua Zimmelman, president of Westwood Tax & Consulting LLC in Rockville Centre, New York, says, "Employees have limits to what expenses they can deduct, but if you're an independent contractor or freelancer, you can deduct many business expenses. For a Lyft or Uber driver, for example, this might include gas, parking fees, car maintenance and repairs, auto insurance and any other expenses you incur for your car."

localnewsbuzz.com
localnewsbuzz.com
localnewsbuzz.com
localnewsbuzz.com
localnewsbuzz.com
localnewsbuzz.com
localnewsbuzz.com
localnewsbuzz.com
localnewsbuzz.com
localnewsbuzz.com
John P. Ketchum said…
There are puzzles on the app you solve using code, and there's also a blank coding editor to create your own apps and games. You can even submit your creation directly to the App Store using Swift Playgrounds. The application is 100% free and guides you step-by-step on how to create better apps using Swift.

Gadgets
Gaming
Health & Fitness
Lifestyle
Movie
Music
News
Nutrition
sam said…
Riverside Chevrolet is one of the best Used Car Dealers Peoria Il. Visit our website and check out our inventory of new and used vehicles. We are your one-stop shop for all your automobile needs.
Visit here - Used Car Dealers Peoria Il
Edison hope said…
Awesome! I thank you your input to this matter. It has been insightful. Thanks. Just keep posting like this.. Eligible travelers can obtain electronic Visa Turkey by filling an online form with their personal details and passport information.
Tom said…
Thank you for this work.. The UK citizens traveling to Azerbaijan can apply for online Azerbaijan visa for UK citizens. Get visa just in 3 steps apply online, upload document, and make payment.
essay writer said…
Your style is very unique compared to other folks I've read stuff from. Many thanks for posting when you have the opportunity, College online
essay writer said…
Thanks for Sharing this Valuable Information with us: this is very useful for me. Keep it Up
writing tutor
deepaks42 said…
Hello there, I discovered your site by way of Google at the same time as looking for a
comparable matter, your site got here up, it seems to be good.
clean and clear face wash
manmeet1 said…
51 Lakhs distributed over prize money. Gandhi Brother’s lottery is Authorized and legal, assuring each player of a fair playing lottery.
nagaland lottery sambad

mounimeera said…
Thank you for giving good services https://caterersinhyderabad.in/"> Caterers in Hyderabad

delvers Allison said…
Wow.. Very informative article thanks for sharing please keep it up.. Pakistan visa fees depend on your visa type and nationality.

Maradona Jons said…
เล่นสล็อตออนไลน์ที่ดีที่สุดในโลก ทั้งหมดในที่เดียวโดยไม่ต้องดาวน์โหลดที่ BK8 – ค้นพบเกมสล็อตที่น่าตื่นเต้นและน่าดึงดูดที่สุด เสริมด้วยกลไกที่ยอดเยี่ยม ภาพที่สวยงาม และมีโอกาสชนะรางวัลแจ็คพอตสุดพิเศษ!สมัคร bk8thai
rom52 said…
Thank You for Providing Such insightful information. If someone is looking for the दिल जलाने वाली शायरी | shayari for true love | शायरी दमदार.
Looking for finance assignment writing help, and it's becoming difficult to find one just try contacting assignment achievers top experts at your help at an affordable price.
https://www.assignmentachievers.com/service-detail/finance+assignment+help
https://www.assignmentachievers.com/service-detail/criminal+law+assignment+help
Maradona Jons said…
We will be adding more UK No Deposit Bonuses soon. Keep a look out as the latest UK no deposit casino bonuses appear here offering free spins with zero deposit and no deposit UK cash bonuses.http://E-Vegas.com
Maradona Jons said…
Dikenal sebagai bandar judi online terbaik tentunya Anda wajib tahu apa saja kelebihan kami. Dibawah ini kami akan menjabarkan satu persatu kelebihan kami. Slot deposit dana


Ronnie said…
Awesome dispatch! I am indeed getting apt to over this info, is truly neighborly my buddy. Likewise fantastic blog here among many of the costly info you acquire. Reserve up the beneficial process you are doing here. hanumanchalisalyrics
Sophia said…

Good morning sir, this is great. If you want to learn more about Turkish visas you can visit on our e visa Turkey website. And read here all info about Turkey evisa online.

Phuwadon said…
ฟีเจอร์การทำเงินผ่านทางเว็ปสล็อตออนไลน์ สล็อต JOKER อีกหนึ่งเว็ปพนันออนไลน์ทำเงินออนไลน์ ที่สามารถทำเงินได้อย่างมากที่สุดเท่าที่เคยมีมา เงินรางวัลโบนัสแตกง่าย โบนัสแจ็คพอตสูง.
Unknown said…
post... thanks For Sharing !!Great information for new guy like Bhagwat Geeta in Hindi PDF
Unknown said…
post... thanks For Sharing !!Great information for new guy like anuvad kise kahate hain
Anonymous said…
This comment has been removed by a blog administrator.
Anonymous said…
Great post, thanks for sharing important information, keep us posted more Python Training in Pune
Best IT Training Provider
charissa said…
i never know the use of adobe shadow until i saw this post.ปลาฉลามวาฬ
pg slot said…
ทดลองเล่น pg slot เว็บเกมสล็อตออนไลน์น้องใหม่จากค่าย พีจีสล็อต ค่ายเกมสล็อตออนไลน์อันดับ 1 ที่ลูกค้าทั่วโลกต่างยกย่องให้เป็นเว็บเกมสล็อตออนไลน์อันดับ 1 ตลอดการ

Popular posts from this blog

Walk On

Featureserver on AppEngine