Friday, February 11, 2011

Extract POIs from OSM PBF ("Protocolbuffer Binary Format") dumps with python

  • I am a GIS n00b - this is my first attempt at handling OSM data.
  • This code would have never been written if it wasn't for Chris Hill's excellent parsepbf.py - blog post here.


I looked at OSM to obtain railway station locations in the country for an in-house project we are running.Parsing through their data dumps sounded like an easy job. I grabbed india.osm.bz2 and india.osm.pbf from Geofabrik. Uncompressing the bz2 file resulted in a 614MB xml whereas the pbf was just 26MB. Intrigued by the small file size of the pbf files ( I never read up on google protocol buffers before) I went to the OSM wiki to read up the format and see if any python libraries are available for this. I found Chris' parsepbf script and ran it with the pbf I had. Turns out running the the script without asking it to spit out osm xml was a bad idea - ended up eating all the memory on my machine [ no swap enabled ] and crashing the system.

I modified the parsepbf file to make as somewhat generic class for picking out nodes with specified tags.


Stats:

  • It took about 5 minutes to pickout all the railway stations on my linode ( 512MB ) VPS.
  • I think a speed up can be achieved by using multiprocessing (?)


Example usage:



Current code can be found here.

21 comments:

Chris Hill said...

Thanks for the mention.

I think I need to put out a warning on the raw code about gobbling memory. The main reason I created the code was to do exactly as you have done - customise it to extract data you are interested in.

Tao said...

Hi


I use your code to see if it works on the map of philadelphia. However, it always return the following error message:

Traceback (most recent call last):
File "", line 1, in
tags = foo.return_tags(refresh=True)
File "osmnodepbf.py", line 216, in return_tags
self.parse()
File "osmnodepbf.py", line 113, in parse
self.processDense(pg.dense,tag)
File "osmnodepbf.py", line 199, in processDense
self.tags[node["sky"]] = [node["svl"]]
KeyError: 'svl'



I tried it on some other maps and it keep prompting the same error

rose said...

Nice blog has been shared by you. it will be really helpful to many peoples who are all working under the technology.thank you for sharing this blog.

selenium training in chennai|

Adam lee said...

Nice Blog!

big data training in chennai

Careen joseph said...

Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
Besant technologies Marathahalli

Inigo joseph said...

Thanks a lot very much for the high your blog post quality and results-oriented help. I won’t think twice to endorse to anybody who wants and needs support about this area. rprogramming training in bangalore

ASO Services said...

Thanks a lot for sharing this with all of us, I like it and we can communicate. Do you need buy app ratings and reviews. To boost app ranking and double app downloads now.

SLA said...

Thank you for providing such a valuable information.

Big Data Training in Chennai
Hadoop Training in Chennai

Dhileepan K said...

Awesome Blog, I Loved it, Me first time here in the Blog. Totally Impressed.
Andorid Training in Chennai | Android Training Institute in Chennai

Unknown said...

Thank you for sharing wonderful information with us to get some idea about that content. check it once through Best AWS Training in chennai | AWS Training institute in velachery | AWS Certification course

Sakthi Murugan said...


Excellent content!!! your content is a great inspiration for writers.


Hadoop Admin Training in Chennai
big data training institute in chennai
hadoop big data training in chennai
Hadoop Training in Chennai
Big Data Training in Chennai
JAVA Training in Chennai
Selenium Training in Chennai

yuva prithika said...


Learned a lot from your blog. Good creation and hats off to the creativity of your mind. Share more like this.
Loadrunner Training in Chennai
French Classes in Chennai
Android Training in Chennai
Digital Marketing Course in Chennai
selenium certification in chennai

Chris Hemsworth said...

The article is so informative. This is more helpful. Thanks for sharing


software testing training institute
Selenium training in chennai

Naveen S said...

Great Blog!!!
Digital Marketing Course Training Institute in Chennai

Big Data Course Training Institute in Chennai

Data Science Course Training Institute in Chennai

Android Training Institute in Chennai

AngularJS Training Institute in Chennai

Anonymous said...

The article is Awesome!!!!

Digital Marketing Course Training Institute in Chennai

SEO Course Training Institute in Chennai

Chris Hemsworth said...

The article is so informative. This is more helpful. Thanks for sharing.

Learn best software testing online certification course class in chennai with placement
Best selenium testing online course training in chennai
Best online software testing training course institute in chennai with placement

zuan said...

I like the helpful info you provide in your articles. I’ll bookmark your weblog and check again here regularly. I am quite sure I will learn much new stuff right here! Good luck for the next!
Web Designing Training Institute in Chennai | web designing and development course | web design and programming courses
Web Designing Course in Chennai | Web Designing Training in Chennai
Web Designing Training Institute in Chennai | Web Designing courses in Chennai with Placement
Mobile Application Development Courses in chennai

zuan said...

I like the helpful info you provide in your articles. I’ll bookmark your weblog and check again here regularly. I am quite sure I will learn much new stuff right here! Good luck for the next!
Web Designing Training Institute in Chennai | web designing and development course | web design and programming courses
Web Designing Course in Chennai | Web Designing Training in Chennai
Web Designing Training Institute in Chennai | Web Designing courses in Chennai with Placement
Mobile Application Development Courses in chennai

Lovable vicky said...



Nice infromation
Selenium Training In Chennai
Selenium course in chennai
Selenium Training
Selenium Training institute In Chennai
Best Selenium Training in chennai
Selenium Training In Chennai

Lovable vicky said...




Rpa Training in Chennai
Rpa Course in Chennai
Rpa training institute in Chennai
Best Rpa Course in Chennai
uipath Training in Chennai
Blue prism training in Chennai

Data Science Training In Chennai
Data Science Course In Chennai
Data Science Training institute In Chennai
Best Data Science Training In Chennai

Lovable vicky said...


Python Training In Chennai
Python course In Chennai
Protractor Training in Chennai
jmeter training in chennai
Loadrunner training in chennai