Friday, February 11, 2011

Extract POIs from OSM PBF ("Protocolbuffer Binary Format") dumps with python

  • I am a GIS n00b - this is my first attempt at handling OSM data.
  • This code would have never been written if it wasn't for Chris Hill's excellent - blog post here.

I looked at OSM to obtain railway station locations in the country for an in-house project we are running.Parsing through their data dumps sounded like an easy job. I grabbed india.osm.bz2 and india.osm.pbf from Geofabrik. Uncompressing the bz2 file resulted in a 614MB xml whereas the pbf was just 26MB. Intrigued by the small file size of the pbf files ( I never read up on google protocol buffers before) I went to the OSM wiki to read up the format and see if any python libraries are available for this. I found Chris' parsepbf script and ran it with the pbf I had. Turns out running the the script without asking it to spit out osm xml was a bad idea - ended up eating all the memory on my machine [ no swap enabled ] and crashing the system.

I modified the parsepbf file to make as somewhat generic class for picking out nodes with specified tags.


  • It took about 5 minutes to pickout all the railway stations on my linode ( 512MB ) VPS.
  • I think a speed up can be achieved by using multiprocessing (?)

Example usage:

Current code can be found here.


Chris Hill said...

Thanks for the mention.

I think I need to put out a warning on the raw code about gobbling memory. The main reason I created the code was to do exactly as you have done - customise it to extract data you are interested in.

Tao said...


I use your code to see if it works on the map of philadelphia. However, it always return the following error message:

Traceback (most recent call last):
File "", line 1, in
tags = foo.return_tags(refresh=True)
File "", line 216, in return_tags
File "", line 113, in parse
File "", line 199, in processDense
self.tags[node["sky"]] = [node["svl"]]
KeyError: 'svl'

I tried it on some other maps and it keep prompting the same error