You are not logged in.

Announcement

*** NOTICE: forum.openstreetmap.org is being retired. Please request a category for your community in the new ones as soon as possible using this process, which will allow you to propose your community moderators.
Please create new topics on the new site at community.openstreetmap.org. We expect the migration of data will take a few weeks, you can follow its progress here.***

Pages: 1

#1 2016-06-22 09:34:48

will#
New Member
Registered: 2016-06-22
Posts: 1

Hadoop and OSM

Hello,

Some of you might be interested in a project I've been working on, processing OSM planet files on Hadoop:

https://github.com/willtemperley/osm-hadoop

It's geared toward a quite specific task: from a planet.pbf file, extracting and rasterizing the linear features with a specific tag. We needed to do this for all highways and railways as an input to an accessibility model.

We've leveraged the Osmosis pbf2 library to perform the deserialization which has just worked, excepting that seeking between file blocks is impossible - see the readme for the workaround.

I'd be interested to hear anyone else's experience processing OSM data using big data tech.  I'd be interested in working on a more generic framework, perhaps with support for Apache Hive or other analytic frameworks.

Any ideas on how to take things forward, formats to support etc would be of interest.

Best, Will

Offline

#2 2016-06-26 19:20:24

mmd
Member
Registered: 2010-11-06
Posts: 2,150

Re: Hadoop and OSM

Also posted here: https://lists.openstreetmap.org/piperma … 29334.html

Offline

Pages: 1

Board footer

Powered by FluxBB