Managing Trieste OSM planet data

From Open source mapmaking technologies
Jump to: navigation, search

The goal

For many regions and countries of the world, there are fresh planet extracts from OpenStreetMap, from well known services, like Geofabrik

Sometimes, this planet file is too large and too complex.

In this exercise, we will learn how to manage such data.

Our goal is to have a daily update of the information of OpenStreetMap regarding the Trieste province.

Basically, we will need a clipping polygon (of Trieste area) and the OpenStreepMap from Italy.

We will create a script that will run every day, fetching the data and clipping it, to keep just the Trieste data.

To create the clipping polygon we will use QGIS with one additional plgin.

Afterwards, we only use command line instructions. Command line instructions are more adequated to create scripts, that can be scheduled to run at specific hours or intervals.

Create the clipping polygon

The Administrative Divisions were downloaded from Global Administrative Areas. You can download China Administrative divisions from GDA (in the Geopackage file format).

Alternatively, you can downloaded a copy from

Using QGIS

After download the archive, go to the Downloads folder and extract the files there. We will use the extracted CHN_adm.gpkg file.

Drag the CHN_adm.gpkg file into a new QGIS project.

Select just the CHN_adm1 layer on the Select vector layers to add... dialog.

Select the Beijing municipality polygon

Open the attribute table, and select the Beijing polygon.

Create a buffer around it

Around the selected polygon, create a buffer of 0.009 (map units). This means a distance of about 1 km. Use Vector → Geoprocessing Tools → Fixed Distance Buffer

QGIS buffer around Beijing.png

We can simplify the buffer created, to get a polygon with less points.

Using the Simplify tool (Vector → Geometry Tools → Simplify Geometries...), we create another shapefile, using the 0.002 value for the tolerance.


Export buffer as POLY

Install osmpoly_export plugin in QGIS.

  • select the feature to export to poly
  • select the column NAME_2 to supply the name to the POLY file header

Using JOSM

The shapefile created can be opened in JOSM, and then exported as POLY. To do so, you need two JOSM plugins: Opendata (to read shapefiles) and poly (to export to POLY file format).

Launch JOSM (Geospatial → Navigation and Maps → OpenStreetMap → JOSM).

Open the terminal. Get the last JOSM version and open it with:

java -jar josm-latest.jar

Go to Edit → Preferences, and then plugins, and then Download list.

Find Opendata, select it and press Update plugins.

Find Poly, select it and press Update plugins.

Finally, open the shapefile with the simplified geometry. Save it, using the *.poly format.

JOSM creating beijing poly.png

The resulting file is beijing.poly.

Clip the planet file

The clipping can be done using osmosis.

osmosis --read-pbf file=../italy-latest.osm.pbf --bounding-polygon file=Trieste.poly --write-pbf trieste-latest.osm.pbf
osmosis --read-pbf file=../italy-latest.osm.pbf --bounding-polygon file=Trieste.poly --write-xml trieste-latest.osm

The resulting file is just the OpenStreetMap data from Trieste province. It is much more faster to process than the data from all the country.

Automate it

You can create a small script to run this process every day.

mkdir ~/bin
mkdir ~/backup
cd ~/bin

Contents of

cd ~/backup
find . -type f -mtime +7 -name '*'.osm.pbf -exec rm -f \{\} \;
rm flat-nodes.bin
data=`date +%Y%m%d%H%M`
wget -O $planeta
osmosis --read-pbf file=$planeta --bounding-polygon file=Trieste.poly --write-pbf trieste-latest.osm.pbf
osm2pgsql -H localhost -s -U user -W -d trieste --latlong -c -S /usr/share/osm2pgsql/ --extra-attributes trieste-latest.osm.pbf


chmod +x
crontab -e

Add the following line:

37 5 * * * $HOME/bin/

Save and exit. The script will run every day at 5:37. Take a rest. Celebrate. The computer will do the work for you.