Home | Posts | OSM Tile Server | Install OSM Tile Server

OSM Tile Server (Part 1)

Chris laundy, 13/05/2020, O s m Tile server
OSM Tile Server (Part 1)

In this post we’re going to get a functional OpenStreetMap tile server up and running. We’ll deploy a small map initially just to make sure everything is working correctly.

These instructions are have been written and tested against a fresh install of Ubuntu 20.04 server and follow on from the steps outlined in Secure Ubuntu 20.04. If you have upgraded from an earlier version of Ubuntu this guide probably won’t work.

Install the dependencies

An OSM tile server has 5 main components:

  1. mod_tile - an apache module that serves cached tiles and decides which tiles need re-rendering.
  2. renderd - a priority queueing system for different sorts of requests to manage and smooth out the load from rendering requests.
  3. mapnik - is the software library that does the actual rendering.
  4. osm2pgsql - loads the raw OSM data into the database.
  5. postgresql/postgis - The database!

First up we will install all the tools and utilities we are going to need.

    sudo apt install libboost-all-dev git tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev munin-node munin protobuf-c-compiler libfreetype6-dev libtiff5-dev libicu-dev libgdal-dev libcairo2-dev libcairomm-1.0-dev apache2 apache2-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-0-dev acl npm tmux

Hit y when prompted to install. It might take a while.

Install the DB

Ubuntu 20.04 has pre-packaged versions of postgresql and postgis (some graphical extensions) so we can simply install from the package manager

sudo apt-get install postgresql postgresql-contrib postgis postgresql-12-postgis-3 postgresql-12-postgis-3-scripts

Again, hit y to install.

See Optimise Postgres for some tips on configuring postgres for a large (full planet) import. You don’t need to do this now, as we’re only going to import a small extract to prove the server is working.

Set-up gis database

Now we need to create a postgis database, the convention is to call it gis so we’ll follow along. We also need a user that Mapnik will use to render maps later. We’ll call it osm but you can call it what ever you like.

sudo -u postgres -i
createuser osm
createdb -E UTF8 -O osm gis

Next we create the postgis and hstore extensions

psql -c "CREATE EXTENSION postgis;" -d gis
psql -c "CREATE EXTENSION hstore;" -d gis

Then set osm as the table owner:

psql -c "ALTER TABLE spatial_ref_sys OWNER TO osm;" -d gis
psql -c "ALTER TABLE geometry_columns OWNER TO osm;" -d gis

And exit from the postgres user

exit

Create an osm system user so the tile server can run as osm.

sudo adduser --system osm

While we are here, let’s give our user id permission to write to the osm user’s home directory.

sudo setfacl -R -m u:chris:rwx /home/osm/

remember to replace chris with your real user name

Install osm2pgsql
sudo apt install osm2pgsql

Install Mapnik

Again we will install the default version in Ubuntu 20.04

sudo apt-get install autoconf apache2-dev libtool libxml2-dev libbz2-dev libgeos-dev libgeos++-dev libproj-dev gdal-bin libmapnik-dev mapnik-utils python3-mapnik

Install mod_tile and renderd

  • “mod_tile” is an Apache module that handles requests for tiles;
  • “renderd” is a daemon that actually renders tiles when “mod_tile” requests them.

note: We’re using the “switch2osm” branch of https://github.com/SomeoneElseOSM/mod_tile, which is itself forked from https://github.com/openstreetmap/mod_tile, but modified so that it supports Ubuntu 20.04, and with a couple of other changes to work on a standard Ubuntu server rather than one of OSM’s rendering servers.

mkdir -p /home/osm/src && cd /home/osm/src
git clone -b switch2osm git://github.com/SomeoneElseOSM/mod_tile.git
cd mod_tile
./autogen.sh

That should complete with autoreconf: Leaving directory `.'

./configure

That should complete with config.status: executing libtool commands

make

You might see some warnings and it might look like it has hung, but stick with it and it should complete with make[1]: Leaving directory '/home/osm/src/mod_tile'

sudo make install

That should finish with make[1]: Leaving directory '/home/osm/src/mod_tile'

sudo make install-mod_tile

That should finish with chmod 644 /usr/lib/apache2/modules/mod_tile.so

Lastly:

sudo ldconfig

which wont reply at all.

Configure Stylesheet

Now we need to download and configure a stylesheet. We’re going to use the ‘standard’ map style from OSM.

cd /home/osm/src
git clone git://github.com/gravitystorm/openstreetmap-carto.git
cd openstreetmap-carto

Next we install a “carto” compiler

sudo npm install -g carto
carto -v

It should respond with a version number of at least 1.2.0

Now we convert carto into something mapnik can use.

carto project.mml > mapnik.xml

There will be warnings, but you now have a mapnik XML stylesheet at /home/osm/src/openstreetmap-cart/mapnik.xml

Loading Some data

Importing the full planet will take a long time, exactly how long will depend on your machine. So we will start with a small extract:

mkdir /home/osm/data && cd /home/osm/data
wget https://download.geofabrik.de/asia/azerbaijan-latest.osm.pbf

Next comes the fun part; importing the data, we switch to the postgres user first:

sudo -u postgres -i

osm2pgsql --database gis --create --slim  --multi-geometry --hstore --tag-transform-script /home/osm/src/openstreetmap-carto/openstreetmap-carto.lua --cache 35000 --number-processes 10 --style /home/osm/src/openstreetmap-carto/openstreetmap-carto.style /home/osm/data/azerbaijan-latest.osm.pbf

where:

  • --database gis: The database to use
  • --create: Load into an empty db rather than --append to existing data
  • --slim: Store temporary data in the database. This greatly reduces the RAM usage but is much slower. Required if you want to update with --append later.
  • --multi-geometry: Generate multi-geometry features in postgresql tables.
  • --hstore: Allows tags for which there are no explicit database columns to be used for rendering.
  • --tag-transform-script <path>: Defines the lua script used for tag processing. This an easy is a way to process OSM tags before the style itself processes them, making the style logic potentially much simpler.
  • --cache <size>: The size of the in memory cache in Mb.
  • --number-processes <n>: Specifies the number of parallel processes used for certain operations. I have 10 cores on my server so I use 10.
  • --style <path>: Location of the style file.
  • Lastly the location of the data itself.

The command will finish with something like:

osm2pgsql took 32s overall
node cache: stored: 3008359(100.00%), storage efficiency: 50.08% (dense blocks: 9, sparse nodes: 2966494), hit rate: 100.00%

Once the import completes you need to grant all privileges to osm user on the gis database:

psql -c "GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO osm;" -d gis

Custom Indexes

Custom indexes are not required, but will speed-up rendering, particularly on full-planet databases or heavy load environments, probably won’t be as helpful for out initial small extract.

cd /home/osm/src/openstreetmap-carto/
psql -d gis -f indexes.sql

The indexes may take a little while to create, especially with a large extract.

Exit from postgres user

exit

Note about the --cache option

Basically the bigger your cache the quicker the import, however it’s a balancing act, too big a value may cause the process to terminate because it runs out of memory. Try with 80% of your free ram and see how you get on. If osm2pgsql runs out of memory you’ll need to start over with a smaller cache.

free --mega

Output:

              total        used        free      shared  buff/cache   available
Mem:          61840       17305       44284           7         250       43969
Swap:          2097           0        2097

So I’d start with --cache 35000 (44284 * 0.8 = 35427)

Shapefile Download

Most of the map data needed comes from the OSM data file we down loaded earlier, but we need to get some shape files for things like low-zoom country boundaries.

cd /home/osm/src/openstreetmap-carto/
scripts/get-shapefiles.py

When complete it will display …script completed in xx.x seconds..

Fonts

Install the fonts to display place names around the world:

sudo apt-get install ttf-dejavu fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted ttf-unifont

Setting up the Webserver

Configure renderd

sudo nano /usr/local/etc/renderd.conf

  • If you are short on ram (2Gb or so) then num_threads=4 to num_threads=2

  • Change the XML line to point to the correct location

  • The URI=/hot/ was chosen so that the tiles generated here can more easily be used in place of the HOT tile layer at OpenStreetMap.org. I’ll change it to osm but you can use what ever you like.

  • For my server I don’t need 20x zoom so I’m going to set MAXZOOM to 18, but you can leave it at 20 if you want

    [renderd]
num_threads=2
tile_dir=/var/lib/mod_tile
stats_file=/var/run/renderd/renderd.stats

[mapnik]
plugins_dir=/usr/lib/mapnik/3.0/input
font_dir=/usr/share/fonts/truetype
font_dir_recurse=1

[ajt]
URI=/osm/
TILEDIR=/var/lib/mod_tile
XML=/home/osm/src/openstreetmap-carto/mapnik.xml
HOST=localhost
TILESIZE=256
MAXZOOM=20

Configure Apache

First we create the paths from [renderd] config above and set the osm user as owner:

sudo mkdir /var/lib/mod_tile
sudo chown osm /var/lib/mod_tile

sudo mkdir /var/run/renderd
sudo chown osm /var/run/renderd

Now we need to tell apache about mod_tile

sudo nano /etc/apache2/conf-available/mod_tile.conf

add the following line:

LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so

save it and run:

sudo a2enconf mod_tile

It will say you need to run systemctl reload apache2 but we won’t do that just yet. We need to tell apache about renderd

sudo nano /etc/apache2/sites-available/000-default.conf

add the following between the ServerAdmin and DocumentRoot lines;

LoadTileConfigFile /usr/local/etc/renderd.conf
ModTileRenderdSocketName /var/run/renderd/renderd.sock
## Timeout before giving up for a tile to be rendered
ModTileRequestTimeout 0
## Timeout before giving up for a tile to be rendered that is otherwise missing
ModTileMissingRequestTimeout 30

Now we can reload apache

sudo service apache2 reload

note: you might need to do it twice, I’m not sure why.

If you have enabled the firewall as described in [Part 1 - setup] you’ll need to open the http port now:

sudo ufw allow http

If you point a web browser at: http://yourserveripaddress/index.html you should get Ubuntu / apache’s “It works!” page.

Running renderd for the first time

Initially we’ll run renderd in the foreground so we can see any errors if they occur:

sudo -u osm renderd -f -c /usr/local/etc/renderd.conf

This command uses sudo to switch to the osm user and run the command

Point your web browser at http://yourserveripaddress/osm/0/0/0.png (if you didn’t choose osm as the uri when you configured renderd make sure you use the uri you chose instead). If all goes well you should see a map of the world in your browser and output on the command line that includes DEBUG: START TILE and DEBUG: DONE TILE

If you see any DEBUG: Failed to read cmd on fd messages you can safely ignore them, they are not errors.

Assuming everything worked you can now ctrl-c to stop the foreground process.

If you get errors and the tile is not displayed then copy the output and ask about the problem somewhere like stackoverflow or help.openstreetmap.org

Running renderd in the background

First edit the /home/osm/src/mod_tile/debian/renderd.init file and set the RUNASUSER property to osm

sudo nano /home/osm/src/mod_tile/debian/renderd.init

...
SCRIPTNAME=/etc/init.d/$NAME
RUNASUSER=osm

## Exit if the package is not installed
...

Next copy it to the system directory, set permissions and add the service:

sudo cp /home/osm/src/mod_tile/debian/renderd.init /etc/init.d/renderd
sudo chmod u+x /etc/init.d/renderd
sudo cp /home/osm/src/mod_tile/debian/renderd.service /lib/systemd/system/

And start it:

sudo /etc/init.d/renderd start

To make it start automatically when the system reboots:

sudo systemctl enable renderd

Slippy Map

There is an example html file in mod_tile’s extras folder which we can use as the index.html on our server. It uses leafletjs to display a ‘slippy map’ centered on Azerbaijan.

nano /home/osm/src/mod_tile/extra/sample_leaflet.html
  • replace the 127.0.0.1 with the ip address of your server.
  • replace hot with osm (or the uri you have chosen)
  • Note if you loaded a different osm data file you will want to change the coordinates [40.36629, 49.83335] to center on your map.

Now replace the index.html file in /var/www/html

sudo cp /home/osm/src/mod_tile/extra/sample_leaflet.html /var/www/html/index.html

Tail the logs so we can see when a tile is requested:

sudo tail -f /var/log/syslog | grep " TILE "

Note: the spaces around TILE are important

Now you can point your browser at http://yourserveripaddress/index.html and you will see a line printed to your console every time a tile is requested.

Conclusion

If you made it this far you now have a functional tile server… Nice!

Next up:

  • The ‘Big Import’
  • SSL with LetsEncrypt
  • Imporve performance with HTTP2 and Pre-rendering tiles.
...

Let’s get started

Whether you are starting a new project, you’ve hit a snag or you simply need some advice…

I can help