usse/documentation/build/osm.html

194 lines
11 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Open Street Maps &mdash; Usse 1 documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
<script src="_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/sphinx_highlight.js"></script>
<script src="_static/js/theme.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="prev" title="Google Maps API" href="googlemaps.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="index.html" class="icon icon-home"> Usse
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="funda.html">Scraping Funda</a></li>
<li class="toctree-l1"><a class="reference internal" href="googlemaps.html">Google Maps API</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Open Street Maps</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#osrm">OSRM</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#setup">Setup</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#nominatim">Nominatim</a></li>
<li class="toctree-l2"><a class="reference internal" href="#ip-firewall">IP Firewall</a></li>
<li class="toctree-l2"><a class="reference internal" href="#tile-server">Tile Server</a></li>
</ul>
</li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">Usse</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html" class="icon icon-home"></a></li>
<li class="breadcrumb-item active">Open Street Maps</li>
<li class="wy-breadcrumbs-aside">
<a href="_sources/osm.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="open-street-maps">
<h1>Open Street Maps<a class="headerlink" href="#open-street-maps" title="Permalink to this heading"></a></h1>
<p>Much like Google Maps, <code class="docutils literal notranslate"><span class="pre">Openstreetmaps</span></code> is a map that allows you to navigate the world.
The main difference being that OSM is free and can be self hosted.
However there is not a <em>plug-and-play</em> solution that will host your own maps.
It turns out that hosting your own maps, including geolocation and calculating distances is quite a big task, especially if you want to suppor the whole world.</p>
<section id="osrm">
<h2>OSRM<a class="headerlink" href="#osrm" title="Permalink to this heading"></a></h2>
<p><strong>O</strong>pen <strong>S</strong>ource <strong>R</strong>outing <strong>M</strong>achine (OSRM) is a project that allows you to calculate distances from a OSM map.
See more about the project <a class="reference external" href="https://project-osrm.org/">here</a></p>
<section id="setup">
<h3>Setup<a class="headerlink" href="#setup" title="Permalink to this heading"></a></h3>
<p>Setting OSRM up is straightforward, but it requires you to the same docker container several times. In short the commands for the Netherlands are as follows:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">#</span>Download The Netherlands
<span class="go">mkdir -p data/</span>
<span class="go">cd data/</span>
<span class="go">wget https://download.geofabrik.de/europe/netherlands-latest.osm.pbf</span>
<span class="go">cd ../</span>
<span class="go">docker run --name osrm-backend -t -v &quot;${PWD}/data:/data&quot; osrm/osrm-backend osrm-extract -p /opt/car.lua /data/netherlands-latest.osm.pbf</span>
<span class="gp"># </span>Process the data
<span class="go">docker run -a osrm-backend -t -v &quot;${PWD}/data:/data&quot; osrm/osrm-backend osrm-contract /data/netherlands-latest.osrm</span>
<span class="go">docker run -t -v &quot;${PWD}/data:/data&quot; osrm/osrm-backend osrm-partition /data/netherlands-latest.osrm</span>
<span class="go">docker run -t -v &quot;${PWD}/data:/data&quot; osrm/osrm-backend osrm-customize /data/netherlands-latest.osrm</span>
<span class="gp"># </span>Finaly run the routing engine
<span class="go">docker run -t -i -p 5998:5000 -v &quot;${PWD}/data:/data&quot; osrm/osrm-backend osrm-routed --algorithm mld /data/netherlands-latest.osrm</span>
</pre></div>
</div>
<p>Now you can use curl to do requests to port <code class="docutils literal notranslate"><span class="pre">5998</span></code>.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>When using Google Maps coordinates to send requests to OSRM, make sure that the Longitude and Latitude coordinates are not swapped.</p>
</div>
</section>
</section>
<section id="nominatim">
<h2>Nominatim<a class="headerlink" href="#nominatim" title="Permalink to this heading"></a></h2>
<p>As it turns out, <code class="docutils literal notranslate"><span class="pre">OSRM</span></code> is a very good routing engine but it cant do geolocation. To solve this <code class="docutils literal notranslate"><span class="pre">Nominatim</span></code> was used to resolve postal codes to coordinates.</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>cat run_nominatim.sh
<span class="gp">#</span>! /bin/bash
<span class="go">docker run -it \</span>
<span class="go">-e PBF_PATH=/nominatim/data/netherlands-latest.osm.pbf \</span>
<span class="go">-v &quot;${PWD}/data:/nominatim/data&quot; \</span>
<span class="go">-e REPLICATION_URL=https://download.geofabrik.de/europe/netherlands-updates/ \</span>
<span class="go">-p 5999:8080 \</span>
<span class="go">--name nominatim \</span>
<span class="go">mediagis/nominatim:4.2</span>
</pre></div>
</div>
<p>This will result in a website that is accessible on port 5999 to which you can send queries.</p>
</section>
<section id="ip-firewall">
<h2>IP Firewall<a class="headerlink" href="#ip-firewall" title="Permalink to this heading"></a></h2>
<p>Procesing routing data is hard.
In order to prevent bots from misusing these interfaces a firewall rule was added to only allow incomming traffic from specific ip addresses.</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>sudo ufw allow from &lt;ip_addr&gt; to any port <span class="m">5998</span>
<span class="gp">$ </span>sudo ufw allow from &lt;ip_addr&gt; to any port <span class="m">5999</span>
</pre></div>
</div>
</section>
<section id="tile-server">
<h2>Tile Server<a class="headerlink" href="#tile-server" title="Permalink to this heading"></a></h2>
<p>In order to also serve tiles and be fully independent from <code class="docutils literal notranslate"><span class="pre">Google</span> <span class="pre">Maps</span></code> a tileserver was also started.
For this the previously donwloaded .pbf file needs to be imported and a postgresql database was used to <em>hopefully</em> speed up the performance.</p>
<p>First we need to import the data, then start a tileserver.</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">Import data</span>
<span class="gp">$ </span>docker run -v <span class="s2">&quot;</span><span class="si">${</span><span class="nv">PWD</span><span class="si">}</span><span class="s2">/data/netherlands-latest.osm.pbf:/data.osm.pbf&quot;</span> -v openstreetmap-data:/var/lib/postgresql/12/main overv/openstreetmap-tile-server:1.3.10 import
<span class="go">Run tile server</span>
<span class="gp">$ </span>docker run -p <span class="m">5997</span>:80 -v openstreetmap-data:/var/lib/postgresql/12/main -d overv/openstreetmap-tile-server:1.3.10 run
</pre></div>
</div>
<p>This will open a map server at port <strong>5997</strong>. When navigating to <em>http://www.herreweb.nl:5997/tile/0/0/0.png</em>, a world map is shown.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The import step took over 2 hours on a 6-core <em>slow</em> server(Intel Xeon E5-2620 v2). The map server is not fast either, but faster after loading all the map tiles.</p>
</div>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="googlemaps.html" class="btn btn-neutral float-left" title="Google Maps API" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2023, Eljakim.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>