``Funda`` is a real estate housing market that tries to keep track of all houses that are currently for sale.
Scraping is not allowed, but on github there are several projects that still try to do this.

 is a real estate housing market that tries to keep track of all houses that are currently for sale.

A quick test from several github projects landed us with `this project <https://github.com/whchien/funda-scraper>`_.

This project still works, but is very limited in the filtering methods.
A few patches to code allows us to inject a URL that will be used and no other filters will be applied.
Next we can setup a basic filter in the browser and copy the URL in order to do scraping.

if self.url != "":
    # https://www.funda.nl/koop/gemeente-huizen/0-350000/tuin/+10km/
    # gemeente-huizen/0-350000/tuin/+10km/
return {
    "close": f"{self.base_url}/koop/verkocht/{self.url}/",
    "open": f"{self.base_url}/koop/{self.url}/",
}

def get_funda_data():
    scraper = FundaScraper(url="nijkerk/beschikbaar/100000-400000/woonhuis/tuin/eengezinswoning/landhuis/+30km/", find_past=False, n_pages=81)
    df = scraper.run()
    return df

``Google Maps`` has a *free* api that can be used to calculate distance matrixes as well as show point on a map.
In order to use this API you will need to request an API key from the `google cloud console <https://cloud.google.com/>`_.

 api that can be used to calculate distance matrixes as well as show point on a map.
In order to use this API you will need to request an API key from the 

This API can be used at a cost for each request, but Google gives ~200 dollars worth of request at the start of every month.

For a quick test to see if it was possible to geolocate each house and calculate the distance. For this a simple python script was used.

origin_locations = {
"nfi_location" : (52.044867266861466, 4.3585175985355225),
"hoogstraat_location" : (52.08725689123654, 5.147180442716177),
"bakkersdijk_location" : (51.85802695253161, 4.482033956202426),
"korhoen_location" : (52.5219455005375, 5.732514040876346),
"harde_location" : (52.41650138296019, 5.870995170999243)
}

def get_distances(out_dict, destination_location):
    for key in origin_locations:
        distance = gmaps.distance_matrix(origin_locations[key], destination_location, mode = 'driving')
        out_dict[key] = distance['rows'][0]['elements'][0]

We can do about 40.000 request for free each month. With about 1200 houses and 6 requests per house that would mean up to 7000 requests.
That is enough to build a quick PoC to see if this method would work.

This works and a lot of houses are shown on google maps with markers.

Another result of this action was a Ban from google.
The reason for this was that I did not read the terms of use.
You are allowed to do 40K requests for free every month, just not several thousand in a few minutes. Oops.

Because we want to do more than 40k requests and dont want to pay for these requests, the project was switched to **O**\pen **S**\treet **M**\ap(OSM)

Because we want to do more than 40k requests and dont want to pay for these requests, the project was switched to 

Usse documentation master file, created by
sphinx-quickstart on Mon Feb  6 21:46:43 2023.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.

The current state of the project is that it is ``Frozen`` until at least June.

The Goal of this project is to, automatically, scrape `real estate housing website Funda <https://www.funda.nl>`_ for houses with some specifications(garden, non-apartment).
Next, we must be able to view these houses and filter them based on the distance to several locations(NFI, Hooghstraat etc.).

 for houses with some specifications(garden, non-apartment).

Much like Google Maps, ``Openstreetmaps`` is a map that allows you to navigate the world.
The main difference being that OSM is free and can be self hosted.
However there is not a *plug-and-play* solution that will host your own maps.
It turns out that hosting your own maps, including geolocation and calculating distances is quite a big task, especially if you want to suppor the whole world.

 is a map that allows you to navigate the world.
However there is not a 

 solution that will host your own maps.

**O**\pen **S**\ource **R**\outing **M**\achine (OSRM) is a project that allows you to calculate distances from a OSM map.
See more about the project `here <https://project-osrm.org/>`_

Setting OSRM up is straightforward, but it requires you to the same docker container several times. In short the commands for the Netherlands are as follows:

#Download The Netherlands
mkdir -p data/
cd data/
wget https://download.geofabrik.de/europe/netherlands-latest.osm.pbf

cd ../
docker run --name osrm-backend -t -v "${PWD}/data:/data" osrm/osrm-backend osrm-extract -p /opt/car.lua /data/netherlands-latest.osm.pbf

# Process the data
docker run -a osrm-backend -t -v "${PWD}/data:/data" osrm/osrm-backend osrm-contract /data/netherlands-latest.osrm
docker run -t -v "${PWD}/data:/data" osrm/osrm-backend osrm-partition /data/netherlands-latest.osrm
docker run -t -v "${PWD}/data:/data" osrm/osrm-backend osrm-customize /data/netherlands-latest.osrm

# Finaly run the routing engine
docker run -t -i -p 5998:5000 -v "${PWD}/data:/data" osrm/osrm-backend osrm-routed --algorithm mld /data/netherlands-latest.osrm

When using Google Maps coordinates to send requests to OSRM, make sure that the Longitude and Latitude coordinates are not swapped.

As it turns out, ``OSRM`` is a very good routing engine but it can't do geolocation. To solve this ``Nominatim`` was used to resolve postal codes to coordinates.

 is a very good routing engine but it can’t do geolocation. To solve this 

$ cat run_nominatim.sh
#! /bin/bash

docker run -it \
-e PBF_PATH=/nominatim/data/netherlands-latest.osm.pbf \
-v "${PWD}/data:/nominatim/data" \
-e REPLICATION_URL=https://download.geofabrik.de/europe/netherlands-updates/ \
-p 5999:8080 \
--name nominatim \
mediagis/nominatim:4.2

This will result in a website that is accessible on port 5999 to which you can send queries.

Procesing routing data is hard.
In order to prevent bots from misusing these interfaces a firewall rule was added to only allow incomming traffic from specific ip addresses.

$ sudo ufw allow from <ip_addr> to any port 5998
$ sudo ufw allow from <ip_addr> to any port 5999

In order to also serve tiles and be fully independent from ``Google Maps`` a tileserver was also started.
For this the previously donwloaded .pbf file needs to be imported and a postgresql database was used to *hopefully* speed up the performance.

 a tileserver was also started.
For this the previously donwloaded .pbf file needs to be imported and a postgresql database was used to 

Import data
$ docker run -v "${PWD}/data/netherlands-latest.osm.pbf:/data.osm.pbf" -v openstreetmap-data:/var/lib/postgresql/12/main overv/openstreetmap-tile-server:1.3.10 import

Run tile server
$ docker run -p 5997:80 -v openstreetmap-data:/var/lib/postgresql/12/main -d overv/openstreetmap-tile-server:1.3.10 run

This will open a map server at port **5997**. When navigating to *http://www.herreweb.nl:5997/tile/0/0/0.png*, a world map is shown.

The import step took over 2 hours on a 6-core *slow* server(Intel Xeon E5-2620 v2). The map server is not fast either, but faster after loading all the map tiles.

 server(Intel Xeon E5-2620 v2). The map server is not fast either, but faster after loading all the map tiles.

Unexpected possible title overline or transition.
Treating it as ordinary text because it's so short.

Treating it as ordinary text because it’s so short.

In order to use the scraped data a *react app* was developed.
The main thing this app does is show a map, with icons for each house, while also showing the distance and travel time:

 was developed.

Copy the generated *out.json* file that was generated in the *usse scrape* project to ``src/locations.json``, next you can run the app:

A dockerfile was created in order to serve the application on ``Wintergreen``.
To use the dockerfile you can simply use the bash script: ``use_docker.sh`` and run it to run the docker container: