EljakimHerrewijnen/usse
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Added files to .gitignore

Browse Source
This commit is contained in:
Eljakim 2024-03-31 16:01:38 +00:00
parent 8edcbe043f
commit 43fbf2bf22
7 changed files with 26 additions and 151893 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
automate.sh Executable file
View File

@ -0,0 +1,14 @@
#! /bin/bash
CURR_DIR=$(pwd)
SCRAPE_DIR=${CURR_DIR}/scrape
REACT_DIR=${CURR_DIR}/react_usse
# Move panda_dump.bin
mv ${SCRAPE_DIR}/panda_dump.bin ${SCRAPE_DIR}/_panda_dump.bin
source ${SCRAPE_DIR}/venv/bin/activate && python ${SCRAPE_DIR}/usse.py && deactivate
cp ${SCRAPE_DIR}/out.json ${REACT_DIR}/src/locations.json
# Deploy new container
cd ${REACT_DIR}
./use_docker.sh recreate

6
react_usse/Readme.md
View File

@ -1,3 +1,9 @@
# React Usse
Simple ``React`` application to use the results generated by the scraper in an *interactive* map.
## Usage:
Update the container
```bash
$ sudo ./use_docker.sh recreate
```
This will rebuild the container and deploy it.

151890
react_usse/src/locations.json
View File

File diff suppressed because one or more lines are too long

2
react_usse/use_docker.sh
View File

@ -44,7 +44,7 @@ fi
if [[ ! "$(docker images -q ${IMAGE_NAME})" ]]; then
echo "Building the Docker container..."
cp ${OUT_JSON_PATH} src/locations.json
docker build -t ${IMAGE_NAME} --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy .
docker build -t ${IMAGE_NAME} .
check_ret "Docker build failed!"
fi

2
scrape/.gitignore vendored
View File

@ -171,3 +171,5 @@ external/
out.json
panda_dump.bin
_panda_dump.bin
_out.json

BIN
scrape/panda_dump.bin
View File

Binary file not shown.

5
scrape/usse.py
View File

@ -22,10 +22,11 @@ ORIGIN_LOCATIONS = {
saved_locations = []
out = []
URL = "https://www.funda.nl/zoeken/koop?selected_area=%5B%22utrecht,15km%22%5D&price=%22-400000%22&object_type=%5B%22house%22%5D"
# URL = "https://www.funda.nl/zoeken/koop?selected_area=%5B%22utrecht,15km%22%5D&price=%22-400000%22&object_type=%5B%22house%22%5D"
URL = "https://www.funda.nl/zoeken/koop?selected_area=%5B%22utrecht,30km%22%5D&price=%22-500000%22&object_type=%5B%22house%22%5D"
def get_funda_data():
# scraper = FundaScraper(url="nijkerk/beschikbaar/100000-400000/woonhuis/tuin/eengezinswoning/landhuis/+30km/", find_past=False, n_pages=81)
scraper = FundaScraper(url=URL, find_past=False, n_pages=20)
scraper = FundaScraper(url=URL, find_past=False, n_pages=150)
df = scraper.run()
return df