From 23a53661e05761a475a5f032755cca148df24682 Mon Sep 17 00:00:00 2001 From: Marc Tobias Metten Date: Thu, 25 Jun 2015 04:21:32 +0200 Subject: [PATCH 1/5] first version of Vagrant installation instructions --- VAGRANT.md | 167 +++++++++++++++++++++++++++++++++++++++++++ Vagrantfile | 49 +++++++++++++ vagrant-provision.sh | 166 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 382 insertions(+) create mode 100644 VAGRANT.md create mode 100644 Vagrantfile create mode 100755 vagrant-provision.sh diff --git a/VAGRANT.md b/VAGRANT.md new file mode 100644 index 00000000..26b0ee75 --- /dev/null +++ b/VAGRANT.md @@ -0,0 +1,167 @@ +# Install Nominatim in a virtual machine for development and testing + +This document describes how you can install Nominatim inside a Ubuntu 14 +virtual machine on your desktop/laptop (host machine). The goal is to give +you a development environment to easily edit code and run the test suite +without affecting the rest of your system. + +The installation can run largely unsupervised. You should expect 1-2h from +start to finish depending on how fast your computer and download speed +is. + +## Prerequisites + +1. [Virtualbox](https://www.virtualbox.org/wiki/Downloads) + +2. [Vagrant](https://www.vagrantup.com/downloads.html) + +3. Nominatim + + git clone --recursive https://github.com/twain47/Nominatim.git + + If you haven't used `--recursive`, then you can load the submodules using + + git submodule init + git submodule update + + + +## Installation + +1. Start the virtual machine + + vagrant up + +2. Log into the virtual machine + + vagrant ssh + +3. Import a small country (Monaco) + + You need to give the virtual machine more memory (2GB) for an import, see `Vagrantfile`. + + See the FAQ how to skip this step and point Nominatim to an existing database. + + ``` + # inside the virtual machine: + cd Nominatim + wget --no-verbose --output-document=data/monaco.osm.pbf http://download.geofabrik.de/europe/monaco-latest.osm.pbf + utils/setup.php --osm-file data/monaco.osm.pbf --osm2pgsql-cache 1000 --all | tee monaco.$$.log + ./utils/specialphrases.php --countries > data/specialphrases_countries.sql + psql -d nominatim -f data/specialphrases_countries.sql + ``` + + To repeat an import you'd need to delete the database first + + dropdb --username postgres -if-exists nominatim + + + +## Development + +Vagrant maps the virtual machine's port 8089 to your host machine. Thus you can +see Nominatim in action on [locahost:8089](http://localhost:8089/nominatim/). + +You edit code on your host machine in any editor you like. There is no need to +restart any software: just refresh your browser window. + +PHP errors are written to `/var/log/apache2/error.log`. + +With `echo` and `var_dump()` you write into the output (HTML/XML/JSON) when +you either add `&debug=1` to the URL (preferred) or set +`@define('CONST_Debug', true);` in `settings/local.php`. + + + + +## Running functional tests + +Tests in `/features/db` and `/features/osm2pgsql` have to pass 100%. Other +tests might require full planet-wide data. Sadly even if you have your own +planet-wide data there will be enough differences to the openstreetmap.org +installation to cause false positives in the other tests (see FAQ). + +To run the full test suite + + cd ~/Nominatim/tests + NOMINATIM_SERVER=http://localhost:8089/nominatim lettuce features + +To run a single file + + NOMINATIM_SERVER=http://localhost:8089/nominatim lettuce features/api/reverse.feature + +To run specific tests you can add tags just before the `Scenario line`, e.g. + + @bug-34 + Scenario: address lookup for non-existing or invalid node, way, relation + +and then + + NOMINATIM_SERVER=http://localhost:8089/nominatim lettuce -t bug-34 + + +## Running unit tests + + cd ~/Nominatim/tests + phpunit + + + + + + +## FAQ + +##### Will it run on Windows? + +Yes, Vagrant and Virtualbox can be installed on MS Windows just fine. You need a 64bit +version of Windows. + + +##### Why Monaco, can I use another country? + +Of course! The Monaco import takes less than 30 minutes and works with 2GB RAM. + +##### Will the results be the same as those from nominatim.openstreetmap.org? + +No. Long running Nominatim installations will differ once new import features (or +bug fixes) get added since those usually only get applied to new/changed data. + +Also this document skips the optional Wikipedia data import which affects ranking +of search results. See [Nominatim instllation](http://wiki.openstreetmap.org/wiki/Nominatim/Installation) for details. + +##### Why Ubuntu, can I test CentOS/CoreOS/FreeBSD? + +In general Nominatim will run in all these environment. The installation steps +are slightly different, e.g. the name of the package manager, Apache2 package +name, location of files. We chose Ubuntu because that is closest to the +nominatim.openstreetmap.org production environment. + +You can configure/download other Vagrant boxes from [vagrantbox.es](http://www.vagrantbox.es/). + + +##### How can I connect to an existing database? + +Let's say you have a Postgres database named `nominatim_it` on server `your-server.com` and port `5432`. The Postgres username is `postgres`. You can edit `settings/local.php` and point Nominatim to it. + + pgsql://postgres@your-server.com:5432/nominatim_it + +No data import necessary, no restarting necessary. + +If the Postgres installation is behind a firewall, you can try + + ssh -L 9999:localhost:5432 your-username@your-server.com + +inside the virtual machine. It will map the port to `localhost:9999` and then +you edit `settings/local.php` with + + pgsql://postgres@localhost:9999/nominatim_it + + +##### My computer is slow and the import takes too long. Can I start the virtual machine "in the cloud"? + +Yes. It's possible to start the virtual machine on [Amazon AWS (plugin)](https://github.com/mitchellh/vagrant-aws) or [DigitalOcean (plugin)](https://github.com/smdahlen/vagrant-digitalocean). + + + + diff --git a/Vagrantfile b/Vagrantfile new file mode 100644 index 00000000..042148f7 --- /dev/null +++ b/Vagrantfile @@ -0,0 +1,49 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +Vagrant.configure("2") do |config| + + # Ubuntu LTS 14.04 + config.vm.box = "ubuntu/trusty64" + + # Apache webserver + config.vm.network "forwarded_port", guest: 8089, host: 8089 + + + # If true, then any SSH connections made will enable agent forwarding. + config.ssh.forward_agent = true + + config.vm.synced_folder ".", "/home/vagrant/Nominatim" + + + + # provision using a simple shell script + config.vm.provision :shell, :path => "vagrant-provision.sh" + + + # configure shared package cache if possible + if Vagrant.has_plugin?("vagrant-cachier") + config.cache.enable :apt + config.cache.scope = :box + end + + + config.vm.provider "virtualbox" do |vb| + vb.gui = false + vb.customize ["modifyvm", :id, "--memory", "1024"] + end + + + config.vm.provider :digital_ocean do |provider, override| + override.ssh.private_key_path = '~/.ssh/id_rsa' + override.vm.box = 'digital_ocean' + override.vm.box_url = "https://github.com/smdahlen/vagrant-digitalocean/raw/master/box/digital_ocean.box" + + provider.token = 'e74c5c15501bd2782e689b0805ad577f954bfd39ea1fde3f17ff62835f2fc6c1' + # provider.token = 'YOUR TOKEN' + provider.image = 'ubuntu-14-04-x64' + provider.region = 'nyc2' + provider.size = '512mb' + end + +end diff --git a/vagrant-provision.sh b/vagrant-provision.sh new file mode 100755 index 00000000..aad75443 --- /dev/null +++ b/vagrant-provision.sh @@ -0,0 +1,166 @@ +#!/bin/bash + + +## During 'vagrant provision' this script runs as root and the current +## directory is '/root' +USERNAME=vagrant + +### +### maybe create ubuntu user +### + +# if [[ ! `id -u $USERNAME` ]]; then +# useradd $USERNAME --create-home --shell /bin/bash +# +# # give sudo power +# echo "$USERNAME ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/99-$USERNAME-user +# chmod 0440 /etc/sudoers.d/99-$USERNAME-user +# service sudo restart +# +# # add basic .profile +# cp -r .ssh .profile .bashrc /home/$USERNAME/ +# chown -R $USERNAME /home/$USERNAME/.* +# chgrp -R $USERNAME /home/$USERNAME/.* +# +# # now ideally login as $USERNAME and continue +# su $USERNAME -l +# fi + + +sudo apt-get update -qq +sudo apt-get upgrade -y +# sudo apt-get install -y git-core screen +sudo apt-get install -y build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev \ + libtool automake libproj-dev libboost-dev libboost-system-dev \ + libboost-filesystem-dev libboost-thread-dev +sudo apt-get autoremove -y + +# get arrow-keys working in terminal (e.g. editing in vi) +echo 'stty sane' >> ~/.bash_profile +echo 'export TERM=linux' >> ~/.bash_profile +source ~/.bash_profile + + +### +### PostgreSQL 9.3 + PostGIS 2.1 +### + +sudo apt-get install -y postgresql-9.3-postgis-2.1 postgresql-contrib-9.3 postgresql-server-dev-9.3 +# already included: proj-bin libgeos-dev + +# make sure OS-authenticated users (e.g. $USERNAME) can access +sudo sed -i "s/ident/trust/" /etc/postgresql/9.3/main/pg_hba.conf +sudo sed -i "s/md5/trust/" /etc/postgresql/9.3/main/pg_hba.conf +sudo sed -i "s/peer/trust/" /etc/postgresql/9.3/main/pg_hba.conf +sudo /etc/init.d/postgresql restart + +# creates the role +sudo -u postgres createuser -s $USERNAME + + + +### +### PHP for frontend +### +sudo apt-get install -y php5 php5-pgsql php-pear +sudo pear install DB + + +# get rid of some warning +# where is the ini file? 'php --ini' +echo "date.timezone = 'Etc/UTC'" | sudo tee /etc/php5/cli/conf.d/99-timezone.ini > /dev/null + + + +### +### Nominatim +### +sudo apt-get install -y libprotobuf-c0-dev protobuf-c-compiler \ + libgeos-c1 libgeos++-dev \ + lua5.2 liblua5.2-dev + +# git clone --recursive https://github.com/twain47/Nominatim.git + + +# now ideally login as $USERNAME and continue +su $USERNAME -l +pwd +ls -la /home/vagrant +cd /home/vagrant/Nominatim + +# cd ~/Nominatim +./autogen.sh +./configure +make +chmod +x ./ +chmod +x ./module + + +# IP=`curl -s http://bot.whatismyipaddress.com` +IP=localhost +echo " settings/local.php + + + + + + + +### +### Setup Apache/website +### + +createuser -SDR www-data + +echo ' +Listen 8089 + + # DirectoryIndex index.html + # ErrorDocument 403 /index.html + + DocumentRoot "/var/www/" + + + Options FollowSymLinks MultiViews + AddType text/html .php + + +' | sudo tee /etc/apache2/sites-enabled/nominatim.conf > /dev/null + + +sudo apache2ctl graceful + + +sudo mkdir -m 755 /var/www/nominatim +sudo chown $USERNAME /var/www/nominatim +./utils/setup.php --threads 1 --create-website /var/www/nominatim + + +# if you get 'permission denied for relation word', then try +# GRANT usage ON SCHEMA public TO "www-data"; +# GRANT SELECT ON ALL TABLES IN SCHEMA public TO "www-data"; + +## +## Test suite (Python) +## https://github.com/twain47/Nominatim/tree/master/tests +## +sudo apt-get install -y python-dev python-pip python-Levenshtein tidy +sudo pip install lettuce nose pytidylib haversine psycopg2 shapely + +## +## Test suite (PHP) +## https://github.com/twain47/Nominatim/tree/master/tests-php +## +wget --no-clobber -q https://phar.phpunit.de/phpunit.phar +chmod +x phpunit.phar +sudo mv phpunit.phar /usr/local/bin/phpunit + + From d7c3f04e04b709a989bdf76987b8843ce949cfe3 Mon Sep 17 00:00:00 2001 From: Marc Tobias Metten Date: Sun, 28 Jun 2015 23:54:48 +0200 Subject: [PATCH 2/5] Vagrant: remove nonworking Digitalocean configuration --- .gitignore | 1 + Vagrantfile | 20 ++++++++++---------- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/.gitignore b/.gitignore index a1332064..b6625912 100644 --- a/.gitignore +++ b/.gitignore @@ -30,3 +30,4 @@ data/wiki_import.sql data/wiki_specialphrases.sql data/osmosischange.osc +.vagrant \ No newline at end of file diff --git a/Vagrantfile b/Vagrantfile index 042148f7..1b165327 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -34,16 +34,16 @@ Vagrant.configure("2") do |config| end - config.vm.provider :digital_ocean do |provider, override| - override.ssh.private_key_path = '~/.ssh/id_rsa' - override.vm.box = 'digital_ocean' - override.vm.box_url = "https://github.com/smdahlen/vagrant-digitalocean/raw/master/box/digital_ocean.box" + # config.vm.provider :digital_ocean do |provider, override| + # override.ssh.private_key_path = '~/.ssh/id_rsa' + # override.vm.box = 'digital_ocean' + # override.vm.box_url = "https://github.com/smdahlen/vagrant-digitalocean/raw/master/box/digital_ocean.box" - provider.token = 'e74c5c15501bd2782e689b0805ad577f954bfd39ea1fde3f17ff62835f2fc6c1' - # provider.token = 'YOUR TOKEN' - provider.image = 'ubuntu-14-04-x64' - provider.region = 'nyc2' - provider.size = '512mb' - end + # provider.token = '' + # # provider.token = 'YOUR TOKEN' + # provider.image = 'ubuntu-14-04-x64' + # provider.region = 'nyc2' + # provider.size = '512mb' + # end end From df7d3b747885bcefe5f90754243bf96c386e1656 Mon Sep 17 00:00:00 2001 From: Marc Tobias Metten Date: Sat, 4 Jul 2015 19:16:09 +0200 Subject: [PATCH 3/5] correct documentation on how to start PHP tests --- VAGRANT.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/VAGRANT.md b/VAGRANT.md index 26b0ee75..d97fb3f7 100644 --- a/VAGRANT.md +++ b/VAGRANT.md @@ -102,8 +102,8 @@ and then ## Running unit tests - cd ~/Nominatim/tests - phpunit + cd ~/Nominatim/tests-php + phpunit ./ From 9b7d8db038b2cf595ca8399fb7b1914b5824a044 Mon Sep 17 00:00:00 2001 From: marc tobias Date: Thu, 9 Jul 2015 03:43:59 +0200 Subject: [PATCH 4/5] assume PHP test suite is run from the tests-php directory. Matches documentation in VAGRANT.md --- tests-php/Nominatim/NominatimTest.php | 2 +- tests-php/README.txt | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/tests-php/Nominatim/NominatimTest.php b/tests-php/Nominatim/NominatimTest.php index 68519e48..7ab68934 100644 --- a/tests-php/Nominatim/NominatimTest.php +++ b/tests-php/Nominatim/NominatimTest.php @@ -1,7 +1,7 @@ Date: Thu, 9 Jul 2015 03:45:31 +0200 Subject: [PATCH 5/5] Vagrant: use 2GB by default, dont overwrite local.php which would be unexpected --- VAGRANT.md | 5 +++-- Vagrantfile | 2 +- vagrant-provision.sh | 12 ++++++++---- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/VAGRANT.md b/VAGRANT.md index d97fb3f7..c662d5e4 100644 --- a/VAGRANT.md +++ b/VAGRANT.md @@ -38,7 +38,8 @@ is. 3. Import a small country (Monaco) - You need to give the virtual machine more memory (2GB) for an import, see `Vagrantfile`. + You need to give the virtual machine more memory (2GB) for an import, + see `Vagrantfile`. Otherwise 1GB is enough. See the FAQ how to skip this step and point Nominatim to an existing database. @@ -46,7 +47,7 @@ is. # inside the virtual machine: cd Nominatim wget --no-verbose --output-document=data/monaco.osm.pbf http://download.geofabrik.de/europe/monaco-latest.osm.pbf - utils/setup.php --osm-file data/monaco.osm.pbf --osm2pgsql-cache 1000 --all | tee monaco.$$.log + ./utils/setup.php --osm-file data/monaco.osm.pbf --osm2pgsql-cache 1000 --all 2>&1 | tee monaco.$$.log ./utils/specialphrases.php --countries > data/specialphrases_countries.sql psql -d nominatim -f data/specialphrases_countries.sql ``` diff --git a/Vagrantfile b/Vagrantfile index 1b165327..843dd668 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -30,7 +30,7 @@ Vagrant.configure("2") do |config| config.vm.provider "virtualbox" do |vb| vb.gui = false - vb.customize ["modifyvm", :id, "--memory", "1024"] + vb.customize ["modifyvm", :id, "--memory", "2048"] end diff --git a/vagrant-provision.sh b/vagrant-provision.sh index aad75443..68e279c5 100755 --- a/vagrant-provision.sh +++ b/vagrant-provision.sh @@ -84,8 +84,6 @@ sudo apt-get install -y libprotobuf-c0-dev protobuf-c-compiler \ # now ideally login as $USERNAME and continue su $USERNAME -l -pwd -ls -la /home/vagrant cd /home/vagrant/Nominatim # cd ~/Nominatim @@ -96,6 +94,12 @@ chmod +x ./ chmod +x ./module +LOCALSETTINGS_FILE='settings/local.php' +if [[ -e "$LOCALSETTINGS_FILE" ]]; then + echo "$LOCALSETTINGS_FILE already exist, writing to settings/local-vagrant.php instead." + LOCALSETTINGS_FILE='settings/local-vagrant.php' +fi + # IP=`curl -s http://bot.whatismyipaddress.com` IP=localhost echo " settings/local.php +" > $LOCALSETTINGS_FILE @@ -118,7 +122,7 @@ echo "