Merge pull request #290 from mtmail/vagrant

first version of Vagrant installation instructions

.gitignore

@@ -30,3 +30,4 @@ data/wiki_import.sql
 data/wiki_specialphrases.sql
 data/osmosischange.osc
 
+.vagrant

VAGRANT.md

@@ -0,0 +1,168 @@
+# Install Nominatim in a virtual machine for development and testing
+
+This document describes how you can install Nominatim inside a Ubuntu 14
+virtual machine on your desktop/laptop (host machine). The goal is to give
+you a development environment to easily edit code and run the test suite
+without affecting the rest of your system. 
+
+The installation can run largely unsupervised. You should expect 1-2h from
+start to finish depending on how fast your computer and download speed
+is.
+
+## Prerequisites
+
+1. [Virtualbox](https://www.virtualbox.org/wiki/Downloads)
+
+2. [Vagrant](https://www.vagrantup.com/downloads.html)
+
+3. Nominatim 
+
+        git clone --recursive https://github.com/twain47/Nominatim.git
+
+    If you haven't used `--recursive`, then you can load the submodules using
+    
+        git submodule init
+        git submodule update
+
+
+
+## Installation
+
+1. Start the virtual machine
+
+        vagrant up
+
+2. Log into the virtual machine
+
+        vagrant ssh
+
+3. Import a small country (Monaco)
+
+    You need to give the virtual machine more memory (2GB) for an import,
+    see `Vagrantfile`. Otherwise 1GB is enough.
+    
+    See the FAQ how to skip this step and point Nominatim to an existing database.
+
+  ```
+  # inside the virtual machine:
+  cd Nominatim
+    wget --no-verbose --output-document=data/monaco.osm.pbf http://download.geofabrik.de/europe/monaco-latest.osm.pbf
+    ./utils/setup.php --osm-file data/monaco.osm.pbf --osm2pgsql-cache 1000 --all 2>&1 | tee monaco.$$.log
+    ./utils/specialphrases.php --countries > data/specialphrases_countries.sql
+    psql -d nominatim -f data/specialphrases_countries.sql
+    ```
+
+  To repeat an import you'd need to delete the database first
+
+        dropdb --username postgres -if-exists nominatim
+
+
+
+## Development
+
+Vagrant maps the virtual machine's port 8089 to your host machine. Thus you can
+see Nominatim in action on [locahost:8089](http://localhost:8089/nominatim/).
+
+You edit code on your host machine in any editor you like. There is no need to
+restart any software: just refresh your browser window.
+
+PHP errors are written to `/var/log/apache2/error.log`.
+
+With `echo` and `var_dump()` you write into the output (HTML/XML/JSON) when
+you either add `&debug=1` to the URL (preferred) or set
+`@define('CONST_Debug', true);` in `settings/local.php`.
+
+
+
+
+## Running functional tests
+
+Tests in `/features/db` and `/features/osm2pgsql` have to pass 100%. Other
+tests might require full planet-wide data. Sadly even if you have your own
+planet-wide data there will be enough differences to the openstreetmap.org
+installation to cause false positives in the other tests (see FAQ). 
+
+To run the full test suite
+
+    cd ~/Nominatim/tests
+    NOMINATIM_SERVER=http://localhost:8089/nominatim lettuce features
+
+To run a single file
+
+    NOMINATIM_SERVER=http://localhost:8089/nominatim lettuce features/api/reverse.feature
+    
+To run specific tests you can add tags just before the `Scenario line`, e.g.
+
+    @bug-34
+    Scenario: address lookup for non-existing or invalid node, way, relation
+
+and then
+
+    NOMINATIM_SERVER=http://localhost:8089/nominatim lettuce -t bug-34
+
+
+## Running unit tests
+
+    cd ~/Nominatim/tests-php
+    phpunit ./
+
+
+
+
+
+
+## FAQ
+
+##### Will it run on Windows?
+
+Yes, Vagrant and Virtualbox can be installed on MS Windows just fine. You need a 64bit
+version of Windows.
+
+
+##### Why Monaco, can I use another country?
+
+Of course! The Monaco import takes less than 30 minutes and works with 2GB RAM.
+
+##### Will the results be the same as those from nominatim.openstreetmap.org?
+
+No. Long running Nominatim installations will differ once new import features (or
+bug fixes) get added since those usually only get applied to new/changed data.
+
+Also this document skips the optional Wikipedia data import which affects ranking
+of search results. See [Nominatim instllation](http://wiki.openstreetmap.org/wiki/Nominatim/Installation) for details.
+
+##### Why Ubuntu, can I test CentOS/CoreOS/FreeBSD?
+
+In general Nominatim will run in all these environment. The installation steps
+are slightly different, e.g. the name of the package manager, Apache2 package
+name, location of files. We chose Ubuntu because that is closest to the
+nominatim.openstreetmap.org production environment.
+
+You can configure/download other Vagrant boxes from [vagrantbox.es](http://www.vagrantbox.es/).
+
+
+##### How can I connect to an existing database?
+
+Let's say you have a Postgres database named `nominatim_it` on server `your-server.com` and port `5432`. The Postgres username is `postgres`. You can edit `settings/local.php` and point Nominatim to it.
+
+    pgsql://postgres@your-server.com:5432/nominatim_it
+    
+No data import necessary, no restarting necessary.
+
+If the Postgres installation is behind a firewall, you can try
+
+    ssh -L 9999:localhost:5432 your-username@your-server.com
+
+inside the virtual machine. It will map the port to `localhost:9999` and then
+you edit `settings/local.php` with
+
+    pgsql://postgres@localhost:9999/nominatim_it
+
+
+##### My computer is slow and the import takes too long. Can I start the virtual machine "in the cloud"?
+
+Yes. It's possible to start the virtual machine on [Amazon AWS (plugin)](https://github.com/mitchellh/vagrant-aws) or [DigitalOcean (plugin)](https://github.com/smdahlen/vagrant-digitalocean).
+
+
+
+

Vagrantfile

@@ -0,0 +1,49 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure("2") do |config|
+
+  # Ubuntu LTS 14.04
+  config.vm.box = "ubuntu/trusty64" 
+
+  # Apache webserver
+  config.vm.network "forwarded_port", guest: 8089, host: 8089
+
+
+  # If true, then any SSH connections made will enable agent forwarding.
+  config.ssh.forward_agent = true
+
+  config.vm.synced_folder ".", "/home/vagrant/Nominatim"
+
+
+
+  # provision using a simple shell script
+  config.vm.provision :shell, :path => "vagrant-provision.sh"
+
+
+  # configure shared package cache if possible
+  if Vagrant.has_plugin?("vagrant-cachier")
+    config.cache.enable :apt
+    config.cache.scope = :box
+  end
+
+
+  config.vm.provider "virtualbox" do |vb|
+    vb.gui = false
+    vb.customize ["modifyvm", :id, "--memory", "2048"]
+  end
+
+
+  # config.vm.provider :digital_ocean do |provider, override|
+  #   override.ssh.private_key_path = '~/.ssh/id_rsa'
+  #   override.vm.box = 'digital_ocean'
+  #   override.vm.box_url = "https://github.com/smdahlen/vagrant-digitalocean/raw/master/box/digital_ocean.box"
+
+  #   provider.token = ''
+  #   # provider.token = 'YOUR TOKEN'
+  #   provider.image = 'ubuntu-14-04-x64'
+  #   provider.region = 'nyc2'
+  #   provider.size = '512mb'
+  # end
+
+end

tests-php/Nominatim/NominatimTest.php

@@ -1,7 +1,7 @@
 <?php
 
 namespace Nominatim;
-require 'lib/lib.php';
+require '../lib/lib.php';
 
 
 class NominatimTest extends \PHPUnit_Framework_TestCase

tests-php/README.txt

@@ -6,6 +6,7 @@ https://phpunit.de/manual/4.2/en/
 installed.
 
 To execute the test suite run
+$ cd tests-php
 $ phpunit
 
 It will read phpunit.xml which points to the library, test path, bootstrap

vagrant-provision.sh

@@ -0,0 +1,170 @@
+#!/bin/bash
+
+
+## During 'vagrant provision' this script runs as root and the current
+## directory is '/root'
+USERNAME=vagrant
+
+### 
+### maybe create ubuntu user
+### 
+
+# if [[ ! `id -u $USERNAME` ]]; then
+#   useradd $USERNAME --create-home --shell /bin/bash
+#   
+#   # give sudo power
+#   echo "$USERNAME ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/99-$USERNAME-user
+#   chmod 0440 /etc/sudoers.d/99-$USERNAME-user
+#   service sudo restart
+#   
+#   # add basic .profile
+#   cp -r .ssh .profile .bashrc /home/$USERNAME/
+#   chown -R $USERNAME /home/$USERNAME/.*
+#   chgrp -R $USERNAME /home/$USERNAME/.*
+#   
+#   # now ideally login as $USERNAME and continue
+#   su $USERNAME -l
+# fi
+
+
+sudo apt-get update -qq
+sudo apt-get upgrade -y
+# sudo apt-get install -y git-core screen
+sudo apt-get install -y build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev \
+                        libtool automake libproj-dev libboost-dev  libboost-system-dev \
+                        libboost-filesystem-dev libboost-thread-dev
+sudo apt-get autoremove -y
+
+# get arrow-keys working in terminal (e.g. editing in vi)
+echo 'stty sane' >> ~/.bash_profile
+echo 'export TERM=linux' >> ~/.bash_profile
+source ~/.bash_profile
+
+
+###
+### PostgreSQL 9.3 + PostGIS 2.1
+###
+
+sudo apt-get install -y postgresql-9.3-postgis-2.1 postgresql-contrib-9.3 postgresql-server-dev-9.3 
+# already included: proj-bin libgeos-dev
+
+# make sure OS-authenticated users (e.g. $USERNAME) can access
+sudo sed -i "s/ident/trust/" /etc/postgresql/9.3/main/pg_hba.conf
+sudo sed -i "s/md5/trust/"   /etc/postgresql/9.3/main/pg_hba.conf
+sudo sed -i "s/peer/trust/"  /etc/postgresql/9.3/main/pg_hba.conf
+sudo /etc/init.d/postgresql restart
+
+# creates the role
+sudo -u postgres createuser -s $USERNAME
+
+
+
+###
+### PHP for frontend
+###
+sudo apt-get install -y php5 php5-pgsql php-pear 
+sudo pear install DB
+
+
+# get rid of some warning
+# where is the ini file? 'php --ini'
+echo "date.timezone = 'Etc/UTC'" | sudo tee /etc/php5/cli/conf.d/99-timezone.ini > /dev/null
+
+
+
+###
+### Nominatim
+###
+sudo apt-get install -y libprotobuf-c0-dev protobuf-c-compiler \
+                        libgeos-c1 libgeos++-dev \
+                        lua5.2 liblua5.2-dev
+
+# git clone --recursive https://github.com/twain47/Nominatim.git
+
+
+# now ideally login as $USERNAME and continue
+su $USERNAME -l
+cd /home/vagrant/Nominatim
+
+# cd ~/Nominatim
+./autogen.sh
+./configure
+make
+chmod +x ./
+chmod +x ./module
+
+
+LOCALSETTINGS_FILE='settings/local.php'
+if [[ -e "$LOCALSETTINGS_FILE" ]]; then
+  echo "$LOCALSETTINGS_FILE already exist, writing to settings/local-vagrant.php instead."
+  LOCALSETTINGS_FILE='settings/local-vagrant.php'
+fi
+
+# IP=`curl -s http://bot.whatismyipaddress.com`
+IP=localhost
+echo "<?php
+   // General settings
+   @define('CONST_Database_DSN', 'pgsql://@/nominatim');
+   // Paths
+   @define('CONST_Postgresql_Version', '9.3');
+   @define('CONST_Postgis_Version', '2.1');
+   // Website settings
+   @define('CONST_Website_BaseURL', 'http://$IP:8089/nominatim/');
+" > $LOCALSETTINGS_FILE
+
+
+
+
+
+
+
+###
+### Setup Apache/website
+###
+
+sudo -u postgres createuser -SDR www-data
+
+echo '
+Listen 8089
+<VirtualHost *:8089>
+    # DirectoryIndex index.html
+    # ErrorDocument 403 /index.html
+
+    DocumentRoot "/var/www/"
+ 
+    <Directory "/var/www/nominatim/">
+      Options FollowSymLinks MultiViews
+      AddType text/html   .php     
+    </Directory>
+</VirtualHost>
+' | sudo tee /etc/apache2/sites-enabled/nominatim.conf > /dev/null
+
+
+sudo apache2ctl graceful
+
+
+sudo mkdir -m 755 /var/www/nominatim
+sudo chown $USERNAME /var/www/nominatim
+./utils/setup.php --threads 1 --create-website /var/www/nominatim
+
+
+# if you get 'permission denied for relation word', then try
+# GRANT usage ON SCHEMA public TO "www-data";
+# GRANT SELECT ON ALL TABLES IN SCHEMA public TO "www-data";
+
+##
+## Test suite (Python)
+## https://github.com/twain47/Nominatim/tree/master/tests
+##
+sudo apt-get install -y python-dev python-pip python-Levenshtein tidy
+sudo pip install lettuce nose pytidylib haversine psycopg2 shapely
+
+##
+## Test suite (PHP)
+## https://github.com/twain47/Nominatim/tree/master/tests-php
+##
+wget --no-clobber -q https://phar.phpunit.de/phpunit.phar
+chmod +x phpunit.phar
+sudo mv phpunit.phar /usr/local/bin/phpunit
+
+