mention php-symfony-dotenv under advanced installations

The SQL code uses array_agg with arrays which is only supported
for Postgresql 9.5. We could provide a workaround but given that
9.3/9.4 are not tested at all there is a good chance that there
are other problems.

See #2394.

Vagrantfile

@@ -30,7 +30,7 @@ Vagrant.configure("2") do |config|
     lv.memory = 2048
     lv.nested = true
     if ENV['CHECKOUT'] != 'y' then
-      override.vm.synced_folder ".", "/home/vagrant/Nominatim", type: 'nfs'
+      override.vm.synced_folder ".", "/home/vagrant/Nominatim", type: 'rsync'
     end
   end
 

docs/admin/Advanced-Installations.md

@@ -5,6 +5,7 @@ your Nominatim database. It is assumed that you have already successfully
 installed the Nominatim software itself, if not return to the 
 [installation page](Installation.md).
 
+
 ## Importing multiple regions
 
 To import multiple regions in your database, you need to configure and run `utils/import_multiple_regions.sh` file. This script will set up the update directory which has the following structure:
@@ -27,6 +28,14 @@ update
 
 The `sequence.state` files will contain the sequence ID, which will be used by pyosmium to get updates. The tmp folder is used for import dump.
 
+### Installing PHP prerequisites
+
+The scripts this section still use the old PHP utils. They require
+the [Symphony dotenv](https://symfony.com/doc/4.1/components/dotenv.html)
+parser to work. To install it on Ubuntu/Debian, run:
+
+    sudo apt install php-symfony-dotenv
+
 ### Configuring multiple regions
 
 The file `import_multiple_regions.sh` needs to be edited as per your requirement:

docs/admin/Deployment.md

@@ -1,7 +1,7 @@
 # Deploying Nominatim
 
 The Nominatim API is implemented as a PHP application. The `website/` directory
-in the build directory contains the configured website. You can serve this
+in the project directory contains the configured website. You can serve this
 in a production environment with any web server that is capable to run
 PHP scripts.
 
@@ -13,10 +13,11 @@ to run a web service. Please refer to the documentation of
 for background information on configuring the services.
 
 !!! Note
-    Throughout this page, we assume that your Nominatim build directory is
+    Throughout this page, we assume that your Nominatim project directory is
-    located in `/srv/nominatim/build` and the source code in
+    located in `/srv/nominatim-project` and that you have installed Nominatim
-    `/srv/nominatim/Nominatim`. If you have put it somewhere else, you
+    using the default installation prefix `/usr/local`. If you have put it
-    need to adjust the commands and configuration accordingly.
+    somewhere else, you need to adjust the commands and configuration
+    accordingly.
 
     We further assume that your web server runs as user `www-data`. Older
     versions of CentOS may still use the user name `apache`. You also need
@@ -29,7 +30,7 @@ web server user. You can check that the permissions are correct by accessing
 on of the php files as the web server user:
 
 ``` sh
-sudo -u www-data head -n 1 /srv/nominatim/build/website/search.php
+sudo -u www-data head -n 1 /srv/nominatim-project/website/search.php
 ```
 
 If this shows a permission error, then you need to adapt the permissions of
@@ -40,11 +41,11 @@ web server access. At a minimum the following SELinux labelling should be done
 for Nominatim:
 
 ``` sh
-sudo semanage fcontext -a -t httpd_sys_content_t "/srv/nominatim/Nominatim/(website|lib|settings)(/.*)?"
+sudo semanage fcontext -a -t httpd_sys_content_t "/usr/local/nominatim/lib/lib-php(/.*)?"
-sudo semanage fcontext -a -t httpd_sys_content_t "/srv/nominatim/build/(website|settings)(/.*)?"
+sudo semanage fcontext -a -t httpd_sys_content_t "/srv/nominatim-project/website(/.*)?"
-sudo semanage fcontext -a -t lib_t "/srv/nominatim/build/module/nominatim.so"
+sudo semanage fcontext -a -t lib_t "/srv/nominatim-project/module/nominatim.so"
-sudo restorecon -R -v /srv/nominatim/Nominatim
+sudo restorecon -R -v /usr/local/lib/nominatim
-sudo restorecon -R -v /srv/nominatim/build
+sudo restorecon -R -v /srv/nominatim-project
 ```
 
 ## Nominatim with Apache
@@ -65,13 +66,13 @@ Make sure your Apache configuration contains the required permissions for the
 directory and create an alias:
 
 ``` apache
-<Directory "/srv/nominatim/build/website">
+<Directory "/srv/nominatim-project/website">
   Options FollowSymLinks MultiViews
   AddType text/html   .php
   DirectoryIndex search.php
   Require all granted
 </Directory>
-Alias /nominatim /srv/nominatim/build/website
+Alias /nominatim /srv/nominatim-project/website
 ```
 
 After making changes in the apache config you need to restart apache.
@@ -110,7 +111,7 @@ Tell nginx that php files are special and to fastcgi_pass to the php-fpm
 unix socket by adding the location definition to the default configuration.
 
 ``` nginx
-root /srv/nominatim/build/website;
+root /srv/nominatim-project/website;
 index search.php;
 location / {
     try_files $uri $uri/ @php;

docs/admin/Installation.md

@@ -37,7 +37,7 @@ For compiling:
 
 For running Nominatim:
 
-  * [PostgreSQL](https://www.postgresql.org) (9.3+ will work, 11+ strongly recommended)
+  * [PostgreSQL](https://www.postgresql.org) (9.5+ will work, 11+ strongly recommended)
   * [PostGIS](https://postgis.net) (2.2+)
   * [Python 3](https://www.python.org/) (3.5+)
   * [Psycopg2](https://www.psycopg.org) (2.7+)

docs/admin/Migration.md

@@ -17,6 +17,17 @@ breaking changes. **Please read them before running the migration.**
 
 ## 3.6.0 -> 3.7.0
 
+### New format and name of configuration file
+
+The configuration for an import is now saved in a `.env` file in the project
+directory. This file follows the dotenv format. For more information, see
+the [installation chapter](Import.md#configuration-setup-in-env).
+
+To migrate to the new system, create a new project directory, add the `.env`
+file and port your custom configuration from `settings/local.php`. Most
+settings are named similar and only have received a `NOMINATIM_` prefix.
+Use the default settings in `settings/env.defaults` as a reference.
+
 ### New location for data files
 
 External data files for Wikipedia importance, postcodes etc. are no longer

docs/admin/Setup-Nominatim-UI.md

@@ -10,12 +10,11 @@ installation. For more details, please also have a look at the
 
 ## Installing nominatim-ui
 
-nominatim-ui does not need any special installation, just download, configure
+We provide regular releases of nominatim-ui that contain the packaged website.
-and run it.
+They do not need any special installation. Just download, configure
-
+and run it. Grab the latest release from
-Clone the source from github:
+[nominatim-ui's Github release page](https://github.com/osm-search/nominatim-ui/releases)
-
+and unpack it. You can use `nominatim-ui-x.x.x.tar.gz` or `nominatim-ui-x.x.x.zip`.
-    git clone https://github.com/osm-search/nominatim-ui
 
 Copy the example configuration into the right place:
 

docs/mkdocs.yml

@@ -1,4 +1,4 @@
-site_name: Nominatim Documentation
+site_name: Nominatim 3.7.2
 theme: readthedocs
 docs_dir: ${CMAKE_CURRENT_BINARY_DIR}
 site_url: https://nominatim.org

vagrant/Install-on-Centos-7.sh

@@ -42,7 +42,7 @@
                         python3-pip python3-setuptools python3-devel \
                         expat-devel zlib-devel libicu-dev
 
-    pip3 install --user psycopg2 python-dotenv psutil Jinja2 PyICU argparse-manpage
+    pip3 install --user psycopg2 python-dotenv psutil Jinja2 PyICU
 
 
 #
@@ -113,30 +113,22 @@ sudo chown vagrant /srv/nominatim  #DOCS:
 # Building and Configuration
 # --------------------------
 #
-# Get the source code from Github and change into the source directory
+# Get the source for the release and unpack it
 #
 if [ "x$1" == "xyes" ]; then  #DOCS:    :::sh
-    cd $USERHOME
+    wget https://nominatim.org/release/Nominatim-3.7.2.tar.bz2
-    git clone --recursive git://github.com/openstreetmap/Nominatim.git
+    tar xf Nominatim-3.7.2.tar.bz2
-    cd Nominatim
 else                               #DOCS:
     cd $USERHOME/Nominatim         #DOCS:
 fi                                 #DOCS:
 
-# When installing the latest source from github, you also need to
-# download the country grid:
-
-if [ ! -f data/country_osm_grid.sql.gz ]; then       #DOCS:    :::sh
-    wget --no-verbose -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
-fi                                 #DOCS:
-
 # The code must be built in a separate directory. Create this directory,
 # then configure and build Nominatim in there:
 
 #DOCS:    :::sh
     mkdir $USERHOME/build
     cd $USERHOME/build
-    cmake $USERHOME/Nominatim
+    cmake $USERHOME/Nominatim-3.7.2
     make
     sudo make install
 

vagrant/Install-on-Centos-8.sh

@@ -22,6 +22,7 @@
 
     sudo dnf -qy module disable postgresql
     sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
+    export PATH=/usr/pgsql-12/bin/:$PATH
 
 # Now you can install all packages needed for Nominatim:
 
@@ -33,9 +34,9 @@
                         php-pgsql php php-intl php-json libpq-devel \
                         bzip2-devel proj-devel boost-devel \
                         python3-pip python3-setuptools python3-devel \
-                        expat-devel zlib-devel libicu-dev
+                        expat-devel zlib-devel libicu-devel
 
-    pip3 install --user psycopg2 python-dotenv psutil Jinja2 PyICU argparse-manpage
+    pip3 install --user psycopg2-binary python-dotenv psutil Jinja2 PyICU
 
 
 #
@@ -107,21 +108,12 @@ sudo chown vagrant /srv/nominatim  #DOCS:
 # Building and Configuration
 # --------------------------
 #
-# Get the source code from Github and change into the source directory
+# Get the source code for the release and unpack it
 #
 if [ "x$1" == "xyes" ]; then  #DOCS:    :::sh
     cd $USERHOME
-    git clone --recursive git://github.com/openstreetmap/Nominatim.git
+    wget https://nominatim.org/release/Nominatim-3.7.2.tar.bz2
-    cd Nominatim
+    tar xf Nominatim-3.7.2.tar.bz2
-else                               #DOCS:
-    cd $USERHOME/Nominatim         #DOCS:
-fi                                 #DOCS:
-
-# When installing the latest source from github, you also need to
-# download the country grid:
-
-if [ ! -f data/country_osm_grid.sql.gz ]; then       #DOCS:    :::sh
-    wget --no-verbose -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
 fi                                 #DOCS:
 
 # The code must be built in a separate directory. Create this directory,
@@ -130,7 +122,11 @@ fi                                 #DOCS:
 #DOCS:    :::sh
     mkdir $USERHOME/build
     cd $USERHOME/build
-    cmake $USERHOME/Nominatim
+if [ "x$1" == "xyes" ]; then  #DOCS:
+    cmake $USERHOME/Nominatim-3.7.2
+else                               #DOCS:
+    cmake /home/vagrant/Nominatim
+fi                                 #DOCS:
     make
     sudo make install
 
@@ -179,10 +175,10 @@ sudo sed -i 's:#.*::' /etc/httpd/conf.d/nominatim.conf #DOCS:
 # with a web server accessible from the Internet. At a minimum the
 # following SELinux labeling should be done for Nominatim:
 
-    sudo semanage fcontext -a -t httpd_sys_content_t "/usr/local/nominatim/lib/lib-php(/.*)?"
+    sudo semanage fcontext -a -t httpd_sys_content_t "/usr/local/lib/nominatim/lib-php(/.*)?"
     sudo semanage fcontext -a -t httpd_sys_content_t "$USERHOME/nominatim-project/website(/.*)?"
     sudo semanage fcontext -a -t lib_t "$USERHOME/nominatim-project/module/nominatim.so"
-    sudo restorecon -R -v /usr/local/lib/nominatim
+    sudo restorecon -R -v /usr/local/lib64/nominatim
     sudo restorecon -R -v $USERHOME/nominatim-project
 
 

vagrant/Install-on-Ubuntu-18.sh

@@ -30,8 +30,7 @@ export DEBIAN_FRONTEND=noninteractive #DOCS:
                         postgresql-server-dev-10 postgresql-10-postgis-2.4 \
                         postgresql-contrib-10 postgresql-10-postgis-scripts \
                         php php-pgsql php-intl libicu-dev python3-pip \
-                        python3-psycopg2 python3-psutil python3-jinja2 python3-icu git \
+                        python3-psycopg2 python3-psutil python3-jinja2 python3-icu
-                        python3-argparse-manpage
 
 # The python-dotenv package that comes with Ubuntu 18.04 is too old, so
 # install the latest version from pip:
@@ -98,29 +97,22 @@ export DEBIAN_FRONTEND=noninteractive #DOCS:
 # Building and Configuration
 # --------------------------
 #
-# Get the source code from Github and change into the source directory
+# Get the source code for the release and unpack it
 #
 if [ "x$1" == "xyes" ]; then  #DOCS:    :::sh
     cd $USERHOME
-    git clone --recursive git://github.com/openstreetmap/Nominatim.git
+    wget https://nominatim.org/release/Nominatim-3.7.2.tar.bz2
-    cd Nominatim
+    tar xf Nominatim-3.7.2.tar.bz2
 else                               #DOCS:
     cd $USERHOME/Nominatim         #DOCS:
 fi                                 #DOCS:
 
-# When installing the latest source from github, you also need to
-# download the country grid:
-
-if [ ! -f data/country_osm_grid.sql.gz ]; then       #DOCS:    :::sh
-    wget -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
-fi                                 #DOCS:
-
 # The code must be built in a separate directory. Create this directory,
 # then configure and build Nominatim in there:
 
     mkdir $USERHOME/build
     cd $USERHOME/build
-    cmake $USERHOME/Nominatim
+    cmake $USERHOME/Nominatim-3.7.2
     make
     sudo make install
 

vagrant/Install-on-Ubuntu-20.sh

@@ -33,8 +33,7 @@ export DEBIAN_FRONTEND=noninteractive #DOCS:
                         postgresql-server-dev-12 postgresql-12-postgis-3 \
                         postgresql-contrib-12 postgresql-12-postgis-3-scripts \
                         php php-pgsql php-intl libicu-dev python3-dotenv \
-                        python3-psycopg2 python3-psutil python3-jinja2 python3-icu git \
+                        python3-psycopg2 python3-psutil python3-jinja2 python3-icu
-                        python3-argparse-manpage
 
 #
 # System Configuration
@@ -96,29 +95,22 @@ export DEBIAN_FRONTEND=noninteractive #DOCS:
 # Building and Configuration
 # --------------------------
 #
-# Get the source code from Github and change into the source directory
+# Get the source code for the release and unpack it
 #
 if [ "x$1" == "xyes" ]; then  #DOCS:    :::sh
     cd $USERHOME
-    git clone --recursive git://github.com/openstreetmap/Nominatim.git
+    wget https://nominatim.org/release/Nominatim-3.7.2.tar.bz2
-    cd Nominatim
+    tar xf Nominatim-3.7.2.tar.bz2
 else                               #DOCS:
     cd $USERHOME/Nominatim         #DOCS:
 fi                                 #DOCS:
 
-# When installing the latest source from github, you also need to
-# download the country grid:
-
-if [ ! -f data/country_osm_grid.sql.gz ]; then       #DOCS:    :::sh
-    wget -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
-fi                                 #DOCS:
-
 # The code must be built in a separate directory. Create this directory,
 # then configure and build Nominatim in there:
 
     mkdir $USERHOME/build
     cd $USERHOME/build
-    cmake $USERHOME/Nominatim
+    cmake $USERHOME/Nominatim-3.7.2
     make
     sudo make install