================
Access
================

The repository of the MatD\ :sup:`3` database is hosted by GitHub:

  https://github.com/HybriD3-database/MatD3

It's a public project, so anybody is able to clone the source code and create issues and merge requests. In order to contribute to the project directly, contact one of the MatD\ :sup:`3` team members who will then grant you the developer status. For pull/push privileges you must upload your RSA public key to GitHub. If you already have one, it is likely located at ``~/.ssh/id_rsa.pub``. If not, the RSA public/private key pair can be generated by issuing

.. code:: bash

  ssh-keygen

It is sufficient to press enter at each query until the keys have been generated. However, protecting the key with a password would be a good idea if you intend to keep it on a public server. Then, at the GitHub website, go to your settings, find "SSH keys", and enter the public key. You can upload several keys in order to have pull/push access from more than one computer. 

From a command line prompt (i.e., using a terminal application, which is usually provided along with any Unix/Linux distribution or with MacOS), you can now clone the Git repository with

.. code:: bash

  git clone https://github.com/HybriD3-database/MatD3.git

The result of this command should be the creation of a directory ``MatD3`` that contains the software needed to operate the MatD3 database. 

================
Setup
================

A detailed set of setup notes is included in the file ``Readme.md`` that is provided directly with the source code. You can access these instructions here:

https://github.com/HybriD3-database/MatD3/blob/master/Readme.md

Before proceeding, you will typically need a few other software packages to be installed. For instance, you will typically want to install a SQL database, program langage headers and a driver that provides a HTTP interface, needed in order to display the output of MatD3 in a browser.

For example, on Ubuntu, you may need:

.. code:: bash

  sudo apt install libmysqlclient-dev
  sudo apt install python3-dev
  sudo apt install firefox-geckodriver

On MacOS, open source software is provided by the Homebrew project, so you may need to install Homebrew and, once that is done, install

.. code:: bash

  brew install mariadb
  brew install geckodriver

With the prerequisites installed, change to the directory ``MatD3`` by indicating

.. code:: bash

  cd MatD3

In order to ensure compatibility with the correct Python packages, it is important to do any development in a Python virtual environment. You can create a virtual environment with

.. code:: bash

  python3 -m venv venv

and activate the environment with

.. code:: bash

  source venv/bin/activate

In some instances of python, the relevant command might be found in a directory called ``venv/Scripts/activate`` instead.

You'll notice that the shell's prompt has changed to remind you that you are in a virtual environment. Any packages installed with Python's ``pip`` command are now part of the current project only. 

Typically, you will also need to update the pip command to a current version:

.. code:: bash

  pip install --upgrade pip

The correct versions of the packages that are required for developing the MatD\ :sup:`3` database are listed in ``requirements.txt``, which is a file located in the ``MatD3`` directory. From a command line prompt with the ``MatD3`` directory as the working directory, install the requirements by issuing

.. code:: bash

  pip install -r requirements.txt

When you make changes to the requirements, such as upgrading a Python package, use

.. code:: bash

  pip freeze > requirements.txt

to record the new list of requirements. Since this will overwrite the current requirements file, it is important to apply the old requirements first before making any changes.

Before starting the server, it is necessary to configure it in order to reflect your specific environment. The configuration variables are read from a file called ``.env`` in the root directory. You need to create this file or make a copy of ``env.example`` in the root directory and edit it.

  **SECRET_KEY**
    Used for cryptographic signing. The default value is only for quickly setting up the server and should not be used in practice.
  **MATD3_NAME**
    Name of this instance of the server
  **MATD3_URL**
    URL of this instance of the server
  **ALLOWED_HOSTS**
    List of host/domain names that this instance can serve
  **EMAIL_HOST**
    The host to use for sending email.
  **EMAIL_HOST_USER**
    Username to use for the SMTP server defined in EMAIL_HOST
  **EMAIL_HOST_PASSWORD**
    Password to use for the SMTP server defined in EMAIL_HOST
  **SELENIUM_DRIVER**
    Which driver to use for running tests with Selenium. Options are "Firefox" and "Chrome" (case insensitive). If not present, Firefox is used.
  **USE_SQLITE**
    Whether to use the SQLite database. If false or not present, mySQL is used instead.
  **DEBUG**
    Whether to run MatD\ :sup:`3` in debug mode. This is useful for quickly setting up and testing the website but should be removed when serving on a production server.
    
================
Some troubleshooting notes
================

The packages listed in ``requirements.txt`` are many. The ``pip`` command should handle incompatibilities, but it does not handle all of them and package incompatibilities can still creep in. If that happens, the output of ``pip`` can, unfortunately, be rather cryptic.

One common consequence of such incompatbilities is that the web server does not start up properly and all you see from your browser is a "502 error" without any useful details.

In this case, the next way forward is to manually log in to the server in question and attempt to understand errors that the different components of Django produce. 

Specifically, Django can run based on a combination of servers called ``nginx`` and ``gunicorn``. The following blog post explains their interaction and, importantly, where the log files are and which commands can be used for troubleshooting:

  https://www.datadoghq.com/blog/nginx-502-bad-gateway-errors-gunicorn/

At this point, the exercise becomes one of Linux administration and starting and stopping services on the server that runs the database. For example, the ``sysctl`` command (which controls services running on a particular server) becomes involved.

For instance, this command here, executed at the command line, will try to start a MatD3 service:

.. code:: bash

  sudo systemctl status (name).service -l

where "(name)" needs to be replaced by the specific name chosen during the server setup of your particular MatD3 instance.

The following command, executed in the directory "/var/www/MatD3", restarts the gunicorn service manually:

.. code:: bash

  sudo /var/www/MatD3/venv/bin/gunicorn -t 3600 --workers 2 --bind unix:/run/(name).sock mainproject.wsgi

(Again, "(name)" needs to be replaced by the specific name chosen during the server setup.)