May 2, 2020

Setting up multiple Django projects with Nginx and Gunicorn

1. Prerequisites:

OS:

Ubuntu 18.04 with user account having sudo access. (root user access should be disabled, in general, for security and can be done using the command: sudo passwd -l root )

Update the sources

sudo apt-get update
sudo apt-get upgrade

Install Build-essentials:

sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \ 
libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ 
xz-utils tk-dev libffi-dev liblzma-dev python-openssl git

Setup Python environment:

Install pyenv and pyenv virtualenv

git clone https://github.com/pyenv/pyenv.git ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bashrc

source .bashrc

git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv

echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc

Install python 3.6.5 and set it as the default python interpreter

pyenv install 3.6.5 
pyenv global 3.6.5
pip install --upgrade

2. Project setup

All the automation scripts for the manual process described below can be obtained from the following git repository

First we make our sites folder to contain all our projects, create two sample projects and set the virtual environments

mkdir sites && cd sites
django-admin startproject site1
django-admin startproject site2

Create a virtualenv for each project

pyenv virtualenv site1
pyenv virtualenv site2

Set virtualenvs, install Gunicorn & Django

cd site1 && pyenv local site1 
pip install gunicorn Django && cd ..
cd site2 && pyenv local site2 
pip install gunicorn Django && cd ..

3. Localhost setup

Use the following setup to allow different sites to be addressed on the localhost. Open the /etc/hosts file

sudo nano /etc/hosts

and add to it the hosts site1.test site2.test such that the line reads

127.0.0.1 localhost site1.test site2.test

This will be useful in testing on a local development environment and should only be done for this purpose, you should skip this step in a production server.

4. Gunicorn setup

The below steps can be executed automatically by running the python script from the linked repository:

python gunicorn_setup.py USERNAME SITEX

(replace the CAPITAL text with appropriate text, USERNAME with your account username and replace sitex by site1, site2 etc.)

The above script implements the following manual process:

Create a file

sudo nano /etc/systemd/system/sitex.socket

and add to it the contents

[Unit]
Description=gunicorn socket
[Socket]
ListenStream=/run/sitex.sock
[Install]
WantedBy=sockets.target

Create a service in systemd to start a gunicorn service for sitex

sudo nano /etc/systemd/system/sitex.service

Add to the file:

[Unit]
Description=gunicorn daemon
Requires=sitex.socket
After=network.target
[Service]
User=yourUsername
Group=yourUsername
WorkingDirectory=/home/yourUsername/sites/sitex
ExecStart=/home/yourUsername/.pyenv/shims/gunicorn \ 
--access-logfile - \ 
--workers 3 \ 
--bind unix:/run/sitex.sock \
sitex.wsgi:application
[Install]
WantedBy=multi-user.target

Configure the python paths for the gunicorn services

mkdir /home/yourUsername/.sitex
sudo nano /home/yourUsername/.sitex/gunicorn_config.py

To which add:

command = '/home/yourUsername/.pyenv/shims/gunicorn'
pythonpath = '/home/yourUsername/sites/sitex'
bind = '127.0.0.1:PORT'
workers = 3
user = 'yourUsername'

Next, setup the Nginx server

5. Nginx setup

Nginx installation:

sudo apt-get install nginx

The steps below can be executed automatically by simply running the script:

sudo python nginx_setup.py USERNAME SITEX SITEX.test

Alternatively, the manual process looks as follows:

Create for each site, the following file

sudo nano /etc/nginx/sites-available/sitex

Add to it the contents

server {
 listen 443 ssl http2;
 root /home/yourUsername/sitex;
 index index.html index.htm; 

 server_name siteurl; 

 #ssl_certificate /etc/ssl/certs/sitex.crt;
 #ssl_certificate_key /etc/ssl/private/sitex.key;
        
 #ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
 
 error_log /var/log/nginx/sitex.error.log;
 access_log /var/log/nginx/sitex.access.log;
 
 location /media/ {
 alias /home/yourUsername/sitex/media/;
 }
 
 location /static/ {
 alias /home/yourUsername/sitex/static/;
 }
 
 # Below is the juice. We're proxying the request to the sock we created
 # earlier via the gunicorn process via SystemD
 location / {
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header Host $http_host;
 proxy_redirect off;
 proxy_pass http://unix:/run/sitex.sock;
 }
 
 location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
 expires 30d;
 }
 
 location /robots.txt {
 alias /home/yourUsername/sitex/static/robots.txt;
 }
 
 location /favicon.ico {
 alias /home/yourUsername/sitex/static/favicon.ico;
 }
}
 
server {
 listen 80;
 server_name siteurl;
 return 301 https://$server_name$request_uri;
}

6. Securing with https access (SSL setup)

6.1 Certbot for the production environment

The above Nginx configuration is already the kind you would use in the production environment. It redirects all HTTP traffic to https and thus requires an SSL setup to work.

If this setup is being performed on the production environment and you already have a domain name, you can follow the steps in https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04 to get the ssl certificates and configure SSL in Nginx automatically.

6.2 Self-signed-certificate for the development environment

If you are working in a development environment, the configuration for SSL needs to be done manually with self-signed certificates. This is done as shown in https://www.humankode.com/ssl/create-a-selfsigned-certificate-for-nginx-in-5-minutes . The process is automated by the script ssl_setup.py .

sudo python ssl_selfsigned_setup.py sitex sitex.test

The manual process:

Create a configuration file for the site

sudo nano /etc/ssl/sitex.conf

Add to it the contents

[req]
default_bits       = 2048
default_keyfile    = sitex.key
distinguished_name = req_distinguished_name
req_extensions     = req_ext
x509_extensions    = v3_ca

[req_distinguished_name]
countryName                 = Country Name (2 letter code)
countryName_default         = US
stateOrProvinceName         = State or Province Name (full name)
stateOrProvinceName_default = New York
localityName                = Locality Name (eg, city)
localityName_default        = Rochester
organizationName            = Organization Name (eg, company)
organizationName_default    = localhost
organizationalUnitName      = organizationalunit
organizationalUnitName_default = Development
commonName                  = Common Name (e.g. server FQDN or YOUR name)
commonName_default          = localhost
commonName_max              = 64

[req_ext]
subjectAltName = @alt_names

[v3_ca]
subjectAltName = @alt_names

[alt_names]
DNS.1   = localhost
DNS.2   = 127.0.0.1

Then create the certificate and key files:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout /etc/ssl/private/sitex.key -out /etc/ssl/certs/sitex.crt \
-config /etc/ssl/sitex.conf

Add the certificate to list of trusted certificates

certutil -d sql:$HOME/.pki/nssdb -A -t "P,," -n "sitex.test" -i sitex.crt

Finally, uncomment the ssl configuration lines from the Nginx configuration file created previously for sitex (/etc/nginx/sites-available/sitex).

7. Get the server up and running:

Start the gunicorn service

sudo systemctl start sitex.socket
sudo systemctl enable sitex.socket

Check the status of the gunicorn service

sudo systemctl status sitex.socket

Reload the daemon

sudo systemctl daemon-reload

This allows services to discover the newly added gunicorn services

Start the nginx server:

First, enable the sites we want to use. https://github.com/perusio/nginx_ensite.git has a useful script for doing this (note that this can also be done manually by creating a symlink to the sitex file in the sites-availble folder to appear in the sites-enabled folder, but symlinks can be notorious for management in the long run and for disabling sites. The convenience script from the nginx_ensite makes this easier)

git clone https://github.com/perusio/nginx_ensite.git
cd nginx_ensite
sudo make install

To enable a site, use:

nginx_ensite sitex

To disable a site:

nginx_dissite sitex

To start the nginx server

sudo service nginx start

To check the status of nginx:

sudo service nginx status

To reload/restart the services after any configuration changes, use:

sudo service nginx reload

sudo service nginx restart

sudo service sitex restart

8. Django settings:

Add to ALLOWED_HOSTS : 'sitex.test' and django will start serving the project though the gunicorn + nginx setup.

Test the webpage by visiting sitex.test from your browser. In case you did not add the self signed certificate to trusted certificates already, the browser will display a security warning which you can accept and proceed to see you site.

Both site1.test and site2.test will be now serving the corresponding django projects (you can make them serve different content to see the difference)

9. Debug logs:

The nginx logs information to

error_log /var/log/nginx/sitex.error.log;
access_log /var/log/nginx/sitex.access.log;

you can view these files to see any encountered errors.