Getting started guide for the server development

If you want to start to develop own features for the server, then follow the following steps to setup your own development environment.

Preamble

This whole guide is based on Ubuntu 16.04 LTS with python 3.5. Ubuntu 12.04+ LTS and Debian based systems should be similar if not even identical. Our current supported python versions are python 2.7 and python 3.5+.

We assume that you already have somewhere a Postgres database running. If not follow the guide to setup postgres.

Installation

  1. Install some generic stuff

    sudo apt-get update
    sudo apt-get install -y \
            git \
            libyaml-dev \
            libpython3-dev \
            libpq-dev \
            libffi-dev \
            python3-dev \
            python3-pip \
            python3-psycopg2 \
            postgresql-client
    sudo pip3 install --upgrade pip
    
  2. Clone git repository

    git clone https://gitlab.com/psono/psono-server.git ~/psono-server
    
  3. Checkout new branch

    cd ~/psono-server
    git fetch
    git checkout develop
    git checkout -b [name_of_your_new_branch]
    
  4. Install python requirements

    sudo pip3 install -r requirements.txt
    sudo pip3 install -r requirements-dev.txt
    
  5. Create settings folder

    mkdir ~/.psono_server
    
  6. Create a settings.yaml in ~/.psono_server/ with the following content

    # generate the following six parameters with the following command
    # python3 ~/psono-server/psono/manage.py generateserverkeys
    SECRET_KEY: 'SOME SUPER SECRET KEY THAT SHOULD BE RANDOM AND 32 OR MORE DIGITS LONG'
    ACTIVATION_LINK_SECRET: 'SOME SUPER SECRET ACTIVATION LINK SECRET THAT SHOULD BE RANDOM AND 32 OR MORE DIGITS LONG'
    DB_SECRET: 'SOME SUPER SECRET DB SECRET THAT SHOULD BE RANDOM AND 32 OR MORE DIGITS LONG'
    EMAIL_SECRET_SALT: '$2b$12$XUG.sKxC2jmkUvWQjg53.e'
    PRIVATE_KEY: '302650c3c82f7111c2e8ceb660d32173cdc8c3d7717f1d4f982aad5234648fcb'
    PUBLIC_KEY: '02da2ad857321d701d754a7e60d0a147cdbc400ff4465e1f57bc2d9fbfeddf0b'
        
    # The URL of the web client (path to e.g activate.html without the trailing slash)
    # WEB_CLIENT_URL: 'https://www.psono.pw'
        
    # Switch DEBUG to false if you go into production
    DEBUG: True
        
    # Adjust this according to Django Documentation https://docs.djangoproject.com/en/1.10/ref/settings/
    ALLOWED_HOSTS: ['*']
        
    # Should be your domain without "www.". Will be the last part of the username
    ALLOWED_DOMAINS: ['psono.pw']
    
    # If you want to disable registration, you can comment in the following line
    # ALLOW_REGISTRATION: False
    
    # If you want to restrict registration to some email addresses you can specify here a list of domains to filter
    # REGISTRATION_EMAIL_FILTER: ['company1.com', 'company2.com']
        
    # Should be the URL of the host under which the host is reachable
    # If you open the url you should have a text similar to {"detail":"Authentication credentials were not provided."}
    HOST_URL: 'https://www.psono.pw/server'
        
    # The email used to send emails, e.g. for activation
    # Not necessary if you do not plan to develop around the user activation
    EMAIL_FROM: '[email protected]'
    EMAIL_HOST: 'localhost'
    EMAIL_HOST_USER: ''
    EMAIL_HOST_PASSWORD : ''
    EMAIL_PORT: 25
    EMAIL_SUBJECT_PREFIX: ''
    EMAIL_USE_TLS: False
    EMAIL_USE_SSL: False
    EMAIL_SSL_CERTFILE:
    EMAIL_SSL_KEYFILE:
    EMAIL_TIMEOUT:
        
    # In case one wants to use mailgun, comment in below lines and provide the mailgun access key and server name
    # EMAIL_BACKEND: 'django_mailgun.MailgunBackend'
    # MAILGUN_ACCESS_KEY: ''
    # MAILGUN_SERVER_NAME: ''
        
    # In case you want to offer Yubikey support, create a pair of credentials here https://upgrade.yubico.com/getapikey/
    # and update the following two lines before commenting them in
    # YUBIKEY_CLIENT_ID: '123456'
    # YUBIKEY_SECRET_KEY: '8I65IA6ASDFIUHGIH5021FKJA='
    
    # If you have own Yubico servers, you can specify here the urls as a list
    # YUBICO_API_URLS: ['https://api.yubico.com/wsapi/2.0/verify']
        
    # Cache enabled without belows Redis may lead to unexpected behaviour
        
    # Cache with Redis
    # By default you should use something different than database 0 or 1, e.g. 13 (default max is 16, can be configured in
    # redis.conf) possible URLS are:
    #    redis://[:password]@localhost:6379/0
    #    rediss://[:password]@localhost:6379/0
    #    unix://[:password]@/path/to/socket.sock?db=0
    # CACHE_ENABLE: False
    # CACHE_REDIS: False
    # CACHE_REDIS_LOCATION: 'redis://127.0.0.1:6379/13'
        
    # Disables Throttling (necessary for unittests to pass) by overriding the cache with a dummy cache
    # https://docs.djangoproject.com/en/1.11/topics/cache/#dummy-caching-for-development
    # THROTTLING: False
    
    # Enables the management API, required for the psono-admin-client / admin portal
    # MANAGEMENT_ENABLED: False
    
    # Allows that users can search for partial usernames
    # ALLOW_USER_SEARCH_BY_USERNAME_PARTIAL: True
    
    # Allows that users can search for email addresses too
    # ALLOW_USER_SEARCH_BY_EMAIL: True
    
    # Only necessary if the psono-client runs on a sub path (no trailing slash) e.g. "https://wwww.psono.pw"
    # WEB_CLIENT_URL: ''
    
    # Prevents the use of the last X passwords. 0 disables it.
    # DISABLE_LAST_PASSWORDS: 0
        
    # Your Postgres Database credentials
    DATABASES:
        default:
            'ENGINE': 'django.db.backends.postgresql_psycopg2'
            'NAME': 'psono'
            'USER': 'psono'
            'PASSWORD': 'password'
            'HOST': 'localhost'
            'PORT': '5432'
    # for master / slave replication setup comment in the following (all reads will be redirected to the slave
    #    slave:
    #        'ENGINE': 'django.db.backends.postgresql_psycopg2'
    #        'NAME': 'YourPostgresDatabase'
    #        'USER': 'YourPostgresUser'
    #        'PASSWORD': 'YourPostgresPassword'
    #        'HOST': 'YourPostgresHost'
    #        'PORT': 'YourPostgresPort'
        
    # Update the path to your templates folder
    TEMPLATES: [
        {
            'BACKEND': 'django.template.backends.django.DjangoTemplates',
            'DIRS': ['/home/psono/psono-server/psono/templates'],
            'APP_DIRS': True,
            'OPTIONS': {
                'context_processors': [
                    'django.template.context_processors.debug',
                    'django.template.context_processors.request',
                    'django.contrib.auth.context_processors.auth',
                    'django.contrib.messages.context_processors.messages',
                ],
            },
        },
    ]
    
  7. Test E-Mail (optional)

    The most tedious step is usually for me to get e-mail working. To make this step easier, we offer a small test script which will send a test e-mail.

    To send a test e-mail to [email protected] execute:

    python3 ~/psono-server/psono/manage.py sendtestmail [email protected]
    

    If you receive this test e-mail, e-mail should be configured proper.

  8. Create our database

    python3  ~/psono-server/psono/manage.py migrate
    

Run the dev server

From this point on forward, you can develop it like any other django application

To start the server you can do for example:

python3 ~/psono-server/psono/manage.py runserver 0.0.0.0:10100

This will start the Psono server on port 10100. If you open now http://your-ip:10100 you should see something like this:

{"detail":"Authentication credentials were not provided."}

If not, please make sure you have no firewall on the server blocking you.

Update database model

If you ever change parts of the model, then you can create the migration script with the following command

python3  ~/psono-server/psono/manage.py makemigrations restapi

and apply it then to your postgres installation with:

python3  ~/psono-server/psono/manage.py migrate

Create user on the commandline:

If you are developing and don’t want to setup email or go to the database to activate a user, you can just create a user on the command line with the following command:

  python3 ./psono/manage.py createuser [email protected] myPassword [email protected]

Other useful commands can be found under Commands

Run Unit Tests (with coverage)

To run unit tests, the database user needs CREATEDB rights.

coverage run --source='.' ./psono/manage.py test restapi.tests administration.tests

To get a nice report one can do:

coverage report --omit=psono/restapi/migrations/*,psono/restapi/tests*,psono/administration/migrations/*,psono/administration/tests*

or:

coverage html --omit=psono/restapi/migrations/*,psono/restapi/tests*,psono/administration/migrations/*,psono/administration/tests*

The output of this command can be shown on https://your-ip/htmlcov/

Run local dummy smtp server

If you want to debug e-mails, like those sent during the registration, one can start a local dummy smtp server with the following command

sudo python -m smtpd -n -c DebuggingServer localhost:25
Edit me