Running a Python webapp with Caddy and Gunicorn as a user

Sep 11, 2016   #caddy  #gunicorn  #server  #tutorial 

Introduction

I recently coded a little Python app (with Flask) that allows me to easily redirect subdomains from a catch-all (A record for *.example.com) to arbitrary URLs, and I called it Subdomain Redirector. At first, I just ran it via Flask’s built-in development server using lighttpd as a reverse proxy, but then decided I needed (or rather wanted) a more robust solution, maybe even something able to handle more than one request at once!

Researching how to go about this, I found out about Gunicorn, which should allow me to run my app with higher performance. Incidentally I also learned about Caddy, a webserver that has a lot of features while being fast and easy to configure. So I decided to try both of these at once.

So here’s a tutorial for setting up a Caddy server to serve Python webapps (and static websites, too!), using my webapp as an example.

Step-By-Step

This assumes a blank server and some Linux knowledge. I’m using the generic editor command here, you can change what editor it uses with sudo update-alternatives --config editor. Furthermore I use the domain example.com, you should obviously change it to whatever domain you have pointing at the server.

1: Set up a basic server

I’ll let you google how to install Ubuntu on a server, how to set up SSH and a firewall (just make sure you let through ports 80 and 443 for HTTP and HTTPS). Other people know more about this than I do, and there are many tutorials out there already.
Don’t forget to run sudo apt-get update and sudo apt-get uprade to make sure everything is up to date!

2: Creating a new user

sudo adduser www

3: Install required software with apt-get and set up systemd

sudo apt-get install virtualenv libpam-systemd git
sudo loginctl enable-linger www

4: Install software as user

sudo su - www

First we install caddy. This is a bit unorthodox but it works fine (change the arch= parameter to 386 or arm depending on your system):

curl -# "https://caddyserver.com/download/build?os=linux&arch=amd64&features="| tar -xzf - caddy

Notice: This has been updated in the follow-up post, you might want to have a look at that!

Addons can be installed by adding them to the URL above as a comma-seperated list, e.g. &features=git,minify.

Next we set up the virtualenv and install the python dependencies:

mkdir python
cd python
virtualenv venv
source venv/bin/activate
pip install gunicorn flask pyyaml
deactivate

(Note: The pyyaml dependency is only for my little app, if you have your own app, install the corresponding dependencies, same goes for flask if you use a different web framework.)

Now we install our webapp. In my case I run:

git clone https://gitlab.com/tobold/subdomain_redirector.git
cd subdomain_redirector

Edit whatever files you need to configure your app, in this example the domain variable in main.py.

5: Configure gunicorn

editor gunicorn.conf

Enter the following into the file:

bind = "127.0.0.1:9001"
workers = 3 
pidfile = "/home/www/subdomain_redirector.pid"

The number of workers depends on how many cores your server has (the gunicorn documentation recommends 2 * $Cores + 1 workers, which is what I use for this single-core example).

6: Configure caddy

cd
mkdir html
editor Caddyfile

Caddyfile is the default configuration file for caddy. Enter into it:

example.com {
    gzip
    root html
}

*.example.com:80 {
    proxy / localhost:9001 {
        transparent
    }
}

Replace example.com with your domain and put your static website in the html directory. The :80 is necessary so Caddy doesn’t try to get a SSL certificate for every possible subdomain, which would be pointless and run straight into Let’s Encrypt’s rate limiting.

Finally we need to allow caddy to use ports 80 and 443 (for which we switch back to the sudo-user):

exit
sudo setcap CAP_NET_BIND_SERVICE=+eip /home/www/caddy
sudo su - www

Notice: This has also been updated in the follow-up post, you might want to have a look at that!

7: Check if we did everything right so far

We open a screen, to run 2 servers in parallel:

screen

In here we start gunicorn:

cd python/subdomain_redirector/
../venv/bin/gunicorn -c gunicorn.conf main:app

My app has a main.py and the Flask object is called app, so change that to whatever your app uses.

Next we press Ctrl+A and then C to open another terminal, where we start caddy (change email option to your email-address):

./caddy -agree -pidfile=/home/www/caddy.pid -email youremail@example.com

Ignore the warning about the file descriptor limit for now, we will fix this in the next step.

Now test your app. In this example we point a web browser at http://xkcd.example.com and it redirects us to Explain XKCD.

If everything works, press Ctrl+C to stop the server and type exit to close the terminal. Repeat for the other server.

If it doesn’t work you’ll have to debug. See if one of the servers throws an error, press Ctrl+A followed by N to switch between the two terminals.

8: Make it all reboot-safe with systemd

mkdir -p .config/systemd/user
cd .config/systemd/user/
editor caddy.service

Into it enter:

[Unit]
Description=Caddy webserver
Documentation=https://caddyserver.com/

[Service]
WorkingDirectory=/home/www
LimitNOFILE=8192
PIDFile=/home/www/caddy.pid
ExecStart=/home/www/caddy -agree -pidfile=/home/www/caddy.pid -email youremail@example.com
Restart=on-failure
StartLimitInterval=60

[Install]
WantedBy=default.target

The LimitNOFILE line fixes the warning about the file descriptor limit. Don’t forget to change the email.

Next:

editor redirect.service

Into which we enter:

Description=Subdomain Redirector
Documentation=https://gitlab.com/tobold/subdomain_redirector

[Service]
WorkingDirectory=/home/www/python/subdomain_redirector/
PIDFile=/home/www/redirect.pid
ExecStart=/home/www/python/venv/bin/gunicorn -c gunicorn.conf main:app
Restart=on-failure
StartLimitInterval=60

[Install]
WantedBy=default.target

We activate those services:

systemctl --user enable caddy
systemctl --user enable redirect

Now we can reboot:

exit
sudo reboot

When the machine comes back up, everything should work as it did when we tested it before!

9: Addendum

When something DOES break, you probably want to see the logs, but by default you can’t. So add the user www to the systemd-journal group:

sudo usermod -a -G systemd-journal www

Now the user can see the logs for systemd units with this command:

journalctl --user-unit=unitname

E.g.

journalctl --user-unit=caddy