How to Deploy Fast.ai Machine Learning Models to DigitalOcean for Production

With Machine Learning being all the rage right now, there are a lot of managed services that you could use to deploy your deep learning models.

However if you already have a DigitalOcean VPS running, why not just use it?

Before we begin...

Source codes and sample model are available on Github.

This tutorial is based on Fast.ai - 2019 Lesson 2 Download notebook.

1. Requirements

To make this work, we will be using the following tools:

• Supervisor
• Flask
• Fastai + Pytorch

What is Supervisor?

Supervisor is a tool that would monitor your processes and if they are terminated, helps you restart them again.

Why do we need Supervisor?

Because we want an easy way to serve our models without setting up WSGI + NGINX, one way is to execute our Flask app directly with Python and restart the process as necessary with Supervisor.

Is there any caveat?

Yup. This works best for low traffic app.

What is Flask?

Flask is a popular micro-framework in Python that helps us easily load our models and start serving requests.

2. Project Stucture

Our project structure would look like this:

myapp
|--model.pkl
|--main.py
|--requirements.txt
|--start.sh
|-tmp

requirements.txt details a list of Python packages that are required for the project to run:

flask
fastai>=1.0
torch
torchvision

main.py houses all the Flask codes needed to start serving traffic to our model. Basically it will take a GET parameter called image, downloads the image locally and predicts it using our model.

import json
import urllib.request

from flask import Flask
from flask import request
from fastai.vision import *

app = Flask(__name__)

@app.route('/')
def handler():
    defaults.device = torch.device('cpu')

    // Load the deep learning model locally
    path = Path('.')
    learner = load_learner(path, 'model.pkl')

    // Download the image based on the GET parameter from URL
    image = request.args['image']
    urllib.request.urlretrieve(image, './tmp/image.jpg')

    // Make Predictions
    img = open_image('./tmp/image.jpg')
    pred_class,pred_idx,outputs = learner.predict(img)

    // Return response
    return json.dumps({
        "predictions": sorted(
            zip(learner.data.classes, map(float, outputs)),
            key=lambda p: p[1],
            reverse=True
        )
    })

tmp is a folder for us to store our downloaded image for our model to make predictions.

model.pkl would be our model file, as created in Jupyter Notebook via the method learn.export(). You should prepare this beforehand while training your desired model.

Finally start.sh is the Supervisor setup script for our VPS that would restart Flask if it was detected to have exited, thus keeping the app alive.

source myappenv/bin/activate
export FLASK_APP=main.py
flask run
# flask run --host=0.0.0.0 --port=8080
deactivate

You can view the source codes directly at Github.

3. Install Dependencies

We are going to start by first creating a virtual environment for our project and activate it. Then, we will install the list of requirements from the requirements.txt file.

$ python3 -m venv myappenv
$ source myappenv/bin/activate
$ pip3 install -r requirements.txt --no-cache-dir

This would create a myappenv folder to install all the local dependencies required for our project.

Because some of the packages required like torch requires a lot of memory, despite the --no-cache-dir flag you might still hit a MemoryError problem. In solve it, either resize your VPS to have more memory or you can try create a larger swap size on the VPS.

4. Test running Flask

Now let's see if our project is set up correctly. Execute the following commands and check if Flask is able to run properly.

$ export FLASK_APP=main.py
$ flask run

You should see the following response:

 * Serving Flask app "main.py"
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Yup that was a warning. In our case, we expect the traffic to be internal and low so it should be alright. If you expect otherwise then you may want to read up on WSGI server instead.

Now that we are sure Flask is running fine. Quit it and deactivate the virtual environment.

(Press CTRL+C to quit Flask)
$ deactivate

5. Set up Supervisor

Let's install Supervisor and run it:

$ sudo apt-get install supervisor
$ sudo supervisord

Next, make sure our start.sh is executable:

$ chmod +x start.sh

Create a Supervisor config file for our Flask app:

$ sudo nano /etc/supervisor/conf.d/myapp.conf

Inside myapp.conf, copy and paste the following codes:

[program:myapp]
directory=/home/username/bears/
command=/bin/bash -E -c ./start.sh
process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
user=username
numprocs=1
redirect_stderr=true
stdout_logfile=/home/username/bears/supervisor.log

And restart Supervisor to load our new configurations:

$ sudo systemctl restart supervisor.service

Verify that Flask is now running via Supervisor:

$ netstat -l | grep 5000

You should a response similar to this:

tcp        0      0 localdomain:5000     0.0.0.0:*               LISTEN

6. Success!

Now our Flask app is running on port 5000 and is ready to serve traffic! You can give it a try by executing the follow curl command:

$ curl http://127.0.0.1:5000/\?image\=http://58.26.9.230/edrive_new/cctv/cctv_image/C165.jpg

You should see this in response:

{"predictions": [["black", 0.9545639753341675], ["grizzly", 0.043770089745521545], ["teddys", 0.0016658934764564037]]}

Awesome! It's now working perfectly and would make predictions based on the input image.

But...

This only works with other web apps that are on the same server because it's only listening for local traffic.

7. Open Up to External Services

To open up to external service, simply uncomment the line in start.sh and replace with an appropriate port value, then restart Supervisor. Make sure your specified port value is not already being used by another service to avoid conflicts.

flask run --host=0.0.0.0 --port=8080

And if you are looking to add HTTPS to the service, you can continue with this guide here.

Closing

That was very easy isn't it? Now you can have a fully working Flask app serving your models on your existing DigitalOcean VPS without paying extra!