Daemonizing Celery and FastAPI

GraphAI ships with two bash scripts used for launching the server: deploy_celery.sh (for Celery) and deploy.sh (for FastAPI). However, these two scripts should only be used in testing. The easiest, most reliable way to put GraphAI into production is to create systemd daemons based on these scripts. This guide helps you in the conversion.

Celery queue daemons

Each line of deploy_celery.sh deploys one of the Celery queues, plus the first one that deploys Celery Beat (for scheduled tasks) and the final one that deploys Flower (the monitoring interface). Each of these should be put in a separate .service file in ~/.config/systemd/user/. For example, the line that deploys the low priority queue (used for video processing):

nice -n 20 celery -A main.celery_instance worker --hostname workerLow@%h -l info -P threads --prefetch-multiplier 1 -c 16 -Q celery,video_2,ontology_6,text_6 -D --without-gossip

Becomes the following file (which we will call celery-low-prio.service:

[Unit]
Description=Deploy celery queues of low priority

[Service]
Restart=always
RestartSec=3
Environment="PATH=/data/venvs/test_venv/bin:/home/ubuntu/.nvm/versions/node/v20.14.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
ExecStart=nice -n 20 /data/venvs/test_venv/bin/celery -A main.celery_instance worker --hostname workerLow@%h -l info -P threads --prefetch-multiplier 1 -c 16 -Q celery,video_2,ontology_6,text_6 --without-gossip
WorkingDirectory=/home/ubuntu/graphai-test/graphai/api/main

[Install]
WantedBy=default.target

The Environment variable must contain the path to your Python virtual environment, and WorkingDirectory points to the api.main folder of GraphAI. All daemons must have Restart=always, so that in case of a crash, the server will auto-recover.

All the other lines in deploy_celery.sh can be converted to services in the same way.

FastAPI daemon

The API daemon relies on all the Celery daemons to run. Therefore, it becomes:

[Unit]
Description=Graph AI API
Wants=celery-beat.service celery-high-prio.service celery-low-prio.service celery-mid-prio.service celery-caching.service celery-flower.service

[Service]
Restart=always
RestartSec=3
ExecStart=/data/venvs/test_venv/bin/gunicorn main:app -b localhost:28800 -w 1 -k uvicorn.workers.UvicornWorker --timeout 240
WorkingDirectory=/home/ubuntu/graphai-test/graphai/api/main

[Install]
WantedBy=default.target

The Wants field establishes the dependencies, and the FastAPI service will restart if any of the Celery services do.