Operations ยท Deployments
Deployments & operations
How code gets to Stage and Production, the exact commands, and the everyday runbooks. The short version: Stage is automated, Production is manual, and the automated tests never deploy anything.
Start here
The big picture
One repository, two branches, two destinations. develop goes to Stage; stable goes to Production. The two paths are deliberately different.
Stage โ automated
A tool (Ansible) does everything: pulls the code, installs dependencies, runs database migrations, rebuilds the sites, updates the web server, and restarts services. You trigger it with one command โ or a comment on a pull request.
Production โ manual
Someone logs into each of the two servers and updates them by hand: pull stable, run migrations, restart services. There is no script and no automatic rollback.
CircleCI runs tests and linting on every change, but it never deploys. Deploying is always a separate, deliberate action. (Moving to fully automated production deploys is blocked partly by the two-server split and tooling decisions.)
Automated
Deploying to Stage
Two ways to deploy Stage. Both run the same automated playbook on the server.
Option A โ a command from your machine
# normal deploy (also refreshes service config)
make stage_deploy
# deploy a specific branch instead of develop
BRANCH=my-branch make stage_deploy_backend
Needs a couple of secrets in your environment (the database password and the server’s sudo password); the make targets pick them up automatically. Full command list is in the reference below.
Option B โ a comment on a pull request
If you don’t want a local setup, just comment on the pull request. A bot deploys it for you and posts the result to Slack. See the next section.
The friendly shortcut
The PR-comment bot
Type a short command as a comment on a pull request, and a bot deploys that branch to a non-production environment โ no local tools required. You need write access to the repository.
| Comment | Deploys | Where |
|---|---|---|
/cs | This pull request’s branch | Staging (comingsoon.rocks) |
/cs develop | A named branch | Staging |
/ip | develop | In-progress (inprogress.rocks) |
/ip my-branch | A named branch | In-progress |
What happens behind the scenes
You comment
/csGitHub sends the comment to a small AWS function (via API Gateway), with a signature so it can’t be faked.
The function works out the deploy
It reads the command, picks the target server, and fetches the needed passwords from AWS Secrets Manager. It posts a “starting” message to Slack.
AWS Systems Manager runs the playbook
It runs the same Ansible deploy directly on the target server โ exactly what
make stage_deploywould do.Slack gets the result
When it finishes, another function posts success or failure to Slack, with the duration and a link.
This only works for non-production environments โ it’s the same playbook that doesn’t run on Production.
Manual
Deploying to Production
Production has no automated deploy. You update two servers, one after the other: the web server, then the jobs server. Production runs the stable branch โ assume everything you want live is already merged there.
ubuntu key access
The everyday login (diviac) owns the code and can pull & migrate, but lacks sudo to restart most systemd services (Nginx and the app/Node services). The service restarts run as the ubuntu user over its own SSH key. Arrange that key access first (see Who can do what) so you’re not stuck mid-deploy. If the deploy is disruptive, turn on maintenance mode now.
1 ยท Web server
Two logins, one machine. Pull and migrate as diviac (it owns the code folder); restart the services as ubuntu (diviac doesn’t have the sudo rights for most of them).
# โ as diviac: owns the code, pulls and migrates โ
make ssh_to_prod # logs in as diviac ยท 54.214.11.210
git pull stable
./manage.py migrate # run the database changes by hand
exit
# โ as ubuntu: has sudo to restart the services โ
ssh -i ~/path/to/Axelerant-key.pem ubuntu@54.214.11.210
sudo systemctl restart padi.service long.service \
travel_next.service travel_ssr.service extranet_ssr.service
sudo systemctl reload nginx # only if the Nginx config changed
make ssh_to_prod logs you in as diviac โ fine for pulling code and running migrations, but it can’t restart the services. The ubuntu login is a plain SSH with the shared Axelerant EC2 key (Axelerant-key.pem) โ point -i at wherever you saved it.
2 ยท Jobs server (separate machine, its own copy of the code)
make ssh_to_celery_prod # 54.203.134.44
git pull stable
sudo systemctl restart padi_celery_low.service padi_celery_mid.service \
padi_celery_high.service padi_celery_beat.service padi_celery_crowdin.service
Check the site responds, both servers look healthy, the workers are up (Flower), and Sentry is quiet. If you turned on maintenance mode, turn it off last: make prod_maintenance_off.
Permissions
Who can do what on Production
Two logins exist on the production web server, with deliberately split powers. Neither one alone can finish a full deploy: diviac pulls the code, ubuntu restarts the services.
| Login | Can | Cannot |
|---|---|---|
diviac | Pull and deploy code (owns the code folder); run migrations | Restart most systemd services โ no sudo rights (Nginx and the app/Node services) |
ubuntu | Full service control โ systemctl restart/reload anything, apply Nginx config | Pull code (doesn’t own the code folder) |
ubuntu access
Get the shared Axelerant EC2 key (Axelerant-key.pem) โ or add your own public key to the ubuntu login via AWS Session Manager. Then connect directly (there is no make target for it):
ssh -i ~/path/to/Axelerant-key.pem ubuntu@54.214.11.210
A full production deploy needs both logins โ diviac to pull code & migrate, ubuntu to restart the services (and Nginx).
How the servers exist
How servers are built
Two tools provision the infrastructure. Neither runs during a normal deploy โ they set things up.
Ansible โ configures the servers
Defines how a server is set up and deployed: dependencies, the app, the sites, Nginx, certificates. This is the engine behind make stage_deploy and the PR bot.
AWS CDK โ creates the AWS resources
Code that creates the AWS pieces: the servers, the OpenSearch domain, the job-queue cache, and the serverless functions. There’s a separate CDK app per piece.
| CDK app | Creates |
|---|---|
| EC2 | The main application server |
| Celery | The jobs server + the ElastiCache job queue |
| OpenSearch | The fast read-store domain |
| Lambdas / API | The serverless functions + their API gateway |
| gh_deploy | The PR-comment bot (function, queue, Slack) |
The CDK code is the intended setup; the live setup has been changed by hand in places (e.g. the production web server was resized). Always reconcile CDK against the live console before re-running it โ a blind re-deploy could shrink production.
Cheat sheet
Command reference
All from the repository root. Swap stage_ for dev_ / dev2_ for the other non-production servers.
Connect
| Command | Connects to |
|---|---|
make ssh_to_prod | Production web server โ as diviac (code/migrations, no service restarts) |
ssh -i ~/path/to/Axelerant-key.pem ubuntu@54.214.11.210 | Production web server โ as ubuntu (to restart services / Nginx) |
make ssh_to_celery_prod | Production jobs server |
make ssh_to_stage | Stage server |
Deploy (Stage / non-production)
| Command | What it does |
|---|---|
make stage_deploy | Normal deploy + refresh service config |
make stage_deploy_backend | Force a backend rebuild |
make stage_deploy_frontend | Force a frontend rebuild |
make stage_deploy_full | Force both |
make stage_managepy | Run migrations / collect static files |
make stage_nginx | Update only the Nginx config |
make stage_reset_lock | Clear a stuck deploy lock |
Maintenance & data
| Command | What it does |
|---|---|
make prod_maintenance_on / _off | Show / hide the maintenance page on Production |
make stage_maintenance_on / _off | Same for Stage |
make restore_from_prod | Download Production’s latest backup and load it into your local database |
make restore_from_stage | Same, from Stage |
Shedding load safely
Maintenance mode
Maintenance mode shows visitors a holding page (a 503) while keeping the partner API and static files working. Use it for disruptive deploys.
make prod_maintenance_on # before disruptive work
# โฆ do the deploy โฆ
make prod_maintenance_off # the very last step
Running an Nginx update or a full deploy resets the maintenance switch back to off. Toggle maintenance around the work, not in the middle of it.
Step by step
Runbooks
Release to Production
Pre-flight
Confirm everything is merged into
stableand tests are green. Arrangeubuntukey access if Nginx will change. Turn on maintenance mode if the deploy is disruptive.Web server
As
diviac(make ssh_to_prod):git pull stableand run migrations. Then asubuntu(SSH with the key): restart the app + Node services, and reload Nginx only if its config changed.Jobs server
SSH into the separate jobs server,
git pull stable, restart the Celery services.Verify & finish
Check the site, both servers, the workers, and Sentry. Turn maintenance off last.
Roll back a bad release
There’s no one-click rollback. Prefer to fix forward (a new release on stable). Rolling back code alone is risky because database migrations may not reverse cleanly โ only undo a migration if you’re sure it’s safe.
Stuck deploy lock (Stage)
If an automated deploy hangs at “Wait for lockfile”, clear it with make stage_reset_lock and re-run.
Debug with real data
Pull a copy of Production (or Stage) data into your local database with make restore_from_prod โ it downloads the latest nightly backup and loads it locally.
Incident / on-call
to document No formal on-call process yet. Watch Sentry (errors), Flower (workers) and CloudWatch (logs); use maintenance mode to shed load and fix forward.
Quality gate
Automated checks (CI)
Every change runs through CircleCI, which checks the backend only โ it never deploys.
- WhenOn every push / pull request that touches backend code
- RunsThe test suite (split across 4 runners), type checks (mypy), and linting (ruff)
- SkipsFrontend-only or docs-only changes finish green without running backend tests
- GateTests and type-checks must pass before a change can be merged
Be aware
Deployment risks
| Risk | What it means |
|---|---|
| No rollback | If a release breaks something, there’s no undo button โ fix forward. |
| No deploy record | Nothing tracks which version is live; you check the server. Tagging releases on stable would help. |
| Production is manual | Two servers by hand; easy to miss the jobs server or an Nginx restart. |
| Automation can’t do Production | The automated deploy works for non-production only โ the two-server split isn’t handled. |
| CDK drift drift | Live servers differ from the infrastructure code; reconcile before re-running CDK. |