Operations ยท Deployments

Deployments & operations

How code gets to Stage and Production, the exact commands, and the everyday runbooks. The short version: Stage is automated, Production is manual, and the automated tests never deploy anything.

automatedStage โ€” one command or a PR comment
manualProduction โ€” by hand, server by server
tests onlyCI runs checks, never deploys
noneRollback โ€” fix-forward instead

Start here

The big picture

One repository, two branches, two destinations. develop goes to Stage; stable goes to Production. The two paths are deliberately different.

Stage โ€” automated

A tool (Ansible) does everything: pulls the code, installs dependencies, runs database migrations, rebuilds the sites, updates the web server, and restarts services. You trigger it with one command โ€” or a comment on a pull request.

Production โ€” manual

Someone logs into each of the two servers and updates them by hand: pull stable, run migrations, restart services. There is no script and no automatic rollback.

The tests don’t deploy

CircleCI runs tests and linting on every change, but it never deploys. Deploying is always a separate, deliberate action. (Moving to fully automated production deploys is blocked partly by the two-server split and tooling decisions.)

Automated

Deploying to Stage

Two ways to deploy Stage. Both run the same automated playbook on the server.

Option A โ€” a command from your machine

# normal deploy (also refreshes service config)
make stage_deploy

# deploy a specific branch instead of develop
BRANCH=my-branch make stage_deploy_backend

Needs a couple of secrets in your environment (the database password and the server’s sudo password); the make targets pick them up automatically. Full command list is in the reference below.

Option B โ€” a comment on a pull request

If you don’t want a local setup, just comment on the pull request. A bot deploys it for you and posts the result to Slack. See the next section.

The friendly shortcut

The PR-comment bot

Type a short command as a comment on a pull request, and a bot deploys that branch to a non-production environment โ€” no local tools required. You need write access to the repository.

CommentDeploysWhere
/csThis pull request’s branchStaging (comingsoon.rocks)
/cs developA named branchStaging
/ipdevelopIn-progress (inprogress.rocks)
/ip my-branchA named branchIn-progress

What happens behind the scenes

  1. You comment /cs

    GitHub sends the comment to a small AWS function (via API Gateway), with a signature so it can’t be faked.

  2. The function works out the deploy

    It reads the command, picks the target server, and fetches the needed passwords from AWS Secrets Manager. It posts a “starting” message to Slack.

  3. AWS Systems Manager runs the playbook

    It runs the same Ansible deploy directly on the target server โ€” exactly what make stage_deploy would do.

  4. Slack gets the result

    When it finishes, another function posts success or failure to Slack, with the duration and a link.

This only works for non-production environments โ€” it’s the same playbook that doesn’t run on Production.

Manual

Deploying to Production

Production has no automated deploy. You update two servers, one after the other: the web server, then the jobs server. Production runs the stable branch โ€” assume everything you want live is already merged there.

Before you start โ€” get ubuntu key access

The everyday login (diviac) owns the code and can pull & migrate, but lacks sudo to restart most systemd services (Nginx and the app/Node services). The service restarts run as the ubuntu user over its own SSH key. Arrange that key access first (see Who can do what) so you’re not stuck mid-deploy. If the deploy is disruptive, turn on maintenance mode now.

1 ยท Web server

Two logins, one machine. Pull and migrate as diviac (it owns the code folder); restart the services as ubuntu (diviac doesn’t have the sudo rights for most of them).

# โ€” as diviac: owns the code, pulls and migrates โ€”
make ssh_to_prod            # logs in as diviac ยท 54.214.11.210
git pull stable
./manage.py migrate         # run the database changes by hand
exit

# โ€” as ubuntu: has sudo to restart the services โ€”
ssh -i ~/path/to/Axelerant-key.pem ubuntu@54.214.11.210
sudo systemctl restart padi.service long.service \
     travel_next.service travel_ssr.service extranet_ssr.service
sudo systemctl reload nginx   # only if the Nginx config changed

make ssh_to_prod logs you in as diviac โ€” fine for pulling code and running migrations, but it can’t restart the services. The ubuntu login is a plain SSH with the shared Axelerant EC2 key (Axelerant-key.pem) โ€” point -i at wherever you saved it.

2 ยท Jobs server (separate machine, its own copy of the code)

make ssh_to_celery_prod     # 54.203.134.44
git pull stable
sudo systemctl restart padi_celery_low.service padi_celery_mid.service \
     padi_celery_high.service padi_celery_beat.service padi_celery_crowdin.service
Then verify and finish

Check the site responds, both servers look healthy, the workers are up (Flower), and Sentry is quiet. If you turned on maintenance mode, turn it off last: make prod_maintenance_off.

Permissions

Who can do what on Production

Two logins exist on the production web server, with deliberately split powers. Neither one alone can finish a full deploy: diviac pulls the code, ubuntu restarts the services.

LoginCanCannot
diviacPull and deploy code (owns the code folder); run migrationsRestart most systemd services โ€” no sudo rights (Nginx and the app/Node services)
ubuntuFull service control โ€” systemctl restart/reload anything, apply Nginx configPull code (doesn’t own the code folder)
Getting & using ubuntu access

Get the shared Axelerant EC2 key (Axelerant-key.pem) โ€” or add your own public key to the ubuntu login via AWS Session Manager. Then connect directly (there is no make target for it):

ssh -i ~/path/to/Axelerant-key.pem ubuntu@54.214.11.210

A full production deploy needs both logins โ€” diviac to pull code & migrate, ubuntu to restart the services (and Nginx).

How the servers exist

How servers are built

Two tools provision the infrastructure. Neither runs during a normal deploy โ€” they set things up.

Ansible โ€” configures the servers

Defines how a server is set up and deployed: dependencies, the app, the sites, Nginx, certificates. This is the engine behind make stage_deploy and the PR bot.

AWS CDK โ€” creates the AWS resources

Code that creates the AWS pieces: the servers, the OpenSearch domain, the job-queue cache, and the serverless functions. There’s a separate CDK app per piece.

CDK appCreates
EC2The main application server
CeleryThe jobs server + the ElastiCache job queue
OpenSearchThe fast read-store domain
Lambdas / APIThe serverless functions + their API gateway
gh_deployThe PR-comment bot (function, queue, Slack)
CDK can drift from reality

The CDK code is the intended setup; the live setup has been changed by hand in places (e.g. the production web server was resized). Always reconcile CDK against the live console before re-running it โ€” a blind re-deploy could shrink production.

Cheat sheet

Command reference

All from the repository root. Swap stage_ for dev_ / dev2_ for the other non-production servers.

Connect

CommandConnects to
make ssh_to_prodProduction web server โ€” as diviac (code/migrations, no service restarts)
ssh -i ~/path/to/Axelerant-key.pem ubuntu@54.214.11.210Production web server โ€” as ubuntu (to restart services / Nginx)
make ssh_to_celery_prodProduction jobs server
make ssh_to_stageStage server

Deploy (Stage / non-production)

CommandWhat it does
make stage_deployNormal deploy + refresh service config
make stage_deploy_backendForce a backend rebuild
make stage_deploy_frontendForce a frontend rebuild
make stage_deploy_fullForce both
make stage_managepyRun migrations / collect static files
make stage_nginxUpdate only the Nginx config
make stage_reset_lockClear a stuck deploy lock

Maintenance & data

CommandWhat it does
make prod_maintenance_on / _offShow / hide the maintenance page on Production
make stage_maintenance_on / _offSame for Stage
make restore_from_prodDownload Production’s latest backup and load it into your local database
make restore_from_stageSame, from Stage

Shedding load safely

Maintenance mode

Maintenance mode shows visitors a holding page (a 503) while keeping the partner API and static files working. Use it for disruptive deploys.

make prod_maintenance_on     # before disruptive work
# โ€ฆ do the deploy โ€ฆ
make prod_maintenance_off    # the very last step
Don’t deploy while it’s on

Running an Nginx update or a full deploy resets the maintenance switch back to off. Toggle maintenance around the work, not in the middle of it.

Step by step

Runbooks

Release to Production

  1. Pre-flight

    Confirm everything is merged into stable and tests are green. Arrange ubuntu key access if Nginx will change. Turn on maintenance mode if the deploy is disruptive.

  2. Web server

    As diviac (make ssh_to_prod): git pull stable and run migrations. Then as ubuntu (SSH with the key): restart the app + Node services, and reload Nginx only if its config changed.

  3. Jobs server

    SSH into the separate jobs server, git pull stable, restart the Celery services.

  4. Verify & finish

    Check the site, both servers, the workers, and Sentry. Turn maintenance off last.

Roll back a bad release

There’s no one-click rollback. Prefer to fix forward (a new release on stable). Rolling back code alone is risky because database migrations may not reverse cleanly โ€” only undo a migration if you’re sure it’s safe.

Stuck deploy lock (Stage)

If an automated deploy hangs at “Wait for lockfile”, clear it with make stage_reset_lock and re-run.

Debug with real data

Pull a copy of Production (or Stage) data into your local database with make restore_from_prod โ€” it downloads the latest nightly backup and loads it locally.

Incident / on-call

to document No formal on-call process yet. Watch Sentry (errors), Flower (workers) and CloudWatch (logs); use maintenance mode to shed load and fix forward.

Quality gate

Automated checks (CI)

Every change runs through CircleCI, which checks the backend only โ€” it never deploys.

  • WhenOn every push / pull request that touches backend code
  • RunsThe test suite (split across 4 runners), type checks (mypy), and linting (ruff)
  • SkipsFrontend-only or docs-only changes finish green without running backend tests
  • GateTests and type-checks must pass before a change can be merged

Be aware

Deployment risks

RiskWhat it means
No rollbackIf a release breaks something, there’s no undo button โ€” fix forward.
No deploy recordNothing tracks which version is live; you check the server. Tagging releases on stable would help.
Production is manualTwo servers by hand; easy to miss the jobs server or an Nginx restart.
Automation can’t do ProductionThe automated deploy works for non-production only โ€” the two-server split isn’t handled.
CDK drift driftLive servers differ from the infrastructure code; reconcile before re-running CDK.