Taking a Dockerised API Live: Every Step, and Everything That Broke( Part 2&3)

Taking a Dockerised API Live: Every Step, and Everything That Broke( Part 2&3)

Part 2 — Three decisions to make before deploying

Don't share an existing database server

The server already had a PostgreSQL container, and reusing it was tempting. Don't, if your application depends on a least-privileged database role — and especially if it uses row-level security.

A PostgreSQL superuser bypasses row-level security entirely. Applications built this way usually connect as a restricted role created by an initialisation script, and the official PostgreSQL image only runs those scripts on the first boot of an empty data volume. Attach to an existing server and the script never runs. The obvious shortcut — connecting as postgres — then silently switches off the isolation your application depends on.

Let your Compose project bring its own database, on its own volume and its own network. Whatever else the server runs stays untouched.

Bind the app to loopback, and put a front door in front of it

The production overlay should publish the app on 127.0.0.1:$PORT only. Nothing on the internet reaches it directly; something that terminates TLS forwards requests to it. This comes back at Step 9.

Decide what terminates TLS — after checking what owns port 80

If your app server is Caddy (FrankenPHP is Caddy with PHP embedded, and Traefik works similarly), it can obtain its own Let's Encrypt certificate. Alternatively, an nginx already on the server can hold the certificate and proxy inward. You cannot do both, because both need port 80. Look first:

sudo ss -ltnp | grep -E ':80 |:443 '

I made this decision in the wrong order. Step 11 covers what that cost.

Part 3 — On the server, step by step

Step 1 — SSH in, and make sure you can drive Docker as yourself

ssh "$SERVER"

docker ps

What I saw

permission denied while trying to connect to the docker API at unix:///var/run/docker.sock

Why: my user was not in the docker group.

Fix:

sudo usermod -aG docker "$USER"

exit

ssh "$SERVER"

id -nG # must now include: docker

docker ps # must work without sudo

What I got wrong: I ran usermod and retried in the same session. Group membership is read once, at login, so nothing changed and it looked as though the fix had failed. If exit doesn't seem to start a new session, the SSH connection is being multiplexed — close it with ssh -O exit "$SERVER". To carry on in the current shell instead, run newgrp docker.

Don't work around this with sudo docker compose. Files end up owned by root, and scripts that write to $HOME write into /root.

Step 2 — Copy the project up

rsync -az --delete \

--exclude vendor --exclude var --exclude node_modules --exclude .env --exclude .git \

./ "$SERVER:~/myapp/"

What I saw

rsync error: error in rsync protocol data stream (code 12) at io.c(232) [sender=3.2.7]

Why: code 12 is almost never about your files. Either rsync is missing on the server — it has to exist at both ends — or a shell startup file on the server prints something when you log in, which corrupts rsync's binary protocol.

Diagnose:

ssh "$SERVER" 'command -v rsync || echo MISSING'

ssh "$SERVER" 'echo ok' # must print exactly "ok" and nothing else

Fix: install rsync on the server, or skip it — tar is on every Linux machine:

tar czf - --exclude=./vendor --exclude=./var --exclude=./node_modules \

--exclude=./.env --exclude=./.git . \

| ssh "$SERVER" 'mkdir -p ~/myapp && tar xzf - -C ~/myapp'

Tar has no equivalent of --delete, so clear the old copy first when replacing one.

Why exclude those directories: dependencies (vendor/, node_modules/) are rebuilt inside the image, var/ holds runtime state, and .env holds your laptop's secrets along with the wrong environment setting.

Lesson: make the server a git clone early. I copied files up by hand four more times during this deployment before that became obvious. .env is ignored by git, so it survives every git pull.

Step 3 — Create .env first, then build

What I got wrong: I built before creating .env.

What I saw

WARN[0000] The "POSTGRES_USER" variable is not set. Defaulting to a blank string.

WARN[0000] The "APP_KEY" variable is not set. Defaulting to a blank string.

WARN[0000] The "OAUTH_ENCRYPTION_KEY" variable is not set. Defaulting to a blank string.

... thirty-odd more

These are harmless for a build, but thirty lines of noise hide the one warning that matters. Copy the template first:

cd ~/myapp

cp .env.example .env

If your Dockerfile creates a user from build arguments — a common pattern so files written through bind mounts belong to you — set them to your user before building:

sed -i "s/^HOST_UID=.*/HOST_UID=$(id -u)/" .env

sed -i "s/^HOST_GID=.*/HOST_GID=$(id -g)/" .env

The image bakes that user ID in at build time. The default is usually 1000, which happened to be my laptop user — so it had never mattered, and those two variables weren't even in .env.example. I only found them at Step 6, after they had already caused two failures.

Now build:

dcp build

Watch out: docker compose build silently skips any service behind a profile (profiles: [...]). Check what a build actually covers:

dcp config --services

dcp --profile migrate config --services

Step 4 — Generate secrets on the server

Never copy your development .env up. Generate every secret on the server itself:

openssl rand -base64 32 # application key

openssl rand -hex 24 # database superuser password

openssl rand -hex 24 # the application's least-privileged database role

Some secrets need the app's own libraries to generate. Run those from the image you just built rather than installing anything on the server — for example, the encryption key a PHP OAuth server needs:

docker run --rm myapp-app php -r \

'require "/app/vendor/autoload.php";

echo Defuse\Crypto\Key::createNewRandomKey()->saveToAsciiSafeString(), "\n";'

Then set the production switches and public URLs:

APP_ENV=production

APP_DEBUG=false

HTTP_PORT=8090

APP_URL=https://example.com

Every public URL is the domain with no port — https already implies 443. Never a loopback port, and never a raw IP address.

What I got wrong: I left one required key empty. Nothing complained until Step 8, and when it did, it did so with an empty response.

Check before moving on:

grep -E '^(APP_KEY|OAUTH_ENCRYPTION_KEY|POSTGRES_PASSWORD|DB_PASSWORD)=' .env

Every required secret must have a value. Don't choose passwords by hand, either.

Step 5 — Start it

dcp up -d

What I saw

Bind for 0.0.0.0:8080 failed: port is already allocated

Why: pgAdmin already had port 8080.

sudo ss -ltnp | grep :8080

docker ps --format '{{.Names}}\t{{.Ports}}' | grep 8080

Fix: set HTTP_PORT=8090 in .env and bring it up again.

Then check what is actually published — not what you think is published:

dcp config | grep -A3 published

The app must show host_ip: 127.0.0.1. The database must have no published port at all. If the database is published, the production overlay isn't being applied.

Lesson: write the port down. Changing it here came back to bite me twice more — at Step 11 in the nginx configuration, and at Step 12 inside the seed script.

Step 6 — Run the database migrations

This step cost more time than any other, through three wrong turns. All three are worth seeing, because the pattern is common.

Attempt one — the project's migrate service:

docker compose --profile migrate run --rm migrate migrate

Output

exec: "vendor/bin/phinx": stat vendor/bin/phinx: no such file or directory

Why: that service was a development convenience. It bind-mounted the project directory from the host and ran the migration tool out of the host's dependency folder — which Step 2 deliberately didn't upload.

Attempt two — install the dependencies on the host through that container:

docker compose --profile migrate run --rm --no-deps --entrypoint composer migrate install

Output

/app/vendor does not exist and could not be created

Why: the container ran as a user fixed at build time — uid 1000 — and my server user wasn't 1000, so it couldn't write into my own directory.

Attempt three — set HOST_UID and HOST_GID, rebuild, try again. The same error.

Why: remember the warning in Step 3 — docker compose build skips services behind a profile. The rebuild produced new images for everything except the migrate service, whose image stayed exactly as it was, still built for uid 1000. The correct fix never reached the one image that needed it.

What actually worked — stop using the development service, and run migrations from the production image, which already contains the dependencies and the migration tool. The database administrator credentials are passed in explicitly:

cd ~/myapp

set -a; . ./.env; set +a

dcp run --rm --no-deps \

-e POSTGRES_USER -e POSTGRES_PASSWORD -e POSTGRES_DB \

-e DB_HOST -e DB_PORT -e DB_USERNAME \

--entrypoint vendor/bin/phinx app migrate

No dependencies on the host, no user-ID question, nothing extra to rebuild. The same approach works for any migration tool — Doctrine, Laravel, Alembic, Prisma, Knex — as long as it is inside the production image.

The administrator credentials are needed because migrations create tables, enable security policies and grant the application role its access — none of which a least-privileged role should be allowed to do. The running app never holds them.

Keep in mind: migration files are copied into the production image at build time. A new migration file on the host is invisible to this command until you rebuild with dcp build app. That caught me at Step 12.

Lesson: development conveniences — bind mounts, dependencies installed on the host, override files — don't carry over to production. Run operational commands from the same image production runs.

Step 7 — Generate signing keys and other runtime files

If your app signs tokens (JWTs, for example), the keys usually have to be generated once on the server:

dcp exec app php bin/generate-keys.php

Output

PHP Warning: mkdir(): Permission denied in /app/bin/generate-keys.php on line 22

Could not create /app/var/keys

Why: /app/var was a named Docker volume, and a named volume takes its ownership from the image the first time it is mounted — and keeps it forever. My first up happened before I corrected the build-time user ID, so the volume belonged to uid 1000 while the rebuilt container now ran as me.

Fix: give the volume to the container's user, then generate:

dcp exec -u root app chown -R app:app /app/var

dcp exec app php bin/generate-keys.php

dcp exec app ls -l var/keys/

Or delete the volume and let it be recreated with the right owner, if it holds nothing worth keeping yet:

dcp down

docker volume rm myapp_app_var

dcp up -d

What I got wrong: I ran the chown and moved on without re-running key generation. It surfaced at Step 8 as a 500 from the key endpoint, and ls: var/keys/: No such file or directory.

That same volume held a cache the app writes on its very first request. The keys were only the first thing to fail.

Step 8 — Check health properly

curl -s https://example.com/healthz

curl -s https://example.com/readyz

curl -s https://example.com/.well-known/jwks.json

What I saw: /healthz returned {"status":"ok"}. The other two printed nothing at all.

Two lessons in that one result.

A health check that never touches the application lies by omission. My /healthz was answered by the web server directly and never reached the application code. It proved the web server was running and nothing whatsoever about the app. /readyz went through the application and checked the database; that is the one that means "working".

curl -s hides failures. Blank output is not success. Always make the status code visible:

curl -i localhost:8090/readyz

curl -s -o /dev/null -w '%{http_code}\n' https://example.com/readyz

The logs named the real problem:

dcp logs --tail=50 app

Output

Fatal error: Uncaught RuntimeException: OAUTH_ENCRYPTION_KEY is required in production.

Set it, or set OAUTH_ENCRYPTION_KEY_FILE to a mounted secret.

Fix: generate the key (Step 4), put it in .env, then recreate the container:

dcp up -d

Lesson: use up -d, not restart. restart reuses the existing container with its old environment; only up -d recreates it with the new .env.

After that /readyz returned 200 — and /.well-known/jwks.json returned a 500, because the keys from Step 7 had never actually been generated.

Step 9 — Reach it from outside the server

Calling http://SERVER_IP:8090 from Postman on my laptop timed out.

There were two separate causes.

1.The app is bound to 127.0.0.1 on purpose. sudo ss -ltnp | grep 8090 shows 127.0.0.1:8090. Nothing outside the server can reach it, by design.

2.The cloud provider's firewall.

Lesson — read the failure itself. "Connection refused" means the packet reached the host and nothing was listening. A timeout means the packet never arrived — a firewall dropping it. A timeout tells you to fix the firewall, whatever else is also wrong.

What I got wrong: to get the demo reachable quickly, I first published the app over plain HTTP on every interface. It worked — and sent passwords across the internet in clear. I deleted that the moment the domain was ready. If you ever need it, restrict the firewall rule to your own IP address and remove it afterwards.

(A side note: Compose appends ports lists across files, and you cannot bind both 127.0.0.1:P and 0.0.0.0:P — so a public port has to be a different number from the loopback one.)

Step 10 — Point the domain at the server

dig +short example.com # from anywhere

curl -s ifconfig.me; echo # on the server

These two must match.

What I saw: dig returned a different address from the IP I had been using all along. Running ifconfig.me on the server settled it — DNS was right, and my noted IP was stale.

Why: cloud VM external IPs are often ephemeral by default — released when the instance stops, with a new one assigned on the next start. One restart after a domain points at the server breaks DNS, and with it every certificate renewal.

Fix: reserve a static IP before pointing a domain at it.

Then open the firewall for web traffic. Port 80 is not optional — certificate validation arrives there, and so does the redirect to HTTPS. On Google Cloud, for example:

gcloud compute firewall-rules create web \

--allow tcp:80,tcp:443 --source-ranges 0.0.0.0/0 --target-tags your-vm-tag

Step 11 — HTTPS

Attempt one — let the application's own Caddy obtain the certificate. I made Caddy's auto_https off setting configurable and added an overlay publishing ports 80 and 443.

Output

failed to bind host port 0.0.0.0:80/tcp: address already in use

sudo ss -ltnp | grep ':80 '

The output listed process IDs belonging to nginx — so it was nginx running on the host itself, not in a container.

What I got wrong: I checked what owned port 80 after choosing an approach, instead of before (Part 2).

Attempt two — nginx as the front door, holding the certificate through certbot, with the app left on loopback.

There was already an nginx site for the domain, and certbot had already added HTTPS to it — but it served static files rather than proxying to the app. Here is the careful way to edit a certbot-managed site.

1. Back it up.

sudo cp /etc/nginx/sites-available/example.com /etc/nginx/sites-available/example.com.bak

2. Find the right block.

Certbot leaves two server blocks: the real one with listen 443 ssl, and a small one on port 80 that only redirects to HTTPS.

Edit the 443 block only.

sudo grep -n "server_name\|listen\|location\|root" /etc/nginx/sites-available/example.com

3. Replace that block's location / with:

location /.well-known/acme-challenge/ {

root /var/www/html;

}

location / {

proxy_pass http://127.0.0.1:8090;

proxy_http_version 1.1;

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header X-Forwarded-Proto $scheme;

client_max_body_size 25m;

proxy_read_timeout 120s;

}

Leave every line marked # managed by Certbot exactly as it is.

Four details in there that a generic proxy snippet usually leaves out:

• The acme-challenge location. Without it, the proxy swallows certbot's renewal challenge and the certificate simply fails to renew.

• X-Forwarded-Proto. Without it, the application believes it is being served over plain HTTP and generates http:// links about itself.

• client_max_body_size. nginx defaults to 1 MB. Uploads, imports and mobile clients syncing in bulk all exceed that.

• The proxy_pass port must match the app's port. I had changed it to 8090 at Step 5.

4. Test, then reload.

sudo nginx -t && sudo systemctl reload nginx

What I saw next:

Output

<center><h1>404 Not Found</h1></center>

<hr><center>nginx/1.28.3 (Ubuntu)</center>

Read it before acting on it. An HTML page signed by nginx means nginx answered the request itself — it never reached the app, whose errors were JSON. And a wrong upstream port would have produced 502 Bad Gateway, not 404. So nginx wasn't proxying at all.

I checked with sudo nginx -T, and proxy_pass was right there. The catch:

nginx -T prints the configuration on disk, not the configuration the running process has loaded. A reload fixed it.

If a reload doesn't, check for a second server block claiming the same name — nginx silently uses the first one it finds:

sudo nginx -T 2>/dev/null | grep -c "server_name example.com"

Finally, start the app without the Caddy HTTPS overlay, so it stays on loopback:

dcp up -d

The two approaches are alternatives, not layers. Running both is what produced the port 80 error in the first place.

Step 12 — Seed the data

Signing in as the demo user returned:

Output

{"type":"…/invalid-credentials","title":"The email address or password is incorrect.","status":401}

Why: migrations create the schema, not the data. The server's database had never been seeded, so the account didn't exist there.

set -a; . ./.env; set +a

dcp exec -T postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT email FROM users"

Zero rows. So, seed it — and that failed twice.

Seed attempt one:

Output

ERROR: insert or update on table "organisation" violates foreign key constraint "organisation_country_code_fkey"

DETAIL: Key (country_code)=(KE) is not present in table "country".

Why: the reference data from Part 1 §5. Nothing had ever populated the country table.

Fix: a migration that inserts it. Copy the new migration up, rebuild — migrations live inside the image — then run the migrate command from Step 6 again.

Seed attempt two:

Output

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Why: the seed script talked to PostgreSQL directly for its first half, then called the API over HTTP for the rest. That address was hardcoded to 127.0.0.1:8080 — and I had moved the app to 8090 back at Step 5. curl reached nothing, returned an empty body, and the failure surfaced several steps later inside a JSON parser.

Fix: read the port from .env, and check the API is actually answering before the first HTTP call, printing the address it tried and what to change.

It was also a partial failure worth noticing: the first half succeeded, so some records existed without the rest. Seed scripts need to be safe to run again from the top.

Step 13 — Get the client-side files, without breaking them

Three mistakes here, all on my own laptop, after the server was already working.

Mistake one — a one-line command that destroyed the file it was meant to fill.

# Don't do this:

ssh "$SERVER" "... ./bin/demo-seed.sh >&2 && cat collection.json" > ~/Desktop/collection.json

The shell opens — and truncates — the destination file before the remote command runs. When the seed failed, && skipped the cat, and I was left with a zero-byte file exactly where a working collection had been.

Fix: two steps. Seed, read what it prints, and only then fetch the file:

ssh "$SERVER" "cd ~/myapp && \

DEMO_PUBLIC_URL=https://$DOMAIN DEMO_OUT=\$HOME/demo \

./bin/demo-seed.sh"

scp "$SERVER:~/demo/postman_collection.json" ~/Desktop/

scp writes only what it actually received.

Mistake two — importing the laptop's collection and pointing it at the server. I exported my working local collection into Postman's cloud workspace. Its base URL said http://127.0.0.1:8080 — nothing had changed it; it had been generated on my laptop. More importantly, nearly half its requests carried the IDs of rows in my laptop's database. Against the server those rows didn't exist. Editing the base URL by hand can't rescue that.

Lesson: any file that embeds database IDs belongs to the database that produced it. Generate it where the data lives. (Better still, make the collection discover those IDs at runtime, so one file works against any environment.)

Mistake three, nearly — changing the OAuth redirect URI to the new domain. A redirect URI belongs to the web front end that receives the sign-in, not to the API, and it must exactly match what the OAuth client is registered with on the server. Postman reads the authorization code out of the response body rather than following the redirect, so the URI never needs to resolve at all. Changing only the Postman variable produces:

Output

{"type":"…/invalid-authorization-request","title":"The authorization request was refused.","detail":"Client authentication failed"}

When the front end exists at a real address, re-register the client and update the collection together — registration first.

One last detail: if your collection includes a request that changes a demo user's password or PIN, Postman keeps the new value only in its runtime variables, not in the file. If a sign-in suddenly fails after a full run, re-seed rather than hunting for a bug.