Why does your site go down when you deploy? Zero-downtime releases and schema changes

Short answer: if your site drops during a release, the cause is almost always one of three things. A schema change grabs an exclusive lock and a queue of queries piles up behind it, the old and new code run side by side for a few minutes and break each other's assumptions, or the load balancer keeps sending requests to a process that is not ready yet or has already started shutting down. The fix is not a longer maintenance window. It is designing every release to run next to the one before it. Teams that do this ship in the middle of the working day; teams that don't wait for 3am and then discover they have no way back. All of it sits on top of a deployment pipeline that already works.

During a rolling update, two versions serve traffic

That single sentence dictates everything else. In a rolling update, part of your fleet runs the new build and the rest still runs the old one. The window might last eight seconds or ten minutes. The duration doesn't matter, the existence of it does. Two different code versions are writing to the same database, the same queue and the same cache.

The rule that follows: every release has to be compatible with the release directly before it. Ship a migration that drops a column together with the code that stopped using it, and the next query from an old process returns "column does not exist". Your user sees a 502. The application didn't crash, it just contradicted itself.

Where a schema change actually blocks

In PostgreSQL, most forms of ALTER TABLE need an ACCESS EXCLUSIVE lock. The lock itself is rarely the problem, since most DDL finishes in milliseconds. The problem is queue behaviour. If a long reporting query is holding the table, your DDL waits, and from that moment every new query against that table lines up behind the DDL. A three-second schema change can freeze the site for five minutes because of the five-minute query in front of it. Where those long queries come from is a separate hunt, covered in our piece on diagnosing slow queries.

There is a one-line antidote: set lock_timeout in the migration session. Two or three seconds is a normal value. If the lock can't be acquired in that time the statement fails and you retry a few seconds later. A cancelled deploy is cheaper than a stalled application.

The safe form of the common operations is well documented by now:

• Adding a column. Since PostgreSQL 11, adding a column with a constant default no longer rewrites the table; the default lives in the catalog and is applied on read. On older versions the same statement rewrites every row.
• Creating an index. CREATE INDEX CONCURRENTLY does not block inserts, updates or deletes. In exchange it scans the table twice and takes noticeably longer than a plain build. If it fails it leaves an invalid index behind, which you have to drop before retrying. It also cannot run inside a transaction block, so if your migration tool wraps each migration in one, this step needs to run outside it.
• Making a column NOT NULL. A direct SET NOT NULL scans the whole table while holding ACCESS EXCLUSIVE. Instead, add a CHECK (column IS NOT NULL) NOT VALID constraint, run VALIDATE CONSTRAINT (that step takes only a SHARE UPDATE EXCLUSIVE lock, so writes keep flowing), and then issue SET NOT NULL. From PostgreSQL 12 onward, the existing valid constraint lets the scan be skipped.

MySQL tells a different story. In InnoDB, adding a column has used ALGORITHM=INSTANT by default since 8.0.12, changing only metadata in the data dictionary, and since 8.0.29 you can place the new column anywhere rather than only at the end. The limits are real: a table tops out at 64 row versions before MySQL demands a rebuild with COPY or INPLACE, and instant additions do not work on ROW_FORMAT=COMPRESSED tables or tables carrying a FULLTEXT index. For changes outside that envelope, gh-ost (GitHub) and pt-online-schema-change (Percona) build a shadow table, copy the data across and swap at the end. Both are well proven, but do not run either without watching the cut-over moment and replica lag.

Expand, migrate, contract

Any change that looks incompatible can be split across three releases. The pattern is called expand and contract, and it is the database version of the Parallel Change pattern Martin Fowler wrote up.

Say you are moving a phone column from free text to E.164 format. Renamed in one step, old code breaks the instant the migration lands. Split apart:

1. Expand. Add phone_e164 as a nullable column. The schema changed, behaviour didn't, no running code noticed.
2. Write both. The new release writes to both columns and still reads the old one. This release is safe to roll back, because the old one keeps working too.
3. Backfill. Move historical rows into the new column in batches.
4. Flip the read. Ship the release that reads from the new column. If something breaks, rollback is still free, because the old column is populated and current.
5. Contract. A few days later remove the dual write, then drop the old column in a separate release.

Five steps looks like a lot for one rename. What you buy is that all five are reversible, and the real cost of a migration is never the number of deploys. It is standing in front of one at midnight that cannot be undone.

Don't backfill with a single UPDATE

Rewriting millions of rows in one statement breaks three things at once: the long transaction holds locks, WAL generation spikes, and replicas fall behind. In an application that serves reads from a replica, users experience that as "the record I just saved isn't there".

Batch instead. Walk the table by primary key in slices of a few thousand rows, commit each slice in its own transaction, pause briefly between slices, and watch replication lag so you can slow down when it crosses your threshold. Write the backfill as a resumable job rather than a step inside the migration itself. A backfill that runs for an hour is normal. A backfill that holds locks for an hour is an incident.

Traffic: processes that aren't ready, and ones already leaving

With the schema side handled, what remains is the most common and most fixable cause. If you define a Kubernetes Deployment without a strategy, maxUnavailable and maxSurge both default to 25 percent, which means a quarter of your capacity can disappear mid-update. Production workloads that want zero downtime usually want maxUnavailable: 0 with a small maxSurge, typically 1: the new pod comes up first, then an old one leaves.

Without a readiness probe, Kubernetes treats a pod as ready as soon as the container starts. Traffic arrives at a process whose JVM hasn't warmed up and whose connection pool isn't built. The probe needs to check the dependencies the application actually needs to answer a request, not return an empty 200.

Shutdown is the sneakier half. When a pod is deleted, two things start at the same moment: the kubelet begins termination, and the pod is removed from the service endpoints (EndpointSlice). Nothing orders those two events. Until the endpoint update propagates through the cluster, the load balancer can still send requests to a pod that is on its way out. The standard remedy is a short sleep in the preStop hook, so the pod stays alive while endpoint removal spreads, and only then receives SIGTERM. Watch the accounting: preStop and post-SIGTERM shutdown share one budget. terminationGracePeriodSeconds defaults to 30, and the clock starts when termination begins, not when SIGTERM is sent. Spend 15 seconds sleeping and your application has 15 left to finish its work.

On virtual machines the equivalent is connection draining. An AWS Application Load Balancer target group has a deregistration delay of 300 seconds by default, configurable from 0 to 3600. A draining target receives no new requests but finishes the ones in flight. None of that helps if your application ignores SIGTERM and keeps accepting new connections.

Blue-green, canary and feature flags

These three solve the same problem from different angles, and they are not substitutes for each other.

Blue-green keeps two complete environments and switches traffic at the router in one move. It gives you the fastest rollback available, since the old environment is still running. The detail people miss: the database is usually shared, so the schema still has to satisfy both versions. Blue-green without expand and contract is a rollback button that has been painted on.

Canary sends a small percentage of traffic to the new version, compares error rate and latency against the old one, then ramps up. The value is entirely in the comparison. If you haven't written down which metric crossing which threshold triggers an automatic rollback, you don't have a canary, you have a slower deploy.

Feature flags separate deploying from releasing. The code sits in production switched off, and the feature goes live when you flip it. When something misbehaves, you close the flag instead of shipping a new build. On the standards side, OpenFeature was accepted into the CNCF in June 2022 and moved to incubating in November 2023; its vendor-neutral API means swapping flag providers doesn't touch application code. The known cost of flags is that they accumulate. Every flag is a branch that keeps two code paths alive. Put the cleanup in the same sprint as the rollout.

The front end has its own outage mode

Even with a flawless server-side release, the tab a user left open half an hour ago will find trouble. Modern bundlers split JavaScript into hash-named chunks. A user who loaded the page before your deploy holds the old index.html, and when they navigate to a lazily loaded route, the browser asks for a filename that no longer exists. What they see is a chunk load error they have no way to interpret.

Both halves need handling. Keep old build artifacts on the CDN for a few releases, which is free of conflicts because the filenames are content-hashed, and add a version check to the application. The simple form compares a build identifier served by the API against the one the client was loaded with and prompts the user to refresh. The same discipline applies to the API contract: adding a field is safe, removing one or making it required is not. How long old clients survive is something you can estimate from version distribution on mobile and from session length on the web.

Background jobs and queues

This is the part most downtime hunts skip. If a worker takes SIGTERM and dies holding a job, the queue redelivers that message and the work runs twice. If the work charges a card, it charges twice. Worker shutdown behaviour needs to be explicit: stop accepting new messages, finish what's in hand, then exit. For jobs too long to finish inside the grace period, make the job itself idempotent so a second run with the same key produces no new effect.

Message format changes behave exactly like schema changes. Never deploy a producer that emits a new required field before the consumers that understand it. Writing consumers to ignore fields they don't recognise, the tolerant reader approach, removes most of the ordering pain.

When a maintenance window is the right answer

Not every system needs zero downtime. For a B2B application used between 09:00 and 18:00 from a single region, a planned 20-minute window at night costs far less than months of engineering. The test is straightforward: compare the business cost per minute of downtime against the engineering cost of removing it, then weight that by how often you would actually need it.

Even if you choose the window, keep the expand and contract discipline. Its real payoff isn't uninterrupted service, it is reversibility. A migration you cannot undo will still have you stuck there when the window closes.

Where to start

Three concrete steps. Add lock_timeout and retry logic to your migration runner. Give each service a readiness probe that checks its real dependencies and a shutdown path that handles SIGTERM. Then split your next schema change into expand and contract steps and ship it that way. For measurement, DORA's change fail rate and the deployment rework rate added in 2024 are the useful pair, since both answer the same question: how many of our deploys needed someone to intervene.

None of these is a large project, and together they change how a team feels about shipping. Wedevit runs this review remotely: we read the last quarter of migration files, the deployment configuration and the shutdown path, and show you where your downtime is manufactured. From there, pair the measurement side with observability and error budgets and lower the risk each deploy carries with a real test strategy. Zero downtime isn't a tool you buy. It is the habit of designing every change to be compatible with the one before it.