Should a new feature go live for everyone at once? Feature flags and progressive rollout

Turning a feature on for every user at once is a single bet, and undoing it costs you a release. The alternative is straightforward: ship the code switched off, open it to your own team first and then to a small slice of users, watch a metric you agreed on in advance, close the switch if it moves the wrong way, raise the percentage in steps if it does not, and delete the switch from the code once the rollout is done. That switch is a feature flag. The hard part is never flipping it on. It is the lifecycle around it: who is allowed to flip it, what closes it in a hurry, and when it leaves the codebase.

Deploying and releasing are different decisions

In the default workflow a feature ships and becomes visible in the same motion. That bundles delivery risk and product risk into one package, and when something breaks you have exactly one move available: roll back the release and take every unrelated change in it along for the ride.

A flag separates the two. The code sits in production, closed. Engineering owns the deploy, product owns the moment of exposure. A side effect worth having is the end of long-lived feature branches, since unfinished work can land on the main branch behind a closed flag instead of drifting for six weeks and turning into a merge problem. Making the deploy itself seamless is a separate discipline, covered in zero downtime deployments.

Four kinds of flag, four different lifetimes

Pete Hodgson's taxonomy, published on martinfowler.com in October 2017, is still the most useful frame. Flags split along two axes, how long they live and how often the decision changes.

• Release flags carry unfinished work in a closed state. Lifetime measured in weeks, decision mostly static, deleted the moment the rollout completes.
• Experiment flags route users into variants for an A/B test. The same user must stay in the same arm, otherwise the measurement is noise.
• Ops flags shed load by turning off something expensive under pressure. Most are short-lived; a few survive permanently as kill switches.
• Permission flags expose a feature to a defined group, such as beta users, an enterprise tier or internal staff. These can live for years.

Managing all four with one naming scheme and one ownership rule is where the mess starts. A release flag that should be gone in three weeks does not deserve the same treatment as a permission flag that will outlive two rewrites.

What a mismanaged flag costs: Knight Capital

The most expensive lesson on this subject dates to 1 August 2012, and the SEC order of 16 October 2013 reads almost like a flag governance checklist.

Knight Capital stopped using a function called Power Peg in 2003 but left the code sitting on its production servers. In 2005 the cumulative quantity counter inside that code was moved elsewhere in the sequence, and Power Peg was never retested against the change. When a new order routing feature was written in July 2012, the team reused the flag that had previously activated Power Peg. During the deployment that began on 27 July, one technician failed to copy the new code to one of the eight servers, and no written procedure required a second person to check.

On the morning of 1 August, starting around 08:01, internal systems generated 97 automated emails reading "Power Peg disabled." They had never been designed as alerts, and nobody acted on them before the market opened. At the open, the repurposed flag triggered the old code on the eighth server. 212 customer orders became more than 4 million executions in 154 stocks for over 397 million shares in about 45 minutes. The firm booked a loss of more than $460 million, and the SEC imposed a $12 million penalty. During the incident the team, hunting for the cause, removed the new code from the seven servers that had it correctly installed, which spread the faulty behaviour to all eight.

Four rules fall straight out of this. Never recycle an old flag name for new behaviour. Do not leave dead code callable in production. Flag state has to be identical everywhere it runs. And flag-related errors belong in an alerting channel, not in a shared inbox nobody reads.

How a staged rollout actually runs

Instead of jumping to 100%, slice it. A sequence that works in practice: internal staff, then an opt-in beta group, then 1%, 5%, 25%, 100%. Each step needs three things written down beforehand, and written down is the operative phrase: which metric you are watching, how long you wait before the next step, and the threshold at which you close the flag.

Percentage rollouts are not random draws. Hash a stable identifier such as the user or account id and derive the bucket from that, so raising the rollout from 5% to 25% never pushes anyone out of the group that already had the feature. Get this wrong and users see a screen on Monday that has vanished by Tuesday, and your support queue fills with tickets nobody can reproduce.

That staged discipline is not only about application code, as the CrowdStrike outage of 19 July 2024 demonstrated. A content update to a security product, Channel File 291, went out worldwide simultaneously; Microsoft estimated roughly 8.5 million Windows devices were affected. One of the mitigations in the published root cause analysis is blunt about the fix: template instances get staged deployment. Configuration, rule files and model updates deserve the same rollout care as code, because they change behaviour just as thoroughly.

Kill switches, and the flag service itself

The value of a staged rollout comes from how fast you can reverse it. Define a dedicated switch for each expensive or fragile dependency: the payment provider, the search service, the recommendation engine, bulk email. How to close them should not be something anyone researches mid-incident. It belongs in the incident response plan alongside who makes the call.

Next question: what happens when the flag service goes down? Unleash's published principles answer this directly by putting availability ahead of consistency. The SDK should keep evaluating from a local cache when the network to the control service is gone, using the last known values. A flag library that makes a synchronous remote call on every request turns the thing you added to reduce risk into a single point of failure, which is a memorable way to learn the lesson.

A flag is not an authorization mechanism

Permission flags and authorization get conflated constantly. A flag evaluated in the browser is present in the code whether or not it is open: anyone who opens the JavaScript bundle can see the feature exists, and usually the endpoints behind it. A flag hides an interface. It does not deny access.

The split is simple. Who is allowed to see what belongs on the server, inside your authorization model. A flag only decides which audience a shipped feature is exposed to, and it gets evaluated server-side. The same holds for personal data: rather than sending email addresses or identity attributes to a flag vendor to build segments, do the targeting on your side and hand the service an opaque key.

Then there is governance. Being able to flip a flag in production means being able to change behaviour without going through the release process. Keep that permission narrow, log every change with who, when and what value, and require a second approval on high-risk flags. Treat a flag change as a deploy event and annotate your dashboards with it, because a week later that annotation may be the only thing explaining the step in the graph.

Flag debt is the real bill

Every flag forks the code and keeps both paths alive. Ten flags is a thousand combinations in theory, and no team tests that. Hodgson's advice is the practical one: test the configuration you expect in production plus the fallback you would revert to, not the full matrix.

The rules that stop accumulation are boring and they work. Whoever opens a flag owns it. Create the removal ticket and the expiry date at the same time as the flag itself. A release flag gets deleted in the same sprint it reaches 100%, not added to a cleanup list that never gets scheduled. Keep naming uniform and archive removed flags so there is a record of what was opened when. Without this discipline flags become the quietest line item in your technical debt: a new developer reading the code spends an afternoon working out whether a flag nobody has touched in months still means anything.

Build it yourself or buy a service?

You do not need a vendor to start. A table in the database, a toggle in the admin screen and an application that re-reads the value at runtime will carry most teams through their first six months. The signal to move to a managed or self-hosted product is specific: you need percentage rollouts, segment targeting, a change history and consistent evaluation across several services.

On the lock-in question there is now a standard. OpenFeature was accepted into the CNCF sandbox in June 2022 and became an incubating project in December 2023, and it defines a vendor-neutral flag evaluation API. The provider list is broad, including flagd, Unleash, Flagsmith, GrowthBook, ConfigCat, LaunchDarkly and Split. Your application calls the standard interface, so swapping providers does not touch application code. Several of the open source options run on your own infrastructure, which matters when you would rather keep evaluation data in house.

Where to start

Three steps are enough. Ship your next medium-sized feature switched off and open it to internal staff only. Write down one success metric and one rollback threshold for it, two lines of text rather than a verbal understanding. Create the removal ticket at the same moment you create the flag.

What those three buy you is the ability to reverse a bad change in minutes without cutting a release. Pair the measurement side with observability and error budgets and the delivery side with a working CI/CD pipeline, and shipping stops being an event people schedule around. Wedevit runs this review remotely: we inventory your existing flags, check who can flip them and where the dead paths are, then hand back a simpler release flow you can actually operate.