Embedded, Firmware, and OT Debt: When You Cannot Just Deploy a Fix
Last updated . Sources are named and dated inline - how we source claims.
Nearly every piece of technical debt advice ever written assumes you can ship a correction. In embedded, firmware, and operational technology, that assumption is the first thing to break.
Hardware sits in the field for a decade or more. There is no rollback, no canary, no gradual rollout to a small slice of users. A bad flash does not degrade the experience -- it destroys a physical object somebody paid for.
Why the Standard Playbook Breaks
The modern approach to technical debt rests on a quiet assumption: that the cost of correcting a mistake is bounded by the cost of writing the correction. Continuous delivery, feature flags, canary releases, and instant rollback all exist to make that assumption true. If you can push a fix in minutes and undo it in seconds, almost any debt is survivable, because the worst case is a bad afternoon.
Embedded systems delete that assumption. The artifact is not a container image on a server you control; it is a binary burned into a physical device that has been shipped, sold, installed, and possibly cemented into a wall. It may be on a wind turbine, inside a patient, on a ship, or in a country your company no longer operates in. It may have no network connection at all, by design, or one the customer firewalled off on day one and will never reopen.
So the interest on embedded debt is not paid in developer hours. It is paid in truck rolls, recalls, warranty claims, certification re-runs, and units that stay broken forever because nobody will ever touch them again. A shortcut a web team would file as minor cleanup can, in firmware, become permanent -- not because it is hard to fix, but because no mechanism exists by which the fix can reach the thing that needs it.
That changes what prudent debt means. On the web, deliberate debt is a bet that you can pay it back later. In embedded, it is often a bet that you will never need to, because the repayment channel may not exist. Different bet, different conversation, and a different set of people in the room: hardware, manufacturing, service, regulatory, and support, not just engineering.
The Update Path Is Itself a Debt Decision
Over-the-air update capability looks like a feature. It is really a decision about how much future debt the product is allowed to carry, made at a point in the schedule when nobody has met the bugs yet.
No Rollback, No Canary
A server rollback is a pointer change. A firmware rollback needs a second copy of the image, flash to hold it, a bootloader that can choose between them, and a health check that decides which is good. If any of those is missing, the previous version is gone the moment the write begins -- usually discovered during the incident rather than before it.
Fix: Budget flash for dual images before the board is laid out, not after
Bricking Is a Real Outcome
An interrupted write, a power loss at the wrong millisecond, or a bootloader that trusts an image it should have rejected turns a working product into scrap. Unlike a failed deploy, the customer cannot retry, and the remedy is a replacement unit and a shipping label. The update mechanism deserves more scrutiny than the features it delivers.
Fix: Cut power during flash deliberately and repeatedly, on real units
Devices That Are Never Updated
Shipping an update is not the same as installing one. Devices sit behind firewalls, on metered links, in installations where any change needs a permit, or with owners who have no idea the product contains software. Plan for a long tail that stays on the original image indefinitely, and make sure your security posture does not assume otherwise.
Fix: Measure fleet version spread, and treat the laggards as the real fleet
Signing Keys Outlive Teams
Secure boot binds a device to a key for its whole life. Lose the key and the fleet becomes unupdatable. Compromise it with no rotation path in the bootloader and the fleet becomes untrustworthy. Key custody, rotation, and revocation are firmware architecture decisions, not IT housekeeping.
Fix: Design key rotation in while the bootloader is still writable
The Toolchain Trap
In most software the build is a solved problem you stop thinking about. In embedded it is frequently the most fragile asset the product has, and it is almost never inventoried as a risk.
The One Machine That Builds It
There is a PC under a desk, or a virtual machine nobody dares update, that produces the shipping binary. It has a specific vendor IDE at a specific patch level, a licence tied to hardware no longer sold, environment variables discovered by trial and error, and an install order that matters. Nobody has rebuilt it from scratch in years, because attempting to is how you find out that you cannot. The source is safe; the ability to produce a release from it is one dead disk away from gone.
Licences You Can No Longer Buy
Commercial embedded compilers, static analysers, and RTOS kernels are sold per seat, per version, and sometimes per target. Vendors get acquired, products get discontinued, licence servers get retired. A team can end up entitled to keep using a tool it can no longer install, or holding a perpetual licence for a version that will not activate on any operating system it is allowed to run. And because the certified binary depends on the certified toolchain, upgrading is not a free choice either.
Non-Reproducible Builds
If you cannot rebuild the exact image running in the field, you cannot answer basic incident questions: which sources produced this, which library versions are inside it, whether a reported vulnerability applies. Reproducibility is harder in embedded because timestamps, absolute paths, and link ordering leak into the binary -- and worth the effort precisely because the artifact lives for a decade. Pair it with a bill of materials so the fleet can be queried rather than guessed at.
Containerise the Build, Archive the Tools
The remedy is unglamorous: put the compiler, the SDK, and every generator into a versioned container or a checked-in tool archive, build on a machine nobody uses interactively, and make a clean-room rebuild part of the release process rather than a rescue operation. Where a tool cannot be containerised because of licensing, record that explicitly as a risk with an owner and a renewal date, exactly as you would an unsupported third-party dependency.
Vendor SDK and BSP Debt
Almost no embedded product is built purely from its own source. Underneath sits a board support package, a vendor hardware abstraction layer, a vendor-forked kernel, a radio stack delivered as a binary blob, and example code written to demonstrate a peripheral rather than to run for ten years. All of it becomes yours the moment you ship.
The failure mode is not that this code is bad. It is that its maintenance schedule is set by someone whose commercial interest is selling the next part, not supporting the one you designed in. Support for an SDK generation slows, then stops; patches land only on the newest branch; the vendor kernel fork drifts further from upstream every year, so the cost of moving to a maintained base grows exactly as fast as the need to move. Meanwhile a binary blob you cannot inspect sits inside your security boundary, and when a vulnerability is announced against it you depend entirely on the vendor rebuilding it for your generation. Local patches applied straight into the vendor tree make this worse in a predictable way: they turn a future upgrade from a merge into an investigation.
Treat silicon selection as a dependency decision with a support horizon, and record that horizon where procurement and engineering both see it. Vendors do publish something to work from -- Texas Instruments, for example, states that its product life cycles are typically ten to fifteen years and often longer -- but note that these are stated norms rather than guarantees, and that part availability is a separate question from SDK maintenance. Ask what the vendor commits to in writing on both, and design the abstraction boundary so a future change of part is expensive rather than impossible. The discipline that applies to software supply chain risk applies here too, with the added constraint that you cannot swap supplier without respinning a board.
Certification Debt: Priced by Process, Not by Engineering
In a safety-related product governed by a functional safety standard -- IEC 61508 for electrical and programmable safety systems, ISO 26262 in road vehicles, DO-178C for airborne software -- the cost of a change has almost nothing to do with the size of the change. It is dominated by the evidence that must be regenerated: impact analysis, updated requirements traceability, re-run verification, revised safety arguments, and in many cases an independent assessment. A three-line correction to a boundary check can carry weeks of paperwork behind it. That is not bureaucracy for its own sake -- it is what makes the claim "this system is safe" mean anything -- but it produces a specific economic distortion.
The distortion is that small improvements become uneconomic. Refactoring, renaming, and cleanup carry the same certification overhead as a functional fix while delivering no argument you can put in front of an assessor, so they never happen. Code quality tends to freeze at the moment of certification and then degrade as narrowly scoped, minimally invasive patches accumulate around the parts nobody is allowed to touch. Each of those patches is chosen to minimise re-validation surface, which is the opposite of the criterion you would use to keep a codebase healthy.
Two things help. Architectural partitioning that limits blast radius, so a change in one component provably cannot affect another and the re-validation scope is genuinely small rather than nominally small. And batching: when the fixed cost per change is high, queueing cleanup and paying the certification cost once across many improvements is rational, provided the batch is planned rather than accidental. Both have to be decided while the architecture is still soft, because after certification the partitioning you have is the partitioning you keep.
When Standard Clean Code Advice Is Actively Wrong
A great deal of mainstream engineering advice assumes memory is effectively free, allocation is safe, and a few extra microseconds are invisible. On a part with kilobytes of RAM and a hard deadline measured in microseconds, several of those recommendations are not merely unnecessary -- they are defects. Dynamic allocation in a long-running system with no memory management unit invites fragmentation that surfaces weeks after deployment, which is why many embedded and safety coding standards restrict or forbid it and prefer static allocation sized at build time. Deep abstraction layers cost stack, and stack overflow without memory protection is silent corruption rather than a clean crash. Exceptions and unbounded recursion break worst-case execution time analysis. Even a helpful-looking log line can miss a deadline if it blocks on a slow bus inside an interrupt handler.
The consequence for a debt conversation is that "this code looks ugly" is not evidence in embedded the way it is elsewhere. A hand-unrolled loop, a fixed-size pool, a lookup table instead of a calculation, or a deliberately flat call graph may be the correct answer under the constraints. The real debt signal is the undocumented constraint. When the reason for an unusual construct exists only in one engineer's memory, the next person will clean it up and reintroduce a timing or memory fault that takes months to find. Write the constraint next to the code and make it a review criterion, so intentional ugliness stays distinguishable from accidental ugliness.
Hardware and Software Co-Debt
Hardware schedules are unforgiving and respinning a board is expensive and slow. So when a fault is found late -- a pin swapped, a pull-up missing, a peripheral that violates its own datasheet, an errata item that lands after tape-out -- the fix goes into firmware. That is usually the right call. The debt is created by what happens next.
The workaround is written under deadline pressure, described in a commit message rather than a design document, and often expressed as a magic delay or an extra register write that looks arbitrary to anyone reading it later. Over the following years it spreads: other drivers are written against the compensated behaviour rather than the datasheet behaviour, calibration data is tuned around it, and test fixtures encode it. By the time a later board revision fixes the underlying fault, removing the workaround breaks three things that now depend on it, and the firmware carries both paths forever -- one for old units in the field, one for new ones coming off the line.
Make compensation visible and bounded. Keep hardware workarounds in a clearly named, isolated layer rather than scattered through drivers, and tie each one to the errata item or board revision that caused it. Make board revision a first-class runtime concept so conditional behaviour is explicit and testable rather than implied. And keep a shared register of active workarounds that hardware and firmware both review, because the most common failure here is a hardware fix nobody tells firmware about, and a firmware workaround nobody tells hardware is still load-bearing.
OT and ICS: Debt You Cannot Take Offline
Operational technology inverts the priorities of IT. The system must keep running; a controller that reboots at the wrong moment can ruin a batch, damage a machine, or hurt someone. Every remediation strategy has to survive that constraint first.
The Maintenance Window Problem
Patching requires downtime, and downtime is scheduled around production, not around vulnerability disclosure. In continuous process industries a planned outage may come around once or twice a year, and the queue for that window is long and contested. A patch released today may genuinely have to wait, and the honest answer is compensating controls in the meantime rather than pretending it will land promptly.
Flat Networks and Lost Isolation
Plant networks were often built flat because flat was simple and the network was assumed to be physically isolated. That isolation eroded quietly: a remote support link here, a data historian bridging to the business network there, a contractor laptop, a cellular modem in a cabinet. The architecture diagram still shows an air gap that stopped being true years ago, and nobody owns the discrepancy.
Equipment Older Than the Team
Controllers commissioned decades ago still run because they work, and replacing them means requalifying a process. NIST puts the position plainly in SP 800-82r3: an OT system can outlive twenty years of service, so many legacy systems contain hardware and software no longer supported by vendors that "cannot be patched or updated to protect against new vulnerabilities", and compensating controls should be considered where security controls are unsupported. The debt is not one system; it is a stack of coupled systems that can only be replaced together.
What Actually Works
Start with an accurate asset inventory, because you cannot protect or plan around equipment you cannot list. Then segment: put unpatchable devices behind boundaries that constrain what can reach them, monitor those boundaries passively so monitoring cannot disturb the process, and bring remote access paths under deliberate control. Sequence replacement against the production calendar over years, and say plainly that segmentation is risk reduction, not remediation.
The Long Tail Now Has a Legal Deadline
For most of the history of embedded products, the obligation to keep supporting a shipped device was commercial and reputational. It is becoming statutory. The EU Cyber Resilience Act, Regulation (EU) 2024/2847, applies horizontally to products with digital elements -- a description that fits most connected hardware rather than one narrow sector.
Two provisions matter most here. Article 13(8) requires manufacturers to determine a support period reflecting how long the product is expected to be in use, and its third subparagraph states that the support period shall be at least five years, shorter only where the expected time in use is shorter. Article 13(9) then requires each security update to remain available for a minimum of ten years after it is issued, or for the remainder of the support period, whichever is longer. The general obligations apply from 11 December 2027, with the vulnerability and incident reporting duties in Article 14 applying earlier, from 11 September 2026.
Read that against everything above and the collision is obvious. A support period expressed in years is a commitment to keep a toolchain alive, keep a vendor SDK patchable, keep a signing key usable, and keep an update path working for longer than many product teams currently plan for at all. Debt that used to expire quietly when a product left the catalogue now has a date attached. Our guide to EU regulation debt covers the wider set of instruments and how to turn them into a backlog rather than a panic.
Testing Without the Hardware
Embedded teams cannot spin up a thousand ephemeral environments. Test capacity is physical: a finite number of boards, wired into a finite number of rigs, in a lab with a door. The infrastructure built to work around that scarcity accumulates debt of its own.
Hardware-in-the-Loop Rigs Are Products Too
A hardware-in-the-loop rig is a bespoke machine with its own wiring, control software, calibration, and usually exactly one person who understands it. It drifts out of alignment with the shipping product, breaks in ways that look like firmware bugs, and gets treated as lab equipment rather than engineering infrastructure. Version it, document it, give it an owner, and budget for its maintenance -- an unreliable rig teaches the team to distrust test results, which is worse than no rig.
Simulator Divergence
Simulators and peripheral fakes let logic be tested on a workstation, and should be the default for anything not timing or hardware specific. The risk is that the fake encodes what the team believes the hardware does rather than what it actually does. When the two disagree, tests pass and devices fail. Re-validate the model against real silicon periodically, and treat every field divergence as a defect in the simulator as well as in the firmware.
Separate Logic From Registers
The highest-return structural change in most firmware codebases is a clean boundary between hardware access and decision logic. Protocol handling, state machines, and calculation can then be tested at speed on a host, leaving scarce rig time for what genuinely needs real hardware: timing, power, electrical behaviour, and integration. Codebases that never made this separation pay for it every day in test cycle time. See our testing strategies guide for the general pattern.
Test the Failures Only Hardware Has
Brownouts, marginal supplies, thermal extremes, disconnected sensors, noisy buses, flash wear, and clock drift are the failure modes that reach customers, and they are exactly the ones a workstation cannot produce. Inject them deliberately. A rig that can drop the supply rail mid-write, pull a connector, or heat a board to its limit finds a class of latent debt no amount of unit testing will surface.
Related Resources
Security & Supply Chain
Bills of materials, vulnerability handling, and the supply chain questions a shipped device must answer for years.
EU Regulation Debt
The Cyber Resilience Act and its neighbours, broken into obligations you can put on a backlog with an article number attached.
Testing Strategies
Building a test pyramid that works when environments are physical, scarce, and shared across every team in the building.
Dependency Management
Keeping vendor SDKs, kernels, and third-party components upgradable instead of frozen at the version you first shipped.
Frequently Asked Questions
Because the delivery channel may not exist, and the failure mode is destructive rather than degraded. An update has to travel to devices that may be offline, firewalled, battery powered, or physically inaccessible, and it has to survive being interrupted halfway through. Without a second image slot and a bootloader that can choose between images there is no rollback at all -- the previous version is overwritten the moment the write begins. A failed update does not show an error page; it can leave a customer holding hardware that no longer starts.
Treat it as a decision about how much future debt the product may carry, not as a feature request. Ask how long the device will be in service, whether it will ever be network connected, what a field visit costs, and what your obligations are if a vulnerability appears after launch. With a long service life and any network exposure, the absence of an update path converts every future defect into a permanent property of the hardware. The costs land early and are concrete; the benefits land later and are probabilistic, which is exactly the shape of decision that gets deferred and then regretted.
It is one of the most dangerous kinds, because it is invisible until you urgently need it. Source control protects the source; it does not protect the ability to turn that source into the exact binary running in the field. When the build depends on one machine, one patched compiler version, and a licence tied to hardware nobody sells any more, you have a single point of failure with no monitoring on it. The remedy is to make the toolchain a versioned, archived artifact and prove it with a clean rebuild in every release rather than only during an emergency.
Recognise it early, because the cost of moving grows faster than the pressure to move. Take ownership of the code you have: archive every SDK and board support package release you rely on, keep the vendor tree pristine and hold local changes as an explicit patch series so an upgrade is a rebase rather than an excavation, and inventory the binary components you cannot inspect so you know where you are blind. Then harden the boundary between vendor code and your own so a future part change is expensive rather than impossible, and make support horizon an explicit selection criterion for the next design.
Stop treating patching as the only control and start treating exposure as the thing you manage. Begin with an accurate asset inventory, because equipment you cannot list cannot be protected or planned around, and in most plants the inventory is the missing piece rather than the tooling. Then segment, so unpatchable devices sit behind boundaries that constrain what can reach them; monitor those boundaries passively; and bring remote access paths under deliberate control. Queue patches against the real maintenance calendar, sequence replacement over years alongside production planning, and report honestly that segmentation reduces risk without removing the underlying debt.
For products with digital elements placed on the EU market, yes. Article 13(8) of Regulation (EU) 2024/2847 requires manufacturers to determine a support period reflecting how long the product is expected to be in use, and its third subparagraph states that the support period shall be at least five years, shorter only where the expected time in use is shorter. Article 13(9) requires each security update to remain available for a minimum of ten years after it is issued, or for the remainder of the support period if that is longer. The general obligations apply from 11 December 2027, and the reporting duties in Article 14 from 11 September 2026.
Fix It Before It Ships
In embedded, the cheapest moment to repay debt is always before the hardware leaves the building. Everything after that is a truck roll.