Context Debt

The algorithm is fine, but your company’s duct-taped workflows are a problem

Here is the skinny:

Enterprise AI is collapsing because enterprises keep treating AI like a feature you can bolt onto a workflow that is taped together with policy, habit, and fear.

The evidence is blunt:

• A big share of AI never makes it out of the lab. Gartner projected that at least 30% of GenAI projects would be abandoned after proof-of-concept by end of 2025, citing data quality, risk controls, costs, and unclear value.

• The “pilot-to-production” gap is widening. S&P Global reported the share of companies abandoning most AI initiatives before production jumped from 17% to 42% YoY, and that 46% of projects are scrapped between PoC and broad adoption.

• Value is concentrated in a small minority. Boston Consulting Group found only 5% of companies achieving AI value “at scale,” while 60% report no material value.

• GenAI is often “everywhere” and “nowhere” at once. MIT NANDA’s report argued 95% of organizations see zero return, and that enterprise-grade tools frequently stall due to brittle workflows and lack of contextual learning (i.e., the system doesn’t get better inside real operations).

• Data readiness is the quiet killer. Gartner found 63% of organizations either lack, or aren’t sure they have, the data management practices needed for AI.

If you’re looking for a single sentence to print on a mug, here you go…

The enterprise keeps buying probability engines and deploying them in certainty-based organizations.

What AI looks like after it meets an enterprise

There’s a popular fantasy in boardrooms: you purchase AI the way you purchase office chairs. You choose a vendor, and then you sign things. Soon, the chairs show up, people sit in them, and productivity scooters into the glittering sunset.

Enterprise AI doesn’t behave like chairs, though.

Enterprise AI behaves like releasing a curious raccoon into the ductwork of a hospital, a bank, or a manufacturer and then acting shocked when it returns with:

1. Secrets

2. Incorrect conclusions, and;

3. A deep affection for the worst possible place to nest

The reason is simple and old: AI in enterprises is a socio-technical system, not a model. It is data pipelines, permissions, policies, incentives, interfaces, exceptions, escalation paths, training, audits, and the quiet art of humans deciding when to ignore the machine.

This is why those “it worked in the demo” moments are so common. The demo is a world without the enterprise. In the demo, nobody is out sick, no one copy-pasted last quarter’s spreadsheet template into this quarter’s reality, and no regulator is asking who approved the model and why.

A lot of enterprise AI failure is simply context debt: meaning, the organization doesn’t know (or can’t reliably reproduce) the conditions under which the model is supposed to be right. That’s why research into managing ML lifecycle “context” (metadata, versioning, change tracking) keeps showing up as practical infrastructure, not academic decoration.

Mermaid lifecycle sketch with common failure points (haha)

mermaid

Copy

flowchart LR
    A[Ambition: "We need AI"] --> B[Use-case selection]
    B -->|Failure: vague KPI / wrong problem| X1[PoC theater]
    B --> C[Data access & governance]
    C -->|Failure: data not AI-ready| X2[Data swamp]
    C --> D[Build or buy decision]
    D -->|Failure: vendor hype / "agent washing"| X3[Shiny tool mismatch]
    D --> E[Prototype & offline evaluation]
    E -->|Failure: offline ≠ real workflow| X4[Demo success, prod failure]
    E --> F[Workflow integration + change mgmt]
    F -->|Failure: no owner / no training| X5[Adoption stall]
    F --> G[Production rollout]
    G --> H[Monitoring, audits, retraining]
    H -->|Failure: drift, no surveillance| X6[Silent degradation]
    H --> I[Scale or retire]
    I -->|Failure: costs exceed value| X7[Budget guillotine]

Case files from the real world

A few stories, because stories are what enterprises actually remember when the dashboard is lying.

A hospital-grade model that didn’t survive the hospital

The Epic sepsis prediction model was widely deployed, but in an external validation at Michigan Medicine it showed AUC 0.63 at the hospitalization level, with 33% sensitivity at the commonly used alert threshold and a large alert burden (alerts on 18% of hospitalizations while missing many true sepsis cases).

Later, a multicenter prospective validation of an updated version (v2) reported better discrimination (AUROC roughly 0.82 to 0.92 across sites) but still emphasized low positive predictive value and high alert burden, with substantial institutional variability.

This is enterprise AI failure in its purest form: even when the model improves, the experience can remain punishing unless workflow integration, silencing strategies, and governance are treated as first-class engineering. Hospitals need an operational safety system.

Hospitals need an operational safety system opposed to a tricky demo without solid change management.

A real estate algorithm meets an unpredictable world

Zillow’s SEC filings describe how its home-buying business depended on “data science and proprietary algorithms” to price homes, and how it ultimately decided to wind down operations in light of “home pricing unpredictability,” capacity constraints, and operational challenges.

That’s an enterprise AI lesson you can hang on the wall: forecasting systems fail hardest when the business demands certainty from an uncertain market, and when operational constraints (renovation capacity, supply chain, labor) become the real bottleneck, not the model. I will type it louder for the kids in the back of the room.

Hiring software that quietly broke the law

The U.S. Equal Employment Opportunity Commission described a case where iTutorGroup allegedly programmed its hiring software to automatically reject older applicants (women 55+, men 60+), resulting in a settlement and ongoing compliance obligations.

This is the enterprise encoding a policy choice into software, then acting surprised when regulators treat it like what it is: a decision system with accountability.

“The algorithm went rogue” isn’t a reasonable excuse for encoding bad behaviors and processes.

A chatbot tries to become a separate legal entity

In a 2024 British Columbia CRT decision (summarized by CanLII), the tribunal rejected Air Canada’s argument that it wasn’t responsible for what its chatbot told customers, emphasizing that the chatbot was part of the company’s website and the company didn’t take reasonable care to ensure accuracy.

If a system speaks on your behalf, it is part of your company for users, regulators, and courts.

Why engineering-led development makes it worse

Enterprises love an engineering led approach because it feels like progress: code is shipping, costs are tracked, a dashboard exists, and someone can say “iteration velocity” with a straight face.

AI punishes this approach when it isn’t paired with product, design, and operations discipline because AI is less like a feature and more like a new layer of decision-making inside a living organization.

Here are the organizational patterns that repeatedly show up behind “AI didn’t work”:

Success is measured like software…but it is more like operations.
MIT NANDA’s report argues that adoption of generic tools may boost individual productivity, while enterprise grade systems fail when they can’t integrate into day-to-day workflows or learn from feedback in context.

That’s a product operating model problem.

Ownership is unclear, so reality has no place to report bugs.
NIST’s AI RMF frames AI risk management as a lifecycle responsibility—mapping, measuring, managing, and governing context-specific risks—because deployment is where harms and failures emerge.

If no one owns monitoring and response, your “intelligent system” becomes a ghost story.

Enterprises underinvest in MLOps and monitoring because the ROI is “invisible.”
S&P Global notes increased interest in MLOps tools while also reporting that difficulty proving ROI is a reason organizations don’t invest…right before those same orgs scrap projects in bulk between PoC and adoption.

This is like refusing to buy smoke detectors because you can’t quantify the value of the fires that didn’t happen.

Context drift is treated as a surprise instead of a certainty.
Research across domains keeps emphasizing that data drift can threaten safety and effectiveness after deployment and that monitoring needs to be built in.
Enterprises still behave as if their data will remain polite and stationary, like a portrait in a hallway.

So, yes: engineering-led shipping can build AI fast. It can also build it wrong at scale.

The counterweight is an operating model where product, design, ops, legal/compliance, and data governance share responsibility and where there’s someone paid to care about what happens after launch. BCG frames “future-built” success as an operating model with shared ownership between business and IT and a willingness to reshape workflows end-to-end rather than layering AI on top.

Procurement, regulation, and the economics of disappointment

Enterprise AI fails in a triangle: procurement, compliance, and cost.

Procurement: You can now buy “agents” from thousands of vendors, many of whom are just rebranding existing automation. Reuters summarized Gartner’s warning about “agent washing,” where chatbot-like tools are marketed as agentic systems.
When buyers can’t reliably test capability boundaries (autonomy, tool use, rollback safety), they purchase confusion.

Regulation and compliance: Governments are moving toward enforceable governance requirements for higher-risk AI uses. The U.S. Office of Management and Budget memorandum M-24-10 requires agencies to implement minimum risk practices and, in some cases, to stop using systems that lack required safeguards.
Meanwhile, the EU AI Act establishes risk-tiered obligations and significant penalties, with phased implementation.
Even outside regulated sectors, this shapes enterprise behavior because the compliance muscle tends to grow everywhere.

Economics: GPU bills aren’t your biggest worry. Costs associated with integration, security, audits, vendor management, and the staff time needed to make adoption real are a bigger concern. Gartner’s forecasts about GenAI and agentic AI cancellations cite escalating costs and unclear business value as dominant factors. The “AI value gap” data is a reminder that money doesn’t magically turn into outcomes: most firms still report limited or no value.

Here is the cruel joke: the enterprise buys AI to reduce complexity, then discovers AI requires a mature relationship with complexity...and a skilled workforce.

Practical recommendations for product leaders and executives

This is the part where the article usually says “be strategic.” That’s like telling someone in a flood to “be drier.” Instead, here are actions that change failure odds.

1. Define production as a contract, not a vibe.
   Name the workflow step, the user persona, the success metric, and the rollback plan before building. Tie budgets to adoption and measurable outcomes, not demo completion.

2. Treat data readiness like a prerequisite gate.
   If your data governance can’t answer “where did this value come from?” and “who owns its definition?”, you don’t have AI-ready data but you do have a lot of misplaced hope. Gartner’s data readiness findings are the “check engine light” you shouldn’t ignore.

3. Build monitoring and retraining policies as part of the feature.
   Assume drift, instrument it, define when to recalibrate, when to pause, and who approves changes. Drift isn’t a rare event; in many domains it’s expected behavior.

4. Assign a single accountable owner for the deployed system.
   Not “the data science team.” Not “the vendor.” A named internal owner who is responsible for outcomes, auditability, and user impact across the lifecycle and make it consistent with a risk-management approach.

5. Stop buying labels; buy tested capabilities.
   Run sandbox evaluations that test autonomy boundaries, error handling, data controls, and failure recovery. Contract for audit support and measurable outcomes. Assume “agent washing” exists until proven otherwise.

6. Design the human workflow, not just the model.
   The quickest way to kill adoption is to make humans do extra work to validate the system. The Epic sepsis model story is partly about model performance and largely about operational burden and alert fatigue.

7. Stand up Product Ops for AI systems.
   You need an operating function that coordinates: data governance, model lifecycle, evaluation discipline, release readiness, training, support, and compliance reporting.

AI success is operational success wearing a new hat.

If you do these things, your AI project may still fail, and that is okay because some should. But it will fail for honest reasons, quickly, and with learnings that transfer. That’s the kind of failure you want because that flavor of failure doesn’t poison the organization (or the goal).

Snark coming in 3, 2, 1…

And if you don’t do these things, don’t worry: your AI initiative will still produce something. Probably a dashboard.

It will be so beautiful in spite of it being so, so wrong. Give it a name like “Intelligence 360,” which is the kind of name you give to something right before you quietly stop talking about it.

Sources:
Use AI to look it up.