The launch of an AI system is usually treated as the finish line. It is closer to the starting gun. A conventional application that passes its tests on Friday behaves the same way on Monday unless someone changes the code. An AI system can behave differently on Monday because its users changed what they ask, because the documents it retrieves were updated, or because the model provider shipped a new version over the weekend. AI observability is the discipline of seeing those changes before customers, auditors or the finance team do.
The need is growing because more systems are reaching production. Menlo Ventures' State of Generative AI in the Enterprise report, published in December 2025, found that 47% of AI deals reached production, against 25% in the previous year. Each of those deployments now has to be watched. Yet the instruments most enterprises have, application performance monitoring built for latency, errors and uptime, say little about whether an answer was correct, whether a model has drifted, or whether a single workflow is quietly consuming a disproportionate share of the inference budget.
This article sets out what AI observability should cover, proposes a five-signal monitoring checklist that an operations team can adopt, and maps each signal to the Manage and Measure functions of the NIST AI Risk Management Framework so that monitoring serves governance as well as engineering. Pre-deployment evaluation, security threats and cost controls are covered in separate PanScience Research pieces and are referenced rather than repeated.
Why AI systems fail differently from ordinary software
The engineering literature has warned about this for a decade. In "Hidden Technical Debt in Machine Learning Systems", presented at NIPS 2015, D. Sculley and colleagues at Google argued that it is "dangerous to think of these quick wins as coming for free", and listed risk factors specific to machine learning, including boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies and changes in the external world. Google Cloud's architecture guidance on MLOps makes the operational point directly: "models can decay in more ways than conventional software systems", and "in practice, models often break when they are deployed in the real world." It recommends that teams "track summary statistics of your data and monitor the online performance of your model."
Large language models add a further failure mode that classical machine learning rarely had: the model can change without the enterprise doing anything. Lingjiao Chen, Matei Zaharia and James Zou compared the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on the same tasks. GPT-4's accuracy on identifying prime numbers fell from 84% to 51%, its willingness to answer some sensitive questions declined and code formatting errors increased. The authors' conclusion was that the behaviour of the same LLM service can change substantially in a short period, which makes continuous monitoring necessary.
Put together, these sources describe three sources of change that observability must catch: changes in the inputs (what users ask and what data arrives), changes in the system's own components (prompts, retrieval indexes, tools, model versions), and changes in the world the system is meant to reflect (policies, prices, regulations, product catalogues).
What AI observability adds to monitoring
Monitoring tells an operator that something is wrong. Observability lets an operator work out why, after the fact, from the data the system emitted. For an AI system that means capturing a structured trace of each request: the user input, the prompt template and version, the documents retrieved and their versions, any tool calls and their results, the model and version that answered, the output, token counts and latency, and any feedback or downstream outcome attached later.
Three properties distinguish a useful AI trace from a log line.
It must be linkable to outcomes. A trace that cannot be joined to the eventual human decision, correction or complaint cannot tell the team whether the system was right.
It must be versioned. If the trace does not record which prompt, index and model produced the output, a regression cannot be attributed to a change.
It must be governed. Traces often contain personal data. Under India's Digital Personal Data Protection Rules, notified on 14 November 2025, logging prompts and outputs is itself processing of personal data and attracts the same purpose, retention and security obligations as any other processing. SCC Online's summary of the Rules notes a requirement to retain certain logs for at least one year, which interacts with deletion duties. Observability design therefore needs a data protection review, not only an engineering one.
A five-signal monitoring checklist for AI observability
The checklist below is this article's analytical contribution. It groups what should be watched into five signals: quality, drift, cost, safety and usage. For each, it states what to measure, how to measure it in production, a typical trigger for investigation and the owner. The thresholds are illustrative; each enterprise should set its own from its pre-deployment evaluation baseline.
Two observations make the checklist more than a list.
First, the signals interact. A rise in cost per task with flat quality often indicates an agent looping or retrying. A fall in usage with stable automated quality scores often means the automated evaluator has stopped reflecting what users value, and needs recalibrating against fresh human review. Teams should review the five signals together, not in separate dashboards owned by separate functions.
Second, usage is a quality signal. In many enterprise deployments, ground truth arrives late or never: nobody records whether a drafted email was good. The human override and edit rate, measured as the share of outputs that users materially change before use, is often the earliest available proxy for quality. It costs little to capture if the interface is designed for it from the start.
Quality in production is a sampling problem
Pre-deployment evaluation, covered in a separate PanScience Research piece, establishes whether a system is good enough to launch. Production quality monitoring asks whether it is still good enough, on today's traffic. The two require different methods.
In production, the team cannot label every output. The workable approach is sampling: draw a random sample of traces each week, have domain reviewers grade them against the same rubric used before launch, and use the graded sample to calibrate any automated evaluators. Automated evaluators, including the practice of using one model to grade another, are useful for scale but drift themselves, so they should be checked against the human sample regularly. A fixed golden set, a collection of representative inputs with agreed correct outputs, should be re-run after every change to prompts, retrieval, tools or model version, and the results recorded against the version identifiers in the trace.
Drift has four sources in LLM systems
Classical machine learning distinguishes data drift, a change in the distribution of inputs, from concept drift, a change in the relationship between inputs and the correct output. Google Cloud's MLOps guidance uses closely related terms: data value skews and changes in feature distributions as signals that a model is stale. LLM systems add two more sources.
Input drift. Users start asking about new products, new regulations or in new languages. In India, a shift in the mix of English, Hindi and romanised Hindi queries can change quality materially.
Corpus drift. In retrieval-augmented systems, the documents change. A policy is revised, a price list is updated, and the index lags. Answers become confidently out of date.
Provider drift. A hosted model is updated. Chen, Zaharia and Zou's results show why the model identifier must be logged in every trace and why a change should trigger a golden-set run.
Concept drift. The world changes what a correct answer is, such as a new tax rule or a revised credit policy, even if inputs and model are unchanged.
Each source needs its own detector. Input drift is caught by distribution monitoring; corpus drift by freshness checks on the index; provider drift by version alerts and golden-set re-runs; concept drift mainly by domain owners who know when the rules have changed.
Cost, safety and usage in brief
Cost observability is the measurement half of AI cost management. The essential unit is cost per completed task, not cost per token, because a cheaper model that needs three attempts is not cheaper. A separate PanScience Research piece on AI cost optimisation covers the levers that act on these measurements.
Safety observability detects attacks and leaks in live traffic: prompt injection attempts, personal data appearing in outputs, and agents taking tool actions outside their mandate. The threat taxonomy is set out in a separate PanScience Research piece on AI security; observability supplies the evidence that controls are working.
Usage observability tells the enterprise whether the system is creating value at all. High automated quality scores mean little if employees have stopped using the tool.
Mapping AI observability to the NIST AI RMF
NIST's AI Risk Management Framework 1.0, released on 26 January 2023 for voluntary use, organises AI risk work into four functions: Govern, Map, Measure and Manage. Observability is the operational machinery behind several Manage and Measure subcategories. Making the mapping explicit lets the same telemetry serve the engineering team and the risk committee.
The mapping has a practical consequence. An enterprise that can show versioned traces, a documented review rhythm and incident records joined to those traces already holds much of the evidence an auditor or a regulator will ask for under the NIST framework, ISO/IEC 42001 or sector guidance. A separate article in this cluster sets out a unified AI risk management framework for Indian enterprises.
Where observability sits in the architecture
Observability is easiest when every model call passes through one layer. If each application calls models directly, traces are fragmented, model versions are recorded inconsistently and cost cannot be attributed. A gateway or control plane between applications and models provides a single point at which to log, measure and enforce.
Within the PanScience Innovations portfolio, two ventures work at that layer. PanScience reports that Proteus is an LLM gateway built to route, secure and monitor model traffic, and that Own360 is a control plane for AI-native enterprises (both descriptions company-reported). The general point does not depend on any particular product: whichever layer the enterprise chooses, it should be the place where traces are captured, model versions are pinned or logged, budgets are enforced and the kill switch required by MANAGE 2.4 is wired in. A separate PanScience Research piece on LLM gateways describes that layer's capabilities.
An operating rhythm
Tools do not observe; people do. A simple rhythm keeps the five signals in use.
Daily: automated alerts on safety flags, provider version changes, cost anomalies and error spikes, routed to named on-call owners.
Weekly: human review of the quality sample; review of drift dashboards and override rates; triage of new input clusters.
Monthly: a joint review of all five signals by product, engineering, risk and finance owners; decisions on retraining, prompt changes, index refresh or model change; golden-set results for any changes made.
Quarterly: a report to the governance committee mapping incidents, changes and residual risks to the NIST subcategories above. A separate PanScience Research piece covers AI governance for boards.
Conclusion
An AI system in production is a moving target. Its inputs, its components, its provider's model and the world it reflects all change, and the engineering literature from Sculley and colleagues to recent work on LLM drift says that this change is the norm rather than the exception. AI observability means capturing versioned, outcome-linked and governed traces, watching five signals together, and turning what is seen into decisions on a fixed rhythm. Mapped to the NIST AI RMF, the same telemetry becomes governance evidence. Enterprises that build this before the second or third system goes live will spend less time explaining incidents and more time improving the systems that work.
Methodology and sources
This article draws on sources opened by the author. The technical debt argument is from Sculley et al., "Hidden Technical Debt in Machine Learning Systems", NIPS 2015 (papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html). Model decay and monitoring guidance is from Google Cloud's "MLOps: Continuous delivery and automation pipelines in machine learning" (docs.cloud.google.com/architecture/mlops-continuous- delivery-and-automation-pipelines-in-machine-learning). LLM drift evidence is from Chen, Zaharia and Zou, "How is ChatGPT's behavior changing over time?" (arxiv.org/abs/2307.09009). NIST AI RMF details are from nist.gov (nist.gov/itl/ai-risk-management-framework) and the AI RMF Core on the NIST Trustworthy and Responsible AI Resource Center (airc.nist.gov/airmf-resources/airmf/5-sec- core). The production rate is from Menlo Ventures' 2025 State of Generative AI in the Enterprise (menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise), a US survey. DPDP Rules references are from PIB (static.pib.gov.in) and SCC Online (scconline.com/blog/post/2025/12/26/digital-personal-data-protection-rules-2025-key-highlights). The five-signal checklist, its thresholds, the NIST mapping and the operating rhythm are the author's original analysis; thresholds are illustrative and should be set from each system's own evaluation baseline. PanScience venture descriptions are company-reported from panscience.xyz.