LIVE

Datadog Expands Observability Platform with Production Testing for AI Agents

🧠 AI Analysis 1 min read 👁 8

Datadog has updated its Agent Observability platform to include Online Experiments, enabling engineering teams to test artificial intelligence agent modifications against live user traffic.

Datadog Expands Observability Platform with Production Testing for AI Agents🧠AI Analysis

Photo: infoq.com

Key points

  • Datadog adds Online Experiments to its Agent Observability platform
  • Combines live production traffic testing with operational metrics like cost and latency
  • Links experiment results to individual execution traces to spot hidden tool errors

Datadog has introduced a new feature called Online Experiments to its Agent Observability platform, allowing development teams to test updated AI agent versions using live production traffic instead of relying only on static evaluation datasets. This system routes real user interactions across a control version and a modified variant, merging performance scoring with operational telemetry such as latency, token usage, execution cost, and error rates.

While traditional offline evaluations help spot known regressions using predefined test cases, they often fail to capture unpredictable user requests and evolving workflows. Datadog addresses this gap by utilizing feature flags to allocate production traffic gradually, helping developers catch trade-offs between cost, speed, and accuracy that controlled tests might miss.

During a recent test comparing two model variants, the platform tracked an estimated cost reduction alongside potential increases in execution duration and invalid tool parameters. Engineers linked these experiment results directly to individual execution traces, allowing them to fix behavioral issues and expand their offline regression test suites before wider deployment.

Source infoq.com — infoq.com/news/2026/11/datadog-ai-agent-observability/?utm_campaign… ↗

Contact the newsroom

Got a tip, a proposal or a partnership question? Write to us and we will reply shortly.