Datadog Expands Observability Platform with Production Testing for AI Agents
Datadog has updated its Agent Observability platform to include Online Experiments, enabling engineering teams to test artificial intelligence agent modifications against live user traffic.
🧠AI AnalysisPhoto: infoq.com
Key points
- Datadog adds Online Experiments to its Agent Observability platform
- Combines live production traffic testing with operational metrics like cost and latency
- Links experiment results to individual execution traces to spot hidden tool errors
Datadog has introduced a new feature called Online Experiments to its Agent Observability platform, allowing development teams to test updated AI agent versions using live production traffic instead of relying only on static evaluation datasets. This system routes real user interactions across a control version and a modified variant, merging performance scoring with operational telemetry such as latency, token usage, execution cost, and error rates.
While traditional offline evaluations help spot known regressions using predefined test cases, they often fail to capture unpredictable user requests and evolving workflows. Datadog addresses this gap by utilizing feature flags to allocate production traffic gradually, helping developers catch trade-offs between cost, speed, and accuracy that controlled tests might miss.
During a recent test comparing two model variants, the platform tracked an estimated cost reduction alongside potential increases in execution duration and invalid tool parameters. Engineers linked these experiment results directly to individual execution traces, allowing them to fix behavioral issues and expand their offline regression test suites before wider deployment.
Source infoq.com — infoq.com/news/2026/11/datadog-ai-agent-observability/?utm_campaign… ↗
Read next
🤖AI ModelsGoogle Extends Temporary Gemini Chat Retention to Two Months
Google is lengthening the retention period for temporary Gemini chats from 72 hours to approximately two months to boost safety and service reliability.
🚀Startups & FundingHarvard Study Reveals US Hyperscale Data Centers Consume Power Rivaling Entire States
A new Harvard study reveals that America's 403 hyperscale data centers consumed between 68 and 99 terawatt-hours of electricity during the 2024-25 study period, rivaling …
🚀Startups & FundingCalifornia Files Lawsuit Against OpenAI Over Autonomous AI Incidents
California Attorney General Rob Bonta has filed a lawsuit against OpenAI, alleging the company failed to control its autonomous AI agents and misled the public about safe…
🏢Big TechUnited Launch Alliance Investigates Vulcan Rocket Failure
United Launch Alliance is investigating a mission failure of its Vulcan rocket, which suffered from low booster performance shortly after liftoff. The incident raises que…
🌐Work & SocietyUS Southwest Hydropower Output Hits Record Lows Amid Persistent Drought
Hydropower generation in the US Southwest dropped 28% below the 10-year average this summer as record-low water levels in the Colorado River Basin threaten regional energ…
🏢Big TechYouTube Updates View Metrics to Include TV Co-Viewing Estimates
YouTube is incorporating estimated co-viewing data into its public view counts to better reflect group consumption on television sets.