Datadog

Verified : Code examples on this page have been automatically tested and verified.

Export agentgateway metrics and LLM traces to Datadog.

Datadog collects metrics and OpenTelemetry traces from agentgateway. Choose the setup that matches what you want to observe.

Setup	Telemetry	When to use it
Complete example	Proxy metrics, LLM traces, and a dashboard	Evaluate the integration end to end or use it as a production reference
Direct trace export	LLM traces	Add tracing to an existing deployment with the fewest components

The complete example is the recommended starting point. It runs the Datadog Agent and a demo model with Docker Compose next to your agentgateway installation. The Agent scrapes proxy metrics and receives OpenTelemetry Protocol (OTLP) traces over HTTP directly from the proxy. The direct setup sends traces from the proxy to Datadog’s hosted OTLP endpoint.

Before you begin

For the complete example, you need:

A running agentgateway installation on the Docker host, with metrics on the default port 15020. If you have not installed it, see Install.
Docker with Docker Compose v2, and curl.
Free loopback ports 3000 for the agentgateway LLM listener, 18080 for the demo model, and 4318 for OTLP ingestion.
A Datadog organization, API key, and the correct Datadog site.
Agent Observability enabled in your Datadog organization to view LLM traces.

The direct setup requires a running agentgateway installation and a Datadog API key. If you have not installed agentgateway or configured an LLM provider, complete the LLM quickstart first.

Run the complete example

The Datadog standalone example uses Docker Compose to run the Datadog Agent and the HomeHub demo model. HomeHub returns fixed OpenAI-compatible responses, so the example does not call a paid model or need a provider API key. You merge the example’s config.yaml into your agentgateway configuration to add the demo model and trace export.

Note

The example pins the Datadog Agent and HomeHub versions in versions.env. The example files expect agentgateway to run on the Docker host. For a container or remote gateway, update the backend addresses in config.yaml and the metrics endpoint in openmetrics.yaml. Check the example README for the tested agentgateway version.

Clone the agentgateway repository and change to the example directory.

git clone https://github.com/agentgateway/agentgateway.git
cd agentgateway/examples/datadog/standalone

Create a .env file for your Datadog credentials. The repository’s .gitignore already excludes this file.
```
DD_API_KEY=replace-with-your-datadog-api-key
DD_SITE=us3.datadoghq.com
```
Note
Datadog API keys and application keys are different. This example requires an API key to send telemetry to Datadog.
Restrict the .env file permissions to your user.
```
chmod 600 .env
```
Start the Datadog Agent and HomeHub.
```
docker compose --env-file versions.env --env-file .env up -d
```
The example’s openmetrics.yaml check sets use_latest_spec: true so that the Agent parses the proxy’s output as OpenMetrics. Version 1.6 and earlier need this setting, or the Agent drops every counter.

Merge the example’s config.yaml into your agentgateway configuration. The file adds the homehub-support model on port 3000 and exports traces to the Agent at localhost:4318. Adjust the port and backend addresses to match your deployment.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
frontendPolicies:
  tracing:
    host: localhost:4318
    protocol: http
    randomSampling: true
    attributes:
      env: '"datadog-dev"'
llm:
  port: 3000
  models:
  - name: homehub-support
    provider: openAI
    params:
      baseUrl: http://localhost:18080/v1

For more information, see Update the configuration.

Send a request to the HomeHub model.

curl --fail http://127.0.0.1:3000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"homehub-support","stream":false,"messages":[{"role":"user","content":"How do I factory-reset my HomeHub X2?"}],"max_tokens":96}'

The response describes the HomeHub X2 reset procedure and reports 12 input and 24 output tokens.

Check the proxy’s request and token metrics.
```
curl --fail http://127.0.0.1:15020/metrics \
  | grep -E '^agentgateway_(requests|gen_ai_client_token_usage)'
```
The output includes agentgateway_requests_total and agentgateway_gen_ai_client_token_usage samples for the homehub-support model.
Run the OpenMetrics check twice, and send another request between the two runs. Counter metrics need two scrapes before rate samples appear.
```
docker compose --env-file versions.env --env-file .env \
  exec datadog agent check openmetrics --check-rate
```
The check reports metric samples tagged component:proxy.
In Datadog, open Metrics > Explorer, filter by env:datadog-dev, and search for an exact metric name.
- agentgateway.requests.count
- agentgateway.gen_ai.token.usage.sum
In Dashboards, import the example’s dashboard.json and set the env template variable to datadog-dev. Enable percentile aggregations in Metrics Summary for the latency distributions before you use the p95 widgets. The controller widgets stay empty in standalone mode. The streaming timing, MCP, and guardrail widgets stay empty until the corresponding traffic is present. HomeHub has no real provider pricing, so estimated-cost widgets can be empty or show Cost unavailable.

Traces from the example omit prompt and completion content.

Review redaction, sampling, access controls, and custom-metric cardinality before you adapt the example for production.

Configure direct trace export

Use this smaller setup when you only need traces and do not want to run the Datadog Agent locally. Replace the API key and change the host for your Datadog site if needed.

Add this tracing policy to your config.yaml.

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
frontendPolicies:
  tracing:
    host: otlp.datadoghq.com:443
    protocol: grpc
    randomSampling: true
    policies:
      backendTLS: {}
      requestHeaderModifier:
        set:
          DD-API-KEY: "<your-datadog-api-key>"

Add this tracing policy to your agentgateway values.yaml.

config:
  frontendPolicies:
    tracing:
      host: otlp.datadoghq.com:443
      protocol: grpc
      randomSampling: true
      policies:
        backendTLS: {}
        requestHeaderModifier:
          set:
            DD-API-KEY: "<your-datadog-api-key>"

Apply the change.

helm upgrade -i agentgateway-standalone \
  oci://cr.agentgateway.dev/charts/agentgateway-standalone \
  --namespace agentgateway-system \
  --version v1.7.0 \
  --reuse-values \
  -f values.yaml

For the EU region, use otlp.datadoghq.eu:443. This direct path does not configure OpenMetrics collection or a dashboard.

Note

Keep API keys out of source control.

Verify LLM traces in Datadog

Both setups export LLM traces, so verify them the same way.

Open AI Observability > Applications and select agentgateway. The summary shows LLM calls, token usage, duration, errors, and traces for the selected time range.

Datadog Agent Observability overview for the agentgateway application, showing error rate, duration, token usage, LLM calls, and total traces.

Open AI Observability > Traces and search for ml_app:agentgateway. Inspect a span to verify its model, token counts, errors, timing, and parent-child trace relationships. Allow several minutes for processing. A successful OTLP response or a trace in APM alone does not prove ingestion into Agent Observability.

Datadog Agent Observability span for agentgateway, showing the synthetic model and agentgateway cost and timing tags.

Troubleshooting

The Datadog Agent container is unhealthy

An invalid API key or incorrect DD_SITE can make the Agent unhealthy even when its OpenMetrics check reaches agentgateway. Verify the site and API key, then inspect the Agent.

docker compose --env-file versions.env --env-file .env ps
docker compose --env-file versions.env --env-file .env \
  exec datadog agent status

Do not share docker compose config, container inspection output, or the local .env file; those outputs can contain credentials.

Metrics are missing

A healthy agentgateway.openmetrics.health service check proves that the endpoint responded; it does not prove that counter samples reached Datadog. Run traffic between scrape intervals, repeat the check with --check-rate, allow several minutes for indexing, and search by the exact metric name.

Gauges appear but counters are missing

In version 1.6 and earlier, confirm that the openmetrics check sets use_latest_spec: true. Without this setting, the Agent parses the proxy’s OpenMetrics output as classic Prometheus text and drops every counter without an error, while gauges and histograms still arrive.

Traces are missing

Confirm that Agent Observability is enabled, the Agent is healthy, and DD_SITE selects the correct organization. Search for ml_app:agentgateway and allow several minutes for processing. Confirm that your agentgateway configuration includes the example’s tracing settings and that port 4318 on the Docker host reaches the Agent.

Dashboard percentile widgets are empty

Enable percentile aggregations for the corresponding distribution metrics in Datadog Metrics Summary, allow time for processing, and confirm that more than one scrape occurred.

Cleanup

Remove the HomeHub model and tracing settings that you merged into your agentgateway configuration.

Stop the Datadog Agent and HomeHub.

docker compose --env-file versions.env --env-file .env down

Learn more

Arize Phoenix Azure Monitor

Was this page helpful?