Lookout (Observe)

Every request traced. Every dollar tracked. Every anomaly caught.

Overview

Every proxy request is automatically traced. Lookout records model, tokens, cost, latency, and status for every call. No configuration needed — it starts recording the moment you send your first request.

Traces

# Get recent traces
curl http://localhost:4200/api/observe/traces?limit=20

# Filter by source (real requests only)
curl http://localhost:4200/api/observe/traces?source=real

Each trace includes: model, provider, status, latency (ms), tokens in/out, cost (USD), timestamp, and a unique trace ID.

Cost Tracking

# Daily cost breakdown
curl http://localhost:4200/api/observe/costs

# Time-series data for charts
curl http://localhost:4200/api/observe/timeseries?period=7d

Cost data is broken down by provider and model, so you can see exactly where your spend is going.

Alerts & Anomalies

# Create an alert rule
curl -X POST http://localhost:4200/api/observe/alerts \
  -d '{"name":"cost-spike","metric":"cost","condition":"gt","threshold":10}'

# View detected anomalies
curl http://localhost:4200/api/observe/anomalies

Safety Monitoring

# Safety summary (score, events by category)
curl http://localhost:4200/api/observe/safety/summary

# Recent safety events
curl http://localhost:4200/api/observe/safety?limit=50

See the Observe product page for screenshots of the dashboard.

Traces

Every request through the proxy creates a trace. Query recent traces:

curl "http://localhost:4200/api/observe/traces?limit=10" \
  -H "Authorization: Bearer sy_admin_..."

Response:

{
  "traces": [{
    "id": "tr_abc123",
    "timestamp": "2026-02-28T14:30:00Z",
    "model": "gpt-4o",
    "provider": "openai",
    "status": 200,
    "latency_ms": 1240,
    "tokens_in": 450,
    "tokens_out": 120,
    "cost_usd": 0.0078,
    "cached": false,
    "modules_active": 24
  }],
  "total": 1247,
  "has_more": true
}

Trace Detail

Get full detail for a single trace including the module chain execution:

curl http://localhost:4200/api/observe/traces/tr_abc123 \
  -H "Authorization: Bearer sy_admin_..."

Cost Tracking

View cost rollups aggregated by day, provider, or model:

curl "http://localhost:4200/api/observe/costs?group_by=provider&days=7" \
  -H "Authorization: Bearer sy_admin_..."

{
  "period": "2026-02-21 to 2026-02-28",
  "total_usd": 12.47,
  "breakdown": [
    {"provider": "openai", "cost_usd": 8.30, "requests": 890},
    {"provider": "anthropic", "cost_usd": 3.92, "requests": 312},
    {"provider": "groq", "cost_usd": 0.25, "requests": 445}
  ]
}

Time-Series Data

Fetch time-series metrics for charting:

curl "http://localhost:4200/api/observe/timeseries?metric=latency&interval=1h&days=1" \
  -H "Authorization: Bearer sy_admin_..."

Alert Rules

Create alerts that trigger when thresholds are breached:

curl -X POST http://localhost:4200/api/observe/alerts \
  -H "Authorization: Bearer sy_admin_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "High daily spend",
    "metric": "cost_usd",
    "threshold": 50.00,
    "window": "24h",
    "action": "webhook",
    "webhook_url": "https://hooks.slack.com/..."
  }'

Alerts can trigger webhooks, email notifications, or both. Update or delete alerts by ID:

# Update an alert threshold
curl -X PUT http://localhost:4200/api/observe/alerts/alert_id \
  -H "Authorization: Bearer sy_admin_..." \
  -H "Content-Type: application/json" \
  -d '{"threshold": 100.00}'

# Delete an alert
curl -X DELETE http://localhost:4200/api/observe/alerts/alert_id \
  -H "Authorization: Bearer sy_admin_..."

Dashboard Overview

The /api/observe/overview endpoint returns a dashboard summary suitable for building custom UIs:

curl http://localhost:4200/api/observe/overview \
  -H "Authorization: Bearer sy_admin_..."

{
  "requests_today": 342,
  "cost_today_usd": 2.15,
  "avg_latency_ms": 1180,
  "cache_hit_rate": 0.23,
  "active_alerts": 1,
  "top_model": "gpt-4o",
  "top_provider": "openai"
}

Configuration

# stockyard.yaml
apps:
  observe:
    retention_days: 30
    cost_tracking: true
    trace_request_body: false   # Set true to log full request/response
    alert_check_interval: 60s
    anomaly_detection: true
    anomaly_sensitivity: 0.8

Common Queries

Filter traces by model, provider, status, or date range:

# Traces for a specific model
curl "http://localhost:4200/api/observe/traces?model=gpt-4o&limit=20" \
  -H "Authorization: Bearer sy_admin_..."

# Failed requests only
curl "http://localhost:4200/api/observe/traces?status=error&limit=50" \
  -H "Authorization: Bearer sy_admin_..."

# Costs grouped by model for the last 30 days
curl "http://localhost:4200/api/observe/costs?group_by=model&days=30" \
  -H "Authorization: Bearer sy_admin_..."

Exporting Data

Export traces and cost data for external analysis. The Lookout API supports pagination via offset and limit parameters, making it straightforward to iterate through large datasets.

Built-in Dashboard

Stockyard includes a built-in dashboard at /ui that displays real-time metrics, cost charts, and trace logs. The dashboard uses the same API endpoints documented here, so you can also build custom dashboards.

Alerting Patterns

Common alert configurations for production:

OpenTelemetry Export

Stockyard can export traces in OpenTelemetry format for integration with external monitoring tools like Grafana, Datadog, or Honeycomb. Configure the OTEL exporter in stockyard.yaml:

# stockyard.yaml
apps:
  observe:
    otel_endpoint: "http://localhost:4317"
    otel_protocol: "grpc"
    otel_service_name: "stockyard-proxy"

When configured, every trace is exported to both the internal SQLite store and the external OTEL collector.