01 / What
An observability, debugging, and evaluation platform for AI agents that records traces, child spans, events, metrics, and errors without changing application control flow. The Python SDK ships with automatic LLM instrumentation for OpenAI and Anthropic, a context-managed tracing API (trace/span/observe), PII redaction, and a daemon-thread background worker that batches records to a Supabase Edge Function. The backend uses a private Postgres schema with row-level security, SHA-256-hashed API keys, and idempotent ingestion. A separate Next.js dashboard visualizes sessions, runs, and spans for debugging and rubric-based evaluation.
02 / Mechanics
- 01
Non-invasive auto-instrumentation of OpenAI and Anthropic clients captures request/response payloads, token usage, USD cost, and latency via runtime monkey-patching without altering application control flow
- 02
Background worker thread uses threading.Event wake/sleep batching with httpx bounded retries (408/429/5xx) for resilient async ingestion to Supabase Edge Functions
- 03
Built-in eval framework with deterministic graders (tool_sequence, tool_arguments_match, regex, output) and LLM judges (rubric_judge, faithfulness, python_code) for systematic agent evaluation
- 04
Versioned prompt templates with Jinja/Python variables stored server-side, bound to model calls at runtime with compile-time validation
03 / Components
Python
SDK core with Pydantic models, ContextVar-based trace nesting, and httpx transport
Supabase Edge Functions
Deno/TS ingestion layer handling SHA-256 API key auth, payload validation, and topological span sorting
PostgreSQL
Multi-tenant data store with Row Level Security, ON CONFLICT idempotent ingestion, and private schema tables for sessions/runs/spans/events/scores
Next.js
Web dashboard for trace visualization and eval management
LiteLLM
Token counting and USD cost pricing across model providers
OpenAI / Anthropic SDKs
Auto-instrumented via runtime patching of chat, responses, and messages.create endpoints