Project / evalon

Evalon

Local, terminal-first observability and evaluation for Python agents — traces, versioned datasets, and deterministic or LLM-judged evals with no cloud.

Source

Python / SQLite / AI Observability / LLM Evals / Terminal UI

01 / What

Evalon records agent runs as traces in local SQLite, turns reviewed traces into versioned dataset cases, and runs deterministic, Python, and rubric-based evaluations. A Textual terminal UI covers both trace inspection and evaluation history — there is no hosted service, cloud account, or web dashboard. It wraps existing OpenAI-compatible and Anthropic clients without changing their calling interface, and ships a catalog of static evaluators plus binary LLM judges for CI-gated eval runs.

02 / Mechanics

  1. 01

    Trace API with nested spans, tools, and provider wrappers (evalon.openai() / openrouter() / anthropic()) captures inputs, outputs, events, metrics, token usage, latency, errors, and optional cost estimates without changing control flow

  2. 02

    Versioned dataset workflow — dataset edits publish immutable versions, reviewed traces can be captured as new cases, and suites bind a dataset to a Python target (module:callable) with YAML-declared evaluators

  3. 03

    Evaluator catalog spans static heuristics (exact match, JSON schema/path, tool order and arguments, call counts, latency, token/cost caps) and LLM judges (faithfulness, correctness, user goal completion, tool-use quality) with per-judge traces

  4. 04

    CI-ready execution with concurrency, background workers, thresholds (min pass rate, max cost, max latency), JSON output, JUnit/Markdown/CSV exports, and exit codes that distinguish config, execution, evaluation, threshold, and storage failures

03 / Components

Python 3.11+

SDK core with sync/async tracing runtime, provider wrappers, and an @observe/@tool decorator surface

SQLite

Local persistence for traces, sessions, datasets, suites, and eval runs at ~/.evalon/evalon-runs.sqlite (JSONL also supported)

Textual

Terminal UI for trace inspection and evaluation history with an Evalon orange-on-black theme

OpenAI / Anthropic

Preserving wrappers that turn provider calls into spans with request/response, tokens, latency, and errors

LiteLLM

Optional model cost estimation per provider call

jsonschema

JSON Schema and JSON path static evaluators for structured-output validation

PyYAML

Versioned dataset and suite definitions declared as review-friendly YAML