Skip to content
TilloTech Docs

How to use Compass

Validate a versioned evaluation suite and write a local fixed-pass smoke report. For command flags, suite fields, and the report contract, see the Compass CLI reference.

Compass is the private @tillo/compass workspace package. Run it from this repository after a build. The smoke command writes a fixed-pass result for every case. It does not call models or tools.

Before you start

  • Node.js 24 or later.
  • A complete EOS repository setup with mise install and pnpm install.
  • A JSON evaluation suite that uses schema version 1.

Build Compass

Run this command from the repository root:

sh
nx run @tillo/compass:build

Provide an evaluation suite

The run command reads one JSON file. The file must use schema version 1. The required fields include model defaults and tool defaults.

The smoke fixture is a valid suite:

text
apps/compass/tests/fixtures/smoke-suite.json

Use that file, or provide another file that matches the same schema. The Compass CLI reference lists the required fields.

Write a smoke report

From the repository root, run:

sh
node apps/compass/dist/cli.js run \
  --eval apps/compass/tests/fixtures/smoke-suite.json

The default output directory is .compass/runs in the current working directory. Pass --output to choose a different directory:

sh
node apps/compass/dist/cli.js run \
  --eval apps/compass/tests/fixtures/smoke-suite.json \
  --output /tmp/compass-runs

The command prints one JSON summary on standard output. The summary includes the run id and the absolute path of report.json.

When the current directory is apps/compass, run the built file from that directory:

sh
node dist/cli.js run --eval tests/fixtures/smoke-suite.json

Confirm the report

Check that the process exit code is 0. Check that the summary accepted field is true and that status is completed.

Open the file at the report path. Each case has result fixed-pass and status passed. Usage fields are 0.

Handle an invalid suite

If the suite file is missing, is not a regular file, or fails schema checks, Compass writes an error to standard error. The message starts with Invalid evaluation suite:. The process exit code is 2. Compass does not write a report.

If --eval is missing, Compass writes Missing required flag: --eval to standard error. The process exit code is 2.