Compass CLI

Compass is the @tillo/compass command-line interface. It validates one versioned evaluation suite and writes a local fixed-pass smoke report.

For the smoke run from this repository, see How to use Compass.

Compass requires Node.js 24 or later. The package is private. Run the built dist/cli.js file from this repository.

The smoke command writes a fixed-pass result for every case. It does not call models or tools. It still requires a complete schema version 1 suite. The suite must contain model defaults and tool defaults.

Commands

compass <subcommand> [flags]

Global flags

These flags are available on every command.

--completions accepts bash, zsh, fish, or sh. --log-level accepts all, trace, debug, info, warn, warning, error, fatal, or none.

--version prints compass v followed by the version in package.json.

compass run

compass run --eval <path> [--output <directory>]

Paths may be relative. Compass resolves them against the current working directory.

The command reads --eval as a regular file. It decodes the file as a schema version 1 evaluation suite. It then writes one report under --output.

The command does not use suite defaults, case tasks, or rubric text after validation. Each report case copies the suite case id and records a fixed-pass result.

Standard output

On success the command prints one JSON object, then a newline:

{
  "accepted": true,
  "report": "/absolute/path/to/<runId>/report.json",
  "runId": "20260917T145642099Z-e572269c",
  "status": "completed"
}

report is the absolute path of the written file. runId matches the report directory name and the runId field inside the file.

Report files

The report path is:

<output-directory>/<runId>/report.json

runId is a UTC timestamp with -, :, and . removed, a hyphen, and the first eight characters of a UUID. Example: 20260917T145642099Z-e572269c.

Compass creates the run directory with mode 0700 and the report file with mode 0600. The file contains indented JSON and a trailing newline.

The resolved run directory must stay inside the resolved output directory. If it would not, the command fails as an invalid evaluation suite.

Errors

Invalid evaluation suites write to standard error:

Invalid evaluation suite: <message>

The process exit code is 2. Compass does not write a report file.

Invalid command use also exits with code 2. A missing --eval flag writes Missing required flag: --eval to standard error.

Evaluation suite

--eval must decode as schema version 1.

Empty arrays are valid for allowed, mcpServices, must, and should.

The smoke fixture at apps/compass/tests/fixtures/smoke-suite.json is a valid suite:

{
  "schemaVersion": 1,
  "name": "compass-smoke",
  "defaults": {
    "models": {
      "target": "amazon-bedrock/openai.gpt-5.6-sol",
      "judge": "amazon-bedrock/openai.gpt-5.6-terra",
      "revision": "amazon-bedrock/openai.gpt-5.6-sol"
    },
    "concurrency": 1,
    "judgePasses": 1,
    "maxIterations": 1,
    "tools": {
      "allowed": [],
      "mcpServices": []
    }
  },
  "cases": [
    {
      "id": "single-smoke-case",
      "task": "Return the temporary fixed smoke result.",
      "rubric": {
        "must": ["reports one fixed-pass case"],
        "should": []
      }
    }
  ]
}

Run report

The written report.json uses schema version 1.

Example for the smoke fixture:

{
  "accepted": true,
  "cases": [
    {
      "id": "single-smoke-case",
      "result": "fixed-pass",
      "status": "passed"
    }
  ],
  "runId": "20260917T145642099Z-e572269c",
  "schemaVersion": 1,
  "status": "completed",
  "stopReason": "fixed-pass",
  "suite": "compass-smoke",
  "usage": {
    "costUsd": 0,
    "inputTokens": 0,
    "modelCalls": 0,
    "outputTokens": 0
  }
}

Exit codes

Related documentation

• How to use Compass