OpenAI: harness engineering

Source: Harness engineering: leveraging Codex in an agent-first world

Published: 11 February 2026.

Evidence type: experimental case study of a greenfield internal product built under a deliberate “no manually written code” constraint. The article reports one team’s practices and results; it is not a controlled comparison or a general Codex setup reference.

Experiment and reported results

OpenAI started from an empty repository in late August 2025. Codex CLI using GPT-5 generated the initial repository structure, CI, formatting, package management, application framework, and AGENTS.md, guided by a small set of existing templates. Five months later, OpenAI reported:

• roughly one million lines across product code, infrastructure, tooling, documentation, and developer utilities;
• roughly 1,500 opened and merged pull requests;
• an initial team of three engineers, later seven;
• an average of 3.5 pull requests per engineer per day;
• hundreds of internal users, including daily power users, plus external alpha testers; and
• an estimate that the product took about one tenth of the hand-written development time.

These figures establish the unusual operating context. They do not isolate the effect of Codex, prove that line count or PR count measures value, or establish the same results for an existing codebase.

“Agent-generated” covered the entire repository: product code, tests, CI, release tooling, internal tools, documentation, design history, evaluation harnesses, review comments, repository-management scripts, and production dashboard definitions. Humans chose priorities, translated feedback into acceptance criteria, and validated outcomes, but did not directly contribute code.

Role of the engineer

The team found early progress limited by an underspecified environment rather than model capability. Human work moved toward decomposing goals, building tools and abstractions, specifying intent, and designing feedback loops. The team worked depth-first: prompt Codex to construct smaller design, code, review, and test building blocks, then use those capabilities to unlock larger tasks.

When the agent failed, the operating question became: what capability is missing, and how can it become both legible and enforceable? The response could be a tool, guardrail, abstraction, document, test, or architectural constraint; the team had Codex add that improvement to the repository so later runs gained the same capability.

The described pull-request loop is:

1. An engineer specifies a task through a prompt.
2. Codex gathers context directly through gh, local scripts, and repository-embedded skills.
3. Codex implements the change and reviews its own work locally.
4. It requests additional targeted reviews from local and cloud agents.
5. It responds to human and agent feedback and iterates until the agent reviewers are satisfied.
6. It pushes updates and can squash and merge its own pull request.

This avoids humans acting as a context-transfer layer. Human review remains possible, but OpenAI reports moving most review toward agent-to-agent loops.

Application and operational legibility

The application can boot independently per git worktree, giving each change a runnable and isolated instance. The UI feedback loop uses Chrome DevTools Protocol access plus skills for DOM snapshots, screenshots, and navigation: Codex captures state, exercises a path, observes runtime events, changes code, restarts, and repeats validation.

Each worktree also receives an ephemeral local observability stack. Application telemetry is routed through Vector to VictoriaLogs, VictoriaMetrics, and VictoriaTraces. Codex can query logs with LogQL, metrics with PromQL, and traces with TraceQL; both the application instance and its telemetry are destroyed when the task ends.

This turns quantitative requirements into executable feedback. The article’s examples include completing service startup in under 800 milliseconds and keeping spans in four critical journeys below two seconds. OpenAI reports that individual Codex runs regularly continue on one task for more than six hours.

The transferable requirement is not a particular browser or telemetry product. An agent should be able to start the system, create the relevant state, inspect UI and runtime behaviour, query success signals, and repeat the test inside the isolated environment where it changes the code.

Repository knowledge system

OpenAI reports that one giant AGENTS.md failed because it consumed scarce context, made every rule look equally important, decayed quickly, and could not be mechanically checked. The replacement uses a root AGENTS.md of roughly 100 lines as a table of contents and makes structured repository documentation the system of record.

The article gives this concrete layout:

AGENTS.md
ARCHITECTURE.md
docs/
├── design-docs/
│   ├── index.md
│   ├── core-beliefs.md
│   └── ...
├── exec-plans/
│   ├── active/
│   ├── completed/
│   └── tech-debt-tracker.md
├── generated/
│   └── db-schema.md
├── product-specs/
│   ├── index.md
│   ├── new-user-onboarding.md
│   └── ...
├── references/
│   ├── design-system-reference-llms.txt
│   ├── nixpacks-llms.txt
│   ├── uv-llms.txt
│   └── ...
├── DESIGN.md
├── FRONTEND.md
├── PLANS.md
├── PRODUCT_SENSE.md
├── QUALITY_SCORE.md
├── RELIABILITY.md
└── SECURITY.md

The parts have distinct jobs:

• architecture documentation maps domains and package layers;
• design documents have an index, verification status, and recorded core beliefs;
• small changes use lightweight ephemeral plans, while complex changes use checked-in execution plans with progress and decision logs;
• active plans, completed plans, and known technical debt remain versioned and colocated;
• generated reference material records inspectable system facts such as a database schema;
• product specifications and discipline-specific guides expose product and engineering expectations; and
• quality grades track gaps by product domain and architectural layer.

Linters and CI check structure, cross-links, and freshness. A recurring documentation-gardening agent finds stale or obsolete material and opens repair pull requests. This is progressive disclosure backed by ownership and maintenance, not merely more Markdown.

Repository-local context and dependency choices

The team treats information unavailable to a running agent as effectively non-existent. Architectural decisions left only in chat, hosted documents, or people’s memories are moved into versioned repository artifacts such as Markdown, schemas, code, and executable plans. The point is to expose the right context through a navigable structure, not to inject all context into every prompt.

This goal also affected dependency choices. The team favours stable, composable, well-represented technology that agents can inspect and model. In some cases it implemented a narrow internal abstraction rather than depending on opaque upstream behaviour. The article’s concrete example is an internal map-with-concurrency helper, integrated with OpenTelemetry and covered by tests, instead of a generic p-limit-style package.

This is a contextual trade-off, not a blanket “build instead of buy” rule. The decision criterion is whether the dependency’s behaviour and integration can be understood, validated, and safely modified within the agent’s environment.

Mechanically enforced architecture

Documentation alone did not keep the codebase coherent. Each business domain uses a fixed dependency order:

Types → Config → Repo → Service → Runtime → UI

Cross-cutting concerns such as authentication, connectors, telemetry, and feature flags enter through an explicit Providers interface. Other dependency directions are disallowed by custom linters and structural tests.

The team also mechanically checks:

• parsing data shapes at system boundaries rather than guessing them;
• structured logging;
• schema and type naming conventions;
• file-size limits; and
• platform-specific reliability requirements.

Custom lint errors include remediation instructions, turning a violation into usable agent context instead of a bare rejection. OpenAI’s governing principle is to enforce invariants centrally and allow autonomy locally. Dependency boundaries, correctness, reproducibility, maintainability, and future-agent legibility matter; incidental stylistic preferences need not be uniform.

Human judgement is fed back into the harness. Review feedback, refactors, and user-facing bugs become documentation or tooling, and a rule moves from prose to code when it can be enforced mechanically.

End-to-end autonomy loop

OpenAI reports that the accumulated harness can now take one prompt through this complete sequence:

1. Validate the repository’s baseline state.
2. Reproduce a reported bug.
3. Record a video demonstrating the failure.
4. Implement a fix.
5. Drive the application to validate the fix.
6. Record a second video demonstrating the resolution.
7. Open a pull request.
8. Respond to human and agent feedback.
9. Detect and remediate build failures.
10. Escalate only decisions that require human judgement.
11. Merge the change.

The videos are evidence artifacts, not decoration: they make the before and after states reviewable. OpenAI explicitly says this autonomy depends heavily on the repository’s structure and tooling and should not be assumed to generalise without comparable investment.

Entropy and continuous garbage collection

Agents reproduce existing patterns, including bad ones. The team initially spent every Friday—20% of the working week—manually cleaning up what it calls “AI slop.” It replaced that batch cleanup with repository-encoded “golden principles” and recurring background tasks.

Two concrete principles are:

• prefer shared utility packages over repeated hand-written helpers so invariants remain centralised; and
• validate boundaries or use typed SDKs rather than probing data and building against guessed shapes.

Background Codex tasks scan for deviations, update quality grades, and open small targeted refactoring pull requests. OpenAI reports that most are reviewable in under a minute and can be automerged. The intended feedback loop is to capture a human judgement once, enforce it continuously, and remove bad patterns before agents replicate them across the codebase.

Merge philosophy and safety boundary

The repository uses minimal blocking merge gates, short-lived pull requests, and follow-up runs for some test flakes. The rationale is economic: agent throughput exceeds available human attention, corrections are cheap, and waiting is costly in this specific system.

OpenAI explicitly says this would be irresponsible in a low-throughput environment. EOS should not import this merge policy merely because the other harness practices are useful. Gate strength must follow failure impact, rollback cost, regulatory requirements, signal reliability, and demonstrated recovery capability—not anticipated agent speed.

What remains unknown

The article identifies unresolved questions rather than claiming a finished model:

• whether architectural coherence will persist over years in a fully agent-generated system;
• where human judgement produces the most leverage;
• how to encode that judgement so it compounds; and
• how more capable models will change the required harness.

The reported evidence reaches internal launch and adoption, not long-term maintenance across multiple model and team generations.

Standards implications

• Treat AGENTS.md as a short map; place detailed, owned knowledge in indexed repository documents.
• Make success observable in the same isolated environment as the change, including UI state and logs, metrics, or traces where relevant.
• Give agents direct access to standard developer tools and repository workflows instead of manually relaying context.
• Treat plans, demonstrations, quality state, design history, and debt as versioned operational data.
• Enforce dependency direction and other architectural invariants mechanically with remediation-rich errors.
• Convert repeated failures and review comments into durable improvements to tools, guardrails, documentation, tests, or architecture.
• Establish recurring documentation gardening, quality grading, and targeted cleanup before increasing autonomy.
• Require baseline validation, reproducible failure evidence, fix evidence, review-feedback handling, and explicit judgement escalation in an end-to-end autonomous workflow.
• Set merge gates from risk and proven recovery economics; do not copy the case study’s minimal-gate policy by default.