Skip to content
All case studies

06 · Independent

AI code evaluation & agentic AI training

Evaluating and training frontier coding LLMs: rubrics, adversarial test cases and training data for long-horizon coding agents.

Context

Frontier coding models lose file paths, drop the reason behind a decision and break context on long tasks. They need high-signal training and evaluation data. I work on that data as an independent contributor.

My role

Independent contributor on training and evaluation tasks for frontier coding LLMs.

Architecture

Architecture, left to right. Input: Coding task → Rubric: 50–90 binary criteria → Grade: Score model output → Output: Higher-signal training data.

The work as a pipeline. Each stage feeds the next.

Key decisions

  1. 01

    Binary rubric criteria

    Each task gets 50–90 yes/no criteria. [WHY: ask Saif — why atomic binary criteria beat scored scales]

  2. 02

    Context-compaction artifacts for long-horizon agents

    Models lose file paths, decision rationale and context across sessions. The artifacts keep those, so an agent can resume a multi-session task.

  3. 03

    Adversarial prompts and full-stack test cases

    Prompts and test cases stress-test how models handle ambiguity, edge cases and production trade-offs.

Results

  • 200+ training and evaluation tasks for frontier coding LLMs.
  • Recurring model failure modes rewritten into higher-signal training data.
  • Consistently rated expert tier, with high agreement against senior reviewers.

Stack

  • LLM Evaluation
  • RLHF
  • Rubric Design
  • Agentic AI
  • Prompt Engineering
  • Adversarial Testing