Agent-skills-eval tests whether skills actually improve outputs

agent-skills-eval is a test runner for Agent Skills, the open standard from Anthropic for giving agents domain knowledge. The tool is designed to answer a simple question with evidence: does a skill actually improve model output, or does it just add more instructions?

The project was introduced in a Show HN post as “the missing piece” for the Agent Skills ecosystem. Its main job is to run the same eval twice, once with the skill loaded into context and once without it as a baseline, then have a judge model score both outputs side by side.

The result is a report that shows whether the skill made a measurable difference. According to the project description, if a skill does not improve performance, users will see that directly. If it does, they get a side-by-side record of the lift.

The tool is packaged as both a CLI and a TypeScript SDK. The author says users can start with a one-line command such as npx agent-skills-eval ./skills --target gpt-4o-mini --judge gpt-4o-mini --baseline --strict, which points the evaluator at a folder of skills and produces a workspace with artifacts and an HTML report.

The output includes metadata, pass/fail results per skill, per-eval folders, and a static index.html report. The report is built from on-disk artifacts, so it can be published without extra infrastructure.

The project is not tied to one agent runtime. According to the post, it is separated from any specific runtime so it can work wherever the skills do, and it supports OpenAI-compatible backends by default, including OpenAI, Together, Groq, Anthropic through compatibility layers, and local Llama servers.

For teams that want more control, the SDK also supports YAML configuration and programmatic use in CI pipelines or custom dashboards. The author says users can override config values with CLI flags, stream events as JSONL for downstream analysis, and plug in their own model backend by implementing a Provider interface.

The framework also supports tool-call assertions, which are intended for agents that use tools rather than only generating text. According to the project description, that lets evaluators check deterministic behaviors as well as the model’s final answer.

The minimum skill setup is simple: a folder with a SKILL.md file, plus optional evals/evals.json and supporting files. The post says SKILL.md includes YAML frontmatter with fields like name and description, while evals/evals.json defines prompts, expected output, files, and assertions.

If an eval includes expected_output but no assertions, the SDK promotes the expected output into a judge assertion automatically. That means users can write a minimal eval and still get pass/fail grading without extra setup.

The project says it implements the full agentskills.io specification, including SKILL.md validation, official artifact layout, and baseline comparison between with_skill and without_skill. It is available as an npm package, and the author points users to docs, examples, and a local preview for the documentation site.