arxiv:2602.12670

SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

Published on Feb 13

· Submitted by

Xiangyi Li on Feb 18

BenchFlow

Upvote

Authors:

Xiangyi Li ,

Xiaokun Chen ,

Yifeng He ,

Bingran You ,

Jiankai Sun ,

Qunhong Zeng ,

Xuandong Zhao ,

Roey Ben Chaim ,

Yizhuo He ,

Abstract

SkillsBench evaluates agent skills across 86 tasks and finds that curated skills improve performance significantly but inconsistently, while self-generated skills offer no benefit, indicating that models struggle to create useful procedural knowledge despite benefiting from curated versions.

AI-generated summary

Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers. Each task is evaluated under three conditions: no Skills, curated Skills, and self-generated Skills. We test 7 agent-model configurations over 7,308 trajectories. Curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain (+4.5pp for Software Engineering to +51.9pp for Healthcare) and 16 of 84 tasks show negative deltas. Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming. Focused Skills with 2--3 modules outperform comprehensive documentation, and smaller models with Skills can match larger models without them.

View arXiv page View PDF Project page GitHub 401 Add to collection

Community

xdotli

Paper author Paper submitter about 21 hours ago

GitHub: https://github.com/benchflow-ai/skillsbench
Website: https://skillsbench.ai/

xdotli

Paper author Paper submitter about 8 hours ago

We made the first benchmark that measures the efficacy of agent skills. We collected 86 tasks from 105 domain experts across 11 domains, every task is verifiable, human created and has verified Skills. SOTA model without skills score ~30% without skills.

We found a few interesting things:

Skills substitute for model scale — Haiku 4.5 with Skills (27.7%) beats Opus 4.5 without (22.0%). The right procedural knowledge can be worth more than a bigger model.
Skills' improvement has nothing to do with LLMs' internal knowledge. We have an ablation where no Skills provided for the agent, but the agent is prompted to generate relevant procedural knowledge before solving the task. This isolates the impact of LLMs' latent domain knowledge.

The result is:

Curated Skills: +16.2pp average improvement across all 7 agent configs
Self-generated Skills: -1.3pp: models can't write their own procedural knowledge pre-trajectory feedbacks. This is used to isolate the impact of LLMs' latent domain knowledge.