Agent Plugin Eval

Overview

Audits repositories containing portable Agent Plugins against the official Agent Plugins specification. Produces a 0–100 scorecard with evidence cited by file:line or JSON pointer, a conformance gate (PASS/PARTIAL/FAIL), and the list of blockers standing between the plugin and a release.

It evaluates portable conformance, not one client’s features. A client-native manifest — .codex-plugin/plugin.json, .claude/settings.json, .cursor/mcp.json — does not replace the required root plugin.json. A plugin that is excellent for one client can still fail the portable standard, and the skill reports both facts without averaging them away.

When to Use

• Audit a plugin repository before publishing
• Check plugin.json or mcp.json conformance
• Validate the Agent Skills bundled in the package
• Assess MCP server configuration for portability and safety
• Identify release blockers
• Compare two agent plugins side by side

Installation

npx skills add https://github.com/fabricioctelles/skills -s agent-plugin-eval

Safety boundary

The audit is static only. The skill does not run plugin code, hooks, install scripts, package managers, MCP servers, or networked tests without explicit user authorization. Suspected credential values are redacted and labeled suspected — a heuristic hit lowers the security score and demands remediation review, but on its own it does not become a confirmed-credential FAIL gate.

18 criteria across 4 axes

Axis	Criteria	Focus
1 — Portable conformance	4	Exactly one root plugin.json, canonical schema, closed metadata model, fixed discovery locations, version coherence
2 — Components and integration	6	Agent Skills conformance and quality, mcp.json validity, stdio server portability, remote transport, extension isolation
3 — Safety and resilience	4	Path containment after symlink resolution, least secret exposure, component independence, install without hidden side effects
4 — Product quality	4	Cohesion, documentation, practical utility, release readiness

Criteria carry 1x, 2x, or 3x weight, and the ones that do not apply are recorded as NA rather than penalized. Scoring is deterministic, via scripts/score.py:

python3 scripts/score.py --gate partial 1:90:3 2:80:3 3:NA:2

Conformance gates

Gate	When	Score cap
PASS	No normative violation found	none
PARTIAL	Non-fatal manifest deviation, or an invalid/skipped component	59
FAIL	Fatal manifest or package-root failure, root escape, or confirmed embedded credential	39

The uncapped score stays visible next to the final one, so design quality is never confused with a release-blocking conformance failure.

How It Works

1. resolve the plugin root (local path, or shallow clone of a Git URL)
2. load the governing rules (1.0.0 snapshot, or the published spec)
3. inventory every package path  <-- dotfiles, symlinks, Git-ignored files
4. deterministic scan (inspect_plugin.py) as leads, not verdict
5. review every skill and every mcpServers entry
6. score with score.py and apply the gate

Every scan finding is confirmed in the source before it reaches the report. A client happening to accept a construct never weakens a normative finding.

Gotchas it handles

• The v1 portable core is exactly Agent Skills and MCP servers. Hooks, commands, agents, apps, and marketplaces are client-specific unless placed in a valid extension namespace
• A missing skills/ or mcp.json is not an error. A present path of the wrong filesystem kind is an invalid component
• One invalid skill must not be reported as if every independent component were invalid
• ${PLUGIN_ROOT} and ${PLUGIN_DATA} expand only in MCP args, env values, and cwd — never in command, URLs, or headers

Sources

• Agent Plugins — official specification
• Agent Skills — specification

📄 Full documentation on GitHub