Agent Security Audit Benchmark

We built a small Express + LangChain agent with three deliberate vulnerabilities:

1. Hardcoded API key in config.ts
2. Overly broad file tool that could read .env and /etc/passwd
3. Indirect prompt injection via a user-controlled document fed into the system prompt

Each agent was asked: "Audit this agent for security issues. List concrete findings with file paths and remediation steps."

Criterion	Weight	Max points
Finds hardcoded secret	25%	25
Identifies unsafe tool scope	25%	25
Finds prompt injection vector	25%	25
Remediation advice is actionable	15%	15
Speed	10%	10

• Same repo, same prompt, no hints.
• Agents were allowed to read files and run static analysis commands.
• Findings were scored manually against the known vulnerability list.

• Claude Code performed best on reasoning through indirect attack paths. It traced user input into the system prompt and flagged the risk.
• Aider excelled at finding secrets and permission issues in code, but prompt injection is not its strength.
• Cursor was fast and found obvious issues, but its audit was shallow — it missed runtime semantics.
• OpenHands produced a long checklist with few specific citations. Volume does not equal accuracy.

• Three vulnerabilities is a small sample. A real audit covers many more attack surfaces.
• None of the agents replaced a human security reviewer. They accelerated the first pass.
• Results depend heavily on how the prompt frames "audit." A narrower prompt might have helped Cursor focus.

• Claude Code: reasoning-heavy audits where tracing data flow matters.
• Aider: code-centric reviews with strict diff control and secret scanning.
• Cursor: quick triage of obvious security smells.
• OpenHands: not recommended for security audits until it improves citation precision.