agentaudit-scanner is an open-source CLI tool that scans AI agent skills, rule files, and MCP configurations for security risks like prompt injections and hidden instructions. Below are 39 llm eval & observability apps with similar functionality to agentaudit-scanner, matched by what each product actually does — not ranked or scored. Explore each to find the closest fit for your use case.
agent-audit-kit is an open-source CLI tool that scans MCP-connected AI agent pipelines for security vulnerabilities. It helps AI security engineers ensure the safety and compliance of agent-based workflows.
agentscanner is an open-source CLI tool that performs static security analysis on agentic AI configuration files, including settings, permissions, hooks, and MCP servers. It helps developers and security teams identify vulnerabilities in AI agent stacks before deployment.
agentproof-scan is an open-source CLI security scanner designed to probe AI-agent HTTP endpoints for secret and prompt leakage. It integrates with CI pipelines using exit codes to automate security checks, helping developers and security engineers ensure their AI agents are not exposing sensitive information.
skill-auditor is an open-source command-line tool that scans AI Agent skills for security issues before installation. It acts as an install gate, helping developers and maintainers ensure that only safe and validated skills are added to their AI agents. Ideal for those building or managing AI agent frameworks.
agentaudit-eval is an open-source evaluation framework for multi-agent AI workflows. It provides tools for handoff quality scoring, failure attribution, loop detection, and cost guardrails, helping developers monitor and improve the reliability and efficiency of complex AI agent systems.
agent-skill-scanner is an open-source CLI tool that scans agent skill files for security vulnerabilities, including prompt injection, capability escalation, and data exfiltration. It applies 22 security rules to help AI developers and security researchers identify and mitigate risks in LLM agent skills. The tool is MIT licensed and suitable for anyone building or auditing AI agent systems.
agentsec-cli is an open-source CLI tool that performs static security analysis on AI coding agents and Model Context Protocol (MCP) configurations. It helps developers and security teams identify vulnerabilities and misconfigurations in agent-based AI systems before deployment.
agentsentinel-cli is an open-source command-line tool that provides security scanning, static analysis, supply chain auditing, and multi-agent trust analysis for AI agents and MCP servers. It helps developers and security engineers identify vulnerabilities and ensure compliance with security best practices in agentic systems.
agent-probe-ai is an open-source CLI tool for adversarial resilience testing of AI agents. It evaluates agent robustness against attacks like memory poisoning and tool misuse, helping AI researchers and security engineers identify vulnerabilities.
AgentAz Compliance Scanner is a web-based tool that analyzes AI agent system prompts or agentaz.json files for compliance with Microsoft's agent-governance guidance. It provides deterministic, automated checks, risk analysis, and actionable fixes for AI developers and teams.
cisco-ai-skill-scanner is an open-source CLI tool for scanning Agent Skills packages to detect prompt injection, data exfiltration, and malicious code. It is designed for AI security engineers and developers working with LLM agents and skills.
agchk is an open-source CLI tool that audits the architecture and health of AI agent systems and LLM-integrated projects. It helps developers and auditors ensure the security and robustness of their AI deployments.
ai-agentic-mcpscan is an open-source command-line tool that provides local-first, offline-by-default security posture scanning for Model Context Protocol (MCP) and local-agent environments. It is intended for security engineers and teams managing agent-based infrastructures.
ai-skill-audit is an open-source CLI tool that audits AI skill and role files for quality and trust, helping developers catch bad prompts and configuration issues before deploying them to agents. It is aimed at AI developers and agent builders.
Score any URL for AI-agent readiness — llms.txt, JSON-LD, AI-bot robots.txt, canonical, MCP, meta, sitemap.
aig-skill-scan is an open-source CLI tool that leverages LLMs to perform multi-stage code audits and vulnerability reviews for agent skills. It is designed for security engineers and developers seeking automated, in-depth security analysis of AI agent codebases.
Skill Behavior Mapping for AI agent skill auditing.
AI-native quality gate for agent-generated code — scan, audit, auto-fix, trend
ai-agentguard is an open-source CLI tool that monitors AI coding agents for security threats such as remote code execution, MCP poisoning, and API key theft. It is designed for developers working with AI agent frameworks to enhance security.
scopesafe-aibom is an open-source CLI tool that scans codebases to discover and risk-assess AI agents, models, and MCP servers. It helps developers and security engineers ensure compliance and security in AI-powered applications by generating AI Bill of Materials (AI-BOM) reports.
agentsec-eval is an open-source CLI framework for evaluating the security of AI agents. It provides adversarial test runners, server-side audits, and scoring mechanisms to help researchers and developers identify vulnerabilities such as prompt injection and improve agent robustness.
Open Agent Readiness Scanner is a free tool that audits websites, APIs, and MCP endpoints across multiple categories to assess their compatibility with AI agents. It provides actionable feedback and a public leaderboard for developers and site owners.
agentsift is an open-source CLI tool that scans AI agent plugins, skills, and MCP packages for security vulnerabilities and supply chain risks. It helps AI developers maintain secure and reliable agent ecosystems.
AI Agent Tool for Penetration Testing and Network Forensics
dscan-security is an open-source command-line security suite for AI agents. It enables developers and security engineers to intercept, scan, and test agents for vulnerabilities, including prompt injection and other security risks.
Offline AI agent bill of materials and attack surface scanner
SkillScan was a free, local, and private scanner designed to detect security issues in AI agent skill files. The tool operated without requiring a GPU, incurred no per-scan fees, and did not require users to send their files to a company, emphasizing privacy and local analysis. Its approach involved static analysis and the use of pattern rules or a lightweight classifier to identify potentially malicious behaviors or instructions within skill files. The scanner was developed as an open-source project, with its code remaining publicly available even after its retirement. SkillScan's methodology centered on identifying problematic patterns, but its creator found that such tools, including SkillScan itself, often missed a significant portion of novel attacks or over-blocked to compensate. The tool was benchmarked alongside other similar scanners, and results indicated that static or rule-based approaches detected only a fraction of sophisticated threats, particularly those involving intent or context that simple pattern matching could not discern. SkillScan's development and subsequent benchmarking were aimed at providing detection capabilities that users could fully control—local, private, and open-source—without reliance on external services or cloud-based models. However, the project's findings highlighted the limitations of static and rule-based detection in this domain, especially when compared to more advanced reasoning models. The scanner has since been retired, but its code remains accessible for reference.
autosecscan is an open-source CLI tool that leverages AI to scan networks and codebases for vulnerabilities. It provides automated triage, fix suggestions, and generates structured security reports, making it ideal for security engineers and DevSecOps teams.
agent-audit-logger is an open-source Python middleware that wraps agent frameworks to log every tool call, providing SOC 2 audit trails for small teams and developers needing compliance and traceability.
agent-diagnostics is an open-source framework for analyzing and annotating the behavior and reliability of coding agents. It provides tools for benchmarking, taxonomy creation, and structured evaluation, helping researchers and developers understand agent performance.
compliance-agent is an open-source CLI tool that scans AI projects for compliance with the EU AI Act. It automates the process of checking regulatory requirements and generates compliance reports, helping AI developers ensure their projects meet legal standards.
reaper-scanner is an open-source command-line tool designed to scan AI agents for security vulnerabilities. It helps AI security engineers identify and mitigate risks in agent-based systems and pipelines.
agent-steward is an open-source CLI tool that helps organizations manage and audit the quality and spending of their AI agent fleets. It provides verification, cost optimization, and quality gating features for AI operations teams.
AI model security scanner — scan any HuggingFace model for vulnerabilities, CVEs, misconfigurations, prompt injection, and bias.
argus-panoptes is an open-source CLI agent that uses AI to perform automated security audits on code repositories. It reads code, deploys the app, and simulates attacks to identify vulnerabilities, helping security engineers streamline code review and pentesting.
agent-id-cli is an open-source command-line interface for managing identities of AI agents. It allows developers to create, update, and organize agent identities efficiently. Suitable for AI developers and teams working with multiple agents.
agentlint is an open-source CLI tool that provides real-time quality guardrails and linting for AI coding agents. It helps developers maintain code quality and safety by integrating with agentic coding workflows and enforcing best practices.
An AI agent interrogation framework for identifying attack surface.
abom-cli is an open-source command-line tool that scans, signs, and verifies the components of AI agents, providing a Bill of Materials for transparency and security. It is designed for AI developers concerned with supply chain integrity and provenance. Licensed under Apache-2.0.
