BC ai-system-testing
Test AI/LLM features that ship in your product. Covers prompt regression testing, response quality evaluation, tool-call validation, hallucination and RAG grounding checks, nondeterministic-output strategies, red-team/safety scans, eval frameworks, and agent-as-target injection (indirect injection via tool output / RAG / scan reports, self-propagating payloads, data exfiltration via an agent) plus a bundled detector for untrusted content. Use when: "test our LLM feature," "prompt regression test," "eval framework," "hallucination test," "RAG grounding," "nondeterministic output," "AI feature testing," "red-team our chatbot," "indirect prompt injection," "agent reading untrusted tool output," "production AI quality." Not for: using AI to generate your own test code — use ai-test-generation. Not for: classifying CI failures with AI — use ai-bug-triage. Not for: EU AI Act / GDPR conformity of an AI feature — use compliance-testing. Not for: canary/flag rollout of an AI feature — use testing-in-production. Related: ai-test-generation, ai-qa-review, api-testing, compliance-testing, security-testing, risk-based-testing, test-data-management.
Test AI/LLM features that ship in your product.
As a process C 64/100 · Has gaps — weak spots: inputs and preconditions, running it twice, progress reporting
How to improve
- Shorten the description to 1024 characters.
- The SKILL.md body is over 5,000 tokens: move reference detail into references/ and load it when needed.
- Your own cases (evals/evals.json, 4–6 real requests with expected answers): the full check would then run those instead of a model-drafted suite.
- A spec.yaml with trigger phrases and assertions — a behaviour contract for CI; `skilltest init` writes a template.
Guard findings · 7
✓ No critical or high findings
Medium and low: 7
-
low Instruction override
en-ignore-previousreferences/injection-detector.md:13Instruction-override phrase ("ignore previous instructions") (documentation table row; documentation of a security skill)| `instruction-override` | high | "Ignore all previous instructions / disregard the system prompt / forget everything" — hijack the agent away from its task. |
tablesecurity skill -
low Risky intent
intent-offensive-securitySKILL.md:99Offensive-security / dual-use content (legitimate for authorised testing; review intended use)| **PyRIT** | Microsoft AI Red Team's orchestration framework | Orchestrated multi-turn attacks; complements Garak. https://github.com/Azure/PyRIT |
A further 5 matches are quotations in this security skill's documentation and are not counted as findings.
Files scanned: 7. Evidence is masked. Grey chips explain why severity was lowered.
Against the Agent Skills spec
- error
description-longdescription is 1153 chars, limit 1024 - warning
body-longSKILL.md body ≈ 7805 tokens (recommended < 5000); move details to references/
Process rating: all ten parameters 64/100
- 0Inputs and preconditions. Does not say what the process needs to start
- 0Progress reporting. Says nothing while it works
- 30Running it twice. 7 mutating operations with no state check
- 50When it triggers. No condition that starts the skill
- 50Failures and branches. 0 branches, has a failure section
- 60Tools and files. Uses tools (bash, web, python) that frontmatter does not declare
- 70Execution cost. Instruction body is 7805 tokens
- 100Steps. 68 steps
- 100Result and completion. Output format and completion criterion are stated
- 100Consistency. Name and required fields are in place
- medium Safety rules and hard prohibitions inside a skill: they belong in the system prompt, here they protect nothing
- low 16 top-level sections: this looks like several domains in one skill
Everything here is measured from the skill text rather than judged by a model, so the numbers are checkable. A parameter weighs more when it is a more common reason for the process to stall.
Quality signals
- +3Description length 1153: 120–800 characters recommended
- +2Single-language instructions
- +5Description quotes 11 example trigger phrases
- +4Description says when NOT to use the skill
- +4Structure: 46 headings
- +3Step-by-step instructions: 68 items
- +3Output format is stated explicitly
- +4Has examples (4 code blocks)
- +4Reference files are cited in the instructions (5 of 5)
- +3All 1 scripts are documented
- +1License stated
Quality base 70; lint remarks subtract, signals add up to 100. Result: 66.