CC research-observability
Add file-based observability, tracing, and run analysis to LLM/agent experiments — per-turn transcripts, tool calls and results, reasoning traces, token/cost accounting, per-episode artifacts, plus terminal viewers and a run-comparison report. Platform-agnostic: one small adapter covers OpenAI, Gemini/Vertex, Anthropic, LiteLLM, Tinker, vLLM, or any custom inference stack, and the same tooling then works unchanged. Use this skill whenever the user wants observability, tracing, logging, or run tracking for model experiments; asks why an agent failed a task or wants to replay/debug/diff an episode; wants to compare models, arms, or ablations; wants to instrument a new folder, repo, or platform ("add observability here", "do the same for Tinker/Gemini/OpenAI"); wants to import or tie in existing experiment results so old runs show up alongside new ones; or wants eval runner scripts where many models share one protocol. Reach for this even when they only say "log what the model is doing", "I can't tell what happened in this run", or "set up traces for this experiment".
The skill promises file-based logging for LLM experiments: per-turn transcripts, tool calls, reasoning traces, token and cost accounting, episode artifacts. Claims single-adapter support for OpenAI, Gemini, Anthropic, LiteLLM, and custom stacks.
Holds 14 files and 7 scripts, but quality scored 55/100 with process at 51. No critical findings, though linter flagged an oversized description. No model runs, no sandbox tests. Positioned as a debugging and model-comparison tool, yet actual functionality remains unvalidated.
Install if you need a logging foundation. Run it on your stack first—the scores suggest rough edges ahead.
Add file-based observability, tracing, and run analysis to LLM/agent experiments — per-turn transcripts, tool calls and results, reasoning traces, token/cost…
As a process C 51/100 · Has gaps — weak spots: steps, result and completion, inputs and preconditions
How to improve
- Shorten the description to 1024 characters.
- Your own cases (evals/evals.json, 4–6 real requests with expected answers): the full check would then run those instead of a model-drafted suite.
- A spec.yaml with trigger phrases and assertions — a behaviour contract for CI; `skilltest init` writes a template.
Guard findings · 0
✓ No critical or high findings
Files scanned: 14. Evidence is masked. Grey chips explain why severity was lowered.
Against the Agent Skills spec
- error
description-longdescription is 1081 chars, limit 1024 - warning
frontmatter-yamlSKILL.md: the frontmatter is not valid YAML (YAML parse error: Nested mappings are not allowed in compact mappings at line 2, column 14: description: Add file-based observability, tracing, and run analysis to LLM/age… ^ ); fields were read line by line. The usual cause is a colon inside an unquoted value
Process rating: all ten parameters 51/100
- 0Steps. Prose only: no discrete steps
- 0Result and completion. Does not say what the result is
- 0Inputs and preconditions. Does not say what the process needs to start
- 50Failures and branches. 0 branches, has a failure section
- 70When it triggers. States when to use, but not when not to
- 100Tools and files. No external tools needed
- 100Consistency. Name and required fields are in place
- 100Execution cost. Instruction body is 1635 tokens
- 100Running it twice. No mutating operations
- 100Progress reporting. Reports progress
Everything here is measured from the skill text rather than judged by a model, so the numbers are checkable. A parameter weighs more when it is a more common reason for the process to stall.
Quality signals
- +4Description does not say when NOT to use the skill (false activations)
- +3Description length 1081: 120–800 characters recommended
- +3No numbered steps or checklist
- +3Output format is not stated: the model decides each time
- +1No license
- +2Single-language instructions
- +5Description quotes 5 example trigger phrases
- +4Structure: 7 headings
- +4Has examples (2 code blocks)
- +4Reference files are cited in the instructions (6 of 6)
- +3All 6 scripts are documented
Quality base 70; lint remarks subtract, signals add up to 100. Result: 55.