SKILLEMALL.ai
GitHub ↗ RU Sign in

CB evaluation

Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "run nel". Handles deployment, config generation, and evaluation execution. Always load when validating completed runs or reporting evaluation scores, including standalone launching-evals analysis. Not for quantizing models (use ptq), standalone serving (use deployment), or comparing completed baseline-vs-quantized results (use compare-results).

NVIDIA/Model-Optimizer Agent Skills author: NVIDIA Apache-2.0 34 files · 4 scripts body ≈ 1 920 tokens Open the sourcegithub.com↗ analyzed 4 h ago

Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL).

As a process B 71/100 · Nearly there — weak spots: result and completion, running it twice, progress reporting

AnalyzerInfrastructuretype and topics are labelled automatically from the skill text
Runs in: Claude / Claude Code ChatGPT / Codex Gemini CLI Cursor GitHub Copilot / VS Code Qwen Code Kimi Code CLI Z.ai (GLM) DeepSeek (Deep Code) Mistral Vibe LangChain (deepagents) OpenClaw Goose OpenCode
JSON
Technical rating
C
88/100
safety, quality, tests
Safety 60%
80
Quality 40%
100
Run on models
none yet
Process rating
B
71/100
Nearly there
Result and completion w 14
0
Progress reporting w 2
0
Running it twice w 4
30
the three weakest of ten parameters · all ten

How to improve

    For the model run — optional
    • A spec.yaml with trigger phrases and assertions — a behaviour contract for CI; `skilltest init` writes a template.

    Guard findings · 20

    ✓ No critical or high findings

    Medium and low: 20
    • low Exfiltration read-dotenv recipes/examples/example_eval_next.yaml:9
      Reads a .env file (code comment; test fixture / example file)
      #   set -a && source .env && set +a    # HF_TOKEN, AWS_*, NEL_NEXT_EVAL_IMAGE, HARBOR_*_ECR_REPOSITORY (from modelopttools:eval-config)
      commentfixture
    • low Secrets in code secret-high-entropy-token recipes/examples/example_eval.yaml:78
      High-entropy token-like string (may be an id, hash or a credential) (test fixture / example file)
      # a card-documented toggle — e.g. NVFP4 MoE (NVID…FP4)
      fixture
    • low Dangerous commands cmd-pipe-to-shell-known-host recipes/examples/gym/example_mrcr.yaml:134
      Pipe-to-shell installer from a well-known host (still executes remote code) (test fixture / example file)
      command -v uv >/dev/null 2>&1 || { curl -LsSf https://astral.sh/uv/install.sh | sh; export PATH="/root/.local/bin:$PATH"; }
      fixture
    • low Dangerous commands cmd-background-process recipes/examples/gym/example_mrcr.yaml:265
      Starts a background / autostarted process (test fixture / example file)
      setsid --wait bash /tmp/gym_run.sh &
      fixture
    • low Secrets in code secret-high-entropy-token recipes/tasks/aa/gpqa_diamond.md:29
      High-entropy token-like string (may be an id, hash or a credential) (quoted — discussed, not commanded)
      Result (0-100): `gpqa…ect`
      quoted
    • low Secrets in code secret-high-entropy-token recipes/tasks/aa/ifbench.md:25
      High-entropy token-like string (may be an id, hash or a credential) (quoted — discussed, not commanded)
      Result (0-100): `ifbe…acy`
      quoted
    • low Secrets in code secret-high-entropy-token recipes/tasks/aa/lcr.md:66
      High-entropy token-like string (may be an id, hash or a credential) (quoted — discussed, not commanded)
      Result (0-100): `aalc…ect`
      quoted
    • low Secrets in code secret-high-entropy-token recipes/tasks/aa/omniscience.md:42
      High-entropy token-like string (may be an id, hash or a credential) (quoted — discussed, not commanded)
      **Primary result — Omniscience Index** (-100 to 100): `omni…dex`. This is AA's headline metric: accuracy net of hallucinations, rewarding abstention over guessin
      quoted
    • low Secrets in code secret-high-entropy-token recipes/tasks/aa/omniscience.md:46
      High-entropy token-like string (may be an id, hash or a credential) (quoted — discussed, not commanded)
      - **Accuracy** (0-100): `omni…ect` — % of questions answered correctly.
      quoted
    • low Secrets in code secret-high-entropy-token recipes/tasks/aime_2025.md:42
      High-entropy token-like string (may be an id, hash or a credential) (quoted — discussed, not commanded)
      Result (0-1): `AIME…f_N`
      quoted
    • low Secrets in code secret-high-entropy-token recipes/tasks/livecodebench.md:39
      High-entropy token-like string (may be an id, hash or a credential) (quoted — discussed, not commanded)
      Result (0-100): `live…acy`
      quoted
    • low Dangerous commands cmd-background-process references/gym.md:87
      Starts a background / autostarted process (quoted — discussed, not commanded)
      via a **quoted heredoc** and launched under `setsid`, so the whole server/Ray
      quoted
    • low Dangerous commands cmd-background-process references/gym.md:174
      Starts a background / autostarted process
      the setsid + process-group reap in the task `command:` prevents.
    • low Secrets in code secret-high-entropy-token references/launcher-workflow.md:129
      High-entropy token-like string (may be an id, hash or a credential)
      Some models need extra vLLM backend env vars (model-card research) — e.g. `VLLM…END=marlin` (Nemotron Super), or `VLLM…FP4=1` + `VLLM_FLASHINFER_MOE_BACKEND=throughpu
    • low Exfiltration read-dotenv references/launcher-workflow.md:419
      Reads a .env file
      set -a && source .env && set +a
    • low Exfiltration read-dotenv references/nel-next.md:58
      Reads a .env file (detector / deny-list definition)
      - `set -a && source .env && set +a` before running so `${VAR}` resolves.
      detector
    • low Exfiltration read-dotenv references/nel-next.md:228
      Reads a .env file
      set -a && source .env && set +a; NEL="$SKILL_DIR/scripts/nel-next.sh"
    • low Exfiltration read-dotenv scripts/nel-next.sh:26
      Reads a .env file (code comment)
      # Usage (source .env FIRST so the config's ${VAR}s resolve; this never reads secrets):
      comment
    • low Exfiltration read-dotenv scripts/nel-next.sh:33
      Reads a .env file (detector / deny-list definition; code comment)
      #     emit_traces=false to avoid the per-sample hang). Run after `source .env`.
      detectorcomment
    • low Dangerous commands cmd-pipe-to-shell-known-host scripts/nel-next.sh:132
      Pipe-to-shell installer from a well-known host (still executes remote code) (string literal in code, not executed)
      command -v uvx >/dev/null 2>&1 || { echo "ERROR: 'uvx' not found (curl -LsSf https://astral.sh/uv/install.sh | sh)" >&2; exit 1; }
      code literal

    Files scanned: 31. Evidence is masked. Grey chips explain why severity was lowered.

    Against the Agent Skills spec

    ✓ No remarks against the Agent Skills spec

    Process rating: all ten parameters 71/100

    • 0Result and completion. Does not say what the result is
    • 0Progress reporting. Says nothing while it works
    • 30Running it twice. 8 mutating operations with no state check
    • 60Tools and files. Uses tools (bash, node) that frontmatter does not declare
    • 70Inputs and preconditions. Inputs and preconditions are listed
    • 100Steps. 11 steps
    • 100When it triggers. States when to use and when not to
    • 100Failures and branches. 3 branches, has a failure section
    • 100Consistency. Name and required fields are in place
    • 100Execution cost. Instruction body is 1920 tokens
    • medium 14 test cases, all positive: not one "should refuse" or "should ask first"
    • low No test case covers injection arriving through data

    Everything here is measured from the skill text rather than judged by a model, so the numbers are checkable. A parameter weighs more when it is a more common reason for the process to stall.

    Quality signals

    • +3Output format is not stated: the model decides each time
    • +2Single-language instructions
    • +5Description quotes 5 example trigger phrases
    • +4Description says when NOT to use the skill
    • +3Description length 528: enough signal without eating the budget
    • +4Structure: 16 headings
    • +3Step-by-step instructions: 11 items
    • +4Has examples (1 code blocks)
    • +4Reference files are cited in the instructions (6 of 11)
    • +3All 2 scripts are documented
    • +1License stated

    Quality base 70; lint remarks subtract, signals add up to 100. Result: 100.