Language testing guide

Practical guide for contributors adding or extending a programming language in rgctl: where tests live, what new files to add, and what each suite must prove.

Not this doc: capability bar (Layers A–F) — see tier-1-language-support.md. Scaffold / crate layout — see that doc §3–4. PR workflow hub — contributor-checklist.md.

1. Mental model

Language quality is proven at four distances from the plugin:

Always start with unit + a small fixture. Do not open a Tier 1 PR with only “discover succeeds on one file.”

Tier-1 vs catalog: Tier-1 deep-analysis languages (CFG/PDG + *-ast-coverage.json + single-pass ScopeStack / KindDispatchTable / parse_with_pool) are a subset of the ~60 plugins in languages.toml. New Tier-1 PRs must use single-pass extract_all and keep the coverage JSON in sync (RGCTL_AST_COVERAGE_STRICT=1 fails on drift; RGCTL_AST_COVERAGE_TRACK=1 records dispatch hits in depth tests). Cross-language MRO (Overrides) and DI/route edges are part of the Tier-1 parity bar where the language has inheritance / frameworks.

2. Where to put tests (map)

crates/rgctl-lang-{id}/
  src/…                    # plugin + unit tests (#[cfg(test)]); single-pass extract_all
  {id}-ast-coverage.json   # grammar kind → extraction intent (build-gated)

tests/fixtures/{id}/
  langfeatures/…           # small sources for langfeatures / CFG probes
  …                        # optional taint / edge-case snippets

tests/
  {id}_langfeatures.rs     # GQL / structured-query gates on ecommerce or fixtures
  {id}_cfg_analysis.rs     # CFG/PDG after discover --with-cfg
  {id}_taint.rs            # source→sink (or declarative pack) gate
  dashboard_ecommerce_{id}.rs

rgctl-tests/ecommerce-{id}/
  src/…                    # multi-file sample app
  correctness/
    expected-facts.json    # graph_correctness harness (required for CI coverage)

Cargo.toml                 # [[test]] entries for each new integration binary
.github/workflows/ci.yml   # only if you add a named always-on QE step

Copy from a neighbor language (same family helps):

3. What to add for a new language

Minimum set depends on tier. Checkboxes are “add these artifacts,” not “paste empty stubs.”

Tier 3 (regex) — minimum

• Unit tests in rgctl-lang-{id} for pattern matches / non-matches
• Optional: one file under tests/fixtures/{id}/ + a tiny integration test if discover wiring is non-obvious
• No CFG / taint / dashboard / Gate B requirements

Tier 2 (generic tree-sitter) — minimum

• Plugin / config unit tests (extract_symbols on fixtures)
• {id}-ast-coverage.json + ast_coverage unit test (same drift rule as Tier 1)
• Optional fixture under tests/fixtures/{id}/
• CFG/taint/dashboard only if you claim those capabilities

Tier 1 (custom plugin) — required test matrix

Add all of the following before calling the language “fully supported.” Ship incrementally if needed, but the PR that claims Tier 1 must close the gaps.

Shared graph QE (no new binary required, but your fixtures must pass):

4. What to assert (by suite)

Plugin unit tests (rgctl-lang-{id})

Assert on extracted structures, not CLI:

• Function/method name, file, start_line (line may move; identity must not depend on line — use qualified_name without @L{n})
• At least one Calls edge between in-fixture symbols
• Inheritance / implements edges if the language has them
• Negative: do not invent calls into string literals / comments

{id}_langfeatures.rs

Typical pattern (see tests/ruby_langfeatures.rs, tests/go_langfeatures.rs):

1. Cold discover . -l {id} on rgctl-tests/ecommerce-{id} or tests/fixtures/{id}/langfeatures
2. rgctl -f json gql '…' or structured find / callers
3. Assert symbol presence, callee/caller sets, and language property when set

{id}_cfg_analysis.rs

• Discover with --with-cfg (and -l {id})
• Inspect CFG via CLI (inspect / dashboard index / analysis APIs used by sibling tests)
• Require branching (if) and a loop when the language has them
• Prefer exact block/edge counts on a frozen harness function

{id}_taint.rs

• Declarative pack under crates/rgctl-analysis/rules/taint/{id}.yaml (preferred over hard-coded Rust detectors)
• Fixture with a clear source and sink
• Assert a finding path exists (JSON schema + non-empty path)

expected-facts.json (graph_correctness)

Schema: rgctl-tests/correctness/SCHEMA.md. Policy: QE.md (required failures stay red).

After discover, graph_correctness also runs assert_graph_invariants (sorted unique nodes/edges, Calls endpoint legality, stable-key consistency, index bijection, digest, external stubs). If this fails, fix extraction or stub policy — do not weaken the checker for convenience.

Dashboard ecommerce

• Follow tests/dashboard_harness.rs + an existing dashboard_ecommerce_*.rs
• Release binary; assert discover metrics and CFG/taint index availability as siblings do

Stable keys (mandatory for Tier 1)

StableNodeKey facets are path + name + node_type + qualified_name — not start_line.

• Prepending blank/comment lines must not add/remove nodes or edges in diff_snapshots
• If your plugin puts @L{line} (or similar) in qualified_name, line-shift QE will fail — fix the plugin

Cold / Gate B

• Small fixtures never replace Gate B
• When ready: fetch corpus (./scripts/fetch-profile-repos.sh), record wall_secs / nodes from [profile] discover summary, add ignored gate + baseline (+10%) in tests/cold_profile_gates.rs
• Document corpus path and env override in AGENTS.md Gate B table

5. Wiring a new integration test binary

1. Add tests/{name}.rs.
2. Register in root Cargo.toml:

[[test]]
name = "{id}_langfeatures"
path = "tests/{id}_langfeatures.rs"

3. Run with:

cargo test --test {id}_langfeatures -- --nocapture

4. CI already runs cargo test --workspace --lib --bins --tests, so new [[test]] binaries are included. Add a named step in .github/workflows/ci.yml only for slow release gates (dashboard, scale) that should stay visible in the Actions UI.

6. Commands to run before a language PR

Replace {id} with your language id (ruby, kotlin, …).

# Format / lint (touched crates)
cargo fmt --all -- --check
cargo clippy -p rgctl-lang-{id} -p rgctl-languages -p rgctl-analysis -- -D warnings

# AST coverage drift (single-pass KindDispatchTable manifests)
cargo check -p rgctl-languages
# optional strict / runtime hit tracking:
# RGCTL_AST_COVERAGE_STRICT=1 cargo check -p rgctl-languages
# RGCTL_AST_COVERAGE_TRACK=1 cargo test -p rgctl-lang-{id} --test {id}_depth_extraction

# Plugin units
cargo test -p rgctl-lang-{id}

# Language integration binaries you added
cargo test --test {id}_langfeatures -- --nocapture
cargo test --test {id}_cfg_analysis -- --nocapture
cargo test --test {id}_taint -- --nocapture

# Graph correctness (includes your ecommerce-{id} once registered)
cargo test --test graph_correctness -- --nocapture

# Shared QE (stable keys + determinism)
cargo test --test stable_key_invariance_qe -- --nocapture
cargo test --test graph_determinism_qe -- --nocapture

# Dashboard (release)
cargo build --release --bin rgctl
cargo test --release --test dashboard_ecommerce_{id} -- --nocapture

# Full contributor workflow (when ready)
# see contributor-checklist.md §5

Paste the commands you ran in the PR description.

7. Checklist (copy into PR)

### Language testing ({id})

- [ ] Tier chosen (1 / 2 / 3) and honesty limits noted
- [ ] `rgctl-lang-{id}` unit tests green
- [ ] AST coverage JSON updated / registered
- [ ] `tests/{id}_langfeatures.rs` (Tier 1)
- [ ] `tests/{id}_cfg_analysis.rs` (Tier 1)
- [ ] `tests/{id}_taint.rs` + `rules/taint/{id}.yaml` (Tier 1)
- [ ] `rgctl-tests/ecommerce-{id}/` + `correctness/expected-facts.json`
- [ ] `PROJECTS` entry in `tests/graph_correctness.rs`
- [ ] `dashboard_ecommerce_{id}` (Tier 1)
- [ ] Listed in `stable_key_invariance_qe` `LANGS` (no line-coupled QN)
- [ ] Gate B corpus planned or baselined (Tier 1 scale)
- [ ] Docs: languages coverage page regenerates from JSON; checklist / this guide updated if you invent a new suite pattern

8. Related docs