Local-first
Raw evidence stays on the user's machine unless sharing is explicit.
Aggregator Integrity Playbook
Roadside audit is not route audit.
TokenAuditor turns AI API aggregator, gateway, fallback, model identity, and degradation claims into redacted, repeatable evidence memos.
Transparent to users. Fair to suppliers. We only stand with evidence.
Core Idea
A local evidence layer for infrastructure claims.
TokenAuditor is not a fraud detector or public blacklist. It helps operators convert unverifiable route claims into bounded evidence that users and suppliers can inspect.
Secret-blind
No API keys, raw private prompts, raw responses, or full production logs.
Metadata-first
Start with route, model, fallback, latency, usage, retry, fingerprint, and schema signals.
Evidence before accusation
Findings carry scope, sample size, time window, limitations, and confidence.
Audit Boundary
Two audit modes, two different outputs.
Roadside audit
Screens a local operation before it runs: a tool call, shell command, package install, outbound copy, credential-adjacent action, or route-mediated agent action.
- Output
- allow / warn / require_approval / block
- Limit
- May trigger route audit; cannot prove route integrity.
Route audit
Evaluates a model route over evidence windows: claimed model, observed model, fallback disclosure, baseline drift, quality, latency, token profile, and tool schema behavior.
- Output
- Watch / Sample / Probe
- Limit
- Still an evidence memo, not a public misconduct verdict.
Evidence Tiers
Start with metadata. Escalate only when evidence warrants it.
Metadata
Route, claimed model, returned model, fallback, latency, usage, retry, request id, fingerprint.
Paired eval
Direct provider versus aggregator route on the same task set, rubric, and time window.
Behavioral fingerprint
Response shape, tool schema behavior, output distribution, and deterministic prompt signatures.
Formal audit
Benchmark, logprob, rank-based, or black-box equality tests for serious probes.
Attestation
TEE, provider-supplied verifiable proof, or partner attestation as a longer-term path.
Evidence Memo
The final artifact is reviewable, not theatrical.
A useful memo separates known facts, observed signals, missing evidence, supplier fairness notes, and the next bounded action.
Conclusion: Watch / Sample / Probe
Audit Mode: roadside_check / route_integrity_audit
Evidence Tier: T0 / T1 / T2 / T3 / T4
Identity State: Consistent / Needs Sample / Likely Substitution Signal
Degradation State: Stable / Possible Degradation / Material Degradation
Policy Decision: allow / warn / require_approval / block / n/a
Confidence: low / medium / high
Known Facts
- Route:
- Claimed model:
- Baseline:
- Current window:
- Sample size:
Supplier Fairness Note
-
Recommended Next Actions
-