We wrote a realistic buying scenario and handed the identical brief to two assistants: one grounded on the Vulgate corpus, one relying on the open web. Same evidence rules for both — cite real documentation or say so. This page is the unedited outcome.
Meridian Trust Systems — a fictional payments company. ~400 engineers, PCI-DSS scope, self-managed GitLab, a Java/Kotlin monorepo plus Go and TypeScript services, and a card-processing build enclave with no internet access. They’re consolidating application-security tooling.
Evaluate 10 named vendors against 12 requirements — 8 mandatory, 4 preferred. 120 cells, each labeled with an evidence state and backed by a documentation citation. Then shortlist the top three and justify every exclusion.
Vendor documentation only. Every non-silent cell needs a source URL and a date. No marketing pages, no memory, no guessing — if the docs don’t say it, the answer is silent, not yes.
Claude, restricted to the Vulgate corpus as its only source of vendor facts, returned the complete compliance matrix: every cell labeled, every non-silent claim carrying a documentation URL, a retrieval date, and the corpus file it came from. Qualifiers traveled with the cells.
The scoring goes deeper than yes/no. Every cell lands in one of five evidence states, and qualifiers travel with the claim — “supported, on the SaaS tier only” never flattens to “supported.” Documented negatives surface as conflicts instead of being skipped, and the shortlist rules are mechanical: a missing mandatory excludes, preferred requirements break ties. The full matrix is below.
“I can’t produce that matrix reliably without first searching the vendors’ documentation… That’s on the order of hundreds of documentation lookups.”
To its credit, it refused honestly — held to the no-fabrication rule and proposed evaluating one vendor at a time instead. But the evaluation you asked for never arrived. The lookups it balked at are exactly the work Vulgate does in advance.
Stale sources. Cited a 2021 vendor blog post as evidence — outside the docs-only rule, and five product versions old.
False silence. Reported one vendor’s docs silent on OpenAPI 3.1 support. The documentation exists; it just wasn’t found.
Marketing as evidence. Sourced a performance claim from a vendor’s marketing pages rather than its documentation.
Eight mandatory requirements across the candidate field. Hover any column header for the full requirement; every non-silent cell in the source output carries a documentation URL and retrieval date.
| Candidate | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | Verdict |
|---|---|---|---|---|---|---|---|---|---|
| Endor Labs | E | E | E | E | S | E | E | E | Shortlist #1 — 7/8 mandatory, 4/4 preferred |
| Semgrep | E | E | E | E | S | E | E | E | Shortlist #2 — 7/8 mandatory, 3/4 preferred |
| Aikido Security | E | I | E | E | S | E | E | Q | Shortlist #3 — 6/8 mandatory, 3/4 preferred |
| Snyk | E | S | E | Q | S | Q | E | Q | Excluded — M2, M5 |
| Socket | E | E | E | S | S | E | Q | E | Excluded — M4, M5 |
| Veracode | E | E | E | C | Q | Q | Q | E | Excluded — M4 conflict, M5 |
| Contrast Security | C | S | E | Q | E | Q | S | Q | Excluded — M1 conflict, M2, M7 |
| Mend | E | S | Q | E | Q | Q | Q | Q | Excluded — M2, M5 |
| OX Security | Q | Q | E | Q | Q | Q | Q | E | Excluded — thin evidence across mandatories |
Preferred requirements P1–P4 — license policy, container scanning, malicious-package detection, IaC scanning — scored separately as tiebreakers.
Only gap is M5: scan compute can run on-prem, but the tenant and vulnerability database stay cloud-hosted — no offline DB updates documented.
Only gap is M5: the CLI engine runs locally, but Supply Chain reachability and policies require the SaaS platform. No container scanning (P2).
Gaps at M2 — Go reachability implied by architecture docs, never stated — and M5. License policy support is qualified.
| Cell | Source URL | Corpus file | Retrieved |
|---|---|---|---|
| Endor Labs — M1 | docs.endorlabs.com/scan/sca/reachability-analysis/ | corpus/endor-labs/…/reachability-analysis.md | 2026-08-01 |
| Semgrep — M3 | docs.semgrep.dev/supported-languages.md | corpus/semgrep/supported-languages.md | 2026-08-01 |
| Semgrep — M7 | docs.semgrep.dev/extensions/pre-commit.md | corpus/semgrep/extensions-pre-commit.md | 2026-08-01 |
Two cells came back as conflicts — the docs answered no. Contrast’s reachability requires a runtime agent, which the scenario forbids. That’s why the evidence contract now has a fifth state.
Air-gapped deployment (M5) eliminated nearly the whole field — the kind of finding that usually surfaces three demos into a sales cycle, surfaced before the first call.
The corpus’s failures are visible and fixable — a disclosed gap. The open web’s failures needed an auditor to catch: stale blogs, marketing pages, missed docs.
Methodology: run 2026-08-02. Both arms received the identical scenario, candidate list, and evidence rules. Vulgate arm: Claude (Opus) restricted to the Vulgate documentation corpus as its sole source of vendor facts. Comparison arm: a leading general assistant with web browsing. Meridian Trust Systems is a fabricated evaluation scenario; the vendors, requirements, and citations are real. Evidence states reflect vendor documentation as collected on 2026-08-01 and may change as documentation changes.