RAG security testing: prove the assistant only knows what the user may see.
Retrieval-augmented generation connects a language model to your documents. That makes every permission in your document stores part of your AI security, and every document a possible instruction. This guide covers where RAG pipelines break, how to test each stage, and where permissions must be enforced to hold.
RAG Security Test Cases, by Attack Class
Six ways a RAG feature fails. Pick one to see what to plant, what to ask and what a secure assistant does.
RAG Permission Testing With Canary Documents
Each test document carries a unique canary string and a known permission. Ten questions, three identities. A test fails when a canary reaches someone who shouldn't see it, or when a planted instruction changes what the assistant does.
Illustrative run · synthetic documents and users · 20 s replay
Stale Permissions Example: Access Revoked, Chunks Still Indexed
The contractor lost access to the HR folder on Monday. The vector index synced permissions weekly. Until Sunday, the assistant kept answering from documents the source system already refused to open.
Choosing an AI red teaming tool →Illustrative finding · synthetic data
Where RAG Permissions Hold, and Where They Don't
Four places teams put document access control. Only the first two survive a determined user; the last two belong as extra layers at most.
Query-time check against the source
Before chunks reach the prompt, confirm the caller can access each source document now.
TEST · Revoke access, ask immediately.Index pre-filter on synced ACLs
Store allowed principals with each chunk and filter by the caller's identity at search time.
TEST · Measure lag between source change and index.Post-filter after retrieval
Retrieve broadly, then drop unauthorized chunks. Works, but top-k can fill with dropped results.
TEST · Check answer quality and citations for gaps and leaks.Instruction in the system prompt
Telling the model what the user may see. The content is already in the context and can be extracted.
TEST · Ask it to ignore the rule.Ratings describe the enforcement approach, not any vendor's product.
How to Test a RAG Application: A Repeatable Workflow
Six steps that make RAG testing rerunnable whenever a source, model, prompt or permission changes.
Canary data in multi-tenant testing →Inventory the sources
List every source the assistant can retrieve from, how it's chunked and how often permissions sync.
Mirror permissions in test data
Create test documents with the same permission patterns as production: users, groups, nested groups, shared links.
Seed canaries and poisoned docs
One canary per permission boundary, plus approved injection and exfiltration payloads in shared sources.
Ask as every role
Run the same questions, and paraphrases, as each identity. Compare retrieved chunks, not just answers.
Change permissions and re-ask
Revoke, delete and move documents, then test immediately to measure stale access.
Rerun on every change
Trigger the suite on source, sync, chunking, model, prompt or tool changes, and on a schedule.
RAG Validation, Per User, on Every Index Change
Hayrok seeds canary documents with known permissions, asks the shipped assistant as each user role, and checks what was retrieved and what was said. Poisoned-document scenarios come from an approved library, and every run is recorded as evidence.
Canary seeding
Plants canary documents per permission boundary in your staging sources.
Per-identity questioning
Asks the shipped assistant as each role and tenant, with paraphrase variants.
Retrieval-level evidence
Records which chunks were retrieved, not just what the model said.
Approved poisoning library
Injection and exfiltration documents come from a reviewed, versioned library.
Rerun on change
Reruns when sources, permissions, models or prompts change; Evidence Fabric keeps each run.
RAG Security Testing FAQ
Testing a retrieval-augmented generation feature end to end: whether each user can only retrieve documents they're allowed to see, whether planted instructions in documents change the assistant's behavior, and whether answers can leak data through links, images or citations.