Replayable proof is evidence of a security finding complete enough that anyone can run it again and get the same result. Because AI agents can report actions as successful when they were not, a finding should count only when its evidence was captured and a replay confirms it. A replay package holds the exact requests, responses, timing, scope and environment needed to check it.
When the agent says it worked, did it?
That last question is not hypothetical. Independent research from AppSecSanta found that AI agents sometimes report a command as successful when it was not. A language model is built to produce a plausible next step. Sometimes the most plausible next step is "that worked".
The cost is not one wasted afternoon. Once an engineer chases a finding that is not real, every finding becomes suspect. The team goes back to retesting by hand. The speed that made autonomous testing worth buying disappears.
The answer is not to stop using AI agents. It is to change what counts as a finding.
A finding you cannot replay is a claim, not proof.
Claim, finding, proof: three words people mix up
Most reports use these words as if they meant the same thing. They do not.
| Term | What it means | Can you act on it? |
|---|---|---|
| CLAIM | A tool or person says something happened | Not yet |
| FINDING | A claim backed by captured evidence of what was sent and received | With care |
| PROOF | A finding that was replayed and gave the same result | Yes |
Replayable proof is evidence of a finding complete enough that someone else can run it again, in the same scope, and get the same result. Two checks turn a claim into proof. Try them.
Captured evidence, and a replay gave the same result. Act on it.
A claim that fails either check is not thrown away. It is logged as a claim, so a person can look at it. It is just not reported as a finding.
What a replay package must contain
A replay package is everything someone needs to check a finding without asking the tool, or the agent, what it meant. If any part is missing, the finding cannot be replayed.
Evidence must be captured from the traffic as it happens, not written by the agent afterward. An agent's summary of what it did is a claim.
How to replay a finding, step by step
Separate proof from claims, every week.
The Hayrok Brief covers exploited vulnerabilities, AI security and regulation for security leaders, with what to validate each week.
Replay after a fix: proof of remediation
Replayable proof pays off twice. The same package that proves a finding also proves the fix. When an engineer ships a fix, replay the original package against the patched system.
This turns retesting from a new engagement into a routine check. It also lets you answer a better question than "did you patch it?" The better question is "how do you know the patch worked?"
What auditors and boards should ask for
If you oversee a security program, you do not need to read raw traffic. You need to know it exists. Ask five questions of any testing report, from any tool or provider.
The last question is the most telling. A tool that never drops a claim is not checking its own work.
How Hayrok applies the standard
We built Hayrok on the idea that an agent's word is not evidence.
The agent can still be wrong. Any AI agent can, including ours. The point of the standard is that a wrong claim is caught before it reaches your engineers. See Evidence Fabric and the Autonomous Testing Governance Framework.
A checklist you can use with any tool
0 of 8 metUse this list on the next report you receive, from any vendor or team.
Any unchecked item means some findings in this report are still claims.
Frequently asked questions
What is replayable proof?
Evidence of a finding complete enough that someone else can run it again, in the same scope, and get the same result.
Can AI pentesting agents report false results?
Yes. Research shows agents can report a command as successful when it was not, so every claim needs captured evidence.
What is the difference between a claim and a finding?
A claim is a statement that something happened. A finding is a claim backed by evidence captured during the test.
How do you verify an AI-generated finding?
Replay it from the captured requests, in the same scope and state, and compare the responses with the expected result.
What should a replay package contain?
The target and scope, preconditions, raw requests and responses, timestamps, environment details and the expected result.
Why might a valid finding fail to replay?
The system may have changed: a new version, a different setting or a fix. Record what changed, so the result still tells you something.
How does replay help verify a fix?
Run the original package against the patched system. If the attack now fails, you have evidence the fix holds.
Is replayable proof only for AI tools?
No. The standard works for human testers too. AI agents simply make it more urgent.
Do replays need approval?
Yes. A replay is a test. It should run inside an approved scope, like the original.
What should a board ask about testing results?
Whether critical findings can be replayed, and which claims were dropped because they could not.
There are two ways to work with us.
For security leaders at SaaS, FinTech or cloud platform companies who want to help shape how replayable proof works in practice.
Apply to the program →Prove governed validation in your own environment, starting now.
Get early access →At Hayrok Labs we question everything and Validate what matters.