Testing a suspected vulnerability requires more than sending a request and checking the response. The application needs to be ready, and the request needs to reach the code being investigated.
Consider an endpoint that appears to let one user fetch another user’s document. To test it, you need two authenticated users, a document owned by one of them, and a running application. If the request fails before reaching the endpoint, it tells you little about whether the application properly restricts access to that document.
We built Runtime Verification in CodeRabbit Secure to help developers test Deep Scan findings and inspect what happened.
What we set out to build
We wanted developers to understand the proposed test before running it. They needed to know:
- What setup it required
- Which steps to follow
- What observations would support or challenge the finding
We built the workflow around a Reproducer Guide that brings those details together.
Developers can follow it in their own test environment or use Runtime Analysis to run it in a prepared CodeRabbit coding environment. A Runtime Analysis result includes supporting evidence and indicators of whether the application was ready and the intended code path was exercised.
Turning a finding into a Reproducer Guide
Deep Scan findings provide context about the suspected issue and its location. Because the code may have changed since the finding was reported, CodeRabbit rechecks whether the issue is still present before generating a guide. If the issue is no longer present or the recheck is inconclusive, guide generation stops.
For a finding that remains present, the guide connects the suspected behavior to prerequisites, steps, and expected observations. When you run the guide with Runtime Analysis, it uses the revision pinned in the published guide. Clicking Run does not recheck the finding or automatically test the latest source.
For the document endpoint example, the guide needs to establish which user owns the document and how the other user will attempt to retrieve it. The expected result must distinguish access to another user’s document from a request that fails before reaching the endpoint.
The guide needs to make its assumptions explicit so developers can check them against their own environment. If it cannot establish a required value or setup detail, it should identify what is missing and what the developer must supply before testing.
Unless a team has turned on automatic verification, generating the guide does not start the test. Developers can inspect it first, then decide where and how to run it.
Getting the experiment running
To run the guide with Runtime Analysis, the repository needs a prepared CodeRabbit coding environment with the dependencies, configuration, and startup instructions required to run the application.
Teams can also turn on automatic verification, which generates guides for newly detected findings at a chosen severity after each Deep Scan and runs them in the repository’s prepared default environment.
Starting the application’s process does not establish that it is ready for the test. A service may be listening while a required dependency is unavailable, or an authentication problem may prevent the request from reaching the intended endpoint. Runtime Analysis needs to account for those conditions when assessing the result.
The result includes Application ready and Target exercised indicators alongside the assessment and supporting evidence. These show whether CodeRabbit established that the application was ready and the relevant code path was exercised.
Use a dedicated test environment and avoid production credentials. The agent can read variables configured for the selected environment, including secrets. Avoid printing those values in test scripts or commands because the output may be retained as evidence.
Making sense of the outcome
Runtime Analysis reports one of three verification states:
- Reproduced: The run produced evidence supporting the finding under the tested conditions.
- Not reproduced: The run did not reproduce the finding under those conditions and other inputs or configurations may behave differently.
- Inconclusive: The run could not establish a supported conclusion. The result explains the reason and limitations.
Each result applies to the tested code revision and environment. A successful reproduction does not establish how every deployed version behaves.
In the document-access example, a 403 is meaningful only if the request reached the ownership check; a rejection caused by a missing session does not test that protection.
Likewise, “not reproduced” is narrower than “safe.” If setup problems prevented a meaningful test, the outcome should remain inconclusive rather than suggest the vulnerability is absent.
The decisions behind the workflow
The workflow separates the test plan, the execution, and the assessment so developers can inspect how the result was reached.
When developers generate a guide themselves, publishing it separately gives them a chance to inspect the proposed test before execution. They can check its assumptions and run it in their own environment. It also makes the distinction between expected results and observed behavior explicit.
What runtime evidence changes
A runtime test is useful only when the team can establish what it actually tested. That is why the guide, execution details, and supporting evidence remain available alongside the conclusion.
Set up Runtime Verification to test a Deep Scan finding in your own environment or with Runtime Analysis.
Run a Deep Scan on your own code. Start with a free sample scan.
