What good QA looks like in a penetration testing team

Good QA should establish whether the client can trust the work and act on it.

That means looking closely at technical accuracy, evidence, severity and recommendations. It also means asking whether the report gives a fair account of the engagement: what we tested, what we found, which decisions we made and what we still do not know.

I use a simple guiding principle for this: AC/DC.

Our work should be:

• Actionable: the client knows what to do next.
• Clear: the meaning is precise and unambiguous.
• Defensible: we can justify our findings and the decisions made during testing.
• Correct: the report gives a complete and accurate account supported by evidence.

These principles apply to the whole engagement: how we test, what we record, the advice we give and how we respond when a client challenges a conclusion.

A reviewer can use them to explain why a finding needs further work.

Actionable: give the client a usable next step

A finding needs a recommendation the client can implement.

Think about the person who will implement the recommendation. They may not have been on the debrief call or seen the test notes. The finding needs to give them enough information to understand and make the change.

Imagine a tester demonstrates that an ordinary user can change another user’s contact details through a profile update function.

A recommendation such as “implement appropriate access controls” gives the client a direction, but leaves much of the useful explanation unwritten.

A stronger recommendation would identify the affected function and describe the required behaviour:

Enforce authorisation on the server for each profile update request. Ordinary users should only be able to change their own contact details. Reject requests that attempt to change another user’s details without the required permission.

The accompanying finding should identify the affected endpoint, the accounts used and the evidence showing the unauthorised change.

That gives the remediation team a defined problem and an intended outcome. It also gives the person performing the retest something specific to verify: unauthorised changes are rejected, while permitted changes still work.

We can describe the behaviour the fix needs to achieve while leaving the implementation to the client’s engineers. Where that choice depends on architecture or business rules we have not seen, the report needs to say so.

The QA question is whether the advice gives the client enough information to move forward.

Clear: make the finding easy to understand and verify

The report needs to remain usable when it passes from the client contact to the engineer implementing a fix, and later to the person retesting it.

One common source of confusion is a recommendation that bundles several different weaknesses together.

For example:

Restrict administrators from changing other administrators’ accounts, require reauthentication before password changes, and strengthen the password policy.

This combines authorisation, reauthentication and password policy. Each weakness may require its own change and verification step.

If the client fixes only the authorisation problem, the retester cannot mark the whole finding as resolved. The report needs to distinguish the work completed from the weaknesses that remain.

Consider separate findings for weaknesses that can be fixed and verified independently. Where several actions are needed to resolve one vulnerability, keep them together but make the steps and acceptance criteria explicit.

Clarity also means being precise about the finding itself. “Users can access sensitive information” leaves the reader to guess which accounts and data are involved. Identify them and explain the affected function. State any conditions required for the access to occur.

The executive summary needs the same care. It should explain the significance of the findings in language the intended reader can understand, while preserving the qualifications that affect their meaning.

One useful check is to ask another reviewer to explain the finding back to you. A different interpretation shows where the wording needs more work.

Defensible: be able to explain the reasoning

Clients make decisions based on our work. We should be able to explain the basis for the advice we give them.

For a finding, that means being able to trace the reasoning from the test performed, through the evidence collected, to the conclusion and severity assigned.

A reviewer should be able to understand:

• what the tester did and what access they started with
• what was directly observed
• what was inferred from those observations
• which assumptions or limitations affect the conclusion
• why the severity and recommendation are appropriate.

A screenshot showing an error may support the statement that an error occurred. Further evidence or explanation is needed to support a claim that an attacker could extract customer records.

Defensibility also applies to actions and omissions.

If an in-scope service was left untested, there should be a recorded reason. Perhaps access was unavailable, the client requested a restriction, or testing encountered an operational constraint. The report should make the resulting limitation visible.

Those decisions are much easier to review when the test notes, evidence and delivery records support the report. A polished paragraph on its own leaves the reviewer guessing about what happened.

Severity deserves particular attention. The team should be able to explain how the rating follows from the evidence, preconditions, potential impact and relevant client context. Where impact is inferred rather than demonstrated, the reasoning should be explicit.

Being defensible also requires a willingness to revise a conclusion. If the client supplies evidence that changes the assessment, updating the finding is part of maintaining its quality. This discipline is central to handling severity disputes.

Correct: report the facts accurately and completely

Correctness requires discipline about what we know.

A finding can sound convincing while making a claim the evidence does not support. Templates, familiar vulnerability patterns and confident wording can all make that easier to miss.

I find it useful to break correctness into three parts:

• Factual: statements about what happened are supported by evidence.
• Accurate: the report represents the observations faithfully, including the affected systems, access and results.
• Complete: it includes the relevant conditions, qualifications and limitations needed to interpret the finding.

Consider this statement:

The organisation uses a third-party DNS provider, which provides a secure configuration and prevents security issues.

The observation about the provider does not establish the security claims that follow it.

If all we established was that the DNS service was hosted by a third party, that is what we should report. Any further assessment needs its own supporting evidence.

The same discipline applies to vulnerability impact. Demonstrating access to one other user’s record establishes a specific result. A claim that every customer record is accessible requires additional evidence or a clearly explained basis for that inference.

Completeness matters here too. If exploitation requires a particular role, configuration or sequence of events, that information belongs in the finding. Leaving it out can change how the reader understands the risk.

A complete report gives a fair account of the work performed and the limits of its conclusions. It does not imply that every possible vulnerability has been ruled out.

Use AC/DC throughout the engagement

AC/DC is most useful when testers apply it before work reaches the reviewer.

During testing, use it to check that the evidence supports the intended conclusion and that any constraints have been recorded. During writing, check that the recommendation addresses the described weakness and gives the retester a defined result to verify.

At final review, it provides four practical questions:

Principle	Review question
Actionable	Can the client identify the next step and the outcome it should achieve?
Clear	Can the intended reader understand the issue without having to guess what we mean?
Defensible	Can we explain the evidence and reasoning behind the finding, rating and relevant delivery decisions?
Correct	Does the report accurately represent what we know, including material conditions and limitations?

These questions complement the engagement’s detailed checks for scope, evidence handling, report consistency and release requirements.

OWASP’s reporting guidance similarly emphasises understandable reports, visible scope and limitations, and sufficient detail to reproduce and resolve findings. AC/DC gives the team a memorable way to apply those concerns in everyday work.

Compare two pieces of feedback: “Rewrite this recommendation” and “The client cannot tell which function needs to change or how we will verify the fix.” The second gives the tester a reason for the correction and something they can check in the next finding.

Reviewers should distinguish material quality problems from personal preferences about wording. AC/DC should make the standard easier to understand and apply consistently.

Use recurring defects to develop the team

QA findings can reveal where the delivery process needs attention. They should also inform how leaders measure penetration testing team performance.

Repeatedly vague recommendations may indicate that testers need practice explaining the required change to a remediation engineer. For missing preconditions or unsupported impact claims, examine the test notes with the tester to establish where the account became incomplete or overstated.

Leaders can use those patterns to choose a response:

• improve evidence collection when reviewers cannot follow how a result was obtained
• provide worked examples when recommendations repeatedly lack a usable next step
• hold severity calibration sessions when similar findings receive inconsistent treatment
• review scoping and escalation practices when limitations only become visible at final QA.

The same defect may affect more than one AC/DC principle. An exaggerated impact statement can be both incorrect and indefensible. Record the cause so the development action addresses the defect.

If the same senior people continually rewrite findings, the team needs worked examples and supervised practice so testers can apply the same reasoning. Otherwise, final review will keep absorbing problems that could have been addressed during delivery.

Before the report goes out, try AC/DC on a finding and its supporting record. Can another reviewer follow the conclusion? Can the remediation team see what needs to change? Any point where they have to guess is worth resolving before the client receives it.