The Limitations of External Penetration Testing (And What to Do About Them)

Core argument 

Traditional internal penetration tests gives executives false confidence because it’s typically scope-limited, scheduled, doesn’t reflect real attacker behaviour and ignores the AI threats with user access.

Would you feel comfortable boarding a plane if the pilot had practised emergency landings but had never actually simulated an engine failure? 

So, why do businesses specifically exclude their most critical systems from the testing scope, especially when those systems are part of the very exploitation path attackers use? 

 Key angles across the series: 

• The real-world difference between “can we break in” vs “will we detect and respond when they do” 

• What cannot be seen is not understood 

• Why annual tests create 364 days of unknown vulnerability and exposure 

• The problem with “safe” testing that avoids business disruption 

• Red teaming vs pen test vs continuous security validation 

What is an External Penetration Test?

An external penetration test simulates a cyberattack from outside an organisation’s network to identify vulnerabilities in internet-facing systems such as websites, servers and remote access services.

Security experts attempt to safely exploit weaknesses to assess risk and provide remediation guidance, helping organisations reduce exposure to real-world attacks originating from the internet.

The Monday Morning Question 

Let’s put this into business context. The annual penetration test report arrives.  

• Twenty pages, executive summary up front.  

• “Low risk” findings only.  

• A handful of configuration tweaks recommended.  

Nothing critical. All good.   You feel relieved. £10,000 well spent. Board update will be straight-forward. The auditors will be satisfied.   Here’s what the report doesn’t tell you: You have no idea if you’d detect a real breach. 

The Comfortable Fiction of Scheduled Security 

Let me tell you about a business last year. They’d received clean pen test results for three consecutive years and just completed the latest one.   Security was “validated” annually for ISO 27001. They were confident with minor follow ups. 

An inquisitive new COO posed a question – “Does that really reflect a real-world attack though?”  The CISO reached out. We ran a purple team exercise, a cooperative test, not known to others took place, where attackers and defenders work together to improve detection.  

Within four hours, we had: 

• Compromised a user account via a credential stuffing attack (their MFA wasn’t enforced on a legacy VPN) 

• We found an unpatched development AWS instance with storage services and recovered keys  

• We moved laterally to a domain controller, without detection 

• We exfiltrated unclassified data which transpired was sensitive  

• We established persistent access through three different backdoors 

• We deepfaked the Chief People Officer’s email account with a new employee benefits email to all staff, we got a 23% click through rate and 47 email addresses and contact details.  

The security endpoint and SIEM solutions detected nothing unusual.  

1. One alert fired during the email campaign but it was classified as only medium severity.  

1. The SOC team was monitoring dashboards that were showing green while we methodically progressed. 

Situation Analysis

None of the techniques we used were sophisticated. All real-world. No zero-days. No advanced malware. Just patient, methodical exploitation of common misconfigurations and gaps in visibility with some standard automations. 

The previous pen tests had all been “clean” because those tests never asked the critical question: “If we bypass your prevention, will you notice?” 

Related article: Continuous Security Validation: Why Security Investments Fail Under Real Attack Conditions

What do External Penetration Tests Actually Test?

Traditional penetration testing focuses on “can we get in” rather than “will you notice when someone does?”.  

Some look at how far an exploitation path progression but this needs to be correlated against monitoring services.

This creates a dangerous gap: 

• What external penetration tests validate: Presence of vulnerabilities and exploitation pathways 

• What they rarely validate: Detection capabilities, monitoring and response effectiveness, correlation processes, recovery procedures 

The Bad Actors know these gaps exist and actively target them: 

• 68% of organisations rely primarily on annual penetration testing for security validation 

• Less than 20% test their security operations ability to detect real attack techniques 

• 12% have validated their incident response plan under realistic breach conditions (having an IR plan on paper does not prepare you for the real thing!)

• Average dwell time for breaches: 194 days 

• Persistence is now a primary attack breach head in 24% of attacks 

Put another way, if your Organisation tests for vulnerabilities once a year, but real attackers maintain access for seven months on average, how do you know they are there? 

Attack vectors against large UK companies through 2025 all confirmed persistence and presence was established at least 12 months previously.  

You’ve been breached, can you answer these questions?

The most common questions initially asked of us in the event of a detected breach are: 

1. How did they access our network? 
2. Are they still here? 
3. What have they taken? 

External penetration testing would only seek to validate part of question 1. These are patient, informed (from research) and motivated adversaries who don’t operate to your schedule. 

Next in the Series

If your penetration testing happens once a year, and the average attacker remains undetected for over six months, then for most of the year your organisation is operating on assumption, not evidence. And assumption is not security.

In the next article I’ll break down five ways penetration tests can unintentionally create false confidence, from scope exclusions and pre-announced testing to the fundamental gap between finding vulnerabilities and validating detection. Find out about CloudGuard’s penetration testing UK services. 

Frequently Asked Questions: External Penetration Testing