What Automated Accessibility Scanners Actually Miss (And Why Manual Testing Still Matters)

Quick answer: Automated accessibility scanners — axe-core, WAVE, Lighthouse, and the tools built on them — are rule engines that check a page’s rendered DOM against machine-testable conditions: does this image have an alt attribute, does this button have an accessible name, does this text meet a contrast ratio. Deque, the company behind axe-core, found in a study of over 13,000 pages that automated testing catches 57% of accessibility issues by volume — a meaningful majority, but well short of complete. The remaining issues require human judgment: whether alt text actually describes the image, whether a keyboard user can escape a modal, whether reading order makes sense, whether an error message is actually helpful. No scanner can evaluate any of those on its own, not because the tools are poorly built, but because the questions aren’t machine-testable in the first place.

This article covers technical accessibility testing practice, not legal advice. Meeting WCAG 2.1 AA, with or without automated tooling, is not the same thing as a legal compliance determination — consult qualified counsel for legal risk questions.

What an automated scanner actually checks

Tools like axe-core (which also powers Lighthouse’s accessibility audit and many other scanners), WAVE, and Lighthouse work the same basic way: they parse a page’s rendered HTML/DOM and run a library of rules against it, each rule looking for a specific, well-defined pattern — a missing alt attribute, a form input with no associated <label>, a heading level that skips from <h2> to <h4>, text and background colors that fall under a computable contrast ratio. These checks are fast, consistent, and can run across an entire site in minutes, which is exactly what makes them useful as a first pass. They’re also, structurally, only able to answer questions with a deterministic yes/no answer from the code alone.

The real number: how much do they actually catch?

Deque’s Automated Accessibility Coverage Report analyzed anonymized data from over 2,000 real-world audits — more than 13,000 pages and nearly 300,000 individual issues — and found that automated testing with axe-core identified 57% of accessibility issues by volume. That’s notably higher than the 20–30% figure that circulated for years in the accessibility industry. The difference comes down to methodology, not a change in what the tools can do: the older estimates measured the percentage of WCAG success criteria that can be tested automatically, while Deque’s newer figure measures the percentage of actual issues found on real pages, weighted by how often each issue type occurs in practice. Some failure types (missing alt text, low contrast, unlabeled form fields) are both common and easy to detect automatically, so they make up a large share of real-world issue volume even though they’re a small share of the full WCAG rule set.

Either way you measure it, a meaningful share of issues — everything axe-core’s own documentation describes as needing a result of “incomplete,” where the tool cannot be certain and flags the item for manual review — falls outside what a scanner can resolve on its own.

What’s structurally impossible for a scanner to catch

Some accessibility failures aren’t a matter of the tools not being sophisticated enough yet — they require a judgment call a rule engine cannot make from markup alone:

• Whether alt text is actually meaningful. A scanner can confirm an <img> has an alt attribute. It cannot tell you whether alt="image1234.jpg" or alt="product photo" conveys anything useful to a screen reader user, versus alt="Men's navy wool overcoat, front view", which does. WCAG 1.1.1 requires a genuine text equivalent, not just the presence of the attribute.
• Keyboard operability. Whether a user can Tab into a component and Tab (or Escape) back out of it is a behavioral question that only shows up when someone actually tries it. WebAIM’s evaluation guide is direct about this: “WAVE and other automated tools can only identify some accessibility issues. You should also test the page with a keyboard, screen reader, and/or browser developer tools.”
• Logical reading and focus order. A scanner can check that headings exist; it can’t confirm that content reads in a sensible order for a screen reader user, or that visual layout order matches DOM order, when CSS has repositioned elements.
• Whether ARIA is used correctly, not just present. role="button" on a <div> satisfies a “does this have a role” check. Whether that element also handles Enter/Space key presses the way a real button does is a separate, unautomatable question.
• Whether interactive elements’ state and purpose are clear. Google’s own Lighthouse accessibility docs list a dedicated set of “manual checks” — including whether custom controls carry the right ARIA roles, whether focus gets trapped, and whether visual order follows DOM order — that don’t factor into the automated score at all, because Lighthouse can’t evaluate them.

What tool-specific documentation says about this

This isn’t a Quietramp opinion about scanner tooling — it’s what the vendors say about their own products:

• axe-core’s own README states plainly: “With axe-core, you can find on average 57% of WCAG issues automatically,” and describes its “incomplete” result category as cases the engine deliberately declines to resolve.
• WAVE’s report page, per WebAIM’s own documentation, tells users directly that a clean scan doesn’t mean an accessible page: manual testing is still necessary even when “no errors were detected.”
• Playwright’s accessibility-testing docs state that automated testing “can’t detect all types of WCAG violations” and recommend combining it with manual assessments and testing with people who use assistive technology.

A practical combined-testing approach

None of this makes automated scanning a wasted step — it’s the fastest way to find and fix the large, common category of issues (missing labels, low contrast, missing alt attributes) before a human reviewer spends time on anything else. The order that works:

1. Run an automated scan first (axe-core, WAVE, or Lighthouse) and fix everything it flags as a definite failure. This clears the bulk of common issues quickly and cheaply.
2. Review every “incomplete” or “needs review” result by hand. These aren’t false alarms — the tool is telling you it found something that needs a human decision.
3. Do a keyboard-only pass. Unplug the mouse, Tab through the page, and confirm every interactive element is reachable, operable, and escapable.
4. Do at least one screen reader pass on primary flows (checkout, account creation, search) using a free tool like NVDA or VoiceOver, following WebAIM’s screen reader tutorials.
5. Re-scan after fixes. Automated tools are also useful for regression-checking that a fix didn’t break something else nearby.

FAQ

If a page passes every automated accessibility check, is it accessible? Not necessarily. A clean automated scan means the page passed every rule that tool can evaluate — it doesn’t mean a human hasn’t verified the things scanners structurally can’t check, like whether alt text is meaningful or a keyboard user can complete checkout.

Which automated tool catches the most issues? Since Lighthouse’s accessibility audit and many other scanners run on the axe-core engine, their automated-detection coverage is similar in practice, even though scoring and reporting differ. The gap between what any of them can catch and full WCAG coverage comes from the same structural limits (judgment calls a rule engine can’t make), not from one tool being more thorough than another.

Is it worth using automated scanners at all if they miss almost half the issues? Yes — the 57% they do catch is heavily weighted toward common, high-volume issue types, so an automated first pass removes a large share of real problems before manual review time gets spent on what’s left.

Get a human-reviewed audit, not just a scan

Quietramp’s $890 audits pair an automated axe-core scan with a manual review of exactly the things scanners can’t evaluate — keyboard flows, screen reader semantics, and whether flagged issues are real problems or false positives — delivered as a prioritized, developer-actionable PDF report. See a real sample report or check pricing.

---

This is educational content about technical accessibility testing practice, not legal advice. Passing automated or manual accessibility testing does not constitute a legal compliance determination under the ADA, WCAG, or any other standard — consult qualified counsel for legal risk questions.

This article was drafted with AI assistance and reviewed by a person for accuracy before publication.