Accessibility Testing with Real Users with Visual Impairments
Speaking about Accessibility Testing at UX Camp Europe in Berlin 2022
Introduction
At UIWS (User Interface and Website Solutions) at hmmh Medienagentur Bremen, we aimed to make Decathlon's online shop more accessible. Since usability is best validated by actual users, we conducted an accessibility test with individuals with visual impairments. This journey brought challenges, valuable insights, and eye-opening results.
Overview of the Case Study
Integrating Accessibility in the Development Process
Differences Between Accessibility Tests and Usability Tests
Challenges in Recruiting Participants
Test Setup, Location, Participants, and Key Findings
User Experiences and Stories
Conclusion and Learnings
Why Accessibility Matters
A detailed discussion on the importance of accessibility can be found here
Integrating Accessibility in Development
Using Accessibility in our Quality Assessment Process was a significant win. To enhance accessibility, we adopted the following strategies:
1. Knowledge Exchange
We initiated AccessibleTea, a weekly meeting for developers, designers, and product owners to discuss accessibility best practices. Ensuring buy-in from all stakeholders is key to long-term success.
2. Creating Pragmatic Accessibility Guides
WCAG guidelines can be overwhelming for beginners, so we developed a practical guide with:
Code examples
Step-by-step implementation
Qualitative assessment techniques
3. Accessibility in Acceptance Criteria
We integrated accessibility into our Quality Assessment Process, ensuring:
Accessibility as an acceptance criterion for all tickets
Development with accessibility as a standard practice
Even with these efforts, real-user testing was essential, as intention does not always translate into usability.
Challenges in Recruiting Participants
Nikolai Sehzentrum our a11y partners.
After all, just because users can use your website doesn’t necessarily mean they want to use it.
Our plan, therefore, was to carry out tests with a focus on users with visual impairments. This was not easy, for several reasons, the biggest of them being recruiting. Finding participants with visual impairments was our biggest hurdle due to:
Data Protection Laws: Organizations cannot disclose disability status.
Digital Literacy: Participants needed to be web-literate.
Setup Constraints: Users should be tested in their own familiar setup.
Diversity of Impairments: We needed a range of visual impairment levels.
COVID-19 Restrictions: Recruitment was further delayed due to the pandemic.
After nearly a year, Nikolaus Pflege Zentrum responded, providing participants and a suitable location.
How Accessibility Tests Differ from Usability Tests
Accessibility tests require additional preparation and adjustments:
Pre-Test Familiarization: Users explore the product beforehand.
Interviewer Preparation: Testers must understand screen reader navigation, tab orders, and accessibility-specific interactions.
User-First, Not Mobile-First: Testing is conducted on users' own devices.
Patience & Adaptability: Screen readers operate at high speeds, requiring frequent user feedback.
Extended Duration & Smaller Task List: Tests take longer due to accessibility considerations.
Two-Person Team: One moderator and one dedicated note-taker are essential.
Test Setup and Participants
The testing took place at Nikolaus Pflege Zentrum, equipped with:
Desktop computers
Magnifying tools
Refreshable Braille displays
NVDA and JAWS screen readers
Mobile phones with zoom and assistive apps
Braille Displays
Braile Integrated Keyboards
Magnified Screens, movable screens.
Participant Profiles (Names Changed for Privacy)
| Name | Age | Visual Impairment | Device | Assistive Tech |
|---|---|---|---|---|
| Bernt | 54 | 100% blind (since 15) | Desktop | JAWS + Braille Display |
| Helga | 55 | 100% blind (since 50) | Desktop | NVDA |
| Paul | 30 | 70% blind (progressive) | Desktop | No screen reader |
| Maya | 23 | 50% blind (progressive) | Mobile | Magnification App |
| Mohmd. | 21 | 30% blind (progressive) | Mobile | No screen reader |
Test Tasks
Shopping for Trekking Shoes - A straightforward product search and selection.
Shopping for a Tent or Rain Jacket - Testing a product with detailed specifications.
Onsite moderated testing with a participant.
Key Findings from the Tests
1. Success Stories
Bernt (Experienced User): Navigated at lightning speed using JAWS and Braille Display. Demonstrated how logical website structure improves accessibility.
Paul (Low Vision User): Relied on 300% zoom. Noted disappearing menus at high zoom levels but successfully navigated using common UI patterns.
Maya (Mobile User): Disappointed by the lack of zoom support but overcame it with a third-party magnification app.
2. Pain Points Identified
Non-Navigable Size Selector: A critical issue preventing purchases.
Wishlist Login Popup Not Announced: Users got stuck in an unannounced login screen.
Mobile Site Not Zoomable: A major barrier for users relying on zoom functionality.
Filter Issues: Screen readers announced irrelevant filter results, causing confusion.
Inescapable Menu Lists: Users were forced to listen to full lists without skipping.
Pronunciation Issues: Screen readers mispronounced certain German words, impacting comprehension.
Conclusion and Learnings
Why This Test Mattered
Identified major accessibility flaws that automated tools (like WAVE and Lighthouse) missed.
Proved that real-user testing is essential to ensure usability, not just technical compliance.
Demonstrated that accessibility benefits all users, not just those with disabilities.
Key Takeaways
Accessibility testing requires real users for meaningful insights.
Investing in accessibility prevents loss of customers (~10% of users rely on assistive tech!).
Digital accessibility isn’t just about compliance—it’s about inclusivity and usability.