User Testing BHIM App: Bringing Prototypes to Real Users
Timeline
2 weeks (July 2024)
Responsibilities
Facilitating Testing Sessions, Analyzing Results and Data Synthesis
Role
UX Designer
Tools
G-Suite, Figma, audio and video recording devices
​
​
Millions use UPI transactions, but are they truly seamless? We set out to test our design, uncover usability challenges, and refine the experience based on real-world insights.
The Challenge
​
​
Recruitment & Screening:
​
-
Created a screening questionnaire to identify diverse participants—first-time UPI users, daily wage workers, homemakers and frequent UPI users.
-
Recruited participants through local communities, online groups, and direct outreach.
-
Ensured a mix of demographics, device types, and digital familiarity levels.
Planning the User Testing: How We Prepared
​Test Planning & Execution:
-
Recruited participants from diverse backgrounds across the country.
-
Conducted in-person user testing in three cities.
-
Developed a structured test plan to evaluate key user journeys.
-
Synthesized data into actionable insights and findings.
Testing three key tasks
We focused on four essential UPI interactions to evaluate usability and pain points. Participants were given realistic scenarios to complete, ensuring their interactions were as natural as possible.
1
​Scanning a QR Code to
make payments
2
​Pay to a contact
3
​Pay an electricity bill
​​Cities Tested
Tier 1 city - Bangalore
Tier 2 cities - Mangalore and Nashik
Nashik
Bangalore
Mangalore
14
Young
(18-25 Years)
43
Participants
19
Mid-Age
(25-50 Years)
10
Older
(50+ Years)
​
​
Before diving into the insights, here’s a glimpse of the on-ground user interviews I conducted.
​​​
​​
​
​
​
​
​
​
​
​
​​
​
​
​​
Multilingual Testing Approach
​To ensure participants were comfortable and expressed themselves freely, I conducted interviews in English, Hindi, and Tamil based on the user's preference. This allowed us to get richer, more authentic responses, especially from first-time UPI users who were not fluent in English.
​
Data Collection
-
We used audio and video recording devices to capture user reactions, pain points, and behaviors.
-
These recordings helped us analyze subtle details—pauses, hesitation, and body language—beyond just verbal feedback.​
Capturing Real Reactions: User Interviews in Action
Key UX Principles in Practice​
Avoiding Bias
Asking neutral, open-ended questions to avoid leading the user.
Empathy
Making users feel heard and valued.
Observation
Paying attention to how users interact with the app naturally.
Feedback
Assuring users there were no “right” or “wrong” answers.
​​Insights and Findings
“A bhel puri vendor in Bangalore struggled to find the ‘Scan QR’ button, assuming it should be on the home screen.”
“Users were hesitant to tap the ‘Pay’ button without a confirmation step, fearing accidental payments.”
56% of users preferred the Light Version over the Dark Version. Over 79% of these users are 25 and above
From Raw Data to Actionable Insights: Synthesizing Our Findings
After conducting user interviews across three cities, we had hours of recordings, detailed notes, and observational data—but raw data alone doesn’t tell the full story. The real challenge? Turning these scattered pieces into clear, actionable insights.
We transcribed audio recordings and categorized responses based on user demographics, behaviors, and pain points.
Step 1: Organizing & Structuring the Data
Step 2: Identifying Key UX Themes
Step 3: Crafting the final report
We distilled our findings into a detailed yet concise research report with:
📌 User behavior snapshots (with direct quotes)
📌 Pain points & friction areas
📌 Potential design improvements with recommendations
Personal Takeaways
As a user testing facilitator, I truly enjoyed interacting with a diverse range of users, and being able to communicate in two regional languages was a huge plus in helping participants feel comfortable and at ease. Ensuring that every participant felt like they were in a safe, judgment-free space to share their honest opinions was a top priority.
​
When it comes to user testing, I believe it's crucial to leave biases at the door and stay open to whatever the participant has to share, even if it challenges preconceived ideas. A key part of the process was reviewing video recordings, analyzing notes, and synthesizing all the data to deliver insightful, data-driven recommendations for design improvements.
What I'd do Differently
If I were to do this project again, I would:
-
Allocate more time for analysis to dive deeper into patterns and user behavior.
-
Test in Real-World Contexts—like observing users making payments at a busy vegetable stall—to capture authentic behaviors and challenges faced in everyday scenarios.