Simplifying The Form-filling Experience Via Usability Testing
Legal
MY ROLE
UX Researcher
(Team of 5)
TOOLS
Qualtrics
Miro
Microsoft Office Suite
Zoom
SKILLS
User Research
Heuristics Evaluation
Usability Testing
Data Analysis
Benchmarking
Duration
4 months
(Jan 2022 - May 2022)
OVERVIEW
The study was a formal attempt at evaluating a subset of the divorce filing process for the Commonwealth of Massachusetts, and it was completed on behalf of the Massachusetts Probate and Family Court Department. It specifically looked to analyze and evaluate the usability test results for the No-Fault 1B Divorce Form and the process of navigating to it from the Mass.gov webpage outlining the No-Fault divorce process.
PROBLEM
The client had received feedback from users and noticed a high error rate
Additionally, users often had questions about completing the form and had trouble finding the information necessary to complete it.
GOAL
Primary goals for this project included the following:
-
Determine problems users face when identifying the necessary forms for the process.
-
Identify which questions on the form users have difficulty completing.
-
Understand why users have difficulty completing those questions.
-
Understand how users currently try to overcome problems in completing forms.
-
Make recommendations for improving the process, website, and forms.
APPROACH
Evaluated the form and the webpage to recommend solutions
Conducted a heuristic evaluation and a usability test of the form and the MA Court webpage outlining the divorce process to determine specific issues and make actionable recommendations. The evaluation objectives were determined based on discussions with the client sponsor.
IMPACT
The results of this study were leveraged to convey pain points to the practitioners and other administrators in the court
-
The research helped identify weaknesses and suggest improvements to the filling of the form, as well as the process related to the filling of the form.
-
As a consequence of the research, the MA Probate Court is likely to form a committee to review the case and fine-tune solutions based on the findings and the recommendations; solutions which are likely to have positive ramifications for the people using the forms.
-
The details, enumerated in the report and the presentation, are being leveraged to discuss and decide on certain aspects- such as the requirement of the Military Affidavit.
THE PROCESS
All testing goals were chosen based on considerations of the client’s key business goals for the initiative:
-
Reduce user anxiety during the process
-
Reduce time waiting in line
-
Reduce errors during the divorce process
-
Reduce the time to completion of the divorce process
-
Reduce the number of questions related to the form completion/filing
-
Reduce time on the searching website for forms
Study Brief
To support the project goals, we framed the following research questions:
-
What is the success rate for identifying the correct forms?
-
What is the error rate for completing the Complaint for Divorce form?
-
Which questions get the most errors?
-
What terms are the least understood?
-
-
What is the user’s confidence that they completed the form correctly?
-
What is the user’s satisfaction with completing the form?
-
What are the drivers of satisfaction and dissatisfaction?
-
-
What is the user’s satisfaction with locating the correct forms for the divorce process?
-
What are the drivers of satisfaction and dissatisfaction?
-
Task List
Preparing the task list was a quintessential phase of the project
-
The task list underwent a few iterations.
-
We also constructed a mock scenario that will put the users in a pre-defined setup and help them move through the tasks.
The tasks were chronological in nature
-
We had to ensure that the user navigates through the webpage and the form in such a way that it helps us address all the project goals.
-
There was no counterbalancing because there would be exposure/learning bias if we showed participants the Complaint for Divorce form before asking them what forms they need to complete.
Heuristic Evaluation
The focus of the heuristic evaluation
Was determined by the tasks identified for the usability testing
Jakob Nielsen's 10 Usability Heuristics for User Interface Design
Was used for the review of the Mass.gov web pages that navigated to the 1B Divorce form.
Adobe Best Practices for Form Design by Nick Babich
Was used to analyze the form itself against accepted form design principles.
The Dumas and Redish severity scale
Was used for both sets of heuristics.
An evaluation was initially conducted by each individual researcher
-
For an extensive review of the form and the webpage
-
The findings were combined to develop a group heuristic evaluation.
-
The findings were organized as positive as well as negative findings
The negative findings were prioritized
-
Based on their severity ratings
-
With recommendations included under each negative finding
These findings
-
Provided insight into the areas of the form and the webpage that needed to be focused upon for improvement
-
Established a base for the usability test
Participant Screening
The team prepared a screener to identify and recruit desired participants
-
Social networks, as well as professional networks such as LinkedIn, were used as channels to reach out to the target groups.
-
The screener consisted of an online survey built using Qualtrics.
Participants whose responses aligned with the criteria set for the study
-
Were invited to participate in the usability test.
-
They were communicated via email and prior consent was taken by requesting the participants to sign a consent form.
-
We excluded people who:
-
Were under the age of 18
-
Work in the legal profession
-
N = 11
(1 pilot, 10 participants)
Moderator's Guide
The moderator’s guide was written to provide a script for the session
It included:
-
An introduction to the study
-
Briefing information for the participants
-
Background interview questions
-
Protocols for the sessions
-
The mock scenario, and the corresponding tasks
-
The post-test debriefing.
Usability Testing
Testing was conducted over Zoom and recorded
-
Every session was led by a moderator through their own computers while the participants used their own computers.
-
We used a Qualtrics survey to collect quantitative data on satisfaction and confidence.
Thinking out loud was demonstrated to the participants during the briefing stage
-
The participants were encouraged to do so as they move through the tasks concerning the webpage.
-
However, the participants were requested to not think out loud, or ask questions, and fill out the Complaint for Divorce form as they would in a real-world setup.
Moderated Session
Participants were free to take help of any other resources
-
As they would normally do while facing the task in a real-world setup.
-
This was to facilitate the authenticity of user activity and data.
User behavior, time on task, and issues or highlights were thoroughly noted
Were thoroughly noted by the observer/s throughout the usability sessions.
Data Analysis
"The whole thing confused me"
"It is a lot to take in"
We synthesized the observations noted during the sessions
-
Pulled the gathered quantitative and qualitative data to assemble our data set.
-
Analyzed this data to derive insights and develop recommendations based on our findings.
-
Categorized findings and recommendations based on the webpage containing the form, and filling out the form itself.
Summary
The broad-level themes that emerged for the findings/ issues were related to-
-
Inconsistent labeling and unclear copy
-
Legal jargon
-
Lack of help/ context
-
Confusing field controls
The topmost recommendations involved
-
Clear and jargon-free copywriting
-
Help regarding legal terms and their implications
-
Clear sections on the webpage listing the required forms based on conditions
-
Visual feedback
-
Ability to edit the form contents after saving.
Findings/ Recommendations
Priority of recommendations was determined by their impact
-
The impact of a recommendation was based on the severity of its corresponding finding
-
The report was organized based on which findings were validated and whether there were any new findings.
Issues related to filling the form were dedicated higher priority
Since they directly contributed to complete task failure and/ or severe user frustration.
Benchmarking
A benchmark test was run
To emphasize the importance of the findings and recommendations
The benchmark value was chosen as 78%
Based on a study that involved an analysis of almost 1200 usability tasks by Jeff Sauro.
Overview
For the task of finding the forms
Only being able to find all the forms was considered a success since any missing form would lead to a second attempt and failure to apply currently.
For the task of filling the form correctly
Only being able to fill out all the details correctly was considered a success.
Results
The benchmark comparison values provided a clear indication
Of the current design’s performance, validated through data.
The test helped with better understanding the implications
Of the quantitative data and give an additional perspective into the findings and the recommendations.
REFLECTION
Limitations and challenges
-
There are demographic segments that were underrepresented or not captured in this study, namely people who are not fluent in English and people who do not have a post-secondary education.
-
Additionally, it proved a challenge to find participants who had been through a divorce (at least once) earlier. Out of our 10 participants, only 1 participant fulfilled this criterion. The primary limitation, thus, was finding a high number of participants. especially those that belonged to a specific demographic setup.
Next steps
-
Since some demographics were underrepresented or not captured, we recommend additional usability testing or a qualitative study with the Help Center to capture the problems encountered by these groups.
-
Additionally, conducting a SUS usability survey or examining data in site analytics tools may reveal additional information about the mass.gov site.
Takeaways
-
Jargon can pose an issue for users regardless of their educational level.
-
Users can be blind to cues when acting with assumptions.