Experimental Research
Usability Research
Role
UX Researcher
(Individual work)
Tools
RStudio
Qualtrics
Microsoft Office
Skills
Literature Review
Survey
Data Analysis
Duration
4 months
(Sep 2022 - Dec 2022)
OVERVIEW
The study intends to aid in further understanding the relationship between color and usability. It is an attempt at experiment design to examine the impact of color saturation on the usability of the web interface. The results of the study shall be useful for designing interfaces, especially from an accessibility point of view.
PROBLEM
Improving the usability of web interfaces has always been a challenging problem space and hence, studying the different factors that tend to impact usability has been an ongoing effort. Saturation is one of the attributes of color; and while color is known to affect usability, the impact of color saturation on usability has never been studied in isolation. Hence we do not know whether the saturation level of an interface has any impact on it's usability.
GOAL
For most parts, we usually deal with colors at high saturation levels. So reducing the color saturation level and checking its impact on usability becomes our goal for this study.
APPROACH
The method adopted to achieve this was conducting a literature review to conjure a research question. Hypothesis were formulated to support the research question and an experiment was designed to validate the hypotheses. Participants were randomly assigned to each of the groups- one which is exposed to a colored web interface and another which is exposed to a desaturated web interface.
THE PROCESS
Literature Review
A literature review of 12 sources published between 1995 to 2019 shows suggested that color is one of the factors that determine aesthetics/ visual appeal. Various experiments have studied the impact of color on usability and user satisfaction.
(Brady & Phillips, 2003; Grishin & Gillan, 2019; Richardson et al., 2014; van der Heijden, 2003)
The studies conducted in the domain of color focus on a change in colors, color contrast, and how the combination of colors affects the usability of a website interface. This has been achieved by controlling the hues used for the color, controlling which colors are placed beside each other, controlling the brightness level of the color, and the background and foreground combinations. However, none of the existing studies discuss the impact of saturation level or desaturation of the entire website interface, on the usability of the website interface. This brings us to the research question –
For most parts, we usually deal with colors at high saturation levels. So reducing the color saturation level and checking it’s impact on usability becomes our goal for this study.
Study Design
The method adopted to test this is a between-subjects design where participants are randomly assigned to each of the groups- one which is exposed to a colored web interface and another which is exposed to a desaturated web interface. The independent variable is the color saturation level of the entire interface. No specific sections or elements are targeted to be desaturated but the entire page/ interface itself. The dependent variable is usability. However, usability was identified to be perceived as well as objective, and therefore both aspects were considered as a part of the study. It is important to note that detection and affordance, both were considered to be a part of objective usability and therefore Success was measured in levels instead of being binary. For all tests, the alpha has been decided upon as 0.05 i.e. a 95% confidence interval.
A survey was created on Qualtrics to facilitate this which contained the tasks and questions. Microsoft Excel was used for management while Rstudio was used for the analysis of data. The participants were requested to take the survey only on their desktop or laptop, basically a web-based platform, and not on mobile phones or tablets. This was to ensure consistency and avoid confounding effects from associated factors.
Survey Design
For objective usability, 3 tasks were identified for measuring success and time across them. object-based, text-based, and image-based.
For each task, participants were supposed to detect and click on elements of interest.
The perceived usability section had 2 sets of tasks. The first task was filling out their responses on the standard SUS questionnaire, and the second was giving their responses to 3 open-ended questions which asked the participants more about their experience.
The next section asked the participants about any known vision deficiencies. Along the same lines, the participants then had to look at the plates from the Ishihara test and type in their responses- to rule out participants who had a color deficiency, since handling that as a confounding variable, was out of scope for the purpose of this study.
At the end of the survey, the participants were asked to provide certain demographic information about themselves such as age group, location, etc.
Data Analysis
A total of 51 participants, from various parts of the world, had their responses in at the time of starting the analysis and any responses later were not considered.
A participant failed the Ishihara test module and had to be discarded. Therefore, the further results are based on data from a total of (N = 50) participants.
For the between-subjects test, since the success data were ordinal, the Mann-Whitney U test was administered for each task for the dependent variable. The data was then graphed into a boxplot and a stacked bar chart. This was done for each of the 5 tasks.
Heat maps from Qualtrics were also examined for each task for success data. These helped in visually comparing the data with the bar charts and thus supported understanding participant behavior in a rounded manner.
For time on task, an independent t-test was performed to compare means for each task for the dependent variable. The data was then graphed into a bar chart as shown. This was done for each of the 5 tasks.
Once past the data analysis for objective usability, it was examined how usable the participants perceived each version to be. For the open-ended questions, most users commented on the system being either too complex or the interface being easy. There were varied opinions in terms of the interface. However, there were a few comments that addressed color and related aspects, as represented above.
Moving on to the SUS scores, an independent t-test was used to compare the scores for each version and the data was plotted on a bar chart as shown. SUS scores indicated that the participants perceived bothversions to be less usable and difficult. Hence, task complexity was analyzed with help of a within-subjects analysis for each version.
For the within group analysis, a Friedmann test was used to compare the success and time on task. Variables with significant results were followed up with a post-hoc analysis (Conover test) to determine which tasks were significant. The data was then plotted visually as shown above.
In summary, even though the participants perceived both the versions to be similar in terms of usability, we saw a significant difference for task 2, which was text based, in which the participants performed significantly better on the colored version. Open ended responses also suggested an inclination towards the colored interface. Task 2 performing better in the colored group and worse in the desaturated group as compared to task 3, suggests at the ease of use for text based input in the colored group. Based on the results, we reject the null hypothesis. Color saturation level does have an impact on the usability of the interface atleast when we are looking at text-based elements.
REFLECTION
Limitations &
Challenges
-
The study did not involve any participants from the older age group
-
The study was administered via Qualtrics, and a lot of users expressed confusion about submitting responses via Qualtrics and that at times it was difficult to understand what to do. They were not familiar with responding via Qualtrics and this may have added to the complexity of the tasks, especially in the object-based and image-based tasks
-
Normality of data and homogeneity of variance were assumed
-
Only 100% and 0% saturation levels were tested, and not the intermediate ones
Next Steps
-
Recruiting participants from a broader age group and especially the older age range definitely needs to be considered, to see whether there is any effect of age on the results. Similarly, a gender-specific analysis should be considered as well to check for correlation
-
The study needs to be conducted with varying levels of saturation, potentially to find whether there is a threshold
-
It would be interesting to see the impact of color saturation on usability, when the participants have color deficiencies, as a step toward accessibility
-
The elements involved, such as the SUS and the objects, texts, images, and so on, need to be appropriately weighed. Research needs to be conducted to have accurate estimates in the form of an equation that can be used.
Takeaways
-
The complexity of using the tool and familiarity with interfaces tend to act as confounding variables that potentially affect the results. Future studies need to be designed taking this into account
-
Moreover, the tools available and used for conducting the study can act as a constraint while designing the study itself