By Meghalee Das | STC Student Member
Remote user research and usability testing is an integral part of user experience (UX) research. With the wide availability of testing and video conferencing tools that allow researchers to create moderated or unmoderated tests, remote research methods have become increasingly popular in the UX field among industry practitioners as well as graduate students like myself.
While in-person UX research is recommended by usability experts, such as Jakob Nielsen, to get a better understanding of user ecologies and build rapport with participants, there are many situations where remote methods are preferred to gather user data and test UX. Such circumstances could be due to intentional choices, such as following a tight budget, convenience, testing participants in a different geographical location, etc. Alternatively, those circumstances could be due to unprecedented events like the COVID-19 pandemic, which has resulted in social distancing rules, travel bans, temporary or indefinite work-from-home decisions, and cancellation of on-site classes in higher education, thereby restricting in-person research.
Irrespective of the reason behind remote UX research, it is imperative that technical communicators and UX researchers in the industry and academia are responsive to the changing environment. In this article, I draw on my experience as a graduate student conducting remote UX research with the testing tool Loop11, and I present takeaways in five key areas of interest: flexibility in planning, budget considerations, remote research and testing methods, uncertainties and challenges, and opportunities. These takeaways can help industry practitioners, as well as current graduate students of technical communication and those transitioning to UX jobs, tackle issues associated with remote UX research and anticipate trends that can inform their practices.
Flexibility in Planning
In spring 2020, as part of a UX class assignment, I analyzed the experience of international patrons of the university library. I conducted initial research through interviews, site visits, user journey maps, and paper prototypes of the library website. Based on these research results, I developed an in-person usability test plan.
When the World Health Organization (WHO) declared COVID-19 a pandemic and social distancing rules were enforced, however, I had to be flexible. I quickly modified my research goals and created a remote test plan to replace the original in-person usability test.
As mentioned earlier, remote testing can be by choice or due to a lack of options. Fields where one could have difficulty conducting in-person research in the current environment include health sciences, fitness products, food, travel, education, and hospitality industries. No matter the circumstances, technical communicators and UX researchers in academia and the industry need to be agile and adaptable to changes.
Flexibility when planning a research project helps to navigate any unexpected challenges and gain realistic insights into UX. How can UX researchers cultivate flexibility? Flexibility or adaptability is a mindset; researchers must be willing to let go of rigidity in goals and methods, and realize that a fixed response to any developing situation will stall the project instead of benefiting it. Also, researchers need to do some legwork to understand the user environment and behavior, and to determine if the participant will be able to access a remote test if the original plan is modified.
Flexibility should also be extended to users in the way that they are treated during the test, such as giving them extra time to understand the tools, treating them with empathy, and acknowledging that they might be in unusual workspaces, facing anxieties, experiencing disruptions due to Internet issues, or caring for children. To better understand and adapt to the circumstances of my participants, I sent them a survey to gauge their technical proficiency, website use behavior, and access to the Internet. I explored remote testing tools through UX research forums, selected Loop11 as my testing tool, and created a moderated, remote testing plan. In this way, I was able to join the users online during the test and help assuage any concerns they had about the technology or the test itself. Because my users had varying linguistic and technical proficiency, my presence helped them feel more comfortable during the test.
Such flexibility in different aspects of the test plan, as well as in the way that participants are treated, can help technical communicators be responsive to any changing circumstances without compromising the quality of research.
Budget Considerations
The COVID-19 pandemic has had an unprecedented impact on the economy, and in June 2020 the US National Bureau of Economic Research officially declared that the US economy was in a recession. While there has been a slight improvement in US employment numbers at the time of this writing, industries across all sectors, including UX, continue to be volatile and could be facing shrinking budgets.
In an industry like UX, where projects can stretch over long periods and include participants across geographical boundaries—requiring extensive funding for recruitment, testing, and compensation—financial resources for research might be tight due to the current economic situation. In-person UX research usually costs more, but moving things online can also be expensive if advanced testing tools are used. In such circumstances, discount usability and lean UX approaches might be more feasible, as they use fewer participants and focus on specific issues that improve UX.
To get better access to users who fit the budget, researchers can use online participant recruiting services with varying fee structures and turnaround times. Student researchers usually recruit representative users from a small pool of participants within their personal network, and there is nominal or no compensation due to limited funding.
My only expense was purchasing coffee gift cards, and as I used the testing tool Loop11 during the trial period, I did not have to pay for it. I do plan to replicate this study, however, and therefore need to take testing expenses into consideration.
There are some tools, such as Userlytics, which have offered discounts due to disruptions caused by COVID-19, and researchers can explore which companies are currently offering free trials and affordable pricing plans. These measures help UX research practitioners, as well as student researchers, lower the cost of testing and compensation when facing reduced funding due to budget cuts in their respective organizations.
Remote Research and Testing Methods
In the first Virtual UX Conference hosted by the Nielsen Norman Group in April 2020, Jakob Nielsen said UX cannot be 100% remote if a researcher wants to generate deeper understanding and insights, but he acknowledged that in current circumstances and in the short term, some aspects of UX research, like usability studies, collaboration, and design, can be done remotely. He said, “User research is better in person … [but] if you have already done it, you have built up a kind of understanding of users; you draw from that bank and use that to interpret these somewhat weaker things that we can do right now.” This section addresses some of those challenges by describing contextual inquiry alternatives, diary studies, and remote testing and collaboration methods.
Contextual Inquiry Alternatives
One of the biggest problems that remote UX researchers face, especially at a time of social distancing, is the inability to conduct contextual inquiry through direct observational methods like site visits, shadowing, and fly-on-the-wall techniques. While in-person field studies cannot be completely replaced, researchers can use other methods of varying complexity, ranging from data analysis to self-reporting. These techniques are triangulated to determine recurring patterns, gaps, and issues users face.
In a 2018 Intercom article, Kumar Dhanagopal, a Consulting User-Assistance Developer at Oracle America, did a comparative analysis of methods to understand users on the basis of reliability and availability of sources. He found user forums, technical support teams, and internal experts of organizations to be the most reliable and easily available sources, and content analytics and research reports to be least reliable but most easily accessible. Remote UX researchers can use alternative methods like these to tackle travel, budget, and time restrictions.
Diary Studies
Diary studies reveal how users interact with a product or website over a period of time. Although they fall under self-reporting methods, such as surveys, focus groups, and interviews, they can be adapted into an audiovisual form of think-aloud protocol.
To give a rich description of their environment, the various steps involved in performing tasks, as well as their cognitive and emotional states, test participants can create electronic submissions of work logs, brief narratives, or snippets of tasks—and combine them with photographs or videos taken on phones, tablets, or digital or wearable cameras—to narrate what they are doing, similar to think-aloud protocol. These entries can be sent in real time or at a later time to describe user activities and ecology, and the data can be observed by UX researchers to perform environment and task analyses.
Diary studies are most useful when trying to understand long-term user habits, behavior patterns, user environments, and user journeys, and they might be particularly useful to researchers in remote settings who can’t be physically present with their participants.
Moderated and Unmoderated Remote Testing Tools and Videoconferencing Sites
There are multiple remote testing tools that can be used to evaluate UX with websites, prototypes, and apps (see “Tools and Resources for Remote UX Research” for a sample).
Loop11 allowed me to choose between moderated and unmoderated remote testing, and it provided an extensive list of usability metrics to measure efficiency (task completion rate, average page views, average time spent), learnability (heat maps, participant path analysis), errors, and satisfaction (System Usability Scale).
Because I created a moderated test plan, I could talk to my participants online during the test and monitor their performance in real time. I could also later revisit the audio and video recordings, as well as my time-stamped notes, to verify my observations.
Morae is another popular remote testing software application that offers functions similar to Loop11 and records on-screen activity, mouse clicks, keystrokes, audio, and video. Researchers can choose to run a moderated test or an unmoderated test by using Morae’s AutoPilot feature, which automates the test session and saves valuable human resources and time.
Besides remote-testing tools like Loop11 and Morae that have embedded recording and screen capture, researchers can also use video conferencing sites that have screen share and recording options, such as Zoom, to conduct moderated, remote usability tests. Researchers can take notes in real time and manually calculate user metrics while observing the participants and their screen. These videoconferencing websites are free and can be useful when spending on advanced remote testing tools is not an option. It is especially helpful for graduate student researchers, who might already have a Zoom or similar videoconferencing account for their classes.
Collaboration
Remote collaboration is not new. In a 2016 survey by the Nielsen Norman Group, 82% of the 86 UX professionals surveyed said they collaborated with others at varying levels. UX researchers use file sharing tools like SharePoint, Google Drive, and Dropbox; videoconferencing tools like Zoom and WebEx; design collaboration tools like Figma; and group collaboration tools like Slack and Microsoft Teams to generate ideas, iterate with team members, distribute workload, and uncover blind spots. Just like testing, collaboration can also be synchronous or asynchronous, and I expect the trend of companies adopting and expanding tools for remote collaboration to continue to accelerate.
Uncertainties and Challenges
In remote UX research, it’s not always clear how remote methods will affect participants and results due to less control over the user environment. Moreover, the loss of control is not just with the user environment but also the tester’s workspace. According to a study conducted on UX practitioners by user research platform EnjoyHQ, 40% of 215 participants were not working remotely before the COVID-19 crisis, and 21% were struggling to balance parenting and work. With a growing trend of extending work-from-home policies, or even making them permanent, UX researchers face a blurring of their professional and personal space, which leads to additional labor, such as managing children, dealing with faulty Internet connections, and learning new conferencing, collaboration, and testing tools. This is exacerbated by the anxiety of potential layoffs and the lack of human contact with team members.
A remote test means both participant and researcher have to learn how to communicate using these tools and also make the interaction as natural as possible. If this is a multicultural project, there can be additional challenges, like cultural and linguistic differences, which make it difficult to elicit responses.
In my project, some participants never responded; one test failed because the Internet connection was too slow; and one user had to repeat the test because they got disconnected. Although I had conducted pilot tests to check the software and the final test plan, it is not possible to anticipate all possible issues.
As new graduates, young scholars, and professionals who are trying to establish their ethos, it is better to be overprepared and to overcommunicate. Create a backup plan for technology issues, such as alternative connection options; conduct pilot tests; and communicate clearly with the participant about software installation to alleviate some of the stress associated with the challenges of remote testing.
Opportunities
With the progress of information and communication technology, the world has become smaller and the UX research tool kit has expanded. Remote UX research will continue to open doors for increased collaboration between cultures and spaces which are physically distant.
This will be useful for academic and industry researchers who are interested in intercultural technical communication and multicultural UX. Instead of seeing travel bans and shrinking budgets as barriers, we can see remote research as an opportunity to learn new tools, grow the user base, and reflect on how we can make the most of this challenging situation and derive newer perspectives.
Virtual tests can help us reach a larger pool of participants we normally wouldn’t have considered, and that could give us surprising insights. Many multinational companies with markets and offices in other parts of the world can come together to test products internationally. Besides testing and analysis, UX researchers can also present research online, either in the form of prerecorded videos or live presentations. Videoconferencing platforms like Zoom and GoToMeeting will likely be popular for synchronous online presentations, as will virtual whiteboard and visual collaboration tools like Google Draw, Microsoft Visio, Sketch, Mural, and Miro.
Nielsen says, based on current trends, the scope of many user bases is broadening. For example, m
ore elderly people are using videoconferencing tools during social distancing to prevent isolation and stay in touch with their loved ones. There is limited data available, however, on the UX of senior citizens. As the target users of many products are expanding, Nielsen says, “those interfaces have to be simplified dramatically for the senior citizens to really use them well.” This is a good opportunity for UX researchers to study elderly user behavior and design interfaces that take into consideration their UX, as well as to make more accessible products or explore new research areas.
Conclusion
Remote testing has become a preferred method for many UX researchers who have access to new technologies and testing tools. While there are unique changes happening in workspaces and new user ecologies due to social distancing rules, we are witnessing a significant shift to remote UX research, which is becoming more normalized. Researchers acknowledge the significance of human contact and in-person contextual inquiry but also realize the importance of being responsive to our environment and exploring the benefits and convenience of remote methods.
I believe remote UX research is here to stay, and the sooner researchers equip themselves with remote testing skills, the more successful they will be in continuing with their projects and understanding current UX.
MEGHALEE DAS (Meghalee.Das@ttu.edu) is a PhD student in technical communication and rhetoric at Texas Tech University. She is interested in intercultural technical communication, UX, digital rhetoric, social justice, and creative pedagogical practices.
References
Dhanagopal K. Understanding our users: A comparative analysis of the techniques we can use. Intercom. 2018;7.
https://www.stc.org/intercom/2018/11/understanding-our-users-a-comparative-analysis-of-the-techniques-we-can-use.
Gothelf J. Lean UX: Applying Lean Principles to Improve User Experience. Sebastopol, CA: O’Reilly Media; 2013.
Laubheimer P. How UX professionals collaborate on deliverables. Nielsen Norman Group. 3 January 2016. https://www.nngroup.com/articles
/ux-deliverables-collaboration.
Nielsen J. Discount usability: 20 years. Nielsen Norman Group. 13 September 2009. https://www.nngroup.com/articles/discount-usability-20-years.
Nielsen Norman Group. Can UX be 100% remote? [Video]. YouTube. Published 10 April 2020. https://www.youtube.com/watch?v=MG64QcjRL8c.
Quintero S. How the COVID-19 crisis is affecting UX teams. EnjoyHQ. 27 April 2020. https://blog.getenjoyhq.com/how-the-covid-19-crisis-is-affecting-ux-teams.
What can UX do to help with Covid-19? [Video]. YouTube. Published 3 April 2020. https://www.youtube.com/watch?v=WpfR0XXqcZE&feature=emb_title.
