doi: https://doi.org/10.55177/tc001245
By Satu Rantakokko
ABSTRACT
Purpose: Extended reality (XR) is an umbrella term for the many ways that we can now design 3D, interactive, and real-time environments as in combining virtual and real elements, and experience immersion in a completely virtual reality (VR). The use of XR is increasing in popularity across a range of industries. Although researchers are becoming increasingly interested in the benefits and challenges of using XR to convey technical instructions, more comprehensive research is required. I aim to address this need in the present article by introducing an affordance model of Technical Instructions in Extended Reality, the TIER model.
Two earlier categorizations, the affordances of technical instructions, and the phases of data handling in XR, formed the point of departure for this article. The analysis utilized a four-category model of affordances by Rantakokko and Nuopponen (2019) that comprised: accessing, finding, understanding, and relying on. Data handling in XR drew on a phase-based model by Rantakokko (2022) featuring: collection, processing, storage, transfer, combining, and presentation.
Methods: The two sets of categorizations were combined into a holistic model with an iterative process in order to offer a tool for analyzing and describing the possibilities and challenges that XR brings to designing technical instructions. The iterative process was conducted by adding examples from existing research into matrix tables to understand how the affordances of technical instructions and the phases of data handling in XR are connected.
Results: The TIER model is introduced with examples to illustrate how it can be used to view every phase of XR data handling in terms of the affordances of technical instructions based on the laws, regulations, principles of good guidance, and the design process.
Conclusion: The TIER model can be used as a tool for an organized, step-by-step design process as well as testing XR-based technical instructions to ensure that the features of XR support the intended affordances of technical instructions.
KEYWORDS: technical instructions, affordances, extended reality, mixed reality, virtual reality
Practitioner’s Takeaway:
- XR offers new possibilities (i.e., interactive, 3D, and real-time instructions) and challenges (i.e., privacy-related challenges) for designing technical instructions.
- More research is needed to fully benefit from the possibilities of various types of XR and avoid or minimize the new risks.
- The affordance model of Technical Instructions in Extended Reality (TIER) model is a tool for organizing the design process of XR-based technical instructions.
- The TIER model also offers an organized means for testing the XR-designed technical instructions.
- Although the TIER model is mainly focused on the perspective of the designers of technical instructions, it can also be used as a research framework.
INTRODUCTION
For as long as there have been technical devices, there have been instructions on how to use them. These instructions can take many forms, such as face-to-face guidance, paper manuals, online-instructions, videos, audios, pictures, etc., as long as they fulfill their main purpose: to guide users preparing to use or operate a technological product. However, the recurring challenge is how to encourage users to read the instructions they need, especially if they opt to immediately begin to test the technical device. According to Novick and Ward’s research (2006), paper manuals were the most rejected form of instructions; it was more common for users to give up on the project rather than use the printed manuals.
Professional communicators have noted this challenge and have searched for novel solutions. One of many such efforts is related to emerging technologies. Extended reality (XR) is considered a promising medium for delivering technical instructions and capturing the users’ interest. XR is an umbrella term for all the three-dimensional, interactive, and real-time environments with virtual elements (see, e.g., Fast-Berglund, Gong, & Li, 2018, p. 32). It includes mixed reality (MR) and virtual reality (VR). Furthermore, mixed reality includes augmented reality (AR) and augmented virtuality (AV). These concepts and their relations are illustrated in a reality-virtuality continuum by Rantakokko (2022, adapted from Milgram & Kishino, 1994, p. 1321) in Figure 1.
Actual reality is the natural environment without virtual elements in it. Instead of diving deep in the ever-lasting philosophical debate about the concept of real, I quote the Merriam-Webster online dictionary (2022), which defines real among other things as “having objective independent existence”; “not artificial, fraudulent, or illusory”; “occurring or existing in actuality”; and “existing as a physical entity and having properties that deviate from an ideal, law, or standard.” With these definitions, from the point of view of this research, actual reality can be defined as “the totality of real things that occur or exist in actuality as physical entities which are not artificial, fraudulent, or illusory, and which have objective independent existence.” XR breaks the boundaries of the perception of reality. At its best, added virtual elements are detected and experienced in a similar way to real elements. Yet, instead of existing in physical reality, they only exist as virtual enhancements.
VR aims to create a completely artificial, virtual experience with no real elements included. It is useful, for example, for virtual training if skills need to be practiced in a safe environment or with less costs (Hoedt et al., 2017). MR includes both real and virtual elements. In mixed reality, two types can be separated: augmented reality (AR) and augmented virtuality (AV). AR is based on actual reality that is enriched with virtual elements, while AV is based on a virtual environment enriched with real elements (see, e.g., Ternier et al., 2012, p. 2146). AR can be used, for example, to offer phase-by-phase manual assembly instructions, pointing out to the user which component to assemble next (de Amicis et al., 2017). AV enables, for example, distance meetings in a virtual, interactive meeting room so that participants can see each other live while interacting with virtual objects in the meeting room (Regenbrecht et al., 2003).
In previous research, the focus has typically been on a certain type of XR, in relation to a specific task (see, e.g., Albert et al., 2014; Zauner et al., 2003; Doshi et al., 2017). However, often, optimal benefits are achieved when combining different types of XR. Furthermore, it is important to consider the common and general features of XR as a whole to gain an understanding of how this technology works and how it can be utilized to support the features of well-functioning technical instructions. Consequently, there is a need for a comprehensive and more in-depth view on the subject.
Using XR to deliver technical instructions requires new approaches. For example, according to Burova et al. (2020, p. 3), “existing, traditional technical documentation content does not work when viewed with AR glasses.” This is due to the fact that there is too much text. This characteristic renders relevant passages difficult to find and understand. Commencing the use of an emerging form of technology as a tool requires careful consideration from the designers of technical instructions to fully utilize its possibilities and to avoid mistakes. More knowledge of the features and nature of XR as a medium are required to understand its benefits as well as possible hindrances and effects. These needs can be addressed with a theoretical model that serves as a tool for the designers of XR-based technical instructions to analyze both the possibilities and challenges posed by using XR. In this article, I propose such a model to extend the discussion of using XR as a tool to deliver technical instructions.
The model I introduce in this article, for the purpose of analysis, is for professionals of technical communication who are planning to use XR as a medium for delivering technical instructions for a target user group. This model offers a comprehensive view on XR as a tool and an organized way to view the design process of XR-based technical instructions.
In general, using XR as a medium to deliver technical instructions offers multiple novel possibilities. Through using XR, technical instructions can be made interactive, context-related, and three-dimensional, and they can also combine multiple virtual elements, such as text, pictures, and videos related to the real environment (Azuma, 1997). This increases the likelihood that the user receives vital information when required. Furthermore, since the instructions are available during the tasks and without interrupting the task-related actions in order to read them, it is more likely that users will actually use the instructions.
On the downside, XR requires certain equipment to function; currently this typically includes headsets or smartphones and tablets as a bare minimum. Despite technological advances in recent years, this equipment still has problems, such as the cumbersome nature of the devices and delays (see, e.g., Aukstakalnis, 2017). Another challenge is that the nature of XR is to constantly collect the data of users and their environment, thus bringing new kinds of cyber security challenges into the field (see, e.g., Piumsomboon et al., 2017, p. 36; Reilly et al., 2014, p. 275; de Guzman, Thilakarathna, & Seneviratne, 2019, pp. 8, 13).
With other media, such as paper manuals, video instructions etc., these challenges have not been significant concerning technical instructions since instructions do not usually include the collection of sensitive and personal data. In the case of XR, the collection of sensitive data is difficult, if not impossible, to avoid, and is thus vulnerable to attack. This risk of attack results from the fact that in order to situate virtual objects in the right place at the right time, XR equipment must track, for example, the user’s location, orientation, gestures, eye movements, and surroundings. XR equipment includes multiple sensors to fulfill these tracking demands (see, i.e., Aukstakalnis, 2017, pp. 20–21). For instance, as de Guzman, Thilakarathna, and Seneviratne (2020, p. 1) mentioned, the requirement of spatial understanding of the user environment leads to spatial maps, which may contain sensitive information that the user did not intend to expose. This sensitive information can be further utilized for unintended functionalities, such as aggressive localized advertisements or even theft. De Guzman et al. (2020) also stated that there are no mechanisms in existing MR platforms that would ensure user spatial data privacy. What makes this issue even more concerning is that this spatial data is perceptually unknown to the users. Instead, they are “oblivious about the captured spatial mapping, its resolution, and exactness” (de Guzman et al., 2020, pp. 1–2).
The affordance model of Technical Instructions in Extended Reality, the TIER model, includes two main elements: technical instructions and XR as the medium delivering them. I will discuss the possibilities and challenges of XR for delivering technical instructions by taking an affordance model of technical instructions (Rantakokko & Nuopponen, 2019) and the phases of data handling in XR (Rantakokko, 2022) as two essential elements of the TIER model and relating them to each other.
In the TIER model, the affordances of technical instructions are defined by Rantakokko and Nuopponen (2019) as accessing, finding, understanding, and relying on. Further on, the affordances are seen through the levels of rules, design, possibilities, and actualizing. This categorization offers a comprehensive view on technical instructions and the features they should have in order to fulfill their purpose of guiding a user of a technical device. These levels also enable viewing the design process from these main questions: 1) What laws, regulations, and principles are required to construct good guidance? 2) How should ideal technical instructions be designed? 3) How are designed affordances seen by the target user group? and 4) How will or will not these affordances be actualized in real use situations?
In a similar vein, the TIER model starts from the phases of data handling in XR as introduced by Rantakokko (2022): collection, processing, storage, transfer, combining, and presentation. This process-based view on the features of XR increases understanding on how data is handled in XR and how XR works as a medium for technical instructions. It also enables for focus to be placed on the possibilities and challenges that XR offers for technical instructions.
To assist the designers of technical instructions in creating functional XR-based instructions for their target group, these two essential elements are combined into a model that serves as a practical tool for the thorough examination and analysis of the possibilities and challenges that XR brings in the process of designing technical instructions. Furthermore, the TIER model offers a way to relate the common and central features of XR to the affordances of technical instructions to gain an organized and comprehensive view on designing XR-based technical instructions. With the help of the TIER model, designers of technical instructions can, for example, deliberate on the most effective way to present instructions to users or consider whether an internet connection would be available at the location in which the instructions are used. If the internet connection is not reliable or always guaranteed, this can disable the affordance of accessing and, therefore, make entire XR-based instructions unusable if the instructions are only made available online.
The structure of this article is as follows. First, I briefly introduce the two previous studies upon which the current research is based. This is in order to explain how the affordances of technical instructions and phases of data handling in XR are understood in this study. Second, I present a methodological process of developing the TIER model. Next, I introduce the TIER model in detail and provide examples of how it can be used. Finally, I discuss the conclusions based on the findings of this research.
TWO ESSENTIAL ELEMENTS OF THE TIER MODEL
In the field of technical communication, there are many novel solutions for diverse challenges (see, e.g., Jones & Gouge, 2017 about wearable technologies; Tham, 2018 about wearables and IoT). These solutions utilize different technologies, accumulated research, and special knowledge. However, I focus strictly on XR-based technical instructions as well as the possibilities and challenges of XR. In the next two subsections, I introduce two essential elements of the TIER model: the affordances of technical instructions and the phases of data handling in XR.
Affordances of Technical Instructions
In this article, I view technical instructions through the concept of affordance. As Treem and Leonardi (2012, p. 46) stated, “[affordance] helps to explain why people using the same technology may engage in similar or disparate communication and work practices.” Furthermore, Treem and Leonardi (2012) explained that the concept of affordance is useful in research on relationships between new technologies and social practices. The concept of affordance was coined by Gibson (1979), explaining how animals perceive their environment through uses. For example, rather than a stone itself, a monkey perceives a series of functions that the stone enables (i.e., how the monkey can benefit from the stone and its interaction). Since then, the concept of affordance has been a subject of a wide-ranging discussion and a plethora of research ever since.
In the context of this research, affordances refer to the possible functionalities that objects, such as technical instructions, are designed to possess. Here, I also focus on how these possible functionalities may or may not be perceived, experienced, and actualized for the intended users and potentially unintended parties (e.g., cyber criminals). Furthermore, the affordances can be possible functionalities that are not intentionally designed to be actualized, or are inherent in the functionality, and thus able to be recognized and utilized. Based on the research of Rantakokko and Nuopponen (2019), the affordances of technical instructions are accessing, finding, understanding, and relying on. The tool mediating technical instructions to the user should be capable of delivering the designed affordances.
Based on Lanamäki, Thapa, and Stendal ’s (2016, pp. 127–128) suggestion that affordances can be understood from four alternative stances, Rantakokko and Nuopponen (2019) took a broad perspective on the concept of affordance. In their view, each of the affordances of technical instructions can be seen at four levels: the level of rules, the level of design, the level of possibilities, and the level of actualizing. These levels are all present and valid at the same time, instead of being alternative, which makes the view broader. The affordances and their levels are illustrated in Figure 2.
The main point of this structure of affordances and their levels is that in order for technical instructions to be accessed when needed, they first must be designed and made available to the user. After accessing the instructions, the user should be able to easily find the necessary information. Understanding the information requires that it is well designed for the user, and it is expressed in an understandable way. To keep using technical instructions and operate based on the information received, the user must be able to rely on the instructions (Rantakokko & Nuopponen, 2019, pp. 58–61).
The four levels take the review of affordances deeper. The level of rules defines the necessary features that technical instructions should include according to laws, regulations, and principles of good guidance. The level of design is the level at which the desired affordances are established in technical instructions for the user from the perspective of the designers of technical instructions and their knowledge about the potential users. The level of possibilities focuses on the users and the affordances that the instructions offer for them. Finally, the level of actualizing defines actual situations, where affordances either will or will not be actualized.
From the perspective of Rantakokko and Nuopponen (2019), the concept of affordances covers: 1) all the possible functionalities that technical instructions are required to include to fulfill their purpose; 2) all the possible functionalities that may be designed; 3) all the possible functionalities that the users see; and 4) all the possible functionalities that are actualized in real situations.
Rantakokko and Nuopponen’s (2019) research is the initial point of departure for this article because it focuses on features of functional technical instructions. It offers a view on the possible functionalities of technical instructions that enable instructions to fulfill their purpose in guiding the use of a technological device. This study focuses on various types of technical instructions that can be delivered via the medium of XR. They can, for example, be assembly instructions, teaching instructions on how to use a technological product, or safety-related instructions of products that include real-time warnings.
The Phases of Data Handling in Extended Reality (XR)
Data and how it is handled in XR technology can be divided into two types. Instruction data refers to the technical instructions delivered by XR. Collected data covers the data collected by the XR equipment, i.e., the user’s position and location (Rantakokko, 2022). In interactive XR-based instructions, these two types of data need to be combined. This occurs, for example, in order to enable the positioning of virtual elements in the context of the instructions – pinpointing optimal placement regarding the user’s position and orientation. As such, there is nothing new in collecting information to design the instructions, but this study’s focus on the data that XR equipment, such as VR or AR glasses, collects from the users and their circumstances (personal, environment, context, etc.) is new.
In XR, the data is handled in several phases: collection, processing, storage, transfer, combining, and presentation (Rantakokko, 2022). These phases are illustrated in Figure 3.
Not only must data be collected and processed in order for XR equipment to operate, but it must also be stored somewhere for user access. Furthermore, it must be transferred to the user and sometimes for processing and storage as well. Finally, it must be combined with other data, such as virtual elements with the real environment, and it must be presented to the user. In Figure 3, the transfer phase is presented beneath other phases because it can be performed throughout.
Phases of data handling is the second essential element of this study as it offers an organized, process-based view on the features of XR, which enables analyzing them in relation to the features of technical instructions. With this combination, it is possible to analyze and describe the possibilities and challenges that XR brings in the process of designing technical instructions. In the next section, I introduce the methodology I used to combine these starting points in order to create the affordance model of Technical Instructions in Extended Reality, the TIER model.
METHODS
To combine the two essential elements of the TIER model, I followed an iterative process in which the existing empirical examples were placed into matrix tables until a point of saturation had been reached. Table 1 is an example of such a table, focusing on the phase of data collection. The terms used in this table are explained in the Results section, where the complete table establishing the TIER model is introduced. Examples of the benefits are identified as user-friendly effects in Table 1. Table 1 also acknowledges the challenges faced during the data collection phase in XR, including privacy and security issues; these issues were also highlighted in previous research (Rantakokko, 2022).
In all of the matrix tables, like Table 1, the possibilities of XR-based technical instructions were related to the three-dimensional, real-time, and the interactive nature of XR. At the same time, privacy- and security-related issues were considered to be significant challenges due to continuous data collection by the software and devices.
RESULTS
I developed the TIER model to create a way to analyze the possibilities and challenges that XR as a medium brings to the process of designing technical instructions. The TIER model offers a way to approach this objective by enabling a holistic perspective on the process in two ways. First, the view of the affordances is comprehensive. The four levels of affordances of technical instructions take into account the aspects of rules, design, possibilities, and actualizing. Second, the TIER model views XR as a whole, focusing on the common features rather than differences between its types, such as VR and AR. While the model was constructed on the premise of empirical findings of existing research, the model itself, in its current state, has not yet been applied to further empirical studies.
The TIER model is especially designed for technical communication professionals who are planning to use XR technology in the delivery of technical instructions. The practical benefit of this model is that it offers an organized way to consider both the affordances of technical instructions and the process of XR data handling, phase by phase. Designers of technical instructions may include their own requirements and goals in the matrix, choose one or several types of XR for different tasks, and ensure that all the necessary aspects are taken into consideration. The application of the TIER model can save a lot of time and money while avoiding potential mistakes. As stated, problems that may occur include privacy and security issues that emerge from constant data collection and the lack of user familiarity with the system, such as leakage of passwords or private conversations.
As Burova et al. (2020, p. 2) highlighted in relation to AR-based industrial instructions, “the development of industrial AR solutions should be iterative, with systematic testing, to identify the best strategies of information presentation and interaction for accessing the AR content.” Furthermore, Burova et al. (2020, p. 2) stated “errors in design may lead to dramatic consequences.” These points are valid for XR-based technical instructions as a whole and highlight the importance of an organized design process.
The basic elements of the TIER model are introduced in Figure 4. The levels of rules, design, possibilities, and actualizing are valid for all the affordances, even though, for clarity, they are listed only via the affordance of accessing.
The process of designing XR-based technical instructions are organized as follows:
1) Appraising valid laws, regulations, and recommendations for the type of instructions with each affordance (the level of rules)—for example, what is required for the instructions to be accessed;
2) Considering all the desired features and the ones that should be prevented, compared to affordances (level of design, focusing on technical instructions)—for example, what kind of safety-related guidelines are needed in the instructions;
3) Comparing the results with the phases of data handling in XR in detail to ensure that each phase supports each affordance (level of design, focusing on XR as a medium for instructions)—for example, how the privacy and security issues are related to accessing data at every phase of the XR data handling; and
4) Testing that the users recognize the affordances present in the design (level of possibilities), and that the affordances can be actualized in real situations (level of actualizing)—for example, do the users understand phase by phase instructions in a maintenance task, and can they perform the task with the instructions.
The levels of possibilities and actualizing require empirical research because these levels are contingent on the users and real use situations. Because this study serves as an introduction to the TIER model. This article focuses on the first two levels of the model—rules and design. Still, the levels of possibilities and actualizing, defined earlier are important.
General Structure of the TIER Model
The basic structure of the TIER model is presented in Table 2. It is based on the affordances of technical instructions combined with the phases of XR data handling. The affordances of technical instructions (accessing, finding, understanding, and relying on) are listed in the table heading. The phases of the XR data handling (collection, processing, storage, transfer, combining, and presentation) are introduced in the first column of the table. The table relates these aspects to each other by putting each phase of XR data handling in relation to every affordance of technical instructions as well as illustrates the impact that any phase of the XR data handling has on each affordance to being actualized for the users.
The terms used in the table, and how the concepts they represent are related to each other, are as follows:
Occurs:
When the affordance first emerges, it is marked as “occurs” in the table. The affordances of accessing, finding, and understanding occur when data is collected. The affordance of accessing also occurs for a new data set in the phase of combining.
Exists:
The affordance of relying on is marked as “exists” on every row. This is because the affordance can occur even before the XR-based technical instructions are designed, based on, for example, the users’ preconception about the technology of XR. Furthermore, relying on the instructions is a precondition throughout the process in order for the user to use the instructions in the first place. The effects of the phases of the XR data handling can be either positive or negative on the affordance of relying on, but if it does not exist, the technical instructions will probably not be used.
Improves:
Some phases improve the probability for the affordance to be actualized. For example, when the data is collected, the affordance of accessing occurs, since, at that very moment, there is data to be accessed. However, for the XR equipment to operate, the data needs to be processed. This makes the affordance of accessing more realistic and even more so in the phase of presentation.
No effect:
Affordances of technical instructions, excluding the affordance of relying on, are temporally related to each other. This is because, for example, to be able to find the data in the instructions, the user must have already accessed the technical instructions, and, therefore, data storage has no significant effect on finding the data. Next, I introduce the TIER model in detail and include some examples that focus on the model’s first two levels—the level of rules and the level of design.
Collection Phase of XR Data Handling
The first phase of the XR data handling is data collection. Data collection is an ongoing process because XR equipment needs to constantly collect data about the user and their surroundings to be able to function (see, e.g., Piumsomboon et al., 2017, p. 36; Reilly et al., 2014, p. 275; de Guzman et al., 2019, pp. 8, 13). Table 3 introduces some examples of the various aspects that may be important to consider at this phase. Note: The numbered items within the subsequent tables in this article correspond to the numbered items in the narrative descriptions of the tables.
With the TIER model, this phase can be organized according to the respective levels.
Level of rules:
1) Considering the instructions and the target group, according to laws and regulations, what kind of data is allowed to be collected? For example, if the users are wearing the XR equipment, it may record all passersby, including their conversations, without their knowledge or consent awareness of the users or passersby. Or if the users log into an application or computer, their passwords may be recorded.
2) It may also be important to take measures to collect only the necessary and allowed data and solve the challenge regarding what to do with the rest. Furthermore, you may need to take measures to protect the data collected with, for example, input sanitization techniques.
Level of design:
3) What kind of data needs to be collected, according to the purpose of the instructions? For example, should you track the user’s eye gaze?
4) What kind of equipment can manage the collection of needed data in a reliable way? For example, if eye tracking is needed, which XR headsets enable this?
Regarding the affordance of accessing, data collection is the most important phase of the XR data handling. In order for access to take place, all the necessary data must first be collected. The instruction application could, for example, collect data from safety critical situations in a work environment, such as information regarding increased amounts of radiation or rising heat. In this way, safety-critical data is obtained, saved, and stored for retrieval and review at any time. The affordances of finding and understanding also occur at the phase of XR data handling, but data collection has no significant effect for them. Actions regarding data collection can affect the user’s inclination preference to rely on XR-based technical instructions. If the user is, for example, given options to block some data from being collected, it may have a positive effect on relying on the instruction application. Thus, users can trust that not every word they say, every object they look at, or every physical or emotional reaction they have during the work task, is recorded and in the hands of the employer or an unknown outsider.
Processing Phase of XR Data Handling
After the data has been collected, it is then processed. This is vital for delivering outputs accurately, interactively, and in real-time, as de Guzman et al. (2019, p. 15) explained. Therefore, processing the data makes the affordance of accessing more realistic. In Table 4, there are some examples of considerations of this phase of XR data handling.
Level of rules:
1) According to laws and regulations, who is allowed to have access to data during processing? For example, if the data is processed on the servers of a third party.
2) Balancing with the guidance of good instructions and the limited space. For example, deciding which number of virtual elements is ideal without risking blocking the users’ view of important objects in their environment, so that everything needed is easy to find.
3) Balancing the most easily understood objects with the required processing capacity of the equipment. For example, deciding on acceptable video quality so that the equipment can process data efficiently.
4) To make the XR-based instructions reliable for the users—how can it be ensured that processing the data is not too slow, causing delays? For example, does the equipment have enough processing capacity for the instructions designed?
Level of design:
5) What kinds of virtual objects are needed to be accessed by the users during the task at hand? For example, do they need real-time and interactive recognition of different parts of the technical device?
6) In which places can virtual objects be easily found? If real-time recognition of different parts is needed, how can they be situated in the user’s view? Should the XR-based instructions recognize and point out the parts themselves, or would the recognition process demand the user to isolate and show different parts for them to be recognized? The latter could cause longer assembly processes and frustration, while the first could be less accurate.
7) How can understanding be supported in relation to data processing? For example, how could it be ensured that the objects are presented in the right place and at the right time? Delays could mean that the application indicates the wrong part, which would make understanding difficult.
8) Considering the amount of data and the purpose, is it more reliable to process the data with the equipment or transfer it elsewhere to be processed? For example, is there too much data for the equipment to process, and if sent elsewhere, is the connection guaranteed?
The phase of data processing is a vital enabler of the possibilities of XR delivering technical instructions. This phase raises questions about what kinds of data are needed to be processed in order to make the desired functionalities possible and how this data can be balanced with limitations of the equipment. This matter has an effect on all the affordances. There may be a need to return to this phase during designing the phase of data presentation to ensure that the designed elements and the realities of processing support each other.
Storage Phase of XR Data Handling
The collected and processed data may be stored (see, e.g., Schmalstieg & Höllerer, 2016, p. 3). The phase of data storage allows for continuity; the instruction data and the relevant collected data are preserved for future needs. In Table 5, there are some examples of the issues to be considered at this phase of XR data handling.
Level of rules:
1) According to laws and regulations, what kinds of data are allowed to be stored? Often, unnecessary data about the users and their surroundings is forbidden to be stored.
2) How should the data be protected while it is stored to deny access from unauthorized parties? The data stored is vulnerable for attacks, and therefore, measures for protection are important to consider.
Level of design:
3) According to purpose, what kinds of data need to be stored? For example, the data collected when the XR equipment was used can include conversations. Some conversations could be necessary to store, such as direct instructions for a specific task, but some could be private or unnecessary.
4) Where should the data be stored? If there is a need for data storage solutions by a third party, it is important to consider that this party is reliable and has good protection for the data stored.
The phase of data storage is important for the continuity of using the instructions. Data storage provides an opportunity for access to the needed data whenever it is required. It, therefore, improves the possibility of the affordance of accessing to be actualized. This phase has no significant effect on the affordances of finding and understanding. However, for the affordance of relying on, it is important to take measures to protect the data and decide where to store it. If the users are not aware of what data related to them is stored and where, it could reduce their reliance on the XR-based instructions.
Transfer Phase of XR Data Handling
Data transfer may be needed for the purposes of processing or storage of the data. It is also needed for collaboration, for example, when using distant guidance via XR equipment (see, e.g., Reilly et al., 2014, p. 275; de Guzman et al., 2019, p. 30). Some examples are introduced in Table 6 to demonstrate how to use the TIER model in this phase of XR data handling.
Level of rules:
1) What instructions are required by laws and regulations to be available to the users at all times? For example, safety-related instructions are usually required.
2) How to make sure that the instructions are available when needed? For example, which instructions can be guaranteed via real-time data transfer only, and which instructions should be available despite the success of internet connection?
Level of design:
3) For what purpose is data transfer needed? For example, data processing and storage, or for collaborative use.
4) How can the risk of interruptions and failures be minimized during data transfer? For instance, protecting data transfer, using encrypting, etc.
Data transfer enables the processing and storage of the data in another location and collaboration in XR environments. At this phase, all needs for data transfer should be identified. Data transfer can improve the possibility of accessing the instructions by, for example, enabling distant guidance via XR equipment. However, if a good connection cannot be guaranteed at the location, XR-based instructions should be available offline as well. Transfer has no significant effect on the affordances of finding and understanding the data. When it comes to the affordance of relying on, fluently working instructions that are available when needed can enhance the affordance of relying on.
Combining Phase of XR Data Handling
Combining data is important for the basic functions of XR. The user’s environment, location, position, and movements need to be combined without too much delay with virtual elements in order to enable the XR-based instructions to assist the user in a real-time and interactive manner (see, e.g., Graig, 2013, pp. 51–52). Data combining enables situation-related instructions by recognizing important objects in the user’s environment and combining them with instructions on how to use these objects. These are new possibilities that other formats of technical instructions cannot offer. Therefore, there is a delay in laws, regulations, and guidance in relation to the evolving situation. Some examples of considerations at this phase of XR data handling are presented in Table 7.
Level of rules:
1) Data combining reopens the questions about data collection—after combining data, are there now new kinds of data sets that would not be allowed to be collected? For example, if it is necessary to measure the users’ physical reactions to make sure they are not endangered in challenging conditions, and their eye gaze is measured as well, can there be a new set of data that reveals their reactions or emotions?
2) How can sensitive data be made difficult to find for unauthorized parties? For example, XR equipment helps to prevent passersby from seeing log-in information when it is shown only with the device the user is wearing.
3) How can sensitive data be made difficult to understand for unauthorized parties? For example, encryption technologies could help to protect data even if an unauthorized party obtains access to it.
4) Informing users of all the central aspects concerning data combining and collection, such as what is collected and what kind of data sets can be formed with combining data, where is it stored, who has access to it, and how is it used, can enhance the possibility of relying on the XR as a medium for technical instructions.
Level of design:
5) What kind of data sets need to be combined for the user to access? For example, rising heat in the work conditions and possible actions that the user should take if this happens.
6) How can the instructions needed be rendered most easy to find with the help of data combining? For example, would tracking the user’s eye movements help situate the data needed at the ideal spot?
7) How can data combining improve understanding? For example, would adding a virtual color-coding help realize the structure of a complicated technical device?
8) How can the risks and benefits of data combining be balanced? For example, combining data can add to accessing, finding, and understanding, but it can also add to the risks of private and sensitive data ending up in wrong hands.
Data combining enables and supports some of the most significant possibilities that XR can offer. At the same time, there is a risk with its ability to combine sensitive data to gain more knowledge about the user, an organization, or a bystander. It is important to give deep consideration of data collection from the aspect of data combining to make sure that there will not be any surprises regarding what kinds of leaked information data combining could lead to. Lastly, data combining influences every affordance.
Presentation Phase of XR Data Handling
Finally, the data is presented to the users, and the possibilities that all the previous phases offered become visible to the users (see, e.g., de Guzman et al., 2019, p. 18). The phase of data processing is also important to consider when designing data presentation. For example, it is relevant to realize the processing power of the equipment used and the possibilities and limitations to determine if there is a need for compromises in relation to what is desired in the phase of presentation. For instance, would increased privacy and security measures mean decreased exactness regarding the informational content and accuracy of the instructions? It is important to notice, that unlike any other instruction format, XR-based instructions can cause safety risks by blocking the users’ view and thus making it impossible for them to see relevant objects from their environment. However, XR can decrease the risks by carefully designing the presentation to assist the user in noticing situations in their outside environment that need attention, such as rising heat in the device they are using or poor air quality. This highlights the importance of a carefully designed data presentation and the importance of how XR-based instructions differ from other instructional formats. Table 8 introduces examples of the considerations on the data presentation phase.
Level of rules:
1) How can access to the required instructions be ensured at all times? For example, making sure that the safety-related warnings are always available even if the connection fails.
2) Considering the nature of XR equipment, can the instructions be created in a non-disturbing way, while still making them easy to find? For example, avoiding long texts and blinking images.
3) How can sensory overload that could diminish understanding be avoided? For example, avoiding the use of many restless elements which can cause fatigue to the user’s eyes.
4) How can the data presentation be protected from hostile parties in order to maintain reliability? For example, if the data is attacked at any point of XR data handling, the presentation could be rendered unreliable.
Level of design:
5) How can it be ensured that the needed instructions are easy to access at all times? For example, that the selected equipment is not only suitable for the work conditions, but also comfortable to use.
6) How can the instructions be made easy to find? For example, would visual, auditory, or tactile effects be most effective for the user to notice safety-related warnings most accurately, or possibly a combination of them?
7) What is the most understandable way to guide the user through the task at hand? For example, text, symbols, videos, etc.
8) How can the data presentation be made functional and uninterrupted? For example, designing instructions for different work phases to be activated when they are needed.
Data presentation is the visible part of technical instructions to the user. Well-functioning instructions give the needed data at the right time, but do not overload the user with unimportant data, blocking their view, or demanding their attention, unless it is absolutely crucial. Presentation makes accessing the instructions easy and realistic for the user. Well-designed presentation also has a significant effect on finding and understanding the instructions. Fluent, functional, and uninterrupted presentation of the instructions enhances the chances that a user can actually rely on them.
CONCLUSIONS
In this article, I have introduced the TIER model, a tool that helps analyze the possibilities and challenges that XR brings to designing technical instructions. I introduced this model at a general and theoretical level with the help of examples of what to consider at each phase of the data handling in XR and in relation to every affordance.
I formed the TIER model by combining the two essential elements of the TIER model—the affordances of technical instructions and the phases of data handling in XR. I focused on the TIER model’s first two levels, level of rules and level of design, out of its four. However, the TIER model’s remaining two levels, level of possibilities and the level of actualizing, are important when testing the instructions to see which affordances the users notice and if the affordances are actualized in real use situations. This helps to improve the instructions designed, if needed.
The TIER model aims to benefit the designers of XR-based technical instructions by offering a tool for an organized design process that focuses both on the well-functioning technical instructions and the possibilities and challenges of XR as a medium for the instructions. By making sure that all the necessary aspects are taken into consideration, crucial mistakes and wasted resources could be spared. The strength of the TIER model is its versatility. It can be used to design the instructions as well as testing them, and in addition to the holistic view in XR, it can be used to view the details of different forms of XR related to technical instructions as well. To my knowledge, there are no other models that combine technical instructions and XR to increase the understanding of XR for mediating technical instructions. However, as XR becomes more common due to technological advances and reductions in cost, it is important for technical communicators to conduct more research in this area.
There are some limitations in this research. At this point, it covers only the levels of rules and design, and is, so far, only theoretical. It is yet to be tested with empirical research. However, the concept of affordance is based on the idea that the users can see and act on various possible functionalities in the objects, despite of what they are planned for. This requires empirical studies that focus on the levels of possibilities and actualizing, making it important to test the TIER model in real design processes of technical instructions. It would also be fruitful to undertake studies of the effects of XR-based technical instructions, conducted with different methods but keeping a wide perspective of XR as a whole.
Many challenges, such as privacy and security risks, can be minimized by taking action during the early stages of XR-based technical instruction development. There are signs that privacy and security issues will increase in the future, when the technology becomes more ubiquitous and widespread, and these issues may be exacerbated with AI or data mining. Thus, more research is needed on the issues of privacy and security.
Furthermore, to fulfill its purpose in guiding technical communication professionals who plan to use XR as a tool to deliver technical instructions, research comparing different kinds of solutions and technologies with their benefits and deficits is needed.
REFERENCES
Albert, A., Hallowell, M. R., Kleiner, B., Chen, A., & Golparvar-Fard, M. (2014). Enhancing construction hazard recognition with high-fidelity augmented virtuality. Journal of Construction Engineering and Management, 140(7). doi: 10.1061/(ASCE)CO.1943-7862.0000860
Aukstakalnis, S. (2017). Practical augmented reality: A guide to the technologies, applications, and human factors for AR and VR. Boston: Addison-Wesley.
Azuma, R. (1997). A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4), 335–385. https://www.cs.unc.edu/~azuma/ARpresence.pdf
Burova, A., Mäkelä, J., Hakulinen, J., Keskinen, T., Heinonen, H., Siltanen, S., & Turunen, M. (2020). Utilizing VR and gaze tracking to develop AR solutions for industrial maintenance. CHI ‘20: Proceedings of the 2020 CHI Conference on Human Factors in Computing System, pp. 1–13. doi: 10.1145/3313831.3376405
de Amicis, R., Ceruti, A., Francia, D., Frizziero, L., & Simões, B. (2017). Augmented reality for virtual user manual. International Journal on Interactive Design and Manufacturing (IJIDeM), 12(2), 689–697. doi: 10.1007/s12008-017-0451-7
De Guzman, J. A., Thilakarathna, K., & Seneviratne, A. (2019). Security and privacy approaches in mixed reality: A literature survey. ACM Computing Surveys, 52(6), article 110, 1–37. doi: 10.1145/3359626
De Guzman, J. A., Thilakarathna, K., & Seneviratne, A. (2020). Conservative plane releasing for spatial privacy protection in mixed reality. Cornell University. arXiv:2004.08029
Doshi, A., Smith, R. T., Thomas, B. H., & Bouras, C. (2017). Use of projector based augmented reality to improve manual spot-welding precision and accuracy for automotive manufacturing. The International Journal of Advanced Manufacturing Technology, 89(5–8), 1279–1293. doi: 10.1007/s00170-016-9164-5
Fast-Berglund, Å., Gong, L., & Li, D. (2018). Testing and validating extended reality (XR) technologies in manufacturing. Procedia Manufacturing, 25, 31–38. https://doi.org/10.1016/j.promfg.2018.06.054
Gibson, J. J. (1979). The ecological approach to visual perception. Boston: Houghton-Mifflin.
Graig, A. B. (2013). Understanding augmented reality: Concepts and applications. Amsterdam: Morgan Kaufmann.
Hoedt, S., Clayes, A., van Landeghem, H., & Cottyn, J. (2017). The evaluation of an elementary virtual training system for manual assembly. International Journal of Production Research, 55(24), 7496–7508. doi: 10.1080/00207543.2017.1374572
Jones, J., & Gouge, C. C. (2017). Guest editors’ introduction wearable technologies and communication design. Communication Design Quarterly, 5(4), 4–14. https://doi.org/10.1145/3188387.3188388
Lanamäki, A., Thapa, D., & Stendal, K. (2016). When is an affordance? Outlining four stances. In L., Kavanagh D., Kelly S., Orlikowski W., Scott S. (Eds.). Beyond Interpretivism? New Encounters with Technology and Organization. IFIP Advances in Information and Communication Technology, 489, 125–139, doi: 10.1007/978-3-319-49733-4_8
Merriam-Webster Dictionary (2022). https://www.merriam-webster.com/dictionary/real
Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems, E77-D (12), 1321–1329. https://search.ieice.org/bin/summary.php?id=e77-d_12_1321
Novick, D. G., & Ward, K. (2006). Why don’t people read the manual? Departmental Papers (CS). Paper 15. https://digitalcommons.utep.edu/cs_papers/15
Piumsomboon, T., Lee, G., Lindeman, R. W., & Billinghurst, M. (2017). Exploring natural eye-gaze-based interaction for immersive virtual reality. IEEE Symposium on 3D User Interfaces (3DUI), 36–39. doi: 10.1109/3DUI.2017.7893315
Rantakokko, S. (2022). Data handling process in extended reality (XR) when delivering technical instructions. Technical Communication, 69 (2) 75–96. doi: 10.55177/tc734125
Rantakokko, S., & Nuopponen, A. (2019). Laajennetun todellisuuden tarjoumat tekniselle viestinnälle – kohti teoreettista mallia. VAKKI Publications, 10, 53–66. http://www.vakki.net/publications/2019/VAKKI2019_Rantakokko&Nuopponen.pdf
Regenbrecht, H., Ott, C., Wagner, M., Lum, T., Kohler, P., Wilke, W., & Mueller, E. (2003). An augmented virtuality approach to 3D videoconferencing. The Second IEEE and ACM International Symposium on Mixed and Augmented reality, 2003. Proceedings, pp. 290–291. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1240725
Reilly, D., Salimian, M., MacKay, B., Mathiasen, N., Edwards, W. K., & Franz, J. (2014). SecSpace: Prototyping usable privacy and security for mixed reality collaborative environments. EICS ‘14: Proceedings of the 2014 ACM SIGCHI symposium on Engineering interactive computing systems, 273–282. doi: 10.1145/2607023.2607039
Schmalstieg, D., & Höllerer, T. (2016). Augmented reality: Principles and practice. Addison-Wesley.
Ternier, S., Klemke, R., Kalz, M., van Ulzen, P., & Specht, M. (2012). ARLearn: Augmented reality meets augmented virtuality. Journal of Universal Computer Science, 18(15), 2043–2164. http://www.jucs.org/jucs_18_15/ARLearn_augmented_reality_meets/jucs_18_15_2143_2164_ternier.pdf
Tham, J. C. K. (2018). Interactivity in an age of immersive media: Seven dimensions for wearable technology, internet of things, and technical communication. Technical Communication, 65(1), 46–65. https://www.ingentaconnect.com/content/stc/tc/2018/00000065/00000001/art00006
Treem, J. W., & Leonardi, P. M. (2012). Social media use in organizations: Exploring the affordances of visibility, editability, persistence, and association. Communication Yearbook, 36(1), 143–189. https://doi.org/10.1080/23808985.2013.11679130
Zauner, J., Haller, M., Brandl, A., & Hartman, W. (2003). Authoring of a mixed reality assembly instructor for hierarchical structures. The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings, 237–246. doi: 10.1109/ISMAR.2003.1240707
ABOUT THE AUTHOR
Satu Rantakokko, MA, is a doctoral student in Communication Studies at the University of Vaasa in Finland. Her research interests include technical instructions, extended reality, privacy and security issues, and affordances. Her dissertation focuses on benefits and deficits of extended reality as a medium to deliver technical instructions. She currently continues her research with a grant by South Ostrobothnia Regional Fund. She is available at satu.rantakokko@student.uwasa.fi.
