By Qian Li, Menno D. T. de Jong, and Joyce Karreman
ABSTRACT
Purpose: Although various researchers have paid attention to differences in the way Chinese and Western documents are structured, only few have investigated how Chinese and Western users differ in their way of using instructions. This study experimentally investigates the effects Chinese and Western manual structures have on Chinese and Western users, in terms of task performance, user satisfaction, and information selection.
Method: A 3×2 randomized experiment (N = 127) was conducted, with participants’ cultural background (Chinese living in China, Chinese living in the Netherlands, and Westerners) and manual structure (Chinese versus Western) as independent variables and task performance, user satisfaction, and information selection as dependent variables. Supported by a manual, participants performed tasks with Excel and afterwards filled out a satisfaction questionnaire. To investigate information selection, eye-tracking data were collected.
Results: Regarding task performance and user satisfaction, no significant main and interaction effects were found. Regarding information selection, our research confirmed some of the hypothesized differences between Chinese and Western users: Chinese users pay less attention to structuring elements (table of contents and headings) and more attention to visuals.
Conclusion: Our findings suggest that cultural adaptations of the structure of manuals do not really matter: Chinese and Western users perform equally well and have similar satisfaction scores with Western and Chinese manual structures. Paradoxically, the differences found in users’ information selection behaviors provided some support for the differences in document design practices. However, users of both cultures appear to be sufficiently flexible in using instructions that are not culturally adapted.
KEYWORDS: cross-cultural communication; cultural differences; user instructions; information selection; user satisfaction
Practitioner’s Takeaway:
- Cultural adaptations of the structural characteristics of user manuals do not seem to matter for the effectiveness and efficiency of task performance or users’ satisfaction with the manual or the product.
- Differences between Chinese and Western manual structures uncovered by content analytic research seem to correspond to differences in information selection behaviors of Chinese and Western users.
- Chinese users differ from Western users in two ways: the way they use structuring elements in user manuals (such elements are less important to them) and in the extent to which they rely on visuals in manuals (visuals are more important to them).
INTRODUCTION
In modern societies, technology is more pervasive than ever before. All kinds of technologies, varying from simple household devices to complex professional software applications, profoundly affect our lives. Technical communicators are the professionals responsible for bridging the gap between technological possibilities and users’ needs, desires, and preferences. An important way of doing so is by creating, facilitating, and maintaining forms of user support. The nature of user support has developed in various directions, including video instructions (Van der Meij, 2018; Van der Meij & Van der Meij, 2013), helpdesks (Robles, 2018; Van Velsen et al., 2007), and user forums (Frith, 2014; Swarts, 2015), but written instructions are still one of the prominent ways of helping users.
Technology not only affects our private and professional lives, it also has dramatic effects on society as a whole. One of those effects is that it has globalized the world we live in. People’s mobility all over the world has rapidly increased and national markets for products are developing into global ones. As a result, manufacturers of technical products increasingly have to consider the possible effects of national and cultural differences on their products as well as on the user support for their products. Technical communication therefore has a strong tradition of research into translation and localization (Batova, 2018, 2019; Batova & Clark, 2015; Lentz & Hulst, 2000; Sun, 2006, 2012).
In addition, various authors focused on differences in document design practices between cultures. The research, so far, predominantly aimed at comparing Western and Chinese technical communication practices. This has practical and intrinsic reasons. Practically, technological developments in China and the country’s openness to the rest of the world have gone through spectacular developments in the past decades. China has developed from a closed country to an influential world player, and from a manufacturing to a high-tech developing country. Intrinsically, comparing Western cultures with China is interesting because the cultural distance is assumed to be quite large.
Using variations of content analysis, various researchers tried to make sense of differences in document design practices (Barnum & Li, 2006; Ding, 2003; Dragga, 1999; J. Wang, 2007; Q. Wang, 2000; Y. Wang, & Wang, 2009; Yu, 2009). The results of these studies complement each other and sometimes point in different directions, which might be attributed to their small-scale data collection, variety of documents used, and rather informal analysis approach (without explicit coding schemes and assessments of inter-coder reliability).
To overcome these shortcomings, Li et al. (2020) conducted a comprehensive content analysis, comparing 50 Chinese and 50 Western manuals for household appliances. In their coding scheme they incorporated the insights from all earlier content analytic studies comparing Chinese and Western documents. They focused on three aspects of user instructions: content, structure, and visuals, leaving out style because it would be too hard to reliably assess in a quantitative content analysis. Their results confirmed that there are cultural differences in all three aspects. Regarding content, the overall difference is that Chinese manuals appear to be less confined to the function of supporting end users than Western manuals. Chinese manuals pay more attention to advertising and relationship-building, and also focus on serving technical experts. Regarding structure, the main difference is that Chinese manuals are less strictly organized than Western manuals. The manuals contain comparable numbers of structuring elements such as headings or lists but use them differently: Chinese manuals have less hierarchy in headings, fewer standout elements in the text, and less functional chunking of information. Regarding the use of visuals, the main difference lies in the use of entertaining and human-oriented visuals in Chinese manuals (such as cartoons and visuals with human figures) versus a restriction to entirely instrumental visuals in Western manuals.
The problem of such content analytic research, however, is that it is uncertain whether the differences found correspond to different user preferences or merely reflect the current state of the art in document design habits. For decisive answers about cultural differences, user research is needed. So far, only a few researchers have taken up this challenge, with very different approaches and somewhat scattered results. In this article, we describe a comprehensive experimental study into the effects Chinese and Western manual structures have on Chinese and Western users. Specifically, we focus on task performance, user satisfaction, and information selection.
CULTURAL DIFFERENCES AND THE STRUCTURING OF INFORMATION
The structuring of information is an important aspect in the body of knowledge of technical communication (Van der Meij et al., 2009). One of the basic principles of technical communication as it developed in Western cultures involves the chunking of information, to optimally facilitate the switching between instructions and task execution. Another fundamental principle is that users must be optimally supported to select the information they need in a specific situation. Assumptions of users reading linearly before they start executing tasks are not considered to be tenable. These principles can also be found in the influential minimalist approach of writing user manuals (Van der Meij & Carroll, 1995).
Earlier content analytic studies into cultural differences in technical communication therefore focused strongly on differences in structure between Western countries and China (Barnum & Li, 2006; Ding, 2003; Dragga, 1999; J. Wang, 2007; Q. Wang, 2000; Y. Wang, & Wang, 2009; Yu, 2009). The most comprehensive content analysis, which summarized and quantitatively tested the insights from earlier studies in a sample of 50 Chinese and 50 Western manuals, was conducted by Li et al. (2020). In that study, several conclusions were drawn about structural differences between Chinese and Western manuals for household appliances.
In contrast to several earlier content analyses, Li et al. (2020) concluded that there is no difference in the number of structuring elements between Chinese and Western manuals, but there are fundamental differences in the way they are used. To summarize their main findings, Chinese manuals have a flatter structure (more first-level headings and fewer higher-level headings), use more visual cues to signal headings, do not seem to systematically and rigorously chunk information into meaningful steps, and contain fewer standout elements (such as tips and warnings presented in a box).
Assuming that such structural differences found in content analytic research reflect the needs and preferences of users from both cultures, a number of hypotheses can be formulated regarding task performance and user satisfaction. For task performance, we distinguish between effectiveness and efficiency; for user satisfaction, we distinguish between satisfaction with the structure of the manual, the manual’s overall usability, and the software package. This leads to the following hypotheses:
Task Performance
H1a Chinese participants working with the Chinese manual structure work more effectively than Chinese participants working with the Western manual structure.
H1b Western participants working with the Western manual structure work more effectively than Western participants working with the Chinese manual structure.
H2a Chinese participants working with the Chinese manual structure work more efficiently than Chinese participants working with the Western manual structure.
H2b Western participants working with the Western manual structure work more efficiently than Western participants working with the Chinese manual structure.
User Satisfaction
H3a Chinese participants working with the Chinese manual structure are more satisfied with the structure of the manual than Chinese participants working with the Western manual structure.
H3b Western participants working with the Western manual structure are more satisfied with the structure of the manual than Western participants working with the Chinese manual structure.
H4a Chinese participants working with the Chinese manual structure are more satisfied with the usability of the manual than Chinese participants working with the Western manual structure.
H4b Western participants working with the Western manual structure are more satisfied with the usability of the manual than Western participants working with the Chinese manual structure.
H5a Chinese participants working with the Chinese manual structure are more satisfied with the software package than Chinese participants working with the Western manual structure.
H5b Western participants working with the Western manual structure are more satisfied with the software package than Western participants working with the Chinese manual structure.
In addition, we formulated a number of hypotheses about users’ information selection. On the basis of the aforementioned content-analytic findings we hypothesized that Chinese participants pay less attention to structuring elements in the manual. Based on another content analytic finding regarding the information contained in manuals—suggesting that Chinese manuals contain more contextual information, for instance to establish a good relationship with the user (Li et al., 2020)—we hypothesized that Chinese participants pay more attention to such contextual information. And based on several earlier content analyses (Carroll & Delin, 1998; Y. Wang & Wang, 2009; Zhu & St.Amant, 2007) and a user study by Honold (1999), all suggesting that visual information is more important in Chinese user instructions, we hypothesized that Chinese users pay more attention to visuals and Western users pay more attention to text. This leads to the following hypotheses:
Information Selection
H6 Chinese participants pay less attention to the table of contents than Western participants.
H7 Chinese participants pay less attention to headings than Western participants.
H8 Chinese participants pay more attention to contextual information than Western participants.
H9 Chinese participants pay more attention to visuals than Western participants.
H10 Chinese participants pay less attention to textual information than Western participants.
EARLIER RESEARCH
In the technical communication literature, only four studies can be found addressing the effects of cultural differences on users. Honold (1999) used a combination of focus groups, questionnaires, and usability tests to make sense of the way Chinese and German users would learn how to use a cell phone. The results suggest that German users value paper manuals more than Chinese. In addition, Germans have different preferences for visuals, preferring overviews of the functionality where Chinese want pictorial information about procedures. In relation to the structuring of information, Honold found that Germans have a need for clear overviews and a detailed index, while Chinese want to start with step-by-step information about basic functions, and think that the importance of information should be related to the size of the characters used.
Zhu and St.Amant (2007) used a qualitative evaluation approach to examine how American users perceive the structure of Chinese-created websites about traditional Chinese medicine. They uncovered various problems including a poor connection to users’ prior knowledge, an organization that did not conform to their expectations, and a failure to provide detailed information. These results hint at the existence of cultural differences, although we cannot be sure that Chinese users would not experience the same problems. Moreover, the number of participants in this study was rather low.
- Wang and Wang (2009) conducted user research comparing German and Chinese mechanics using technical automobile documentation, evaluating their comprehension of text and visuals and their navigation. Their results, however, were not very informative about the relationship between document characteristics and cultural background. They found that Chinese mechanics outperformed their German colleagues in in the comprehension test. They also found that the navigation strategies tended to differ between the two cultural groups: Germans considered more potentially important topics than Chinese, and often went from specific to general information, whereas the majority of the Chinese went from general to specific information.
Li et al.(2015) conducted the only experimental research so far. Focusing on the structuring of information, they experimentally compared the performance and appreciation of Western and Chinese users working with a carefully manipulated Chinese or Western version of a software manual. They did not find any significant difference of the users’ cultural background or the manual version. Western and Chinese users appeared to work equally well with both versions of the manual. However, they reported some methodological shortcomings that might have affected the results. First, the manual was rather short and the participants appeared to be reluctant to actually use the manual. A situation in which structure matters more and participants must use the instructions would have been better. Second, the user instructions were based on an existing manual and the limited guidance from the then available content analytic studies—as a result, both the Chinese and the Western manual version may not have been optimal. And third, the Chinese participants were in fact Chinese living and studying in the Netherlands and might have been influenced by the Western environment they had lived in.
The study reported in this article can be seen as a follow-up to the study by Li et al. (2015). We designed a more comprehensive experiment that overcomes the shortcomings of that study.
METHOD
Experimental research is the only way to make causal inferences about the effects different manual versions have on users. An experimental approach would imply that participants are randomly assigned to one of the two versions of the manual and that any differences found can only be ascribed to the manual versions. Our study also had a quasi-experimental variable: the participants’ cultural background. Participants could not be randomly assigned to these conditions, but belonged to them long before the research started.
We used a 3×2 experimental design, with participants’ cultural background (Chinese living in China, Chinese living in the Netherlands, and Westerners) and manual structure as between-subject independent variables. Participants were randomly assigned to one of the two manual structures. Dependent variables were task performance, user satisfaction, and information selection. For the information selection, we collected eye-tracking data.
Data were collected in individual omnibus research sessions, in which we also collected data for two other experiments, about which we will report in separate articles. The tasks and stimulus materials used in the other two experiments differed considerably from those of this study, so that it was unlikely that they would influence each other. The research was approved by the ethical committee of our university. Below, we describe the various aspects of the research more in detail.
Software Package and Tasks
Just like an earlier study by De Jong et al. (2017), we chose Microsoft Excel 2016 as the software package for the experiment. Excel is a widely used spreadsheet application, embodied with several hundreds of features, from fundamental arithmetic calculations to rather complex programming. It is a software package that many people are familiar with, while only few seem to master a broad range of its functionality. It is not hard to develop different assignments with Excel that range in difficulty and are suitable for lay software users (students, in our case).
We created four user tasks, with varying levels of difficulty, which had to be performed within 30 minutes:
- Task 1: Using the “fill handle” function to automatically fill out a short sequence of numbers (15, 13, 11, 9, 7, 5, 3, 1).
- Task 2: Ranking 999 cities according to their population, and adding a rank number using the “fill handle” function.
- Task 3: Using the “flash fill” function to create students’ e-mail addresses based on their first names, last names, and e-mail service providers (sub task 1) and to extract their dates of birth from their ID card numbers (sub task 2).
- Task 4: Creating a “pivot table” based on a company’s sales data given (sub task 1) and then providing an overview of a company’s revenues per region in two subsequent years (sub task 2).
All four tasks were not in the realm of the widely familiar Excel operations, although their usefulness is easily recognizable for Excel users like the participants in our study. The first three tasks were relatively easy and could be accomplished with up to four actions; the fourth task was considerably more complex and time-consuming. We provided the participants with Excel files with the data needed to do the tasks.
Experimental Manipulation: Manual Versions
For this experiment, we designed a shortened Excel manual based on the information in the official manual (shortened because it only provided information about a small selection of the functionality of Excel). The manual consisted of the usual content elements of user instructions (procedural and declarative information, visuals) and was written following existing style guidelines. The instructions were offered on paper in a four-folding manual of eight pages. The first page contained a table of contents, copyright information, and disclaimers. The second page provided contextual information (“About this Guide” and an introduction of new functions in Excel 2016). The remaining six pages consisted of task-related instructions, all of which were relevant for the four tasks.
We created four manual versions. For the experimental manipulation, we needed a variation in manual structure (Chinese versus Western). Given our participant groups, we needed two language versions of both. We did not want our results to be skewed by differences in English language proficiency, especially considering the group of Chinese participants residing in China. In the experiment, all Chinese participants worked with Chinese manual versions, and all Western participants worked with English manual versions.
For creating the two versions of the manual structure, we used the insights from the comprehensive content analysis by Li et al. (2020) as the starting point. As stated before, this content analysis was based on all earlier research into cultural differences between Chinese and Western documents, and tested for differences between the two cultures in a large sample of user manuals. This resulted in the following differences:
- The Chinese structure had a flatter structure than the Western one. All Western second-level headings were changed into first-level headings in the Chinese versions. All Western third- and fourth-level headings were removed in the Chinese structure.
- The headings in the Chinese structure (color, font size, and underlining) were marked with more visual cues than those in the Western one (color and font size).
- The Chinese manual structure was less chunked into procedural steps than the Western one. Numbered lists of steps were presented in normal body text, with three or more separate actions described in the same paragraph, and sometimes the same sentence.
- In contrast to the Western structure, the Chinese structure did not contain any standout elements such as tips and notes highlighted in a box. These types of information were integrated to the text.
Apart from these differences in structure, the content of the two manual versions was exactly the same. Examples of pages in both versions can be found in the Appendix.
The two versions of the manual were translated into Chinese by the first author. After that, the Chinese texts were translated back to English by a professional translator. A comparison of the original and back-translated English texts showed that they were consistent with each other.
Research Instrument and Measures
Regarding task performance, we used two measures: effectiveness (the proportion of the six tasks that were completed correctly) and efficiency (the total number of tasks completed successfully divided by the time taken).
Regarding user satisfaction, two measures were also used: satisfaction with the manual and satisfaction with the software package. Satisfaction with the manual was measured on five-point Likert scales (from strongly disagree to strongly agree) using 24 items based on Li et al. (2015). A factor analysis (with varimax rotation) showed that, after removal of confounding items, there were two underlying constructs: (1) satisfaction with the general usability of the manual (nine items; Cronbach’s alpha = .90) and (2) satisfaction with the structure of the manual (four items; Cronbach’s alpha = .84). Examples of items regarding general usability are: “The manual is of good quality” and “The manual was helpful for conducting the tasks.” Examples of items regarding manual structure are “The structure of the manual is confusing” and “I could easily find the information I need in the manual.”
Satisfaction with the software package was measured on five-point Likert scales (from strongly disagree to strongly agree) using eight of the 10 items of the System Usability Scale (SUS) (Brooke, 1996; Lewis & Sauro, 2009). Two items were removed from the scale because they did not fit in with the focus on limited functions of the software. We asked the SUS questions twice: for the first three tasks (all about filling in data) and for the fourth task (about the pivot table). Examples of items are: “I think that I would like to use the function frequently” and “I found this function was very awkward to use.” Together, these items formed a sufficiently reliable scale (Cronbach’s alpha = .85).
Participants’ information selection was investigated by means of eye-tracking data. Eye-tracking is increasingly used to make sense of users’ behaviors navigating texts or interfaces (Cooke, 2008, 2010; Elling et al., 2012). We used Tobii Pro Glasses 2, a mobile eye-tracker, to collect the eye-tracking data. Tobii Pro Glasses are worn like regular glasses and weigh only 45 grams. We concentrated on participants’ information selection within the manual. Two measures were used: fixation counts (the number of fixations on parts of the manual) and fixation duration (the time used to process information around the fixations). To make sense of the eye-tracking data, we used the following measures:
- Total attention paid to the manual.
- Attention to structuring elements: (a) table of contents, (b) headings.
- Attention to contextual information.
- Attention to visuals versus text in the instructions.
Participants
Our study aimed at comparing Chinese and Western users’ interactions with the two manual structures. To overcome a potential shortcoming of the Li et al. (2015) study, and at the same time test whether it really applied, we decided to include two groups of Chinese participants in our research: Chinese living in China, who were born and raised in China and had never been abroad, and Chinese living in the Netherlands, who had lived in Western countries for at least one year (on average 2.2 years). The Western participants were born and raised in Western Europe or North America and were currently living in the Netherlands. All participants were university students. Chinese participants spoke Mandarin as their first language and Western participants were English native speakers or were fluent in English. Participants were rewarded for participating in the study either with participant credits or a 5-euro compensation.
Overall, 158 students were recruited from a Chinese and a Dutch university. In recruiting participants, we tried to balance gender and the technical versus nontechnical orientation of their study program. Eventually, 31 participants had to be excluded from the analysis for one of the following reasons: problems with the completeness of the eye-tracking data (14), overlooking one or more of the tasks (14), and not using the manual (3). This resulted in 127 participants that were included in the final analysis.
Table 1 gives an overview of the participants’ background characteristics per experimental condition. Overall, the female-male ratio (70:57) and the ratio between technical and nontechnical educational programs (67:60) appear to be more or less in balance. The participants’ age ranged from 18 to 35 (M = 23.4, SD = 3.2).
Testing for possible differences in background characteristics between the experimental conditions, we did not find any differences regarding gender and program type. However, the participants’ ages were not equally distributed. An analysis of variance showed significant differences for participants’ cultural background (F (2,121) = 8.67, p < .001) and manual structure conditions (F (1,121) = 4.52, p< .05); no significant interaction effect was found (F (2,121) = 1.42, p = .25). An LSD post hoc test showed that Chinese participants living in the Netherlands (M = 24.9) were on average older than Western participants (M = 23.2) and Chinese participants living in China (M = 22.3). To judge the potential consequences of these age differences, we examined the correlation between age and all dependent variables. The Pearson product-moment correlation coefficients appeared to be small and nonsignificant (ranging from -.14 to .14). Therefore we decided to analyze the results without age as a covariate.
Procedure
All data were collected in individual sessions. In the sessions, data for three separate experiments were collected. The session started with an experiment on the structure of declarative information (10–15 minutes). After that, the data for this experiment were collected (40–60 minutes). The session ended with an experiment on the appreciation of various types of visuals in manuals (5–15 minutes).
At the start of the session, participants were welcomed, and read and signed an informed consent form. The participants received a separate introduction for each of the experiments. When they were ready, they were asked to put on the Tobii Pro Glasses. After calibration of the eye-tracking device, the facilitator started the Tobii Pro Eye-Tracker Manager to record the participants’ eye-movements.
The participants sat in front of a computer with the Excel program opened. They received a task sheet. When they indicated that they understood what to do, they were given their version of the user instructions. They all did the four tasks in the same order.
The recording was stopped when the participants had completed the four exercises or when the time ran out (after 30 minutes). After that, they were asked to fill out the online questionnaire about their satisfaction with the manual and the software.
Analysis
For the analysis of eye-tracking data, the software Tobii Pro Lab was used. The software can automatically map eye gaze data onto still images (snapshots) of real-world environments and objects of interest (in our case, the experimental manual). To analyze participants’ fixations, areas of interests (AOI) in the manual were defined. All headings and all paragraphs were marked as areas of interest. Fixations outside of the AOI were not counted. The automatic mapping was satisfactory, often accurate to the letter level. However, there were also instances of inaccurate mapping. To ensure correctness, the first author checked all mappings one by one and manually corrected misplaced mappings. All eye-movement data were imported in the same data file as the task performance and user satisfaction data.
The statistical analyses were conducted in SPSS. We used two-way analysis of variance, with LSD post hoc tests to further explore differences between the three cultural background groups. In our analyses, we were interested in the main effects of participants’ cultural backgrounds, main effects of manual structure, and interaction effects between cultural background and manual structure.
RESULTS
Below, we describe the results per dependent variable. We start with task performance, followed by user satisfaction and information selection.
Task Performance
Table 2 provides an overview of the average task performance, subdivided into an effectiveness and an efficiency score, of the participants in all experimental conditions. Regarding effectiveness, participants in all conditions managed to complete between 79 and 88 percent of the tasks correctly. An analysis of variance showed that there were no significant differences between the three cultural groups (F (2,121) = 1.17, p = .31). Chinese participants living in China and abroad as well as Western participants performed equally well on the six tasks. Furthermore, there were no significant differences between the two manual structures (F (1,121) = .06, p = .80). For the effectiveness of participants’ task performance, it did not matter whether they used the Chinese or the Western manual structure. Finally, there was no interaction effect between participants’ cultural background and the manual structure (F (2,121) = .38, p = .68).
Regarding efficiency, the findings are similar. Analysis of variance again showed no significant differences between the cultural groups (F (2,121) = .63, p = .54) and between manual structures (F (1,121) = .22, p = .64). Furthermore, no interaction effect between the two independent variables was found (F (2,121) = .49, p = .62). In all, the efficiency scores confirmed the conclusions drawn about effectiveness: All three groups of participants performed equally well, the manual structure did not make a difference, and there was no relationship between participants’ cultural backgrounds and their efficiency working with either manual version.
In all, no support was found for our hypotheses that Chinese users work more effectively [H1a] and more efficiently [H2a] with the Chinese manual structure, and that Western users work more effectively [H1b] and more efficiently [H2b] with the Western manual structure.
User Satisfaction
Table 3 gives an overview of participants’ user satisfaction scores per condition. Regarding manual structure, no significant differences were found between cultural groups (F (2,121) = .20, p = .82) and manual structures (F (1,121) = .73, p = .40). There was also no interaction effect between the two independent variables (F (2,121) = 1.00, p = .37).
Similar results were found regarding manual usability: There were no significant differences between cultural groups (F (2,121) = 1.66, p = .19) and between manual structures (F (1,121) = .58, p = .45), and no interaction effect was found (F (2,121) = 2.33, p = .10).
Participants’ satisfaction scores for software functionality also did not show significant differences between the experimental conditions. No main effects were found of cultural groups (F (2,121) = .49, p = .61) and manual structure (F (1,121) = .46, p = .50), and no interaction effect between the two was found (F (2,121) = .23, p = .10).
In all, our findings provide no support for the hypotheses that Chinese users are more satisfied with the manual structure [H3a], the manual’s usability [H4a], and the software [H5a] when working with the Chinese manual structure. Likewise, there is no support for the hypotheses that Western users are more satisfied with the manual structure [H3b], the manual’s usability [H4b] and the software [H5b] when working with the Western manual structure.
Information Selection
Tables 4 and 5 present the findings regarding participants’ information selection, subdivided into their fixation counts and their fixation duration, in all experimental conditions.
Overall attention to the manual
In our analysis, we first focused on the overall attention participants paid to the manual, as indicated by the total number of fixation counts and the total fixation duration. Analyses of variances did not show any significant main or interaction effects of the experimental conditions. On average, participants paid attention to the manual for 278 seconds (4.6 minutes), ranging between 40 and 709 seconds (11.8 minutes).
Attention to structuring elements
Looking at participants’ attention for the structuring elements in the manual, we found a number of significant differences. Regarding the usage of the table of contents, participants’ fixation counts (F (2,121) = 37.44, p < .001, partial η2 = .38) and their fixation duration (F (2,121) = 36.48, p < .001, partial η2 = .38) indicated a strong and significant difference between the cultural groups. LSD post hoc tests showed that Western participants consulted the table of contents more often and longer than both groups of Chinese participants. In addition, the fixation duration showed a significant difference between the manual versions (F (1,121) = 5.98, p < .05, partial η2 = .05): Participants working with the Chinese manual structure used the table of contents longer than participants working with the Western manual structure. This may be due to the less explicit structure within the instructions themselves. We did not find a similar effect for fixation counts (F (1,121) = 2.46, p = .12). No interaction effect was found.
Regarding the usage of headings, the fixation counts (F (2.121) = 5.82, p < .005, partial η2 = .09) and fixation duration (F (1,121) = 9.87, p < .005, partial η2 = .14) again showed a significant difference between the cultural groups. For both measures, LSD post hoc tests suggest that Western participants used headings more and longer than Chinese participants living in China. The Chinese participants living in the Netherlands took a middle position: They differed significantly from the Chinese living in China for fixation counts, and differed significantly from the Western participants for fixation duration. Furthermore, the fixation counts (F (1,121) = 5.82, p < .001, partial η2 =.11) and the fixation duration (F (1,121) = 8.32, p < .01, partial η2 = .06) showed that there was also an effect of manual structure: The Chinese manual version urged participants more to pay attention to headings than the Western version did. This might have been caused by an upgrading of second-level headings (which resembled frst-level headings in the Chinese manual version) and the more emphasized layout of the headings. No interaction effect was found.
These findings support our hypothesis that Chinese participants pay less attention to the table of contents [H6] and headings [H7] than Western participants. At the same time, our findings draw attention to an effect we did not expect: The Chinese manual structure simulated participants more to pay attention to the table of contents and headings than the Western manual structure. A possible explanation for this finding is that participants working with the more implicit and less chunked Chinese manual structure were looking for more guidance during the process.
Attention to contextual information
An analysis of the attention paid to contextual information revealed differences in the opposite direction than expected. Participants’ fixation counts (F (2,121) = 15.27, p < .001, partial η2 = .20) and fixation duration (F (2,121) = 32.37, p < .001, partial η2 = .35) indicated a strong and significant difference between the cultural groups. Contrary to our expectations, LSD post hoc tests showed that Western participants paid significantly more attention to this type of information than both groups of Chinese participants. There were no effects of manual structure, neither for fixation counts (F (1,121) = .24, p = .62) nor for fixation duration (F (1,121) = .07, p = .79). Participants’ fixation counts, however, showed a significant interaction effect between the cultural groups and the manual structure (F (2,121) = 3.33, p < .05, partial η2 = .05). An LSD post hoc test revealed that Western participants were more inclined to read contextual information in the Western manual structure (34.3 versus 26.7), whereas Chinese participants living in the Netherlands tended to read more contextual information in the Chinese manual structure (21.6 versus 9.3). There was no interaction effect for fixation duration.
These findings do not support our hypothesis regarding attention to contextual versus task-related information. We expected Chinese participants to pay more attention to contextual information than Western participants [H8], but they appeared to focus less on the contextual information. An explanation for this unexpected finding might be that Chinese users are more used to the presence of non-instrumental elements in a manual and know what to expect from them. Li et al. (2020) showed that non-instrumental elements are more common in Chinese manuals than in Western ones, especially in the introductory parts. When confronted with assignments that solely focus on task execution, Chinese participants may have felt more confident to set the contextual information aside, whereas Western participants, being used to the overall instrumental content of manuals, may have felt less confidence to skip these introductory passages.
The interaction effect found for fixation counts—with Western participants focusing more on contextual information in the Western manual structure, and Chinese participants living in the Netherlands focusing more on contextual information in the Chinese manual structure—is harder to explain. It is imaginable that participants working with the manual structure connecting to their own culture were more positive of the quality of the manual and therefore had higher expectations of the usefulness of the contextual information. It seems not unlikely that this only affected the Chinese participants living in the Netherlands—who had been exposed to user instructions from both cultures—and not the Chinese participants living in China—who know from experience that contextual information in manuals is never relevant for task execution.
Attention to visuals versus text
An analysis of the fixation counts of attention to visuals showed that there was a significant difference between the cultural groups (F (2,121) = 6.47, p <.005, partial η2 = .10). An LSD post hoc test showed that both groups of Chinese participants paid significantly more attention to the visuals than the Western participants did. We did not find a significant difference at this point for fixation duration (F (2,121) = .55, p = .58). We also did not find significant differences of manual structure, neither for fixation counts (F (1,121) = .05, p = .83) nor for fixation duration (F (1,121) = .08, p = .78). No interaction effect was found.
Regarding participants’ attention to text, no significant differences were found between the cultural groups, both in fixation counts (F (2,121) = 2.82, p = .06) and in fixation duration (F (2,121) = 1.08, p = .34). One could say that there is a tendency towards significance regarding fixation counts, but the difference is in the opposite direction than expected: Chinese participants tended to have more fixations on text than Western participants. However, manual structure appeared to matter. We found a tendency towards significance for fixation counts (F (1,121) = 3.59, p = .06, partial η2 = .03) and a significant difference for fixation duration (F (1,121) = 4.90, p < .05, partial η2 = .04). The Chinese manual structure appeared to focus participants’ attention more on the text than the Western manual structure. No interaction effect was found.
These findings provide partial support for the hypothesis that Chinese users pay more attention to visuals than Western users [H9]. This was confirmed for fixation counts but not for fixation duration. No support was found for the hypothesis that Chinese users pay less attention to text than Western users [H10].
The finding that the Chinese manual structure focuses participants more on the textual instructions can be explained by the fact that the less explicit structure with fewer standout elements may require that users read more text to find the information they are looking for.
DISCUSSION
Main Findings
As a follow-up to the study by Li et al. (2015), the research reported in this article is a second attempt to investigate whether culturally adapted manual structures would make a difference for Chinese and Western users. To answer this question, we created a Chinese and a Western version of a manual based on current insights from content analytic studies and tested how these versions worked for Chinese and Western users. In the research design, we tried to overcome the shortcomings of the earlier study by Li et al. (2015). Nevertheless, the results of our study were perfectly in line with Li et al.’s earlier findings: The structural differences between the two manual versions had no effect on Chinese and Western users. All hypotheses regarding task performance [H1-2] and user satisfaction [H3-5] had to be rejected.
Looking at participants’ information selection, our study resulted in a number of interesting observations, mostly in line with our hypotheses, but one time in the opposite direction. Our results confirmed our hypotheses that Chinese users make less use of structuring elements in a manual (the table of contents and headings) than Western users [H6-7]. We also found partial confirmation for our hypothesis that Chinese users pay more attention to visuals than Western users [H9], but did not find confirmation of the counterpart of the latter hypothesis, that Western users pay more attention to textual information than Chinese participants [H10]
The observation in the opposite direction involved the use of contextual information [H8]. On the basis of the content analytic studies, we expected that Chinese users would use contextual information more than Western users. In our study, however, Chinese users appeared to use this type of information less than Western users. The explanation seems to lie in the strict task orientation of our experiment. In line with the cultural differences found in content analyses, Chinese users might have known, based on earlier experiences with manuals, that this part of the manual would not contain useful information for their tasks. Western users, who tend to see the entire manual as a document aimed at supporting users to work with the software, may have had higher expectations of the potential usefulness of the contextual information.
Theoretical and Methodological Implications
Our research contributes to the body of knowledge regarding cross-cultural and intercultural technical communication. The main issue at stake here is whether the insights that have been developed in content analytic research about cultural differences in the structuring of user manuals (Barnum & Li, 2006; Ding, 2003; Dragga, 1999; Li et al., 2020; J. Wang, 2007; Q. Wang, 2000; Y. Wang, & Wang, 2009; Yu, 2009) actually reflect cultural differences in user preferences. If that would not be the case, such differences must be ascribed to local folklore and random habits of technical communicators.
Our research provides a mixed answer to that question. On the one hand, our results seem to relativize the importance of cultural differences in the structuring of user instructions. Our study showed that implementing the structural differences identified in content analyses does not make a noticeable difference to users. Chinese users can work equally well and are equally satisfied with a Western and a Chinese manual structure. And the same applies to Western users. The latter is surprising because the Western manual structure was meant to reflect the current (Western) state of the art in structuring user instructions, and the Chinese structure deviates from some basic principles (creating a clear hierarchical structure, chunking information, making certain elements stand out).
On the other hand, our results indicate that Chinese and Western users differ in their way of using manuals. And these differences appear to correspond to structural and other differences uncovered in content analytic research (Li et al., 2020). For instance, Chinese users pay less attention to the structuring elements than Western users, which somehow supports the looser and less rigid structuring of Chinese manuals. Chinese users pay more attention to visuals in the manual, which corroborates earlier findings about the prominence of visuals in Chinese technical communication. And Chinese users expect more than Western users that contextual information is not task-related, which supports the more multifunctional nature of Chinese manuals.
How do these paradoxical insights align with each other? Li et al. (2015) proposed the existence of a “community of practice” in the domain of user support, in which a more or less universal way of designing and using instructions emerges (cf. Eckert, 2006). Our results seem to further specify that. There are differences in the way Chinese and Western users operate using instructions, but at the same time, Chinese and Western users have developed flexibility in using instructions that enables them to work equally well with different cultural versions. This might indicate that the structuring of manuals can be seen as a less prominent aspect of cross-cultural technical communication than previously assumed.
Methodologically, we wanted to find out whether recruiting Chinese participants living in Western countries, like Li et al. (2015) did, leads to comparable results as recruiting Chinese participants living in China. This relates to the effects of acculturation on cultural differences. We only found one instance in which Chinese participants living in the Netherlands took a middle position between Chinese participants living in China and Western participants. In all other differences found, the two groups of Chinese participants did not differ from each other. Our findings thus did not provide convincing evidence that it is problematic to conduct cross-cultural technical communication research with participants residing in Western countries. Intrinsically, our results suggest that cultural differences in using user instructions can be quite resistant to the effects of acculturation.
Practical Recommendations
Based on our findings, it does not seem justified to spend much time and effort on cultural adaptations of a manual structure. Other aspects of a manual seem to be more promising to make a difference for the usability and user experience of international users. One aspect appears to be the use of visuals. Our research shows that visuals are more important to Chinese users than to Western users. In design processes of individual manuals or of a format for manuals, it would be wise to pay attention to the differential role visuals can play in the perceived quality and usability of a manual. Another aspect involves content, particularly the way manuals may contribute to the relation between brand and user and to user experience in general. Our findings underline that Chinese users have different expectations at this point than Western users.
Limitations and Suggestions for Future Research
Of course, there are limitations that should be kept in mind when interpreting our results. First, it must be acknowledged that the participants in our research were all students. There was a match between participants and software package—students often use Excel for various purposes—but the user group of Excel tends to be broader: It is likely that professionals in the workplace, older people, and people with lower levels of education also use the package. For the purpose of an experimental comparison, a certain degree of homogeneity in the sample is beneficial, as uncovering differences between groups is more feasible when the diversity within groups is limited. For the external validity of the findings, it would be helpful if the research could be replicated within other relevant cross sections in society. One could expect that professionals in the workplace might even have more flexibility in working with the software and the instructions whereas cultural differences might be more prominent among older users and users with lower levels of education.
Second, one could argue that the structural characteristics of manuals will play a more significant role when the document is more substantial. Our study ensured that the manual was sufficiently complex and that participants really used it, but the manual was still relatively short, basically an excerpt from a more comprehensive Excel manual. It is imaginable that the effects of structure, and as a consequence the possible effects of cultural variations of structure, will be clearer in the case of a more substantial manual. Future research could focus on that.
Third, our research only focused on immediate task performance, whereas using the manual may also involve learning and memorizing how to perform certain tasks (reading to learn to do; Redish, 1989). Earlier research shows that manipulations in user instructions might have no effect on immediate task performance and still have an effect on learning and future (memorized) task performance (Eiriksdottir & Catrambone, 2011; Ummelen, 1997). Future research could therefore also focus on the effects of manual structures on learning and future task performance.
Fourth, our experiment was entirely focused on task execution, which relates to the usability of manuals, but leaves out other types of contributions a manual might make, specifically in the broader realm of user experience. Especially with regard to the information selection of users in a manual, it would be very interesting to take a broader perspective on the use of manuals. Our strict focus on usability might be an example of cultural bias in itself. More eye-tracking studies of participants using a manual for a broader range of tasks than immediate task performance would be very interesting.
Fifth, we positioned our research as a comparison of Chinese and Western user instructions, but the labels “Chinese” and “Western” might be too generic. The Western world consists of many different countries and we can by no means be sure that all these countries are culturally homogeneous. Similarly, China is a huge country that may also have differences between different parts. At the same time, it could be argued that the label “Chinese” is too limited, as other East Asian countries (e.g., Korea or Japan) might favor the same structuring principles and might show similar user behaviors. Future research could shed more light on these issues.
A sixth limitation involves the role of translation in our research. We believe we had good reasons to work with two language versions of the instructions. Despite our precautions in developing the translated versions, the quality of the translations might have influenced the results. We cannot be completely sure whether the two Chinese manual versions were of equal quality as the two English versions. More research into the role of technical translation is needed. One can think of a larger project in which translation quality is added as an extra factor. It would also be interesting to investigate in detail how technical translators deal with the structural aspects of user instructions, both in terms of their end products and in terms of their considerations.
Finally, our findings call for more meticulous empirical research into the effects of (combinations of) structuring approaches and elements in both cultures. Carefully and explicitly structuring instructions can be seen as one of the cornerstones of technical communication. Our findings seem to relativize the importance of the two packages of structuring principles. Based on these somewhat puzzling results, it would be relevant to find out which combinations of structuring approaches and elements are vital for the usability of instructions, and which approaches and elements may be redundant. Ideally, such results would make the connection between the various structuring elements and approaches and the user behavior of Chinese and Western users. More fine-grained eye-tracking data could be helpful in this respect.
CONCLUSION
The study reported in this article shows that cultural adaptations of the structuring of a user manual do not have the expected effects on Chinese and Western participants. Both groups of participants perform equally well and are equally satisfied with both manual structures. This does not mean that they behave in the same way. Chinese users pay less attention to structuring elements (table of contents and headings) and more attention to visuals. These differences appear to correspond to some of the insights content-analytic studies have provided about differences between Chinese and Western documents. Task performance and satisfaction scores suggest that Chinese and Western users have the flexibility to use a manual version that does not correspond to the principles of structuring belonging to their culture. In this respect, the structure of user instructions may not be an urgent aspect to pay attention to in cross-cultural technical communication.
REFERENCES
Barnum, C. M., & Li, H. (2006). Chinese and American technical communication: A cross-cultural comparison of differences. Technical Communication, 53(2), 143–166.
Batova, T. (2018). Negotiating multilingual quality in component content management environments: A case study. IEEE Transactions on Professional Communication, 61(1), 77–100.
Batova, T. (2019). Lost in content management: Constructing quality as a global technical communication metric. Technical Communication, 66(1), 30–52.
Batova, T., & Clark, D. (2015). The complexities of globalized content management. Journal of Business and Technical Communication, 29(2), 221–235.
Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas, I. L. McClelland, & B. Weerdmeester (Eds.), Usability evaluation in industry (pp. 189–194). Taylor & Francis.
Cooke, L. (2008). How do users search Web home pages? An eye-tracking study of multiple navigation menus. Technical Communication, 55(2), 176–194.
Cooke, L. (2010). Assessing concurrent think-aloud protocol as a usability test method: A technical communication approach. IEEE Transactions on Professional Communication, 53(3), 202–215.
De Jong, M. D. T., Yang, B., & Karreman, J. (2017). The image of user instructions: Comparing users’ expectations of and experiences with an official and a commercial software manual. Technical Communication, 64(1), 38–49.
Ding, D. D. (2003). The emergence of technical communication in China—Yi Jing (I Ching) the budding of a tradition. Journal of Business and Technical Communication, 17(3), 319–345.
Dragga, S. (1999). Ethical intercultural technical communication: Looking through the lens of Confucian ethics. Technical Communication Quarterly, 8(4), 365–381.
Eckert, P. (2006). Communities of practice. In K. Brown (Ed.), Encyclopedia of language and linguistics (2nd ed., pp. 683–685). Elsevier.
Eiriksdottir, E., & Catrambone, R. (2011). Procedural instructions, principles and examples: How to structure instructions for procedural tasks to enhance performance, learning and transfer. Human Factors, 53(6), 749–770.
Elling, S., Lentz, L., & De Jong, M. (2012). Combining concurrent think-aloud protocols and eye-tracking observations: An analysis of verbalizations and silences. IEEE Transactions on Professional Communication, 55(3), 206–220.
Frith, J. (2014). Forum moderation as technical communication: The social web and employment opportunities for technical communicators. Technical Communication, 61(3), 173–184.
Honold, P. (1999). Learning how to use a cellular phone: Comparison between German and Chinese users. Technical Communication, 46(2), 196–205.
Lentz, L., & Hulst, J. (2000). Babel in document design: The evaluation of multilingual texts. IEEE Transactions on Professional Communication, 43(3), 313–322.
Lewis, J. R., & Sauro, J. (2009). The factor structure of the System Usability Scale. Lecture Notes in Computer Science, 5619, 94–103.
Li, Q., De Jong, M. D. T., & Karreman, J. (2015). Cultural differences and user instructions: Effects of a culturally adapted manual structure on Western and Chinese users. Technical Communication, 62(3), 163–182.
Li, Q., De Jong, M. D. T., & Karreman, J. (2020). Cultural differences between Chinese and Western user instructions: A content analysis of user manuals for household appliances. IEEE Transactions on Professional Communication, 63(1), 3-20.
Redish, J. C. (1989). Reading to learn to do. IEEE Transactions on Professional Communication, 32(4), 289–293.
Robles, V. D. (2018). Resolving discourse at technical-support helpdesks. IEEE Transactions on Professional Communication, 61(3), 275–294.
Sun, H. (2006). The triumph of users: Achieving cultural usability goals with user localization. Technical Communication Quarterly, 15(4), 457–481.
Sun, H. (2012). Cross-cultural technology design: Creating culture-sensitive technology for local users. Oxford University Press.
Swarts, J. (2015). What user forums teach us about documentation and the value added by technical communicators. Technical Communication, 62(1), 19–28.
Ummelen, N. (1997). Procedural and declarative information in software manuals: Effects on information use, task performance and knowledge. Rodopi.
Van der Meij, H. (2018). Cognitive and motivational effects of practice with videos for software training. Technical Communication, 65(3), 265–279.
Van der Meij, H., & Carroll, J. M. (1995). Principles and heuristics for designing minimalist instruction. Technical Communication, 42(2), 243–261.
Van der Meij, H., Karreman, J., & Steehouder, M. (2009). Three decades of research and professional practice on printed software tutorials for novices. Technical Communication, 56(3), 265–292.
Van der Meij, H., & Van der Meij, J. (2013). Eight guidelines for the design of instructional videos for software training. Technical Communication, 60(3), 205–228.
Van Velsen, L. S., Steehouder, M. F., & De Jong, M. D. T. (2007). Evaluation of user support: Factors that affect user satisfaction with helpdesks and helplines. IEEE Transactions on Professional Communication, 50(3), 219–231.
Wang, J. (2007). Some theoretical considerations on the issue if xinghe and yihe. Foreign Language Teaching and Research, 39(6), 409–416.
Wang, Q. (2000). A cross-cultural comparison of the use of graphics in scientific and technical communication. Technical Communication, 47(4), 553–560.
Wang, Y., & Wang, D. (2009). Cultural contexts in technical communication: A study of Chinese and German automobile literature. Technical Communication, 56(1), 39–50.
Yu, H. (2009). Putting China’s technical communication into historical context: A look at the Chinese culinary instruction genre. Technical Communication, 56(2), 99–110.
Zhu, P., & St.Amant, K. (2007). Taking traditional Chinese medicine international and online: An examination of the cultural rhetorical factors affecting American perceptions of Chinese-created Web sites. Technical Communication, 54(2), 171–186.
ABOUT THE AUTHORS
Qian Li is currently a lecturer at Beijing Normal University (China). She received the Ph.D. degree in Communication Science from the University of Twente in 2019. Her research interests focus on cultural differences in user experience and usability. Contact: q.li19@outlook.com
Menno D.T. de Jong received the Ph.D. degree in Communication Science from the University of Twente (Enschede, the Netherlands) in 1998. He is currently a Full Professor of Communication Science with the University of Twente. Between 2009 and 2015, he served as the editor of Technical Communication. His research interests include the fields of technical and organizational communication. He has been the recipient of various awards for his research, including the Ken Rainey Award for Excellence in Research from the STC, and the Alfred N. Goldsmith Award for Outstanding Achievement in Engineering Communication from the IEEE Professional Communication Society.
Contact: m.d.t.dejong@utwente.nl
Joyce Karreman received the Ph.D. degree in Technical Communication from the University of Twente (Enschede, the Netherlands) in 2004. Since then, she has been affiliated with that university. She teaches courses in document design, user support, and user-centered design processes. Her research interests include the design of instructive documents, user-centered design, and intercultural issues related to usability and user experience. She has presented her work at international conferences on technical and professional communication and has published in several journals. Contact: j.karreman@utwente.nl
Appendix: Examples of Pages of the Two Manual Structures
