65.3, August 2018

Cognitive and Motivational Effects of Practice with Videos for Software Training

By Hans van der Meij

Abstract

Purpose: Software makers nowadays regularly post videos on their websites to satisfy their clients’ need for instructional support. Some of these designs include an opportunity for practice. This study investigated whether the presence and timing of practice affected motivation and learning in video-based software training.

Method: An experiment is reported with four conditions: video-practice (VP), practice-video (PV), practice-video-practice (PVP), and video only (V). For motivation, the study assessed mood states and flow experience during training. In addition, task relevance and self-efficacy were measured before and after training. Learning was assessed with several performance tests for trained tasks. In addition, a transfer test was administered.

Results: The findings for presence of practice were mixed. Practice increased training time and led to more negative mood states during training. A clear advantage of practice was found only on the transfer test. The findings for timing of practice favored a sequence in which instruction preceded practice. Perplexing results were found for the PVP condition. The highest learning gains were expected for this condition, but, instead, this condition had the lowest performance scores on a practice test and immediate post-test.

Conclusion: This study indicates that the design issue of whether or not to complement video-based software training with practice is more complex than it seems.

Keywords: instructional video, software training, practice, motivation and cognition

Practitioner’s Takeaway:

  • The present study found mixed results for the presence and timing of practice in learning. Surprisingly, the lowest learning was found in the group that engaged in practice before and after video instruction.
  • Designers may want to mitigate negative mood states of the user that emerge during practice (more than with video only).
  • The study points out several advantages and disadvantages for deciding whether to include practice during training.

Introduction

The development and distribution of instructional video (simply “video” from here on) for software training is rapidly increasing. This growth has been stimulated by technological advances, such as faster computers, better and cheaper video programs, and improved possibilities for distribution (Mogull, 2014). Thus, it is no surprise to find many videos on the websites of software makers, such as Adobe, Apple, IBM, Microsoft, TechSmith, and SAP.

The Adobe website for Premiere Pro tutorials can be used to illustrate what videos are on offer (Adobe, 2017, August 1). Figure 1 shows a section from the website’s homepage with access to five videos for beginners. Each video link shows a screenshot, a title and a characterization of user actity (i.e., “Watch” or “Try it”), and video duration. The website presents three different formats as possible ways to provide instructions about software usage in a video tutorial: Embedded, Stand-alone View, Stand-alone Try-it.

Figure 1. Screenshot of a section of the Adobe homepage for the website with Premiere Pro video tutorials for beginners

Figure 2 shows the embedded video for “Learn five editing basics in Premiere Pro.” In embedded videos, the website serves as the main user guide. It begins with a goal description and picture, and then states that there are five basic editing tasks that users should be able to perform. Thereafter, the website introduces the five tasks, with each task being numbered, titled, and described in a single sentence. The screenshot of the program interface gives access to the video with procedural information for each task.

Figure 2. Screenshot of the embedded video “Learn five editing basics in Premiere Pro” on Adobe’s tutorial website

Figure 3 illustrates the View variant of a stand-alone video. As in the embedded version, there is a title (e.g., “Quickly make expert color adjustments”) and a short description. In contrast to the embedded video, all user guidance comes from the video. The video includes conceptual and procedural information on the task.

Figure 3. The start screen of “Quickly make expert color adjustments,” a View stand-alone video on Adobe’s tutorial website

The homepage for Premiere Pro tutorials also hosts a Try-it variant of a stand-alone video (e.g., “Enhance your project with video effects”). These videos closely resemble the View variant, the main difference being the availability of practice files that allow the user to follow along with the modelled task performances. Before the video opens, the website draws the user’s attention to the possibility of downloading practice files as opportunity for practice (see Figure 4). The video also mentions this possibility. As far as we have been able to establish, this approach is not unique for Adobe. TechSmith also complements its tutorial videos with links to practice files, and there are likely to be other software makers doing so as well. It is this coupling of video and practice that is focal in this paper.

Figure 4. The start screen of “Enhance your project with video effects,” a Try it stand-alone video on Adobe’s tutorial website

All video types consist of recorded demonstrations—a screen capture animation with narration. Beyond that, though, little is known about the design characteristics and effectiveness of the Adobe videos. The same applies to the videos produced by other software companies (compare Alexander, 2013; Morain & Swarts, 2012).

The videos from Adobe and other software companies serve the dual purpose of supporting task performance and learning. Recent research has proposed a theoretical model for constructing videos for software training that serves these aims. Empirical studies have established the effectiveness of video for software training based on this model (e.g., Brar & van der Meij, 2017; H. van der Meij, 2017; H. van der Meij & van der Meij, 2016; J. van der Meij & van der Meij, 2015). In these studies, there was a classic coupling of instruction and practice in which participants first viewed a video about a task and then engaged in practice of that task.

It was assumed but not tested that practice contributes to learning. Testing that assumption is the focus of this paper. The present study investigates whether the presence and timing of practice affects the outcomes of video-based software training. After a review of the literature, an experiment on practice in video-based software training is reported.

Research on the Presence and Timing of Practice

A literature search produced only one empirical study on video-based software training that manipulated the presence of practice. In her second experiment, Ertelt (experiment 2, 2007) presented participants with five videos about RagTime, a desktop publishing program. One group engaged in practice after watching each video while another group only viewed the videos. The findings revealed a significant but small effect of practice on learning on an immediate and a delayed post-test of the trained items. In addition, practice had a positive effect on a transfer test. The author suggested that the inclusion of practice stimulated users to engage in more active and deep processing.

In response to the virtual absence of empirical studies, we conducted our own experiment on practice with video-based software training (H. van der Meij, Rensink, & van der Meij, in press). The study involved videos on formatting tasks in Microsoft Word. There was one control condition in which the users could only view the videos (i.e., video only). There were two experimental conditions with practice, which varied in the timing of the practice (i.e., video-practice and practice-video).

One experimental condition included the traditional sequence of task instructions preceding practice. Complementing the video instruction with practice-after was expected to have two important advantages. One was that practice-after can deepen understanding by stimulating the user to (re)construct a procedure. Another was that practice-after can serve as a check of understanding and consolidate learning (compare van Gog, 2011).

The other experimental condition was designed to reflect the situation in which the user has first attempted to complete a software task and then turns to the video for support after failure or to check on the solution. In this condition, the contribution of practice was expected to be mainly motivational. Confronting the user with a practice task before providing task instructions should stimulate software exploration during practice. In addition, the practice was expected to increase the user’s motivation for studying the video (compare Stark, Gruber, Renkl, & Mandl, 2000).

In short, it was expected that the presence of practice would enhance learning, and also that there was a higher learning outcome in the video-practice condition than the practice-video condition. These predictions were not confirmed. The video-only condition had learning outcomes comparable to those for the experimental conditions on various tests (i.e., immediate post-test, delayed post-test, transfer test). Also, the video-practice condition did not outperform the practice-video condition on these tests. The only significant difference found was on the practice test during training. On this test, the video-practice condition did better than the practice-video condition, achieving an average task performance success of 86% (up from 13% prior to training). What was surprising was that the practice-video condition also improved significantly, raising the participants’ pre-test score of 14% to a practice test score of 45%. The increase was ascribed to the participants becoming acquainted with the interface, either from viewing a related task video or from exploring the interface during task completion attempts.

The design of our video-based software training is similar to a design approach that is known as the worked example strategy. Just as in our video research, worked example designs revolve around an ideal model of problem solving that is complemented with instructional features to enhance learning (Atkinson, Derry, Renkl, & Wortham, 2000; Renkl, 2014b; Sweller & Cooper, 1985). The effects of the presence and timing of practice has also been empirically investigated in worked examples research. The findings from this body of research are somewhat comparable to the outcomes found in our earlier video study. That is, several studies have found that example-only studying is equally as effective as practice-after. Many studies also found that practice-after is more effective for learning than practice-before (e.g., Leppink, Paas, van Gog, van der Vleuten, & van Merriënboer, 2014; Reisslein, Atkinson, Seeling, & Reisslein, 2006; van Gog, Kester, & Paas, 2011).

Research Design and Questions

The present study was designed as an extended replication of the earlier investigation of practice in video-based software training. The present study differs in three ways from that research. First, the present study recorded training time to assess whether the inclusion of practice significantly increases the time the users spend on training. Second, whereas the previous research concentrated on cognitive outcomes, the present study also tracks motivational outcomes. Third, an additional (third) experimental condition was included. In that condition, users had an opportunity to engage in practice both before and after the video (i.e., practice-video-practice = PVP). The condition mimics the situation in which a user first tries to complete a software task independently (with or without success), then consults a video, then has another opportunity for practice.

The set-up of the study was quasi-experimental, with random allocation of participants to conditions within classrooms. In the control condition (Video only = V), participants could only view the videos during training. The three experimental conditions varied in the timing of practice. One condition provided an opportunity for practice after instruction (i.e., video-practice = VP), while another offered the chance to practice before viewing the instruction (i.e., practice-video = PV). The third experimental condition offered the opportunity for practice before and after the instruction (i.e., practice-video-practice = PVP).

Research question 1: “Does practice affect the outcomes for key variables (training time, motivation, learning) for video-based training?” This research question investigates whether there is any effect of the presence of practice (control versus experimental conditions). The inclusion of practice was expected to increase the training time of the users, because the practice is in addition to the video.

For motivation during training, the study looks at the users’ mood states and flow experience. These constructs represent temporary affect states that can mediate the effect of motivation on learning (Vollmeyer & Rheinberg, 1999, 2006). For motivation development, the study looks at appraisals of task relevance and self-efficacy before and after training. These constructs represent the two key factors in the expectancy-value theory of motivation (Eccles & Wigfield, 2002). No specific predictions for the effects of condition on motivation were formulated in advance.

In view of the arguments reported in the worked example literature, a positive effect of practice was expected for learning, as assessed with task performance tests (i.e., immediate post-test, delayed post-test, transfer test).

Research question 2: “Is there an effect of the timing of practice on the outcomes for key variables (training time, motivation, learning)?” No specific predictions of the effect of timing of practice on training time were formulated in advance.

In view of the argument reported in the worked example literature, a stronger effect of practice-before than practice-after on motivation was expected. There were no specific predictions for PV versus PVP.

The tested prediction for learning was that the highest learning gains will be seen for the PVP condition, followed by VP, followed by PV. It was exected that the highest learning gains would be in the condition with practice-before and practice-after (PVP), because it combines the advantages of the two practice timings. The classic instructional paradigm is to start with instruction and follow with practice. This is also the standard recommendation for training on procedural knowledge development (Ertelt, 2007; Grossman, Salas, Pavlas, & Rosen, 2013; Kerr & Payne, 1994; Rosen et al., 2010; H. van der Meij & van der Meij, 2013). On these grounds, it was expected that practice-after (VP) would yield greater learning than practice-before (PV).

Method

Participants

The 93 participants in the study came from two first-year and two second-year classrooms from a middle school in Germany. Students from each classroom were randomly assigned to the four conditions in the experiment. One student was excluded from the database, because he did not take the pre-test. Also excluded were five students who missed training. The mean age of the remaining 45 male and 42 female students was 12.4 years (range 11.3–14.3). All instructional materials, including the software, were in the students’ native German language.

Instructional Materials

Videos

The design of the video for software training was based on the theoretical model shown in Figure 5. The model combined Demonstration-Based Training (DBT) and multimedia learning theory (e.g., Brar & van der Meij, 2017; H. van der Meij, 2017; H. van der Meij & van der Meij, 2016). The constructed videos were recorded demonstrations that illustrated and explained the stepwise progression involved in task completion. Each video showed a single, menu-based method for completing the given task. That demonstration was enhanced with instructional features that support four key observational learning processes, namely, motivation, attention, retention, and (re)production.

Figure 5. Theoretical model for video-based software training. Highlighted instructional features (in bold) were incorporated in the design of the videos in this study.

The videos instructed students how to format Microsoft Word documents. The tasks selected for inclusion represented important features from the school reports that the students must regularly produce. This anchored the instructions in the task domain of the audience (M1 in Figure 5).

Long formatting tasks were split into meaningful subtasks. For instance, the objective of changing the margins of a complete Word document was split into one subtask (and video) for adjusting the right margin and one subtask for the left margin. This segmentation (R1) reduced task complexity. Users needed to remember three main steps in each task, which should be within the limits of their working memory capacity (Doumont, 2002; Sweller, van Merrienboer, & Paas, 1998). In addition, segmentation yielded shorter videos, which contributes to engagement (Guo, Kim, & Rubin, 2014; Wistia, 2012).

The videos were organized by ‘chapters.’ Chapter 1 revolved around formatting a complete Word document. The two task videos in this chapter demonstrated how to adjust the right and left margin, in that order. Although both tasks involved selection of a similar screen object, object selection was far easier to accomplish for the right margin. The videos were thus ordered in a simple-to-complex sequence (R3). Chapter 2 revolved around formatting a section from a Word file. This chapter included four task videos (indenting paragraphs, left indent citation, right indent citation, and formatting lists).

Students could access the videos via a website that presented a table of contents with chapter titles serving as organizers. Paragraph titles described the distinct tasks and were linked to the videos. The links served to address the videos’ weak points of accessibility as compared with a paper-based document. Clicking on a paragraph title changed its color and opened the video on the right side of the website. Students could start, pause, rewind, and stop the video with a standard control panel. With this panel, students had a moderate degree of user control (A4) over video play.

Each video started with a preview (A2) with information about the initial goal state and an explanation of jargon for key concepts (e.g., margin, paragraph, citation). An example of a preview statement (for paragraphs) in the videos was: “You see a text in which the separate sections are inconspicuous. To make these stand out better, we should indent the first part of each section.” These statements frequently used personal pronouns (e.g., you, we) to create a conversational style (M2).

The main content of the video consisted of the action-reaction pattern that characterizes procedural discourse for software tasks (H. van der Meij, Blijleven, & Jansen, 2003; H. van der Meij & Gellevij, 2004). The recorded demonstration of the (changes on the) interface was accompanied by a narrative. The narrative that told the user what to do on the interface followed the preferred format for these statements (Farkas, 1999). That is, each action was presented in a succinct statement built around an imperative verb (e.g., “Click the left mouse button” or “Press the ALT-key to check if the margin is 4 centimeters”). In addition, the narrative informed the user of the (in)visible effect of an action on the interface (“You can see a dotted line appear on your screen” “and a small window with the words ‘Right Indent’ appears”).

Signaling (A1) and zooming techniques were used to draw the user’s attention to screen objects and locations. A warm color (e.g., red) was used to increase the attention-drawing effect (Kosslyn, Kievit, Russell, & Shephard, 2012).

The videos regularly featured 2-second pauses (R4) that coincided with sub-goal achievement in the procedure. These pauses can help the user overcome processing problems caused by transience of information; they give the user a brief period of additional time to digest the information (Spanjers, van Gog, Wouters, & van Merriënboer, 2012).

The pace (A3) of the video was largely determined by the narrative. The male voice-over went at a normal speaking rate. To obtain an objective measure of pace, a mean words-per-minute (wpm) count was computed. The wpm metric is the number of narrated words divided by the total time of the video (in seconds) multiplied by 60. The mean wpm for the videos was 89. This is considerably lower than the average 125 to 150 wpm for conversational speed in English (Fulford, 1992).

There are different recommendations for video length (M3). Plaisant and Shneiderman (2005) proposed a 1-minute duration for recorded demonstrations. An empirical analysis of viewer engagement with MOOC videos (Guo et al., 2014) found best results for short videos of 3 minutes maximum. The mean length of the videos in the present study was 1.13 minutes (range 0.58–1.46). Four of the six videos were 1 minute long or shorter. The total length of the videos was 6.35 minutes.

Practice files

Students who engaged in practice during training were supported in their hands-on experience with practice files (P3) created especially for each task. In addition to facilitating practice, these files also standardized practice; they made task completion efforts comparable across conditions. The practice files in the experiment were superficially different from (but structurally similar to) the file in the video. This was expected to enhance the user’s understanding of the procedure. The PVP condition included two practice files for each task. The training time measure was obtained from saved practice files, as was task performance success during training. In the PVP condition, success was measured on the first as well as the second practice task.

Instruction booklets

For each condition a paper instruction booklet provided students with a training scenario (see Figure 6). The task sequence in the booklet was the same as on the website. For each new task, the booklet first engaged the students in a “reading task” that explained jargon for key concepts (e.g., margin, paragraph, citation) and presented a before-after display of the initial and final goal state for a task. (Note that the booklet contained this preview information to provide students in the PV and PVP condition with the necessary prerequisite information for engaging in task practice before viewing the video.) Next, the booklet directed the student to view the video or to engage in hands-on practice, depending on the student’s assigned condition. A “practice task” was flagged with a keyboard and a list of action steps. A “viewing task” was flagged by a picture of a monitor and instructions to view a particular video. In addition, the booklet included the questionnaires for mood and flow (see Instruments).

Figure 6. A section from the instruction booklet for the practice-video (PV) condition (translated version)

Instruments

Initial experience & motivation questionnaire (IEMQ)

The IEMQ is a paper-and-pencil instrument that measured the student’s experience and motivation before training. The IEMQ presented a screenshot for each training task and asked three questions about that task: (a) “Do you ever have to do this task?” (experience), (b) “How often do you need to complete this task?” (task relevance), and (c) “How well do you think you can complete this task?” (self-efficacy). Answers were given on a 7-point Likert scale, which ranged from never (1) to always (7), or very poorly (1) to very well (7). Reliability analyses, Cronbach’s alpha, indicated satisfactory results for the three measures (i.e., Experience α = ٠.٧٧; Task Relevance α = 0.76; Self-efficacy α = 0.89).

Mood questionnaire (MQ)

The MQ is a paper-and-pencil instrument that asked the student to select the pictogram, a smiley, plus descriptor (i.e., happy, certain, neutral, uncertain, or sad) that best fitted his or her current emotional state (see Read, 2008). The analysis of mood looked at its valence (compare Plass, Heidig, Hayward, Homer, & Um, 2014) by making a distinction between positive, neutral, and negative values. Positive mood states are considered conducive for learning, while negative mood states can disrupt learning (Astleitner, 2000; Um, Plass, Hayward, & Homer, 2012). “Happy” and “certain” were scored as signals of a positive mood; “uncertain” and “sad” were signals of a negative mood. The MQ was administered four times, after the user had completed a major task in the tutorial (i.e., after setting both margins for a complete document, and after formatting paragraphs, citations, and lists). Scores are presented as percentages. Thus, a score of, say, 75% for positive mood indicates that the student selected the happy or certain smiley at three of the four measurement points for mood.

Flow questionnaire (FQ)

The FQ is a paper-and-pencil instrument that was an adapted version of the Flow Short Scale (FKS) from Rheinberg, Vollmeyer, and Engeser (2003). It consisted of four items (e.g., “I had the feeling that I had everything under control,” “The right thoughts came without effort,” “With every step, I knew what to do,” and “My head was completely clear”). Answers were given on a 7-point Likert-scale, which ranged from completely disagree (1) to completely agree (7). The FQ was always presented immediately after the MQ. Hence, there were also four measurement points. Reliability scores, Cronbach’s alpha, were at or above 0.90 for each time point.

Final motivation questionnaire (FMQ)

The FMQ was a paper-and-pencil instrument that asked students to rate the relevance of the trained tasks and to appraise their self-efficacy for completing these tasks in future. There were seven questions about task relevance (e.g., “I find it important to have a wide enough margin for a document” and “I think it is important to present lists in a well-structured manner”), and seven questions about self-efficacy (e.g., “I can now present a nicely structured list” and “I now know how to indent the first line of a new text segment”). Answers were given on a 7-point Likert-scale, which ranged from completely disagree (1) to completely agree (7). Reliability scores, Cronbach’s alpha, were 0.87 for task relevance and 0.90 for self-efficacy.

Performance tests

Five tests (i.e., pre-test, practice, immediate post-test, delayed post-test, and transfer test) assessed the students’ task performance success. With the exception of the transfer test, all test items presented the formatting tasks demonstrated in the videos, differing only in the appearance of the test files. The transfer test included items that differed slightly from the training tasks (e.g., improve a list within a list). For each test, students were awarded a score of 0 points for each task they did not complete correctly. Correct task completion yielded a score of 1. With the exception of the transfer test, the maximum score for each test was 7 (i.e., right and left document margin, right and left citation indent, paragraph indent, list keywords, and list descriptors). For the transfer test, the maximum score was 4 (one main task consisted of two distinct subtasks). Scores were converted to a percentage of possible points.

Procedure

The study was conducted in three sessions that were held in the computer rooms of the school. In the first session, students were told (5 minutes) that they would engage in software training on Microsoft Word to assist them in improving the formatting of their school reports. Next, they were instructed to complete the IEMQ and Pre-test (20 minutes).

The training session followed a week later. This session started with a 10-minute introduction. An explanation was given about using the instruction booklet. This explanation told the students about the different types of activities they were expected to engage in and their sequence. Website navigation and video usage were also illustrated. In all conditions, the students were instructed to watch the video for each task until they felt sure they could complete the task.

Practice (P1) and practice sequence (P2) were both manipulated in the design of the study, leading to slightly different procedures in the experimental conditions. Students in the experimental conditions received instructions about the handling of practice files and video viewing. In the PV condition, students were told they were not allowed to return to a practice task after having seen the video. Instead, they were to continue to the next practice task. In the VP condition, students were told they were not allowed to return to the video once they had started on the practice. After practice, they were to move on to viewing the next video. In the PVP condition, students were told they were to practice first, then view a video, and then engage in another practice attempt (with a different file) at the instructed task.

All students were instructed to work independently for 40 minutes and to call for assistance only when stuck. Students received the audio input from the video via headphones. After training was completed, there was a 10-minute break. Next, they were instructed to complete the FMQ (5 minutes) and the immediate post-test, for which the students were given 20 minutes. Students were not allowed to consult the video during this (or the delayed) test.

The third session followed one week later. In a brief (5-minute) introduction, the students were told that, in addition to a test on the trained tasks, there was another test with three new, untrained tasks they were to try to accomplish. The students were instructed to start with the delayed post-test first and then to work on the transfer test. They received 30 minutes to complete both tests together.

Analysis

A check on the random distribution of student characteristics across conditions revealed no statistically significant differences for age or gender. Also, conditions did not differ on scores on the IEMQ nor on the pre-test scores. The effects of presence and timing of practice were assessed for gain scores. The analyses involved ANOVAs. If the assumption of homogeneity of variance was violated, the analysis involved a non-parametric test (i.e., Mann Whitney (U) test or Kruskall-Wallis (H) test). For significant effects of timing of practice, contrasts were computed. Only the statistics for significant findings will be reported in detail. Tests were two-tailed with alpha set at 0.05. The degrees of freedom occasionally differ due to missing data. For significant differences found on non-parametric tests, I report the r-statistic (Field, 2013). This statistic tends to be qualified as small, medium, and large for respectively the values r = 0.10, r = 0.30, and r = 0.50. For ANOVAs, Cohen‘s (1988) d-statistic is used to report effect size. These tend to be qualified as small for d = 0.20, medium for d = 0.50, and large for d = 0.80.

Results

Training Time

The presence of practice had a significant and large effect on training time, F (1,60) = 28.539, p < 0.001, d = 1.54. Training time was shorter in the control condition compared to the practice conditions (see Table 1).

The timing of practice also had a significant effect on training time, F (2,41) = 14.377, p < 0.001. Detailed analyses showed that VP and PV students finished training significantly and substantially faster than PVP students, respectively p < 0.001, d = 2.08, p < 0.001, d = 2.34.

Table 1. Mean (standard deviation) and range for training time by condition

Training time*
Condition M (SD) Range
Video-Practice (n = 18) 24.00 (4.00) 19-32
Practice-Video (n = 17) 23.29 (3.79) 17-30
Practice-Video-Practice (n = 7) 32.29 (3.95) 29-40
Video only (n = 19) 18.11 (4.00) 15-32
Total (n = 61)** 22.92 (5.72) 15-40

* Training time in minutes.
** Problems with the recording software led to missing data for 26 students, mostly in the PVP condition.

Motivation During Training

The students predominantly experienced a positive mood during training (see Table 2); expressions of positive mood states occurred about twice as often as neutral ones. Negative mood states were reported the least often, with a mean frequency of 18.5%. The presence and timing of practice did not affect positive or neutral moods.

The presence of practice had a significant and small effect on negative moods, U(87) = 900.500, z = 2.68, p = 0.007, r = 0.29. Negative mood states were experienced more often in the practice conditions than in the control condition, d = 0.61. The timing of practice had no effect on negative moods.

With an overall mean score of 4.3 points, the results for flow were just above the scale midpoint. This outcome indicates that all students experienced a moderate level of concentration during training and did not feel taxed beyond their capacities. The presence and timing of practice did not affect flow.

Motivation After Training

The scores for task relevance indicate the presence of a low level of motivation before training. After training, these scores had risen to above mid-scale values (see Table 3). The presence and timing of practice did not affect gain scores for task relevance.

The scores for self-efficacy before training indicate that students began training with a modest level of confidence in their capacity to deal with the training tasks. Self-efficacy scores were much higher after training (see Table 3). The presence and timing of practice did not affect gain scores for self-efficacy.

Table 2. Means (standard deviations) for mood and flow by condition

Mood* Flow**
Positive Neutral Negative
Condition M (SD) M (SD) M (SD) M (SD)
Video-Practice (n = 20) 52.5 (42.0) 25.0 (28.1) 22.5 (32.3) 4.29 (1.87)
Practice-Video (n = 24) 44.1 (28.6) 30.9 (24.1) 25.0 (24.5) 4.19 (1.63)
Practice-Video-Practice (n = 23) 56.5 (40.0 24.6 (31.1) 18.8 (36.4) 4.29 (1.91)
Video only (n = 20) 63.8 (41.7) 30.0 (35.9) 6.3 (19.7) 4.50 (1.59)
Total (n = 87) 53.8 (38.1) 27.7 (29.5) 18.5 (29.4) 4.31 (1.73)

* Scores for mood states are given in percentages.
** Scale maximum is 7. A higher score indicates greater flow.

Table 3. Mean (standard deviation) for task relevance and self-efficacy by condition

Task relevance* Self-efficacy*
Before After Before After
Condition M (SD) M (SD) M (SD) M (SD)
Video-Practice (n = 18) 1.52 (0.45) 4.78 (1.37) 3.77 (1.71) 4.52 (1.55)
Practice-Video (n = 21) 1.84 (1.02) 4.46 (1.51) 3.15 (1.89) 4.69 (1.63)
Practice-Video-Practice (n = 14) 2.07 (0.82) 4.82 (1.85) 3.14 (1.80) 4.61 (1.82)
Video only (n = 20) 1.98 (0.99) 4.52 (1.33) 3.34 (1.82) 5.02 (1.29)
Total (n = 73) 1.85 (0.87) 4.62 (1.48) 3.25 (1.80) 4.72 (1.54)

* Scale maximum is 7. Higher scores indicate higher levels of task relevance or self-efficacy.

Learning: Training Test Outcomes

The test scores indicated that the students began training with relatively low prior knowledge (15.9%) and that there were considerable learning gains during and after training (see Table 4). The timing of practice had a significant effect on gain scores from pre-test to practice, H(67) = 7.897, p = 0.019. Further analyses showed that there was a significantly stronger and moderate gain from pre-test to practice for VP compared with PV and PVP, respectively U(44) = 153.500, z = 2.07, p = 0.038, r = 0.31, and U(43) = 123.500, z = 2.63, p = 0.008, r = 0.4. (When the analysis was done for the practice-after task in the PVP condition, the difference with VP became marginally significant, U(43) = 155.000, p = 0.064.)

The presence of practice had no effect on the gain scores from pre-test to immediate post-test. In contrast, there was a significant effect of timing of practice, F(2,66) = 4.183, p = 0.020. Further analyses showed there was a significantly stronger and large gain from pre-test to immediate post-test for PV compared with PVP, F(1,46) = 8.40, p = 0.006, d = 0.85.

The presence and timing of practice had no effect on gain scores from pre-test to delayed post-test.

Learning: Transfer Test Outcomes

The transfer test scores show there was a moderate level of transfer of training (see Table 4). The presence of practice had a significant and moderate effect on the scores for the transfer test, F(1, 86) = 6.231, p = 0.014, d = 0.65. The results from the control group were lower than the scores in the practice conditions. Detailed analyses showed that the difference between the control and VP condition was significant and large, F(1, 39) = 7.336, p = 0.010, d = 0.86. The difference between the control and PV condition was marginally significant F(1, 43) = 3.987, p = 0.052. There was no difference between the control and the PVP condition. The timing of practice had no effect on the scores for the transfer test.

Table 4. Mean percentages (standard deviations) for performance tests per condition

Pre-test Practice Immediate post-test Delayed post-test Transfer test
Condition M SD M SD M SD M SD M SD
Video-Practice (n = 20) 20.0 13.4 68.6 31.3 45.7 29.2 50.0 23.4 57.5 29.4
Practice-Video (n = 24) 11.3 16.3 45.8 26.0 50.6 33.2 51.2 30.4 50.0 27.6
Practice-Video-Practice (n = 23) 16.8 12.7 42.9* 20.7 32.9 19.5 52.8 22.1 46.7 26.4
Video only (n = 20) 16.4 17.5 42.1 25.6 44.3 23.6 33.8 26.0
Total (n = 87) 15.9 15.1 51.6 28.0 42.9 27.8 49.8 25.0 47.1 28.1

* This is the score for the practice-before task. The score for the practice-after task was 52.8% (21.7)

Discussion

The experiment systematically investigated the effect of practice in video-based software training on training time, motivation, and learning. The main findings for these dependent variables will be discussed for the two manipulations of the independent variable, namely, the presence and timing of practice.

The first research question concerned the effects of the presence of practice. For training time, a significant effect was found. The video-only condition completed training substantially faster than the experimental conditions (18 minutes versus 25 minutes). This finding confirms the expectation that complementing a video with practice increases training time.

The practice conditions reported experiencing significantly more negative moods during training than the video-only condition. This finding signals that practice can rouse unpleasant feelings. Much more so than when merely viewing a demonstration of task performance, students are likely to experience obstacles when they engage in their own task practice.

The influence of negative emotions on learning has been studied relatively infrequently in education. The FEASP approach (Astleitner, 2000) contends that instructions should be designed in such a way that they decrease negative emotions, such as Fear, Envy, and Anger, and increase positive emotions such as Sympathy and Pleasure. Because negative moods can have an adverse affect on learning, they should be reduced or avoided whenever possible. The finding from the present study may therefore prompt designers to address the possible occurrence of negative feelings when practice is included in training (e.g., “You may discover that practice is not as easy as it seems. This is often the case when new knowledge must be applied. So don’t worry if task completion does not go smoothly.”).

The experiment showed that students gave substantially higher ratings for task relevance and self-efficacy after training than before. There was no effect of presence or timing of practice on these motivational gains.

The video-only condition had lower gain scores on both post-tests than the practice conditions, but the difference was not statistically significant. The experiment, therefore, did not support the expectation that practice would yield greater learning for trained tasks. However, there was a significant difference on the transfer test, which favored the practice conditions. Detailed analyses revealed that the video-only condition compared especially unfavorably with the video-practice condition. The finding concurs with the outcome from Ertelt’s (2007) study. In worked examples research, such an effect of practice is also repeatedly reported (see e.g., Renkl, 2014a; Salden, Koedinger, Renkl, Aleven, & McLaren, 2010). The transfer test assessed learning beyond what was explicitly trained. This finding, therefore, tentatively supports the claim that practice during training contributes to better software understanding. Practice appears to increase the user’s ability to accomplish related, untrained tasks.

The second research question concerned the timing of practice. This involved comparisons between the three experimental conditions. The PVP condition was found to have significantly longer training time than the other conditions. It is unclear whether this stems from users spending more time viewing the videos or whether it stems from the repeated practice.

There was no significant effect of timing of practice on motivation. The users in the three practice conditions essentially experienced similar mood and flow states during training and did not differ in the development of their appraisals of task relevance and self-efficacy. The findings, therefore, do not support the claim that practice before training increases motivation (compare Stark et al., 2000).

There was a significant effect of timing of practice on the practice test scores. The finding that practice-after (VP) yielded a gain score on this test than practice-before (PV and PVP) is not very striking. It merely illustrates that the videos helped students achieve greater task success during training. What is more surprising, however, is that the PVP condition also did poorly on the practice-after task. The PVP condition was expected to have the best practice test performance, if only because users could engage in practice twice. Apparently, it did not work that way. What happened? Did users tire from having to attend three times in a row (practice-before, video, and practice-after) to the same formatting task?

The PVP condition also had the worst gain score on the immediate post-test. There was a significant effect of timing on this test. Detailed analyses again showed that the PVP condition compared especially unfavorably to the PV condition.

There was no effect of timing of practice on gain in knowledge as shown on the delayed post-test or on the transfer test.

Conclusion

The coupling of instruction and practice, preferably in that order, is a well-established design approach in education. That a coupling of practice and video is also found in video-based traning for software was illustrated in the introduction of this paper. The “Try it” stand-alone video on Adobe’s website came accompanied with downloadable practice files (see Figure 4). These files accommodate practice, enabling the user to mimic the demonstrated actions on his or her own computer. The present experiment aimed to find out whether there is empirical support for complementing instruction with practice and whether the timing of practice vis-à-vis instruction matters.

The findings from the present study do not unequivocally support the inclusion of practice in video-based software training. Practice was found to increase training time. This finding can be seen as an advantage, because it can increase learning. However, it may also be a disadvantage, because it can cause early drop-out. The inclusion of practice also increased the occurrence of negative mood states. This is clearly a disadvantage, and the suggestion was given to try to mitigate the presence and effect of such states when designing for practice. The findings for learning of trained tasks showed that the video-only condition achieved comparable outcomes on an immediate and delayed performance test. The only finding that clearly favored the inclusion of a practice condition concerned the transfer test. On related but not trained tasks, the practice conditions did significantly better than the video-only condition.

In short, the experiment led to somewhat mixed outcomes for the presence of practice in video-based software training. The finding from this single study (and its immediate predecessor) are not enough to proclaim that practice is not needed, of course. Rather, the outcomes suggest that incorporating a practice component in a video-based training design is a more complex design issue than might initially be thought.

The findings for the timing of practice favor a sequence in which instruction precedes practice. On measures of motivation and scores for learning after training, no difference was found between the video-practice and practice-video conditions. In contrast, there was a significant and substantial advantage for the video-practice condition for the practice test.

The findings for the practice-video-practice (PVP) group were somewhat perplexing. Whereas this condition was expected to have the highest learning gains of the practice conditions, learning was actually lower for these students. Both on the practice and immediate post-test, the students in this condition showed significantly lower gains than the other practice conditions. The PVP condition was included in the experiment to represent a realistic scenario where practice precedes video consultation and is followed by further practice. Additional research is needed to determine why the users in this condition performed relatively poorly during and immediately after practice.

References

Adobe. (2017, August 1). Premiere Pro tutorials. Retrieved from https://helpx.adobe.com/premiere-pro/tutorials.html

Alexander, K. P. (2013). The usability of print and online video instructions. Technical Communication Quarterly, 22, 237-259. doi:10.1080/10572252.2013.775628

Astleitner, H. (2000). Designing emotionally sound instruction: The FEASP-approach. Instructional Science, 28, 169–198.

Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of Educational Research, 70(2), 181–214. doi:10.3102/00346543070002181

Brar, J., & van der Meij, H. (2017). Complex software training: Harnessing and optimizing video instructions. Computers in Human Behavior, 70, 1–11. doi:10.1016/j.chb.2017.01.014

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

Doumont, J.-L. (2002). Magical numbers: The seven-plus-or-minus-two myth. IEEE Transactions on Professional Communication, 45(2), 123–127. doi:10.1109/TPC.2002.1003695

Eccles, J. S., & Wigfield, A. (2002). Motivation beliefs, values, and goals. Annual Review of Psychology, 53, 109–132. doi:10.1146/annurev.psych.53.100901.135153

Ertelt, A. (2007). On-screen videos as an effective learning tool. The effect of instructional design variants and practice on learning achievements, retention, transfer, and motivation. (Unpublished doctoral dissertation), Albert-Ludwigs Universität Freiburg, Germany,

Farkas, D. K. (1999). The logical and rhetorical construction of procedural discourse. Technical Communication, 46, 42–54.

Field, A. (2013). Discovering statistics using IBM SPSS statistics (4 ed.). London, UK: Sage.

Fulford, C. P. (1992). Systematically designed text enhanced with compressed speech audio. Paper presented at the Annual Meeting of the Association for Educational Communications and Technology, Washington, D.C.

Grossman, R., Salas, E., Pavlas, D., & Rosen, M. A. (2013). Using instructional features to enhance demonstration-based training in management education. Academy of Management Learning & Education, 12(2), 219–243. doi:10.5465/amle.2011.0527

Guo, P. J., Kim, J., & Rubin, R. (2014, March). How video production affects student engagement: An empirical study of MOOC videos. Paper presented at the L@S ‘14, Atlanta, GA.

Kerr, M. P., & Payne, S. J. (1994). Learning to use a spreadsheet by doing and by watching. Interacting with Computers, 6(1), 3–22.

Kosslyn, S. M., Kievit, R. A., Russell, A. G., & Shephard, J. M. (2012). PowerPoint presentation flaws and failures: A psychological analysis. Frontiers in Psychology, 3, 1–22. doi:10.3389/fpsyg.2012.00230

Leppink, J., Paas, F., van Gog, T., van der Vleuten, C. P. M., & van Merriënboer, J. J. G. (2014). Effects of pairs of problems and examples on task performance and different types of cognitive load. Learning and Instruction, 30, 32–42. doi:10.1016/j.learninstruc.2013.12.001

Mogull, S. A. (2014). Integrating online informative videos into technical communication service courses. IEEE Transactions on Professional Communication, 57(4), 340–363.

Morain, M., & Swarts, J. (2012). YouTutorial: A framework for assessing instructional online video. Technical Communication Quarterly, 21, 6–24. doi:10.1080/10572252.2012.626690

Plaisant, C., & Shneiderman, B. (2005). Show me! Guidelines for recorded demonstration. Paper presented at the 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC’05), Dallas, Texas. http://www.cs.umd.edu/localphp/hcil/tech-reports-search.php?number=2005-02

Plass, J. L., Heidig, S., Hayward, E. O., Homer, B. D., & Um, E. (2014). Emotional design in multimedia learning: Effects of shape and color on affect and learning. Learning and Instruction, 29, 128–140. doi:10.1016/j.learninstruc.2013.02.006

Read, J. C. (2008). Validating the fun toolkit: An instrument for measuring children’s opinions of technology. Cognition, Technology & Work, 10, 119–128. doi:10.1007/s10111-007-0069-9

Reisslein, J., Atkinson, R. K., Seeling, P., & Reisslein, M. (2006). Encountering the expertise reversal effect with a computer-based environment on electrical circuit analyses. Learning and Instruction, 16, 92–103. doi:10.1016/j.learninstruc.2006.02.008

Renkl, A. (2014a). Toward an instructionally oriented theory of example-based learning. Cognitive Science, 38, 1–37. doi:10.1111/cogs.12086

Renkl, A. (2014b). The worked examples principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 391–412). New York, NY: Cambridge University Press.

Rheinberg, F., Vollmeyer, R., & Engeser, S. (2003). Die Erfassung des Flow-Erlebens [Capturing the flow experience]. In J. Stiensmeier-Pelster & F. Rheinberg (Eds.), Diagnostik von Motivation and Selbstkonzept (pp. 261–279). Gottingen, Germany: Hogrefe.

Rosen, M. A., Salas, E., Pavlas, D., Jensen, R., Fu, D., & Lampton, D. (2010). Demonstration-based training: A review of instructional features. Human Factors, 52(5), 596–609. doi:10.1177/0018720810381071

Salden, R. J. C. M., Koedinger, K. R., Renkl, A., Aleven, V., & McLaren, B. M. (2010). Accounting for beneficial effects of worked examples in tutored problem solving. Educational Psychology Review, 22, 379–392. doi:10.1007/s10648-010-9143-6

Spanjers, I. A. E., van Gog, T., Wouters, P., & van Merriënboer, J. J. G. (2012). Explaining the segmentation effect in learning from animations: The role of pausing and temporal cueing. Computers & Education, 59, 274–280. doi:10.1016/j.compedu.2011.12.024

Stark, R., Gruber, H., Renkl, A., & Mandl, H. (2000). Instruktionale Effekte einer kombinierten Lernmethode: Zahlt sich die Kombination von Lösungsbeispielen und Problemlöseaufgaben aus?. [Instructional effects of a combined learning method: Does the combination of worked-out examples and problem-solving tasks pay off?]. Zeitschrift für Pädagogische Psychologie, 14, 205–217. doi:10.1024//1010-0652.14.4.206

Sweller, J., & Cooper, G. A. (1985). The use of worked examples as s substitute for problem solving in learning algebra. Cognition and Instruction, 2(1), 59–89. doi:10.1207/s1532690xci0201_3

Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251–296.

Um, E., Plass, J. L., Hayward, E. O., & Homer, B. D. (2012). Emotional design in multimedia learning. Journal of Educational Psychology, 104(2), 485–498. doi:10.1037/a0026609

van der Meij, H. (2017). Reviews in instructional video. Computers & Education, 114, 164–174. doi:10.1016/j.compedu.2017.07.002

van der Meij, H., Blijleven, P., & Jansen, L. (2003). What makes up a procedure? In M. J. Albers & B. Mazur (Eds.), Content & complexity. Information design in technical communication (pp. 129–186). Mahwah, NJ: Erlbaum.

van der Meij, H., & Gellevij, M. R. M. (2004). The four components of a procedure. IEEE Transactions on Professional Communication, 47(1), 5–14. doi:10.1109/TPC.2004.824292

van der Meij, H., Rensink, I., & van der Meij, J. (In press). Effects of practice with videos for software training. Computers in Human Behavior.

van der Meij, H., & van der Meij, J. (2013). Eight guidelines for the design of instructional videos for software training. Technical Communication, 60(3), 205–228.

van der Meij, H., & van der Meij, J. (2016). Demonstration-Based Training (DBT) for the design of a video tutorial for software instructions. Instructional Science, 44, 527–542. doi:10.1007/s11251-016-9394-9

van der Meij, J., & van der Meij, H. (2015). A test on the design of a video tutorial for software training. Journal of Computer-Assisted-Learning, 31(2), 116–132. doi:10.1111/jcal.12082

van Gog, T. (2011). Effects of identical example-problem and problem-example pairs on learning. Computers & Education, 57, 1775–1779. doi:10.1016/j.compedu.2011.03.019

van Gog, T., Kester, L., & Paas, F. (2011). Effects of worked examples, example-problem, and problem-example pairs on novices’ learning. Contemporary Educational Psychology, 36, 212–218. doi:10.1016/j.cedpsych.2010.10.004

Vollmeyer, R., & Rheinberg, F. (1999). Motivation and metacognition when learning a complex system. European Journal of Psychology of Education, 14(4), 541–554. doi:10.1007/bf03172978

Vollmeyer, R., & Rheinberg, F. (2006). Motivational effects on self-regulated learning with different tasks. Educational Psychology Review, 18, 239–253. doi:10.1007/s10648-006-9017-0

Wistia. (2012, May 7). Does length matter? Retrieved from http://wistia.com/blog/does-length-matter-it-does-for-video-2k12-edition

Acknowledgements

The author wishes to thank Rafaela Zigoli for her help in conducting the experiment. I would also like to thank Emily Fox for her excellent editorial support.

About the Author

Hans van der Meij is senior researcher and lecturer in instructional technology at the University of Twente in the Netherlands. His research interests are technical documentation and the functional integration of ICT in education. His papers have received awards from STC and IEEE, including a “landmark paper” award for a publication on minimalism (with John Carroll). He recently received the Horace Hockley award from ISTC for his contributions to technical communication over a long period of time. He is available at H.vanderMeij@utwente.nl.

Manuscript received 27 March 2017, revised 8 August 2017; accepted 12 September 2017.