COMPARISON OF FIXED-ITEM AND RESPONSE-SENSITIVE VERSIONS OF AN ONLINE TUTORIAL

：芭芭拉

COMPARISON OF FIXED-ITEM AND RESPONSE-SENSITIVE VERSIONS OF AN ONLINE TUTORIAL
Lyle K Grant, Marni Courtoreille. The Psychological Record. Gambier: Spring 2007. Vol. 57, Iss. 2; pg. 265, 8 pgs

Abstract (Summary)
This study is a comparison of 2 versions of an Internet-based tutorial that teaches the behavior-analysis concept of positive reinforcement. A fixed-item group of students studied a version of the tutorial that included 14 interactive examples and nonexamples of the concept. A response-sensitive group of students studied a different version of the tutorial in which, after an initial round of 10 example and nonexample items, the computer readministered item types the students had answered incorrectly until the students responded correctly to each of 5 example and 5 nonexample item subtypes. A control group studied an online tutorial in biological psychology. Students in all 3 groups took a 10-item pretest and posttest over positive reinforcement. An analysis of the students' change scores indicated that the students learned more using the response-sensitive tutorial than the fixed-item tutorial. Scores of students using both versions of the tutorial were superior to those of the control group. Students using the response-sensitive and fixed-item tutorials required the same amount of study time. Students rated both versions of the positive reinforcement tutorial favorably on several dimensions.

：芭芭拉

Full Text (3495  words)

Copyright The Psychological Record Spring 2007

[Headnote]
This study is a comparison of 2 versions of an Internet-based tutorial that teaches the behavior-analysis concept of positive reinforcement. A fixed-item group of students studied a version of the tutorial that included 14 interactive examples and nonexamples of the concept. A response-sensitive group of students studied a different version of the tutorial in which, after an initial round of 10 example and nonexample items, the computer readministered item types the students had answered incorrectly until the students responded correctly to each of 5 example and 5 nonexample item subtypes. A control group studied an online tutorial in biological psychology. Students in all 3 groups took a 10-item pretest and posttest over positive reinforcement. An analysis of the students' change scores indicated that the students learned more using the response-sensitive tutorial than the fixed-item tutorial. Scores of students using both versions of the tutorial were superior to those of the control group. Students using the response-sensitive and fixed-item tutorials required the same amount of study time. Students rated both versions of the positive reinforcement tutorial favorably on several dimensions.


Online tutorials and demonstrations have become widely available to educators as a source of computer-based instructional materials (e.g., Davidson, 2004; Krantz, 2004; National Institutes of Health, 2004). These resources provide the advantages of computer-based instruction (Alessi & Trollip, 2001; Kulik, 1986; Kulik & Kulik, 1991) to online users at no cost beyond Internet access charges. Many of these online tutorials are not interactive because there is no opportunity to respond to the content and receive guidance or feedback (Alessi & Trollip, 2001). When opportunities for interaction are provided, tutorial exercises often consist of a fixed number of items or problems that the students answer. This use of a fixed number of items does not give students an opportunity for additional practice over item types they answer incorrectly. In the present study, we compared two versions of an Internet tutorial designed to teach the psychology concept of positive reinforcement. One of these versions, which students had used for several years and evaluated highly (Grant, 2004), consisted of a fixed number of items that students answered and received feedback. We compared this fixed-item tutorial with a responsesensitive version in which the computer monitored student answers and gave students more conceptual problems of the type they had answered incorrectly until the students answered those problems correctly.

Rothen and Tennyson (1978) taught positive reinforcement using a response-sensitive procedure that requires pretesting and applications of probability theory that are difficult for many web-based instructors to implement without considerable assistance. In the present research, we used a response-sensitive strategy that web-based instructors can easily understand and use. The response-sensitive strategy we used, which Alessi and Trollip (2001) identify as "flash card queuing," retires item subtypes that have been answered correctly and reschedules item subtypes that have been answered incorrectly for later presentation. Although this flash card strategy is familiar to many educators as a method for drilling students over factual information, it has not been studied extensively (Siegel & Misselt, 1984) and has not been examined as a method to teach concepts.

：芭芭拉

Method

Participants

Two hundred thirty-four students enrolled in an introductory psychology course participated in the study to fulfill a research participation requirement. The students attended a distance-education open university.

Procedure

Students entered a research participation Web page using a password and their university student number. They read a consent form, agreed to participate in the research, and took a 10-item pretest on the concept of positive reinforcement. The pretest consisted of five examples of positive reinforcement and five nonexamples. The students' task was to judge whether the items were examples of positive reinforcement. Following the pretest, a computer program randomly assigned the students to (a) a fixed-item group, (b) a response-sensitive group; or (c) a control group. Each of the three groups consisted of 78 students.

The fixed-item tutorial group entered a Web-based tutorial on the concept of positive reinforcement (Grant, 1996). The tutorial defines and exemplifies the concept with a series of six example-nonexample pairs designed to highlight the critical features of the concept. In an examplenonexample pair, the concept is illustrated in an example and presented along with a nonexample that lacks a single critical feature of the concept (Carnine, 1980; Tennyson, Woolley, & Merrill, 1972).

After reading the example-nonexample pairs, the fixed-item students began an interactive exercise consisting of eight examples and six nonexamples of positive reinforcement. Students decided if each item was an example of positive reinforcement and identified their decision using a mouse click. After mak-ing a selection, the computer told the students whether the selection was correct and explained why the item was an example or a nonexample. Previous work has shown that explaining why items are examples or nonexamples promotes concept learning (Grant, McAvoy, & Keenan, 1982). If students responded incorrectly, they received corrective feedback via a hyperlink based on specific errors.

The response-sensitive tutorial group read the same introductory material as the fixed-item tutorial group, but did a different interactive exercise. For the response-sensitive group, the first round of the interactive exercise consisted of five nonexamples and five examples of positive reinforcement that were randomly selected from a pool of 50 items. The 50 items in the pool included 25 examples and 25 nonexamples. The 25 examples consisted of five item subtypes of five items each. The five example subtypes were (a) items in which the positively reinforcing consequence was ostensibly undesirable (e.g., a slap or a reprimand); (b) items in which the reinforcing consequence was an opportunity to engage in an activity (e.g., free play); (c) items in which the reinforcing consequence was intermittent rather than dependent on each response; (d) items in which the positively reinforced behavior was undesirable (i.e., disruptive classroom behavior); and (e) items illustrating positive reinforcement in which the consequence was ostensibly desirable, the consequence was presentation of a stimulus rather than an activity, reinforcement was continuous rather than intermittent, and the behavior was desirable. Likewise, the 25 nonexamples consisted of five subtypes of five items each. The five nonexample subtypes were (a) items in which there was no increase in response rate; (b) items in which the consequence was presented on a response-independent rather than on a response-dependent basis (Lattal & Poling, 1981); (c) items in which the behavior occurred more frequently due to rule-following rather than the application of reinforcing consequences; (d) items in which the consequence was dependent on the nonoccurence of a behavior rather than the occurrence of a behavior; and (e) items in which the consequence consisted of removal of a stimulus (i.e., negative reinforcement) rather than presentation of a stimulus. The response-sensitive students interacted with the exercise items in the same way as the fixed-item students did. The students decided whether each item was an example of positive reinforcement. After mak-ing a selection, the computer indicated whether the selection was correct and explained why the item was an example or a nonexample. If students responded incorrectly, they received corrective feedback via a hyperlink based on specific errors.

Following the first round of 10 items of the response-sensitive exercise, students received an onscreen summary report indicating which of the 10 subgroups of examples and nonexamples they answered correctly and incorrectly. For students who answered all 10 items correctly, the computer indicated that they had finished the interactive exercise and they could click on a button to send them to the posttest. For response-sensitive students who answered one or more of the items incorrectly during the first round, a second round began in which the student answered one new item from each of the 10 subgroups that they had answered incorrectly on the first round.

After the students had answered all the second-round items, they received another summary of their performance on each of the subgroups. If they answered all the second-round items correctly, the computer informed them that they had completed the exercise. If students answered one or more of the second-round items incorrectly, a third round began that included new items from subgroups answered incorrectly during the second round. Students initiated new rounds of items in this fashion until they had either answered all the items in a round correctly or had exhausted an item subtype pool by answering all five of the subgroup items incorrectly in five successive rounds. If an item subtype pool was exhausted in this way the computer indicated that the program had run out of items and students could click on a button to send them to the posttest.

The control group entered a Web tutorial in biological psychology over the structure and function of the human ear. This tutorial was part of a larger set of tutorials that teach many concepts in biological psychology (Randall & Grant, 2000).

After the students completed the tutorials, they took a 10-item posttest, which consisted of five examples and five nonexamples of positive reinforcement. The posttest items, like the pretest items, consisted of a new item from each of the 10 subgroups of examples and nonexamples. The students' task was to judge whether the items were examples of positive reinforcement.

Following the posttest, students in the response-sensitive and fixeditem groups answered a five-item online evaluation. The first four items were (a) "I found the tutorial helpful in learning the concept," (b) "I would like to make use of other online tutorials like this one in my courses," (c) "I would recommend this tutorial to other students," and (d) "The examples in the tutorial suggested ways of applying the concept in my day-to-day life." Students evaluated these four statements on a 5-point scale (1 = Strongly agree; 5 = Strongly disagree). The fifth item on the evaluation was the question: "Was the tutorial too long or too short?" Students used a 5-point scale to answer this question (1 = Much too long; 5 = Much too short).

Results

We analyzed pretest-posttest change scores using a 3 x 2 ANOVA with the tutorial type as a between-subjects variable and examples/ nonexamples as a within-subjects variable. Analysis of the examples/ nonexamples variable permitted examination of the tutorial's influence on errors of overgeneralization (i.e., identifying nonexamples as examples) and undergeneralization (i.e., identifying examples as nonexamples). The overall effect of the tutorial type was significant, F(2, 231) = 72.55, p < .0001. Students who completed the response-sensitive tutorial had a mean pretest-posttest improvement of 2.35 correct test items (SD = 1.75). Students who completed the fixed-item tutorial had a mean pretest-posttest improvement of 1.65 correct test items (SD = 1.75), and students in the control group had a mean pretest-posttest improvement of .72 correct test items (SD = 1.46). Neuman-Keuls tests indicated that (a) the difference between the response-sensitive tutorial group and the control group was significant, p < .01; (b) the difference between the response-sensitive tutorial group and the fixed-item tutorial group was significant, p < .01; and (c) the difference between the fixed-item tutorial group and the control group was significant, p < .01. Neither the difference between examples and nonexamples, F(1,231) = .04, nor the interaction between tutorial type and examples/nonexamples, F(1, 231) = .21, was significant.

We calculated the time elapsed between the time the students submitted the pretest and the time they submitted the posttest in order to measure the duration of time spent on the tutorial. For the response-sensitive group, the mean elapsed time was 32.83 min (SD = 15.18), whereas this value was 33.00 min (SD = 12.74) for the fixed-item group and 32.92 min (SD = 20.75) for the control group. The difference in elapsed time among groups was not significant, F(2, 231 ) = .02.

Students in the response-sensitive and fixed-item groups rated the Positive Reinforcement Tutorial favorably. These students agreed with the statements "I found the tutorial helpful in learning the concept" (M = 1.82, SD = 0.05), "I would like to make use of other online tutorials like this one in my courses" (M = 1.94, SD = 0.07), "I would recommend this tutorial to other students" (M = 1.87, SD = 0.06), and 'The examples in the tutorial suggested ways of applying the concept in my day-to-day life" (M = 2.19, SD = 0.07). The students answered the question "Was the tutorial too long or too short?" by indicating that the length was "Just right" (M = 3.03, SD = 0.03). We analyzed possible differences in the ratings between the response-sensitive and fixed-item groups using an analysis of variance, but there was no significant difference between the two groups in their ratings of any of the evaluative items.

：芭芭拉

Discussion

The response-sensitive version of the tutorial was more effective than the fixed-item version, producing superior pretest-posttest change scores yet requiring the same amount of learning time and producing the same positive student evaluations. The pretest-posttest change score was over 42% higher for the response-sensitive group compared to the fixed-item group, representing an educationally significant difference.

These results suggest that other Internet-based tutorials that rely on a fixed sequence of interactive items should be altered such that the computer presents students with the type of problems they are answering incorrectly, thereby focusing student attention and study time on the kind of problems the students find difficult. In this respect, the present data strengthen the generality of Rothen and Tennyson's (1978) finding that a response-sensitive strategy is highly effective in teaching concepts by showing that the effect is maintained across procedural variations.

Historically, applications of behavior analysis in higher education (Buskist, Cush & DeGrandpre, 1991; Grant & Spencer, 2003; Johnson & Ruskin, 1977) have made use of mastery criterion on unit quizzes in order to improve students' final exam performance. A disadvantage of this approach is that students find out whether they have met the mastery criterion or are required to restudy the material only after they have taken a quiz. It would be more desirable if the student received ongoing feedback regarding their mastery of the material and opportunities for remedial study, if necessary, in the study situation where most students' learning takes place. The use of response-sensitive computer-based instruction that we examined in this study essentially moves the mastery criterion, and any required remedial instruction, from the unit quiz to the students' study environment. This gives students additional continuous feedback regarding the effectiveness of their studying, automatically implements remedial instruction for those students who require it, and potentially removes the stigma and inconvenience of failure on unit quizzes. In the present study, a relatively modest mastery criterion of one correct answer to each item subtype yielded a substantial improvement in posttest performance. The present data suggest that programmed and semiprogrammed textbook exercises with a fixed number of items (e.g., examples and nonexamples of behavior-analysis concepts, math problems, etc.) would be more effective pedagogically if they were converted to a response-sensitive computer-based format.

In the present study we did not collect data regarding how many total items students in the fixed-item and response-sensitive groups studied. It is possible that the response-sensitive students learned more because they simply studied more example and nonexample items, but this possibility is discounted by the equal learning time durations: If the response-sensitive students received more items, then their learning time ought to have been higher than that of the fixed-item group, but observed learning time durations were equal. A more likely possibility is that the response-sensitive and fixed-item students studied about the same number of items, but that the response-sensitive students studied (a) more of the item types they initially answered incorrectly and (b) fewer of the item types they initially answered correctly than the fixed-item students. Further work would be useful in verifying the precise features of the response-sensitive procedure that were responsible for its superiority.

The extent of the superiority of the response-sensitive tutorial came as a surprise to the author of the original fixed-item tutorial. The selection of original 14 items of the fixed-item tutorial represented the author's intuitions, based on considerable experience in teaching concepts using examples and nonexamples (e.g., Grant, 2004; Grant & Evans, 1994; Grant et al., 1982), regarding what was required to teach the concept effectively. Selection of the items on the basis of these intuitions however, was less beneficial than basing item selection on student errors after the computer had initially presented an appropriate range of items. The present data suggest that a major advantage of computer-based instruction is the capability of computer programs to focus individual student activity on instructional content based on student error patterns. These data further suggest that presenting instructional tasks on the basis of ongoing individualized student error patterns is more helpful than presenting such tasks based on instructor preconceptions regarding what students require.

Authoring response-sensitive versions of computer-based instruction is more time-consuming than writing the equivalent fixed-item version. Although it is seldom addressed in the literature, this is a practical disadvantage of the response-sensitive strategy because instructor time is limited. In the present study, the authors wrote only 14 items for the interactive portion of the fixed-item tutorial compared to 50 responsesensitive items. There is no clear algorithm for determining whether the additional time the instructor spends on a particular set of teaching materials is well spent or could be better spent elsewhere. Nonetheless, the magnitude of the response-sensitive effect in this study suggests that the additional time required to write the response-sensitive tutorial was at least a strong candidate for the instructor's attention.

One solution to the problem of limited instructor time is to make use of the Internet as a means of distributing the increased workload though collaborative interinstitutional ventures (Grant, 2004). The positive reinforcement tutorial, for example, is maintained on one computer server but is generally available to all Internet users. If different institutions were to develop and maintain different response-sensitive tutorials, all students would benefit by having access to better software than any one institution could afford to develop alone. In principle, this distributed workload would allow individual instructors to create and perfect many responsesensitive tutorials, permitting students of all the instructors to use an entire curriculum of tutorials too numerous and time-consuming to be created and maintained by any one individual or one institution. This distributed approach to providing students with computer-based materials demands more interinstitutional coO p e r ation than currently exists as well as a greater sense of collective responsibility for universal student success.

[Reference]
References
ALESSl, S. M., & TROLLIP, S. R. (2001). multi-media for learning: Methods and development (3rd. ed.). Boston: Allyn and Bacon.
BUSKIST, W., CUSH, D., & DEGRANDPRE, R. J. (1991). The life and times of PSI. Journal of Behavioral Education, 1, 215-234.
CARMINE, D. W. (1980). Relationships between stimulus variation and the formation of misconceptions. Journal of Educational Research, 74, 106-110.
DAVIDSON, M. W. (2004). Molecular expressions: Science, optics and you. Retrieved November 7, 2005, from the Florida State University Web site: http://micro.magnet.fsu.edu/optics/tutorials/.
GRANT, L. (1996). Positive reinforcement: A self-instructional exercise. Retrieved November 7, 2005, from the Athabasca University Web site: http://psych. athabascau.ca/html/prtut/reinpair.htm.
GRANT, L., & EVANS, A. (1994). Principles of behavior analysis. New York: HarperCollins.
GRANT, L., MCAVOY, R., & KEENAN, J. B. (1982). Prompting and feedback variables in concept programming. Teaching of Psychology, 9, 173-177.
GRANT, L. K. (2004). Teaching positive reinforcement on the Internet. Teaching of Psychology, 31, 69-71.
GRANT, L. K., & SPENCER, R. E. (2003). The Personalized System of Instruction: Review and applications to distance education. The International Review of Research in Open and Distance Learning, 4. Retrieved November 7, 2005, from http://www.irrodl.0rg/content/v4.2/grant-spencer.html.
JOHNSON, K. R., & RUSKIN, R. S. (1977). Behavioral instruction: An evaluative review. Washington, DC: American Psychological Association.
KRANTZ, J. H. (2004). Psychology tutorials and demonstrations. Retrieved November 7, 2005 from the American Psychological Society web site: http:// psych.hanover.edu/Krantz/tutor.html.
KULIK, C-L. C. (1986). Effectiveness of computer-based education in colleges. AEDS Journal, 19, 81-108.
KULIK, J. A., & KULIK, C-L. C. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers in Human Behavior, 7 (1&2), 75-94.
LATTAL, K. A., & POLING, A. D. (1981). Describing response-event relations: Babel revisited. The Behavior Analyst, 4, 143-152.
NATIONAL INSTITUTES OF HEALTH. (2004). Medline plus: Interactive health tutorials. Retrieved November 7, 2005 from the National Institutes of Health Web site: http://www.nlm.nih.gov/medlineplus/tutorial.html.
RANDALL, S. N., & GRANT, L. K. (2000). Introductory biological psychology tutorials. Retrieved November 7, 2005, from the Athabasca University Web site: http://psych.athabascau.ca/html/Psych289/Biotutorials/.
ROTHEN, W., & TENNYSON, R. D. (1978). Application of Bayes' theory in designing computer-based adaptive instructional strategies. Educational Psychologist, 12, 317-323.
SIEGEL, M. A., & MISSELT, A. L. (1984). Adaptive feedback and review paradigm for computer-based drills. Journal of Educational Psychology, 76, 310-317.
TENNYSON, R. D., WOOLLEY, F. R., & MERRILL, M. D. (1972). Exemplar and nonexemplar variables which produce correct concept classification behavior and specified classification errors. Journal of Educational Psychology, 63, 144-152.

[Author Affiliation]
LYLE K. GRANT and MARNI COURTOREILLE
Athabasca University

[Author Affiliation]
Please address correspondence concerning this manuscript to Lyle K. Grant, Psychology Centre, Athabasca University, 1 University Drive, Athabasca, Alberta, Canada, T9S 3A3. (E-mail: lyle@athabascau.ca). We thank Dean Mah for his assistance with the computer programming for this study.