| |
"Does
Using Technology in Instruction Enhance Learning?
or, The Artless State of Comparative Research"
The Technology Source, "Commentary," June 1998
http://horizon.unc.edu/ts/commentary/1998-06.asp
Jerald Schutte, a Sociology
Professor at CSU-Northridge, has attracted a lot of attention for his
1996 study on the use of technology in instruction. Schutte reported
that students in his "virtual" class performed 20% better than students
in his "traditional" class, and the results of the study were widely
disseminated on listservs and the popular press (including piece in
the Chronicle of Higher Education, Feb. 21, 1997, A23). His article,
Virtual Teaching in Higher Education: The New Intellectual Superhighway
or Just Another Traffic Jam? is available online at: http://www.csun.edu/sociology/virexp.htm
Unfortunately, Schutte's research design and methodology are so flawed
that the results of the study are uninterpretable. His fundamental error
arises from a confusion between teaching methods and delivery systems,
an error which is common to many studies in this field.
The Study
Students enrolling in Schutte's course in Social Statistics (n = 40)
were randomly assigned to a "traditional" class or a "virtual" class
in roughly equal numbers. Students in the traditional class (where he
presumably lectured) met every Saturday during the 14 week semester
and submitted weekly problem assignments. By contrast, here is Schutte's
description of the virtual class:
The virtual class had four assignments each week: 1) e-mail collaboration
among randomly assigned groups of three students in which they generated
weekly statistical reports and sent them to the instructor using e-mail;
2) hypernews discussion in which a weekly discussion topic was responded
to twice a week by each student; 3) forms input via the WWW which allowed
for student submission of the same homework problems being solved by
the traditional class; and 4) a weekly moderated Internet relay chat
(mIRC) in which student discussion and dialogue were carried out in
real time in the virtual presence of the professor.
At the end of the semester, the virtual students scored an average of
20 points higher on the 100 point midterm and final exams, which Schutte
attributes to "virtual" interaction:
This experiment was intended to assess the merits of a traditional,
versus virtual, classroom environment on student test performance and
student affect toward the experience. It was hypothesized that face-to-face
professor-student interaction is crucial to test performance. However,
the data indicate the reverse, that virtual interaction produces better
results. (Emphasis added.)
Schutte goes on to suggest that perhaps the reason the students in the
virtual class performed better is because they compensated for the lack
of face-to-face contact with the professor by forming study groups to
"pick up the slack of not having a real classroom." (He offers no documentation
about these study groups, so his evidence is apparently anecdotal.)
What he apparently fails to understand is that his study compares students
working in groups with students who are not working in groups, regardless
of whether they were in self-created study groups or in virtual discussion
groups.
Weaknesses in the Study
Actually, students in the virtual class experienced a completely different
method of teaching from those in the traditional class. Not only did
they have more opportunities to be involved with each other and with
the teacher, but (very significantly) they were intensively engaged
with the course material over the entire week. It seems clear that Schutte
would have to provide similar small-group activities, discussion opportunities,
and other assignments for the traditional class if he wanted to test
the comparative effectiveness of virtual instruction with traditional
instruction. Schutte's study simply demonstrates that particular teaching
methods (e.g., cooperative learning and exercises that insure more time
on task with the material) yield improved performance, results that
educational researchers established years ago. (See, for example, research
summaries in Johnson & Johnson, 1989, Nilson, L. B. 1998, and Menges,
Weimer, & Associates, 1996.)
There are also a number of methodological weaknesses in this study.
Although Schutte did administer a pre-test to measure the demographic
similarity of the two groups, he didn't test them for statistical knowledge
or aptitude (only "statistical feelings" and "math feelings" on a scale
from one to ten). Pre-existing differences in statistical expertise
could explain the superior performance of the virtual class, especially
given the small number of students in each group. (Indeed, if the virtual
group did have more expertise or aptitude, one might expect them to
out-perform the traditional class by more than 20%.) Furthermore, Schutte
provides only scant information about the course tests and doesn't describe
the grading procedure at all. He does say that both mid-term and final
exams had four parts 1) matching, 2) objective, 3) definitions; and
4) problems. It would be important to know if the tests were "blind
graded," across students as well as across the two groups, especially
since portions of the tests require subjective judgment on the part
of the grader (definitions and problems). An effective research design
would include pre-test and post-test measures and selection of significance
levels prior to collecting the data.
In short, the Schutte study violates so many of the rules of empirical
research that we cannot use it to draw any conclusions about the comparative
effectivess of "real" versus "virtual" teaching. Can we learn anything
useful about teaching with technology from his work? One could argue
that, without technology, Schutte would not have experimented with other
teaching methods. We have all heard anecdotes about professors who,
in trying to incorporate technology into their courses, adopt better
instructional approaches as a consequence. However, we don't know how
many teachers benefit in this way nor whether students in their classes
actually learn more material or master more skills. This argument also
fails to address the question of whether we need the technology at all
if we can be more effective by simply adopting these teaching methods
in the traditional classroom--and it would certainly be cheaper.
Alternative Outcomes
Does Shutte's study demonstrate that it is possible to conduct cooperative
learning in a virtual environment? The answer to this question depends
on one's definition of cooperative learning. At its simplest level,
cooperative learning is "a structured, systematic instructional strategy
in which small groups work together toward a common goal" (Cooper and
Mueck, 1990, p. 68), but authorities on the subject insist that effective
cooperative learning requires more than simply putting students into
small groups and giving them a task to perform. For example, there must
be "positive interdependence" among group members, which means that
each member's success must depend to some degree on the group's success.
"Individual accountability" is also necessary, which means that all
members of a group are accountable for their own learning as well as
for the other students in the group, and group members should share
the work load equally. The group must also be aware of (and consciously
work on) collaborative social skills such as conflict management and
giving constructive feedback. The social aspects of cooperative learning
groups, which are crucial to their effectiveness, are difficult (if
not impossible) to replicate in a virtual environment. Schutte obviously
did not set out to create cooperative learning groups by this definition,
but his study shows that some aspects of student collaboration can be
accomplished in "virtual space." However, even this conclusion is provisional,
and it begs the question of how much more effective the groups could
have been if they had been "real" instead of "virtual."
Finally, one might look at Schutte's project in terms of the effectiveness
of "total instructional redesign," since his methods in the virtual
class were so different from his methods in the traditional class. If
we accept the results of his study as valid (which, for the reasons
given above, we cannot do), his "redesign" could be called effective.
However, we cannot ignore the enormous costs of the technology in this
equation. If he had used these methods in his traditional class, costs
would not increase, but because he and his students needed the networked
technology of a major educational institution, the costs of this change
are extremely high. One could argue that traditional instruction requires
buildings, desks, heat, light, and other resources that cost enormous
amounts. However, buildings don't need to be rebuilt every five years
and desks don't become obsolete because someone has upgraded the design,
nor do traditional classrooms require a platoon of highly-paid experts
to stay "on-line."
Other Comparative Research
Schutte's study is flawed, but many of the research studies in this
area are also weak. Thomas Russellís survey of research on what he characterizes
as "The ëNo Significant Differenceí Phenomenon" (http://tenb.nbcc.nb.ca/phenom/)
is often used to support the argument that it doesnít matter what delivery
system is used, since there is no difference in how students perform.
But if one actually reads the studies on Russell's list, it is obvious
that they suffer from many shortcomings: the research designs are poorly
conceived, the statistical analysis is weak or absent, and/or the sample
size is too small. Many studies donít try to measure learning outcomes
at all, but focus instead on attitudinal outcomes--how the students
felt about the experience rather than what they learned. Attitudinal
outcomes are important, but certainly only in conjunction with cognitive
outcomes. The studies that do try to assess student learning as an outcome
variable often use tests that measure simple recall of information rather
than mastery of higher-order learning. In these cases, it is not surprising
that there is "no significant difference" in performance because it
really doesnít matter to students how they acquire factual information.
Conclusion
Educational researchers face many difficulties in trying to conduct
controlled studies in university settings, since threats to validity
and reliability are often beyond the influence of the investigator.
As a result, a number of people who are interested in the application
of technology to teaching have abandoned traditional research models,
opting instead for "proof in practice." The Flashlight Project of the
American Association for Higher Education is an example of this approach
(a description of the project is available on the AAHE web site: http://www.aahe.org/).
This strategy can work, and might ultimately identify successful applications
of technology in teaching, as long as evaluations of the results are
based on learning outcomes. Unfortunately, most of these projects repeat
the same error that Schutte committed, confusing teaching processes
with instructional outcomes.
For example, the Flashlight Project seeks to discover "whether faculty
and students find the available technology useful (or a hindrance) when
they try to implement each of 'seven principles of good practice in
undergraduate education.'" The seven principles (from Chickering and
Gamson, 1987) are:
Interaction between the student and teacher (or tutor, or other expert);
Student-student interaction;
Active learning;
Time on task;
Rich, rapid feedback;
High expectations of the student's ability to learn; and
Respect for different talents, ways of learning.
Their justification for using this approach reveals an interesting,
if flawed, line of reasoning: "Because so much research indicates that
these practices support better learning, it would be significant to
discover that they were being implemented and that technology was playing
an important role." In short, if we find that these secondary indicators
are present in technology-based instruction, we should assume that learning
occurred. Most teachers know that any teaching method can be performed
well or poorly, so the presence of (for example) active learning techniques
will tell us nothing about the quality of the learning outcomes. Nor
will this project tell us what the students learned (facts, ability
to apply knowledge, critical thinking skills) or whether they could
have learned better or faster without the technology. The evaluation
does not address cost-benefit issues with respect to learning outcomes,
so the question of comparative costs of "delivering" instruction with
and without technology will go unanswered. These are the kinds of questions
that are essential for evaluating any educational program, but especially
programs that place increased demands on the human and financial resources
of higher education.
In the 1960ís, instructional television promised to change teaching
and learning dramatically. College administrators and state legislators,
hoping to "expand educational opportunity" (and ultimately save money)
by using this electronic delivery system, invested millions in closed-circuit
systems, TV production facilities, educational television stations,
and even airborne broadcasting systems. Proponents of ITV in the '60s
predicted that "as much as 50 percent of the college degree program
will be available for credit via television" in the future (Murphey
& Gross, 1966, pp. 83-95). Instructional television failed to achieve
the transformation of higher education, but the 1960's were boom years
for education and the grand experiment with ITV (wasteful as it was)
occurred in a time of flush resources. Today, higher education operates
in a much tighter fiscal environment and we cannot afford to make many
mistakes. We therefore need better studies, focused on significant questions,
to guide us in developing appropriate and cost-effective applications
of the new technology. To make educational policy decisions or base
large investments on anything less is foolhardy.
____________________
Chickering, A. and Gamson, Z. (1987) Seven principles of good practice
in undergraduate education, AAHE Bulletin (March).
Cooper, J. & Mueck, R. (1990). Student involvement in learning:
Cooperative learning and college instruction. Journal on Excellence
in College Teaching, 1, 68-76.
Johnson, D. W. & Johnson, R. T. (1989). Cooperation and competition:
Theory and research. Edina, MN: Interaction.
Nilson, L. B. (1998). Teaching at its best: A research-based resource
for college instructors. Bolton, MA: Anker.
Menges, R. J., M. Weimer, & Associates. (1996). Teaching on solid
ground: Using scholarship to improve practice. San Francisco: Jossey-Bass.
Murphey, Judith, & Gross, Ronald. (1966). Learning by Television.
New York: The Fund for the Advancement of Education.
|