Will the use of computers in classrooms fundamentally change how we educate
children? And can educational technology lead to improved student achievement?
Our review of the research on the effectiveness of educational technology
at the Educational Testing Service (ETS) shows that rudimentary uses of
computers--using drill-and-practice software, for instance, to teach addition
and subtraction--can be effective and efficient. But more pedagogically
complex uses of the computer--using the Internet in small groups to conduct
collaborative research, for example--often show inconclusive results. It
appears that the more complex and sophisticated the instructional design,
the more difficult the evaluation. But sometimes these complex situations
offer promising and inviting vignettes on the future of teaching and learning.
The clear benefits
Studies show computer-based instruction can individualize instruction
and give instant feedback to students, even explaining the correct answer.
The computer is infinitely patient and nonjudgmental, thus motivating students
to continue.
In a presentation to a recent RAND conference, researcher James Kulik
summarized more than a decade's worth of work spent analyzing the effectiveness
of computers used for instruction. A research approach called meta-analysis
allowed him to aggregate the findings of more than 500 individual studies
of computer-based instruction. The studies, conducted independently by research
teams using different methods at eight research centers, focused on different
uses of the computer with different populations. Kulik drew several conclusions
from his work:
* Students usually learn more in less time when they receive computer-based
instruction.
* Students like their classes more and develop more positive attitudes
toward computers when their classes include computer-based instruction.
* Computers do not, however, have positive effects in every area in which
they were studied. In 34 studies that examined students' attitudes toward
subject matter, for instance, the average effect of computer-based instruction
was near zero.
For the most part, the computer programs reviewed in Kulik's analysis
were developed before 1990 and tended to emphasize drill and practice. More
recently, the Software Publishers Association released a report, prepared
by an independent consulting firm, that analyzed 176 studies conducted from
1990 to 1995 on the effectiveness of technology in schools. The report shows
students in technology-rich environments experienced positive effects on
achievement in all major subject areas, preschool through higher education,
for both regular and special-needs students. Most students--although not
necessarily low-achieving students, who tended to require more structure--were
better able to pace themselves when technology was used. Student attitudes
toward learning and the students' own self-concepts improved consistently
when computers were used for instruction. "The use of technology as
a learning tool can make a measurable difference in student achievement,
attitudes, and interactions with teachers and other students," the
report concludes.
Numerous studies have demonstrated that technology is particularly valuable
in improving student writing. The ease with which students can edit their
written work on word processors makes them more willing to do so, which
in turn improves the quality of their writing. Studies have shown that students
are also better at critiquing and editing written work that is exchanged
over a computer network with students they know. And student writing that
is shared with other students over a network tends to be of higher quality
than writing produced for in-class use only.
Other benefits are documented as well. As schools have added computers,
they've reported improvements in their attendance and dropout rates. They've
also reported students are more challenged, more engaged, and more independent.
Encouraged to experiment and explore the new frontiers of knowledge through
the use of technology, students have assumed more responsibility for their
assignments and produced higher-quality work.
More difficult to judge
The RAND report goes on to say that the "more cognitive" applications
of technology--those involving such skills as critical thinking or cooperative
learning--are more difficult to evaluate. The research data are less extensive,
the data that exist are harder to organize, and new evaluation designs are
often needed. Nevertheless, according to the RAND report, programs with
more-cognitive applications can engage students in authentic tasks, often
with other students using computer network software and databases that are
intended not only to improve subject matter learning, but to develop skills
in cooperation, communication, and problem solving.
A recent report by Beatrice Berman and her colleagues at the American
Institutes for Research in Washington, D.C., describes the implementation,
effectiveness, and role of technology in the context of education reform
efforts. Until new and ongoing evaluations of cutting-edge educational technology
projects are available, Berman says, the findings from the following projects
represent the best of currently available research:
* The Role of Online Communications
in Schools: A National Study. This project, conducted by the Center for Applied Special Technologies
(CAST), operates on the premise that the best way to introduce students
to online computing is to do so within the context of what is already happening
in the classroom. The study compared the work of 22 fourth and sixth-grade
classes in seven urban school districts--half with access to online communications
and the Internet and half without. In a semi-structured instructional unit
completed over a two-month period, all classes studied issues of civil rights
by researching specific topics, sharing information, and completing a final
project.
In the end, CAST researchers found that fourth-graders with online access
scored significantly higher on two of nine learning measures, and sixth-graders
with online access scored significantly higher on four of nine learning
measures.
* Technology's
Role in Education Reforms. This four-year study, conducted for the
U.S. Department of Education's
Office of Educational Research and Improvement, by Barbara Means, Kerry
Olson, and colleagues, set out to understand how technology can support
constructivist teaching at the classroom level. The schools or projects
included in the study all served substantial numbers of disadvantaged
students.
Overall, the researchers reported that increases in technology had positive
effects on these schools, leading to increased motivation and improvements
in academic performance. Seven of the eight schools in the study reported
lower teacher turnover, six reported higher student attendance rates, and
five had higher test scores than a comparison group. In addition, fewer
disciplinary incidents were reported.
* Union City Interactive
Multimedia Education Trial. The Union City, N.J., school district
implemented a five-year plan that included a significant investment in technology
to support its curriculum reform goals. In addition, Bell Atlantic worked
with the city, the state board of education, and the Center for Children
and Technology at the Educational Development Center Inc., in Princeton,
N.J., to carry out a technology trial at two schools.
Although the district's entire comprehensive reform program has yielded
substantial gains in student progress, the most impressive results have
occurred at Christopher Columbus Intermediate School. For example, Columbus
students led the district in the percentage of students passing practice
administrations of New Jersey's Early Warning Test. In addition, more Columbus
students qualified for the ninth-grade honors program than did students
from any other city school; Columbus had the district's best attendance
record both for students and for faculty; and Columbus had the highest number
of transfers in and the fewest numbers of transfers out between 1993 and
1995.
* Higher Order Thinking Skills Program (HOTS). Begun in the early
1980s as an alternative to Title I, HOTS is a pull-out program that has
evolved into a widely used and effective intervention for disadvantaged
fourth through seventh-graders. The program's detailed and creative curriculum
aims to build the thinking skills of students through exposure to computers,
drama, and Socratic dialogue.
Recent research shows the combination seems to be working. HOTS students
have outperformed a control group of students in a traditional Title I program
on all measures: They've doubled the national average in gains on reading
and math test scores, and they've shown improvements in, among other things,
reading comprehension, metacognition, writing, and grade point averages.
Ten to 15 percent of the Title I and learning-disabled students enrolled
in HOTS made the honor roll in 1994.
* Assessing the Growth: The Buddy Project Evaluation, 1994-95. The
state of Indiana, along with the Lilly Endowment and Ameritech, sponsored
this project, which supplied students with home computers and modem access
to the school. An assessment of the project indicated significant differences
between seven Buddy Project classrooms and three non-Buddy Project classrooms.
The Buddy Project students showed improvement in all writing skills, a better
understanding and broader view of math, more confidence with computer skills,
an ability to teach others, greater problem-solving and critical thinking
skills, and greater self-confidence and self-esteem.
* Apple
Classrooms of Tomorrow (ACOT). Sponsored by Apple Computer
Inc., ACOT focuses on the changes in instructional practices and
student learning that occur when schools and students have extensive access
to technology. In the initial years of the project, each student and teacher
was given two computers, one for home and one for school.
Over 10 years of evaluation, independent researchers found that ACOT
students continued to perform well on standardized tests and developed a
number of competencies not usually measured. According to this research,
ACOT students explored and represented information dynamically and in many
forms, became socially aware and more confident, communicated effectively
about complex processes, used technology routinely and appropriately, became
independent learners and self-starters, knew their areas of expertise and
shared that expertise spontaneously, worked well collaboratively, and developed
a positive orientation to the future.
* Computers Helping Instruction and Learning Development (CHILD).
This project began in 1987 as a five-year investigation into the use of
educational technology in nine Florida elementary schools serving over 1,400
students. Three to six computers were placed in each classroom, and teachers
received training in both the technology and the concept of establishing
a team environment with other teachers in the project. The project was seen
as a move toward "student empowerment," and much of the students'
daily routine involved self-paced interactions at learning stations.
Standardized test scores indicated positive and significant results across
all grades, schools, and subjects, with the largest effects appearing for
students who had been in the program for more than one year. When surveyed,
none of the nine schools expressed dissatisfaction with the project, five
were planning to expand their level of participation, and nine new schools
were about to become involved.
Looking at the big picture
When we try to determine the effectiveness of educational technologies,
we are confronted by a number of methodological and practical issues. First,
we need to remember that technology is only one component of an instructional
activity. Assessments of the impact of technology are really assessments
of instruction enabled by technology, and the outcomes are highly dependent
on the quality of the implementation of the instructional design.
According to Roy Pea, now the director of SRI's Center for Technology
in Learning in Menlo Park, Calif., the "social contexts" of how
technology is used are crucial to understanding how technology might influence
teaching and learning. Educational technologies cannot be effective by themselves.
The social contexts are all-important. This means more attention should
be paid to the teaching strategies used both "in" the software
and "around it" in the classroom, and to the classroom environment
itself. It is a recurrent finding that the effects of the best software
can be neutralized through improper use, and that even poorly designed software
can be creatively extended to serve important learning goals.
There are also a host of methodological issues to confront. First, standardized
achievement tests might not measure the types of changes in students that
educational technology reformers are looking for. New measures, some of
which are currently under development, would assess areas, such as higher
order thinking skills, that many believe can be particularly affected by
using new technologies.
There is also a need to include outcome measures that go beyond student
achievement, because student achievement might be affected by students'
attitudes about themselves, their schools, the types of interactions that
go on in schools, and the very idea of learning. Another consideration is
pointed out by the U.S. Office of Technology Assessment's Teachers and
Technology: Making the Connection: Technological changes are likely
to be nonlinear, and might show effects not only on student learning, but
also on the curriculum, the nature of instruction, the school culture, and
the fundamental ways that teachers do their jobs.
In a paper presented at this year's meeting of the American Educational
Research Association, Ellen Mandinach and Hugh Cline of ETS explored many
of the challenges to the scientific examination of technology's impact on
education. They suggested the need to focus on longitudinal design, multiple
methods, multiple levels of analysis, and systems analysis in lieu of traditional
methodologies. Traditional research designs are inadequate, inappropriate,
and often ask the naive question, "Does it work?" The impact of
technology is too multifaceted for such a simple question, which cannot
be answered without considering the impact on students' learning and motivation;
classroom dynamics, including interactions among students, teachers, and
technology; and schools as formal organizations. Perhaps the most important
lesson is the absolute necessity for researchers to remain flexible in applying
their methodological knowledge in a field setting--that is, to make continuous
adjustments in all aspects of implementation and in assessment efforts,
in the hope of gaining a more thorough understanding of technology's impact
on teaching and learning activities.
Evaluators of educational technology are hampered by the fact that they
are often chasing a moving target. Policy makers and the public want to
know whether investing in a particular type of statewide computer network
is worthwhile, but by the time evaluation data are collected and analyzed,
the particular network is probably obsolete, and another investment opportunity
needs to be explored.
An example from our files
A real-life example can serve to illustrate some of these thorny evaluation
issues.
ETS was asked to document and evaluate the New Jersey Networking Infrastructure
in Education Project, which is funded by the National Science Foundation
and is aimed at enhancing elementary and secondary science education through
the use of the Internet. The project's goal is to connect 500 schools to
the Internet, to train teachers to access and use the Internet, and, ultimately,
to develop science curricula that draw from the Internet and its wealth
of real-time data.
Gita Wilder, who heads the evaluation portion of the project, has identified
four issues that have arisen from the New Jersey project and probably apply
to any effort to evaluate technology-based innovation in schools and classrooms.
First, she says, it's impossible to systematically assess cognitive and
achievement outcomes for students without addressing variations in their
starting points and differences in program implementation. Second, it is
simplistic to suggest, given the variations in starting points and implementation
in different classrooms and different schools, that technology can produce
comparable outcomes among classes and students.
Third, although the motivational and attentional benefits of technology
have been widely reported, the cognitive and achievement effects have not
been as consistently cataloged. Researchers and teachers need to work together
to develop hypotheses about how students' cognitive processes and school
achievement might be affected by the consistent and innovative application
of technology; to test these hypotheses in small and controlled studies;
and to design larger-scale field studies that test the results under a range
of classroom and school conditions.
Finally, Wilder says, it is time to stop discounting the importance of
changes in teaching practices that accompany the use of technology in the
classroom. So far, attention has focused on the effect of educational technology
on students and the way they learn, but more attention should be paid to
the effects technology has on teachers and the way they teach. After all,
students move on; they are affected by conditions that are both cumulative
and changing, but teachers remain to influence another generation of students.
Richard J. Coley, education policy analyst at the policy
information center at the Educational Testing
Service in Princeton, N.J., is a coauthor of Computers and Classrooms:
The Status of Technology in U.S. Schools, from which this article is adapted.
A complete copy of the report
is available online. A print copy ($9.50), is available by e-mail;
phone (609) 734-5694; or write Policy Information Center, 04-R, ETS, Rosedale
Rd., Princeton, NJ 08541. |
