What is the evidence on effectiveness of e-learning?
The evidence on e-learning as compared to conventional teaching is a mess to interpret. In some studies, students are not randomly assigned to either the e-learning or conventional alternative, and so the quality of students may differ. In other cases, comparing an e-learning class that, say, requires a quiz every week to a conventional class with a midterm and a final may tell you more about the value of weekly quizzes than about e-learning itself. Thus, Bell and Federman turn to "meta-analyses," which is the term for studies that look at the results of dozens or hundreds of different studies, and thus can make statistical adjustments for what kind of e-learning is being done, how it is being evaluated, characteristics of students and teachers,and so on. The overall theme is that e-learning often does just as well as conventional learning. Here are summary statements about a few of the meta-analyses: I'll first give the citation for the study, and then use Bell and Federman's words to summarize the findings.
Robert M. Bernard and others, “How Does Distance Education Compare with Classroom Instruction?
A Meta-Analysis of the Empirical Literature,” Review of Educational Research 74 (2004): 379–80.
Bell and Federman: "In summary, the meta-analysis revealed no significant overall difference between e-learning and traditional instruction in terms of overall achievement, but more negative student attitudes toward synchronous e-learning and higher dropout rates in asynchronous e-learning." This study considers both "asynchronous (mostly correspondence and online courses, in which students participate at different times) and synchronous (mostly teleconferencing and
satellite-based delivery, in which all students participate simultaneously)."
Traci Sitzmann and others, “The Comparative Effectiveness of Web-Based and Classroom Instruction: A Meta-Analysis,” Personnel Psychology 59 (2006): 623–64.
Bell and Federman: "[W]eb-based instruction was 6 percent more effective than traditional classroom
instruction for teaching declarative knowledge (facts and principles), but not procedural knowledge (rules and procedures) or student reactions. Used as a supplement to classroom instruction (blended learning), web-based instruction was 13 percent more effective than classroom instruction for declarative knowledge and 20 percent more effective for procedural knowledge. ... Indeed, the authors found web-based and classroom instruction equally effective for teaching declarative knowledge when the instructional methods used in both were equivalent. They attribute the small overall advantage of web-based instruction to its use of more (and more effective) instructional
methods, rather than to the delivery media per se."
Barbara Means and others, Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies, report prepared for the U.S. Department of Education, Office of Planning, Evaluation, and Policy Development (Washington: U.S. Department of Education, September 2010).
Bell and Federman: "[Students who took a course online did not perform significantly differently than those taking the same course through traditional face-to-face instruction. Students in courses that combined online and face-to-face instruction (blended learning) had stronger learning outcomes than
did those in face-to-face instruction alone. Both instructor-directed and collaborative and interactive online instruction (both fully online and blended) led to stronger outcomes than classroom instruction, but outcomes in independent online learning and face-to-face instruction had no significant difference."
Traci Sitzmann, “A Meta-Analytic Examination of the Instructional Effectiveness of Computer-Based
Simulation Games,” Personnel Psychology 64 (2011): 489–528.
Bell and Federman: "Simulation games were more effective than lectures, assignments, and readings, but less effective than computerized tutorials. Trainees learned more from simulation games when they had unlimited access to the games (presumably leading to more time spent learning) and when
the games were embedded in a program of instruction (blended learning). In fact, when simulation games were the sole instructional method, trainees in the comparison group learned more than those in the simulation game group. Finally, in studies that matched the simulation and comparison groups in terms of the activity level of instruction, learning was similar across conditions. Once again, this finding suggests that the learners in the simulation games condition may have been advantaged not because of the delivery media per se, but rather because they often received more active instruction than those in the comparison group."
I was a little surprised at the findings of these meta-analyses, given that they are looking at studies of e-learning as it existed several years ago. I suspected that e-learning would catch up with classroom learning at some point, but it may already have done so. I'm probably not alone in being surprised: "A survey of the general public conducted by the Pew Research Center using a nationally representative sample of 2,142 adults found that only 29 percent believe online courses are as valuable educationally as courses taken in the classroom."
Given the tone of these study results, Bell and Federman argue that the issue is no longer whether e-learning can be effective. Clearly, it can be. The issue is now one of instructional design: that is, what characteristics of a particular course are especially important. They write:
"[S]tudies designed to evaluate the effectiveness of a particular e-learning technology are of limited value. Indeed, any form of instruction can be effective if it is able to create the conditions necessary for students to learn specific content. ... Empirical research is also shifting away from evaluating whether e-learning works and toward examining the instructional features that influence its effectiveness. Rather than comparing different forms of delivery such as e-learning versus classroom, studies are beginning to compare e-learning programs that differ on important instructional dimensions, including interactivity, engagement and activity, and feedback."
Along with this focus on specific instructional features, what are some of the other main questions facing e-learning at the college level? Here are three that I took away from the essay.
How to deal with cheating? In a pure online course, how do you know who is at the other end of the screen answering questions? One can imagine various security measures, like having a camera snap a series of photos at random times during an exam (but what if someone is whispering answers from off-camera), or having student take their exams in a campus testing center.
What about students who are uncomfortable or unprepared for e-learning? Students learn in all kinds of ways, some by reading, some by listening, some by talking in study groups, some by writing out answers by hand, and so on. Some students won't do well with e-learning, either because it's just not their thing, or because they don't yet have a high comfort level with computers in general. Some students will learn the material quickly in any format. But figuring out how to make e-learning work for as broad an audience as possible, and thinking about alternative versions of e-learning for different audiences, is a big task.
Does e-learning save money? A lot of the hope of e-learning is that it will provide education less expensively, but at least so far, that doesn't seem to be true. Bell and Federman: "[F]ew institutions believe e-learning reduces their costs, and, in fact, most believe that online courses are at least as expensive to provide as traditional courses. This perspective is based largely on the significant start-up costs of e-learning, including investments in technology, course design, and the training of instructors, but also on recurring costs, such as those that result from increased coordination demands and technical support."
Maybe the high costs of e-learning are mainly start-up costs? Maybe as technology improves, the costs will come down and the effectiveness will go up? Maybe some kinds of e-learning courses, after they are developed, can then be scaled up to very large numbers of students ? On the other side, e-learning is going to keep offering new capabilities, which are sure to be expensive to develop, and likely to be costly to maintain. If e-learning is to have the desired qualities and outcomes, it won't come cheap.
ADDED: For answers to the three questions above from Daniel Lemire, a computer science professor with considerable experience in on-line teaching, see here.