Education That Works:
The Child is Always Right, Part Two
How do we test for teaching effectiveness?

Edward L. Anderson, Ph.D.

For thirty years, an unheralded group of “behavior analysts” have been doing just that! Their research has demonstrated that all students (“special ed,” low IQ, disadvantaged, non-English speaking, many problem behavior students, most “slow learners” and many “Attention Deficit Hyperactivity Disorder” (ADHD) children) can learn at roughly the same rate, though they begin and end at different levels. In spite of these results educators still assert that student background, preparation, and even a good self-image are pre-requisite to learning!

These methods have taught both disadvantaged and well-prepared children over two grades in one year and illiterate adults two grades in 20 hours. Later I’ll prove these statements, but for now, let’s just accept them while we pursue two broader questions: How do we test for teaching effectiveness? and Why haven't effective methods been adopted?

How do we test for teaching effectiveness?

The commonly proposed remedies have not worked. We have data on the effects of reduced class size, using magnet schools, and spending more money on facilities and teacher pay. They didn’t work! In spite of their value they, of themselves, cannot improve American education. Only a scientific approach to teaching can, and only if we are ready to accept the results that science gives us.

Today, untested new ideas come from nowhere: There were never any long term data that justified the massive, near-instant switch to “open classrooms” twenty years ago. Old ideas keep reappearing; the “look-say” approach to reading replaced “phonics” in my youth, but it has now reappeared as “whole language.” Santayana was right; unlike real sciences, education repeats the past.

So how can we separate the wheat from the chaff? Julie Vargas at West Virginia University gives us tests to apply to proposals that purport to improve educational effectiveness. There is a hierarchy of five levels:

• Level 1. Test your students. This so-called “post” measurement will at least tell you how good your students are now, relative to national averages. Schools should have these results available.
• Level 2. Pre and post measures. Has the change been already demonstrated to be effective, using measurements made both before and after the intervention? This will document gain, if any. While essential, note that you will have learned nothing about whether a different intervention would have been better or poorer.
• Level 3. Comparison. Compare pre- and post-tests in the new program against those of students not in the program. Is there substantial superiority?
• Level 4. Cost effectiveness. Students may learn better but the increased gain per student must be worth the money. But note that using cost per student may not be as meaningful as cost per successful student. $8,000/year spent on each student is really $16,000 if only half learn. This is much more expensive than $9000/year if every student graduates.
• Level 5. Evidence of long term impact on society, usually in dollar costs or savings due to lower costs of special education, public assistance, and/or crime. It is probably premature to apply this test under present conditions where not even the first three tests are common.

The U.S. Department of Education itself pays only lip service to evaluation. In its booklet “Schools that Work,” 23 exemplar programs are discussed. Of 23 descriptions, only 4 reach even Level 2, pre- and post-test measurement! Of the remaining 19, 9 programs give some kind of post-test measures, but 10 give no data whatsoever! None give evidence at Levels 4 or 5. Far from showing what works, the booklet gives the impression that enthusiasm, rather than results, is what the government endorses.

Note how little in teaching has scientific merit. We honor “exemplary teachers”, but testimonials don’t meet scientific proof standards! Have their classes been tested against others, even in the same school? And how can ephemeral skills be taught to other teachers? Transferring their skills implies a definable, organized, and universally reproducible set of principles that all teachers can master; in short, a “technology.” Could we all become good comedians by hearing even several lectures from Bob Hope? No.

Textbook selection procedures are themselves a scandal. Less than two percent of curricula have been tested prior to adoption to see how effective they actually are with students!

California and Texas, because of their size, determine by their selection what texts will be published and hence, effectively, the curricula for much of the nation. Groups of curricula experts define what the programs should teach and a formal competition is set up in which publishers are given perhaps a year to develop materials designed to win. There obviously is no time for pre-testing! The famous physicist Richard Feynman, in his biography, tells of his disgust when serving on one of the evaluation panels. He surreptitiously prepared two books with all blank pages. Both were evaluated as above average.

California’s Educational Code (its law!) states that effectiveness shall be considered as a factor in selection of curricula, yet in a recent competition to select language arts curriculum materials, California rejected as ineligible a behavior analysis program that demonstrably had taught all students basic language arts effectively for over twenty-five years.

The program, Direct Instruction (“DI”), was the only program in 5,000 submissions that had any data on how well students learned from the submitted materials. It was at Level 4, showing superiority to all other methods, and cost effectiveness. The author of the program, Siegfried Engelmann of the University of Oregon, sued the state over the rejection of his materials. He won in lower court, the state appealed, and Engelmann won again, the appeals court agreeing that California had violated its own law. Rather than reconsider his materials, the state Department of Education persuaded the legislature (without public hearings) to pass a bill suspending the law.(1,2)

In 1990-91, San Diego dropped a reading program that showed average success in favor of one more congruent with the district’s philosophy.(3) In the seven years before the new program, over 55% of first-graders tested above national norms in reading. After the new program, “Whole Language,” was introduced, only 26% of first-graders tested above national norms. The schools created thousands of disadvantaged kids in one year. How will they ever be able to catch them up, even if they go back to the old, only average program?

Note that merely testing to Level 3 would have exposed this program’s deficiencies. But the new program fits most educators’ theoretical ideology and, two years later, is still in use.

Chester Finn, a former Assistant Secretary of Education, observed that

“three promising educational reform ideas are under siege in Washington: national school standards, exams keyed to these standards, and the use of exam results... Trying to cut their supply lines are an army of education establishmentarians, academics and congressmen.”

Edward L. Anderson, Ph.D.
Ed Anderson was among the founders of the Cambridge Center,
and remains one of its most active supporters.
This paper is based upon a talk that Dr. Anderson gave on August 2, 1994,
sponsored by the Chautauqua Institution, Chautauqua, New York.

Return Home