Colleges themselves might want to ask next year's applicants to download a copy of the report and write an essay responding to it. What conclusions the applicants draw will say a lot about their critical thinking skills. If an applicant wholeheartedly accept its assertion that the average SAT gains derived from enrolling in commercial test preparation programs likely are "in the neighborhood of 30 points," perhaps an elite college is not the best fit for him, and he would be better off somewhere close to home, maybe within bicycling distance.
The author of the report is Derek C. Briggs, an associate professor of quantitative methods and policy analysis at the University of Colorado at Boulder and the chairman of the university's Research and Evaluation Methodology Program. The report was commissioned by the National Association for College Admission Counseling, a Virginia-based organization representing high school counselors, for-profit college counseling providers, and college employees involved in admissions and financial-aid decisions. The association tacked onto the report's cover the caveat that its conclusions are Mr. Briggs' own, but it nonetheless has publicized the report as part of NACAC's efforts to advance "the knowledge base and dialogue about test preparation."
The NACAC press release announcing the report declares: "Report Highlights Test Prep Paradox—Paying for Test Prep Doesn’t Yield Big Returns, But Returns May Still Matter in Light of Admission Practice." Among the major media outlets which ran with the "no big return" assertion were the Washington Post (headline: "Study Sees Small Average Gains from College Test Coaching") and the Wall Street Journal (headline: "SAT Coaching Found to Boost Scores--Barely").
Briggs concludes that SAT test preparation increases scores on the math portion of the test by just 10 to 20 points, and on the verbal portion of the test by just 5 to 10 points. He does not base that conclusion on any new research, but on a review of a tall stack of past studies of the impact of test coaching. Actually, to be precise, he bases his conclusion on just three studies in the stack. He discounted the rest--some of which found score increases from coaching of 100 points or more--as based on small samples that were not representative of the nation's population or as otherwise methodologically flawed.
Two of the three studies that Derek C. Briggs characterizes as valid and pointing to "a consensus position" on the effects of SAT test coaching were performed by none other than Derek C. Briggs. (If his name is otherwise recognizable to people in the field, it is because, far from being a neutral arbiter of such research, he already has established himself as a prominent critic of the idea that SAT coaching works.) Briggs not only put himself in a position to pass judgment on his own research and (surprise surprise) declared his own work rock solid, but he also has declared a consensus based on me-myself-and-I vote counting. The third study that he counts toward that consensus, by Donald Rock and Donald Powers, unsurprisingly reaches the same conclusion he had.
About that only other study in the stack that Briggs found methodologically acceptable: It was sponsored by two organizations which are highly invested, financially and otherwise, in the idea that SAT scores cannot be raised significantly by coaching--the College Board, which owns the SAT, and the Educational Testing Service, which administers it. Powers was a principle research scientist at ETS, and, as the book Color and Money shows, both ETS and the College Board have a history of promoting research that makes their case and squelching research that doesn't.
Powers and Rock identified those students who had received SAT test coaching based on whether the students owned up to it in response to a questionnaire on ETS letterhead sent to them after they had taken the SAT and before they got their scores back. Based on their choice of methodology, one wonders if they would have published a study concluding that married men seldom cheat based on a surveys administered to husbands by wives with revolvers in their hands. As Bob Schaeffer, a spokesman for the watchdog group FairTest, noted in an e-mail:
"The notion that students would respond accurately to a testing company's questionnaire asking whether they had been coaching, especially during the period after they had taken the exam and before scores were reported, is ludicrous. At a minimum, they would wonder how that information would be used. Given fears about the secrecy with which ETS/College Board handle data, might have even believed that ETS could 'flag' scores being sent to colleges to indicate that an applicant had been coached, just as they then did for tests taken with extended time."
To his credit, Briggs acknowledges--albeit in a subtle, after-the-fact sort of way--that the three studies he cites, in talking about the average gains derived from SAT coaching by nationwide samples of students of all backgrounds, mask differences in the effectiveness of programs based on their quality, setting, and duration. He also acknowledges the two cited studies of his own that he cites "suggest that coaching is more effective for students with strong academic backgrounds and high economic status who underperformed on the PSAT."
The implications of this concession are huge. His new report, and the hype surrounding it, are wrongly leading the nation to believe that high-priced SAT test preparation services do not substantially raise the scores of students who participate in them in earnest, when, in fact, the average gains produced by some coaching services quite possibly may be the 100 points or more that other studies have claimed. The "average gains" Briggs' report cites are based on studies that included slackers who dropped out of coaching programs or repeatedly skipped sessions or screwed around and paid no attention to their instructors at all, as well as people who enrolled in ineffective, disreputable programs. If one wants to determine how much weight people lose on Weight Watchers, one should base that study on the weight changes of people who enrolled in Weight Watchers and took it seriously, not on the entire universe of people who have ever declared they are going on a diet and, in some cases, sat down to wolf down that "one last piece of pie."
The idea that families should not waste their money on coaching programs that won't raise scores is a perfectly fine one to get out. But, while NACAC may be well-intentioned in trying especially hard this week to get word out to families who are low- or middle-income (and presumably don't have money to spare), one wonders if the group will be contributing to class-linked gaps in access to selective higher education by so targetting its message. After all, its report, read carefully, suggests the rich might in fact be wise to enroll their children in reputable programs costing $1,000 or more, because such programs are likely to produce gains that will make a difference in the admissions office. SAT coaching already is giving the children of the wealthy enough of an unfair edge in getting into top colleges without broadcasting the message that those of lesser means should not bothering trying to put their children on equal footing.
The results of a survey of colleges contained in the new NACAC report makes clear that, even if the gains derived from test coaching are only in the neighborhood of 30 points on a 1600 point scale, those 30 points can make a lot of difference at some colleges and among students who were high scorers to begin with. Of colleges that use the SAT in evaluating applicants, 21 percent have a rigid cut-off scores. And, at the upper end of the SAT score range, well over a third of colleges said a 20 point increase in an applicant's SAT math score or a 10 point increase in an applicant's SAT verbal score would "significantly improve" their likelihood of gaining admission. Although NACAC and the College Board have advised colleges against giving small differences in SAT scores much weight in admissions decisions, their admonitions appear to be falling against deaf ears. Why? Relying heavily on SAT scores offers selective colleges an inexpensive way to sort through an annual barrage of applications. Taking in high scorers helps colleges boost their rankings in U.S. News and World Report and other college guides. And colleges have an additional, and powerful, financial incentive to depend heavily on the SAT, in that the strong correlation between SAT scores and family wealth means that high scorers are more likely than other applicants to get through college without needing financial aid and to donate generously to their alma maters down the road.
The problem with nearly all research on SAT preparation is that it is hard to find a disinterested party to do it. The College Board and ETS, which make huge sums of money off administering the SAT, know that its very existence is threatened by research showing that the test can be beaten by coaching. The test preparation industry has a financial incentive to argue the SAT can in fact be beaten by coaching and their services will give college applicants a big bounce in their scores. Selective colleges--and the members of organizations like NACAC--have incentives, financial and otherwise, to keep the test around. Advocates of low-income minority students see the SAT as a screening device that favors the white and wealthy, even without SAT coaching being factored into the equation.
It is possible to construct an experiment empirically measuring whether SAT coaching works. Doing so would require randomly assigning students to experimental and control groups and comparing test-to-test changes in the scores of those who had gone through coaching and those who had not. Unfortunately, no such experiment has yet been performed, leaving the nation's parents having to base their decisions on advertisements by test-preparation companies and "research" that may be no more trustworthy.
UPDATE: Bob Schaeffer of FairTest points out that a study involving experimental and control groups of students actually was done back in 1988. It was for a Ph.D. dissertation, and, although it had a fairly small sample, the fact a doctoral student pulled it off suggests the possibility of conducting other such studies on a broader scale. A link to it is here.
