Peter Schmidt's AlterNet Essay--and His Response to Its Critics

From Peter Schmidt:

On Saturday, May 23, the Web site AlterNet published an essay in which I argued that selective colleges bear some responsibility for our current economic crisis because their admissions policies reward and encourage unethical behavior and their graduates account for a disproportionate share of those in positions of economic or political power. The essay was widely e-mailed and republished and generated a lot of discussion on the Internet. The responses to it included amens and applause, ad hominem attacks on me and my educational background by people who know little about me and absolutely nothing about my educational background, and critiques that, in some cases, were thoughtful.

The ad hominem attacks don't deserve a response other than to say it is sad to see people who claim Ivy League degrees or positions as tenured professors incapable of coming up with anything better.

Common courtesy demands that I give those who applauded my essay my thanks.

The criticisms that I found relevant and at least somewhat worth taking seriously deserve an answer. The AlterNet piece is reprinted immediately below, and my response to its critics follows.

Elite Colleges Are Promoting a Culture of Selfish, Cutthroat Behavior and We Are All Paying the Price

By Peter Schmidt
Like many of us, the nation's elite colleges and universities have taken a financial beating over the past year.

Among them, Harvard, Yale, Princeton and Stanford all watched their endowments shrink by about 20 percent as a result of investment losses.

Despite all their brainpower, such institutions appear to have failed to learn what every simple farmer knows: you reap what you sow. Elite colleges and professional schools bear a share of the blame for the economic crisis that now plagues them, because it is they who educated and bestowed academic credentials upon many of those who got us into this mess.

It should come as no surprise to them that many on Wall Street and in Washington have proven ethically bankrupt and without regard for people of lesser means, because their admissions policies have done much to ensure such a result.

In determining which applicants they will admit and put on the fast track, most elite higher-education institutions systematically favor people from privileged backgrounds who display selfish, cutthroat behavior. The results are campus environments where disregard for society is socially accepted, where bad people are encouraged to become worse.

Consider, for starters, how most such institutions rely on standardized admissions tests such as the SAT, even though they know perfectly well that the nation's massive test-preparation industry has severely compromised the reliability of such instruments, turning them into tools for measuring, as much as anything, wealth and willingness to seek unfair advantage.

Test-preparation programs make people better test-takers not better prospective students. They raise scores mainly by teaching various test-taking tricks, such as how to quickly spot the "sucker" answers to a multiple-choice question to improve the odds of guessing correctly. Yet many are effective enough to offer those families that can afford their fees -- typically, $500 to $1,000 -- a chance to buy their children enough extra points to transform many from also-rans into shoo-ins.

In turning a blind eye to the widespread tainting of admissions test scores, higher-education institutions argue that they lack better mechanisms for efficiently judging applicants from high schools of sharply varying quality. But many education researchers disagree and say some alternatives to such tests, such as admissions systems that give substantial weight to class rank or samples of each applicant's work, are more reliable predictors of applicants' academic performance.

Moreover, selective colleges have ulterior motives for relying on standardized admissions tests that have nothing to do with academic considerations and everything to do with their bottom lines. The more high-scoring students they admit, the higher their "selectivity" ratings in the college-ranking guides that help determine how many applicants knock on their doors each year. And not only is sifting through applications based on test scores a lot cheaper than hiring enough people to consider each candidate carefully, but relying on such scores helps skew the process in favor of wealthier applicants, who will not need financial assistance and are likely to donate generously down the road.

If young people find that artificially inflating their test scores isn't enough to get them into a choice college, they always have the option of having someone bribe their way in with a big donation.

Selective colleges are so happy to have their palms greased in such a manner that some make little effort to hide how much they lower the bar for applicants connected to generous alumni and other contributors. To improve their odds of having favors done for them by people in positions of power, many selective higher-education institutions also admit mediocre applicants at the request of state and federal officials.

They let their professors and administrators in on the game by lowering the bar for the children of employees, as a job perk. Despite all of their talk about operating athletics programs to promote sportsmanship, they assure recruited athletes the playing field will be tilted in their favor in the competition for freshman-class seats.

Through such admissions policies, colleges end up giving the nation's high school students crash courses in cynicism. They teach young people that money talks, fairness is for losers, who you know matters more than what you know, and some people are simply entitled to what others may never attain, no matter how hard they work.

Considering how much selective colleges and universities favor applicants who take such lessons to heart, should it surprise anyone that about half of all graduate- and professional-school students admit on surveys to having recently cheated?

Investors take note: MBA candidates have been found to be the biggest cheaters of all, with 56 percent admitting to having cheated in the past year, in a 2006 survey published by the Academy of Management Learning and Education. Many business schools have responded to the latest economic crisis by broadcasting their intent to beef up their ethics classes, but they might as well be promoting sobriety in a bar.

Give George W. Bush credit for this much: He admits to having gotten into Yale through his family connections, and he is quite capable of self-effacing humor. In delivering Yale's 2001 commencement address, he declared: "And to the C students I say, You, too, can be president of the United States."

Although he meant the remark as a joke, he stood as living proof that he was absolutely right, that students who have gotten through the doors of a top college need not perform well there to have other doors opened to them.

Historians of education say the Great Depression shook the nation's faith in its leadership and helped inspire many selective colleges to reform their admissions policies to do more to take in the best students and not just the best-connected.

Our latest economic crisis could inspire similar soul-searching and a renewed emphasis on meritocracy in higher education. But it also could have the opposite effect, prompting selective colleges and universities to even more heavily favor those applicants with cash and connections in an effort to repair their own finances.

If the recent devastation of their endowments should teach such institutions anything, it is that basing their admissions policies on the short-term pursuit of monetary gain is likely to cost them -- and the rest of American society -- dearly down the road.

So far, at least, the more serious critiques of the essay have taken one of four forms: 1) assertions that it places too much faith in meritocracy 2) complaints that it paints elite colleges and their students with too broad a brush 3) allegations of faulty logic 4) allegations that it makes assertions based on no evidence. I'll pick them off here one by one. (Full disclosure: I'll get a marginal commission from Amazon every time you buy a book through one of the links I provide below.)

Re: the assertion the essay places too much faith in meritocracy

I take this criticism more to heart than any. I'm a huge fan of Michael Young's landmark satirical essay, The Rise of the Meritocracy, which makes abundantly clear how a society ruled by those considered "the best"--and therefore confident of their superiority--could indeed be a very brutal place. I'm also well aware that most definitions of "academic merit" in college admissions also tend to be measures of economic and cultural advantage and, in some cases, the willingness of parents to give their children an edge by any means necessary--basic fairness and the good of society be damned. (The second chapter of my book gives a thorough overview of research on this topic.)

But I fail to see how admissions decisions based solely on academic merit would be worse for society than admissions practices that suspend considerations of academic merit in the case of people who have displayed unethical behavior. With a merit-only approach, you get some mixture of people who are smart/essentially decent and people who are smart/unethical and, if Young is right, run the risk of everyone involved being corrupted by belief in their own superiority. With our current admissions policies, you have people who are dim/unethical bumping the smart and ethical out of seats in freshmen classes, thereby enlarging the unethical population on campuses and reducing the enrollments of smart/ethical people who might go on to help hold the unethical in check. Meanwhile, the colleges insist everyone on their campus actually belongs there based on merit, so the perceptions of superiority that Young worries about exist anyway.

It's entirely possible that society may be well-served by selective colleges bending or expanding their definitions of academic merit to accommodate certain populations, such as people who have some unusual talent not measured by traditional academic criteria or people who have done remarkably well given their life circumstances. But I'm not sure I see much societal interest in bending or expanding definitions of merit to accommodate people who pull strings, offer bribes, or might perform well enough on the athletic field to help sell stadium seats. Selective colleges often argue that the money derived through such practices can help fund scholarships for the needy, but I am hard-pressed to think of another American institution that tries to justify accepting bribes or giving in to undue political influence on the grounds that whatever it gains by doing so it uses for some societal good.

Re: complaints that the essay paints elite colleges and their students with too broad a brush

Any careful reader will see that the essay does not argue that all students at elite colleges engage in selfish, cutthroat behavior or that all elite colleges engage in every admissions practice cited as favoring the ethically challenged. To suggest otherwise is to construct a straw man. Plenty of elite college students and graduates who profess idealism and concern for the best interests of society have agreed with my characterization of many students on such campuses without taking my essay the least bit personally.

Critics of the essay likewise themselves grasp at too broad a brush in asserting that I argue that all political leaders or business people responsible for our current economic crisis came out of elite colleges. I simply don't.

Based on research to be discussed later in this blog entry, I will say this much, though: Elite higher education institutions credentialed a disproportionate share of the political leaders and business people responsible for our current economic crisis. Nearly all selective colleges engage in at least a few of the admissions practices my essay describes. And the population of selfish, cutthroat, entitled students found on most such campuses is large to provide considerable social support for such thinking and behavior.

Re: allegations of faulty logic

Every allegation of faulty logic so far directed at this piece has been based on straw men constructed by misconstruing my statements. For example, people have claimed the essay argues that all unethical people come out of elite colleges and therefore everyone enrolled at an elite college is unethical. That's nonsense.

At the core of my essay is a sound deductive argument. It goes like this:

Premise 1 (based on extensive research): Elite higher education institutions educate and credential a disproportionate share of our society's leaders and influence even those who do not pass through their doors.

Premise 2 (also based on extensive research): Many of the admissions practices of elite higher education institutions favor applicants based on displays of unethical behavior and send the clear message that those institutions regard at least some unethical behavior as acceptable.

Conclusion: Elite higher education institutions therefore bear some responsibility for the presence of unethical people in our society's leadership positions.

Re: allegations that the essay makes assertions based on no evidence

I'll acknowledge offhand that the essay makes several assertions without expressly citing the research and data they are based on.

This was partly a function of necessity. The article was a journalistic op-ed, not a book or submission to an academic journal. It is a roughly 1,000-word essay in a world where many newspapers and magazines will not print essays over 600 words. Buttressing every assertion with a full discussion of its factual basis likely would have caused the essay to grow to 5,000 words or more, making it unpublishable and damn near unreadable.

For the record, however, every single assertion in the essay is fully supported by extensive research, much of it discussed in the book Color and Money and on this Web site.

Here's a breakdown of how the essay's key assertions are backed:

The assertion that elite colleges train a disproportionate share of people in positions of economic or political power is supported by several books, studies, and legal documents cited in Color and Money, including:

Michael Useem and Jerome Karabel, “Pathways to Top Corporate Management,” 175–207; Charles L. Cappell and Ronald M. Pipkin, “The Inside Tracks: Status Distinctions in Allocations to Elite Law Schools,” 211–30; both in The High Status Track: Studies of Elite Schools and Stratification (Albany: State University of New York Press, 1990)

Paul W. Kingston and John C. Smart, “The Economic Pay-Off of Prestigious Colleges,” in The High Status Track

Jerome Karabel, The Chosen: The Hidden History of Admission and Exclusion at Harvard, Yale, and Princeton (New York: Houghton Mifflin, 2005).

Anthony P. Carnevale and Stephen J. Rose, “Socioeconomic Status, Race/Ethnicity, and Selective College Admissions,” in America’s Untapped Resource: Low-Income Students in Higher Education, ed. Richard D. Kahlenberg (New York: Century Foundation, 2003).

Briefs submitted to the U.S. Supreme Court by a long list of top corporations, business and professional associations, and retired military leaders in the cases Grutter v. Bollinger and Gratz v. Bollinger.

(To update one set of figures given in my book, it is worth nothing that seven of the 19 presidents inaugurated since 1900 earned their bachelor’s degrees from Harvard, Princeton, or Yale, and all but three attended prestigious colleges or professional schools.)


My assertions regarding selective colleges disproportionately serving students from privileged backgrounds are supported by extensive reporting for The Chronicle of Higher Education, the Carnevale and Rose study cited above, and several reports and books cited in the Chapter 1 of Color and Money, including:

Douglas S. Massey, Camille Z. Charles, Garvey F. Lundy, and Mary J. Fischer. The Source of the River: The Social Origins of Freshmen at America’s Selective Colleges and Universities. (Princeton, NJ: Princeton University Press, 2003)

Danette Gerald and Kati Haycock, Engines of Inequality: Diminishing Equity in the Nation’s Premier Public Universities (Washington, DC: Education Trust, 2006).

My assertions regarding student attitudes at selective higher education institutions are supported by extensive reporting by me and other Chronicle reporters, The Source of the River (cited above), several studies cited in Chapter 5 of Color and Money, and by research conducted for the Center for Academic Integrity.

My assertions regarding selective colleges' reliance on the SAT are backed by extensive Chronicle reporting, research presented at the American Educational Research Association's 2008 annual conference, a recent National Association of College Admissions Conference report, and the following authoritative history of the SAT test:

Nicholas Lemann, The Big Test: The Secret History of the American Meritocracy (New York: Farrar, Straus & Giroux, 1999)

Lemann's book also includes a discussion of the SAT test preparation industry and its impact on scores. For more information on that subject, see David Owen's book None of the Above.

My characterizations of selective college admissions policies are backed by extensive Chronicle reporting and nearly all of the studies and books cited in Chapter 1 of Color and Money (see the links under "Chapter 1" on the right side of the screen). Perhaps the best recent work examining how people use cash and connections to get into specific elite colleges was done by Dan Golden, a Pulitzer-prize winning journalist, for the Wall Street Journal and the book The Price of Admission: How America’s Ruling Class Buys Its Way into Elite Colleges, and Who Gets Left Outside the Gates. William Bowen, a former president of Princeton University, has co-authored several exhaustive studies of the impact of various admissions preferences. They include:

William G. Bowen, Martin A. Kurzweil, and Eugene M. Tobin, Equity and Excellence in American Higher Education (Charlottesville: University of Virginia Press, 2005).

William G. Bowen and Sarah A. Levin, Reclaiming the Game: College Sports and Educational Values (Princeton, NJ: Princeton University Press, 2003).


The research cited above provides just a smattering of the data supporting the AlterNet essay's assertions, which rest on a wide body of empirical data gathered in recent decades. Those interested in locating more can find it by surfing around this Web site.

New Report on SAT Test Prep's Effects Is: A) Flawed B) Suspect C) Damning D) All of the Above

A new report on programs that prepare students for college entrance examinations has garnered a lot of attention in the media for two key reasons: It concludes that commercial SAT test preparation programs really don't raise scores all that much, and it finds that many colleges--against the advice of the testing companies and many experts in the college admissions field--are giving substantial weight to small score differences, with some even using certain scores as a cutoff in processing applications.


Colleges themselves might want to ask next year's applicants to download a copy of the report and write an essay responding to it. What conclusions the applicants draw will say a lot about their critical thinking skills. If an applicant wholeheartedly accept its assertion that the average SAT gains derived from enrolling in commercial test preparation programs likely are "in the neighborhood of 30 points," perhaps an elite college is not the best fit for him, and he would be better off somewhere close to home, maybe within bicycling distance.


The author of the report is Derek C. Briggs, an associate professor of quantitative methods and policy analysis at the University of Colorado at Boulder and the chairman of the university's Research and Evaluation Methodology Program. The report was commissioned by the National Association for College Admission Counseling, a Virginia-based organization representing high school counselors, for-profit college counseling providers, and college employees involved in admissions and financial-aid decisions. The association tacked onto the report's cover the caveat that its conclusions are Mr. Briggs' own, but it nonetheless has publicized the report as part of NACAC's efforts to advance "the knowledge base and dialogue about test preparation."


The NACAC press release announcing the report declares: "Report Highlights Test Prep Paradox—Paying for Test Prep Doesn’t Yield Big Returns, But Returns May Still Matter in Light of Admission Practice." Among the major media outlets which ran with the "no big return" assertion were the Washington Post (headline: "Study Sees Small Average Gains from College Test Coaching") and the Wall Street Journal (headline: "SAT Coaching Found to Boost Scores--Barely").



Briggs concludes that SAT test preparation increases scores on the math portion of the test by just 10 to 20 points, and on the verbal portion of the test by just 5 to 10 points. He does not base that conclusion on any new research, but on a review of a tall stack of past studies of the impact of test coaching. Actually, to be precise, he bases his conclusion on just three studies in the stack. He discounted the rest--some of which found score increases from coaching of 100 points or more--as based on small samples that were not representative of the nation's population or as otherwise methodologically flawed.


Two of the three studies that Derek C. Briggs characterizes as valid and pointing to "a consensus position" on the effects of SAT test coaching were performed by none other than Derek C. Briggs. (If his name is otherwise recognizable to people in the field, it is because, far from being a neutral arbiter of such research, he already has established himself as a prominent critic of the idea that SAT coaching works.) Briggs not only put himself in a position to pass judgment on his own research and (surprise surprise) declared his own work rock solid, but he also has declared a consensus based on me-myself-and-I vote counting. The third study that he counts toward that consensus, by Donald Rock and Donald Powers, unsurprisingly reaches the same conclusion he had.


About that only other study in the stack that Briggs found methodologically acceptable: It was sponsored by two organizations which are highly invested, financially and otherwise, in the idea that SAT scores cannot be raised significantly by coaching--the College Board, which owns the SAT, and the Educational Testing Service, which administers it. Powers was a principle research scientist at ETS, and, as the book Color and Money shows, both ETS and the College Board have a history of promoting research that makes their case and squelching research that doesn't.

Powers and Rock identified those students who had received SAT test coaching based on whether the students owned up to it in response to a questionnaire on ETS letterhead sent to them after they had taken the SAT and before they got their scores back. Based on their choice of methodology, one wonders if they would have published a study concluding that married men seldom cheat based on a surveys administered to husbands by wives with revolvers in their hands. As Bob Schaeffer, a spokesman for the watchdog group FairTest, noted in an e-mail:

"The notion that students would respond accurately to a testing company's questionnaire asking whether they had been coaching, especially during the period after they had taken the exam and before scores were reported, is ludicrous. At a minimum, they would wonder how that information would be used. Given fears about the secrecy with which ETS/College Board handle data, might have even believed that ETS could 'flag' scores being sent to colleges to indicate that an applicant had been coached, just as they then did for tests taken with extended time."

To his credit, Briggs acknowledges--albeit in a subtle, after-the-fact sort of way--that the three studies he cites, in talking about the average gains derived from SAT coaching by nationwide samples of students of all backgrounds, mask differences in the effectiveness of programs based on their quality, setting, and duration. He also acknowledges the two cited studies of his own that he cites "suggest that coaching is more effective for students with strong academic backgrounds and high economic status who underperformed on the PSAT."


The implications of this concession are huge. His new report, and the hype surrounding it, are wrongly leading the nation to believe that high-priced SAT test preparation services do not substantially raise the scores of students who participate in them in earnest, when, in fact, the average gains produced by some coaching services quite possibly may be the 100 points or more that other studies have claimed. The "average gains" Briggs' report cites are based on studies that included slackers who dropped out of coaching programs or repeatedly skipped sessions or screwed around and paid no attention to their instructors at all, as well as people who enrolled in ineffective, disreputable programs. If one wants to determine how much weight people lose on Weight Watchers, one should base that study on the weight changes of people who enrolled in Weight Watchers and took it seriously, not on the entire universe of people who have ever declared they are going on a diet and, in some cases, sat down to wolf down that "one last piece of pie."


The idea that families should not waste their money on coaching programs that won't raise scores is a perfectly fine one to get out. But, while NACAC may be well-intentioned in trying especially hard this week to get word out to families who are low- or middle-income (and presumably don't have money to spare), one wonders if the group will be contributing to class-linked gaps in access to selective higher education by so targetting its message. After all, its report, read carefully, suggests the rich might in fact be wise to enroll their children in reputable programs costing $1,000 or more, because such programs are likely to produce gains that will make a difference in the admissions office. SAT coaching already is giving the children of the wealthy enough of an unfair edge in getting into top colleges without broadcasting the message that those of lesser means should not bothering trying to put their children on equal footing.


The results of a survey of colleges contained in the new NACAC report makes clear that, even if the gains derived from test coaching are only in the neighborhood of 30 points on a 1600 point scale, those 30 points can make a lot of difference at some colleges and among students who were high scorers to begin with. Of colleges that use the SAT in evaluating applicants, 21 percent have a rigid cut-off scores. And, at the upper end of the SAT score range, well over a third of colleges said a 20 point increase in an applicant's SAT math score or a 10 point increase in an applicant's SAT verbal score would "significantly improve" their likelihood of gaining admission. Although NACAC and the College Board have advised colleges against giving small differences in SAT scores much weight in admissions decisions, their admonitions appear to be falling against deaf ears. Why? Relying heavily on SAT scores offers selective colleges an inexpensive way to sort through an annual barrage of applications. Taking in high scorers helps colleges boost their rankings in U.S. News and World Report and other college guides. And colleges have an additional, and powerful, financial incentive to depend heavily on the SAT, in that the strong correlation between SAT scores and family wealth means that high scorers are more likely than other applicants to get through college without needing financial aid and to donate generously to their alma maters down the road.

The problem with nearly all research on SAT preparation is that it is hard to find a disinterested party to do it. The College Board and ETS, which make huge sums of money off administering the SAT, know that its very existence is threatened by research showing that the test can be beaten by coaching. The test preparation industry has a financial incentive to argue the SAT can in fact be beaten by coaching and their services will give college applicants a big bounce in their scores. Selective colleges--and the members of organizations like NACAC--have incentives, financial and otherwise, to keep the test around. Advocates of low-income minority students see the SAT as a screening device that favors the white and wealthy, even without SAT coaching being factored into the equation.

It is possible to construct an experiment empirically measuring whether SAT coaching works. Doing so would require randomly assigning students to experimental and control groups and comparing test-to-test changes in the scores of those who had gone through coaching and those who had not. Unfortunately, no such experiment has yet been performed, leaving the nation's parents having to base their decisions on advertisements by test-preparation companies and "research" that may be no more trustworthy.

UPDATE: Bob Schaeffer of FairTest points out that a study involving experimental and control groups of students actually was done back in 1988. It was for a Ph.D. dissertation, and, although it had a fairly small sample, the fact a doctoral student pulled it off suggests the possibility of conducting other such studies on a broader scale. A link to it is here.

Study Finds the Brightest and Wealthiest Increasing Concentrated at Top Colleges

Research findings presented last month show that intensifying competition for admission to selective colleges has led to a rising concentration of top students at such institutions.

As discussed in depth in an article on the Chronicle of Higher Education news blog, the new study, presented at the annual conference of the American Educational Research Association, found that students who are high achievers or from high-income families have become scarcer at two-year colleges and noncompetitive four-year institutions in recent decades as they have focused their attention on getting into the best colleges possible.

Michael N. Bastedo, an assistant professor of education at the University of Michigan at Ann Arbor, and Ozan Jaquette, a doctoral candidate at Michigan, based their analysis on data from three nationally representative, long-term surveys: the High School and Beyond Survey of 1980, the National Educational Longitudinal Survey of 1988, and the Educational Longitudinal Survey of 2002. They focused on students who completed high school in 1972, 1982, 1992, and 2004, analyzing changes over time in the relationship between socioeconomic stratification, precollege academic preparation, and the colleges where students end up.

The researchers’ analysis was rooted in “signaling” theory, which holds that education credentials distinguish their holders from competitors for jobs, and the value of a credential is inversely related to the proportion of job applicants possessing it.

How Researchers Classify Biracial Subjects Skews Study Results

The manner in which researchers classify biracial subjects can seriously skew their results, according to a study presented last month at the American Educational Research Association's annual conference and discussed in depth in a Chronicle of Higher Education article.

The authors of the study are Karen Kurotsuchi Inkelas, an associate professor of college-student personnel at the University of Maryland at College Park; Matthew Soldner, a doctoral student at Maryland; and Katalin Szelényi, an assistant professor of education at the University of Massachusetts at Boston. They conducted their analysis using data on more than 22,000 undergraduate students at 49 colleges gathered as part of the 2007 National Study of Living-Learning Programs, which uses a survey instrument that lets students identify with as many races and ethnicities as they please.

The researchers crunched their numbers using three commonly used approaches to classifying biracial and multiracial students.With one approach, they classified subjects who belong to two or more racial or ethnic groups as simply being “biracial” or “multiracial.” With a second approach, subjects who identity with two groups are classified as belonging to the least prevalent one, so that a student who reports being both white and black is designated as black.

Under a third approach, used by the federal Office of Management and Budget, they gave biracial research subjects dual classifications reflecting their backgrounds, such as “white-black” or “white-Hispanic.” For the sake of keeping the number of categories manageable, they disregarded data from any biracial subset that accounts for less than 1 percent of the total sample studied, with the result being that all dual classifications that did not have "white" on one side of the hyphen were excluded from their analysis.

The researchers found that their choice of classification scheme had a profound impact on their results, with some schemes painting a much bleaker picture than others for certain racial or ethnic groups. In using the second approach, for example, and classifying students who had identified themselves as white and Native American as being Native American, they drastically overestimated the percentage of Native American students who were receiving merit-based aid.

Because each classification scheme had strengths and weaknesses, the researchers concluded “there is no single solution to this empirical dilemma."