Considering What Evidence?
In February, the Brookings Institution released its 2012 Brown Center Report on American Education. These comments concern Part 1 of the Brookings Report, “Predicting the Effect of the Common Core State Standards on Student Achievement,” especially those pertaining to the Common Core State Standards for Mathematics.
A minor oddity is that this section of the report begins by describing the CCSS as “written by teams of curriculum specialists.” This statement has a footnote which gives this URL. But the information at the URL says no such thing and is contradicted by the list of members of the writing groups here at the same web site.
The report continues:
Despite all the money and effort devoted to developing the Common Core State Standards—not to mention the simmering controversy over their adoption in several states—the study foresees little to no impact on student learning. That conclusion is based on analyzing states’ past experience with standards and examining several years of scores on the National Assessment of Educational Progress (NAEP).
At first glance, based on the evidence presented in the report, I would agree with this conclusion. But, closer examination shows omissions and shortcomings. Among the latter is reliance on Andrew Porter’s Education Week account of the CCSS. (This article was based on an analysis of grades 3–6 and 8 published in Educational Researcher which I discuss extensively here.)
My comments fall into three categories:
- Framing of the discussion and its context.
- Flaws in evidence presented.
- Relevant evidence that was not presented.
Framing and context. The report says:
Past experience with standards suggests that each part of this apparatus—a common curriculum, comparable tests, and standardized performance levels—is necessary. No one or two of them can stand alone for the project to succeed.
Does any policy-maker disagree? No evidence is provided.
In fact, the level of national activity suggests that many policy-makers are doing their best to ensure that no two of the three stands alone. That is one way in which they seem to be building on lessons learned from the past decade of No Child Left Behind.
A list of these activities would be a long digression, instead I refer the reader to a web site and a blog:
The Mathematics Common Core Coalition, a coalition of the National Council of Teachers of Mathematics, the National Council of Supervisors of Mathematics, the Association of Mathematics Teacher Educators, the Association of State Supervisors of Mathematics, the Council of Chief State School Officers, the National Governors Association, and the two assessment consortia. This lists a wide variety of national efforts aimed at audiences that range from governors to educators, policy-makers, and parents.
Tools for the Common Core Standards publishes updates and reports on projects to support the implementation of the Common Core State Standards in Mathematics.
Other activities that have been in place for many years, such as the National Science Foundation’s Math Science Partnerships, take on new relevance in light of the CCSS. Combined, these activities seem to form a remarkable and unprecedented national effort.
But, the Brookings Report does not mention any of them.
Instead, on the basis of past events, it considers whether or not the CCSS is likely to affect:
- Curriculum quality. Better curriculum, due to better standards. (“Better,” of course, is in the eye of the beholder and will be discussed further below. The mechanism by which curricula might be chosen is not discussed.)
- Performance expectations. Higher expectations, due to states sharing tests and expectations for performance on those tests. (This alludes to the “race to the bottom” in which states “competed” to set lower standards for proficiency as measured by the state tests required by the No Child Left Behind Act.)
- Standardization. Efficiencies in textbook production (no need to produce different versions for each state) and efficiencies for students or teachers who move from one state to another. (Economies of scale in curriculum and test development are not mentioned. Ironically, Part 2 of the report notes “The goal of assessing higher-order skills is laudable,” but neglects to discuss cost considerations in the development of such items. There’s a reason why many state tests only cover lower-order skills.[i])
The Brookings report’s analysis of these three areas seems to assume that states will repeat past behavior, rather than learning from past decades. (In their blog posts, Kathleen Porter-Magee and Andrew Rotherham make similar points.) That may be, but the remarkable level of current activity described in the Mathematics Common Core Coalition list suggests that many states have no such intention.
Evidence presented. As noted earlier, the Brookings report relies on Porter’s Education Week commentary. (It cites other criticisms, but in order to keep this post relatively short, analysis of these will be deferred to future posts.) This, in turn, purports to be based on the findings of an Educational Researcher article that compared the CCSSM for grades 3–6 with standards from 14 US states and compared the CCSSM for grade 8 with written expectations for Finland, Singapore, and Japan. This results in the following collection of assertions (listed here in reverse chronological order), which are not consistent with each other.
Writing in Education Week in the summer of 2011, Andrew Porter compared the Common Core to existing state standards and international standards from other countries and concluded that the Common Core does not represent much improvement. (Brookings Report, February 2012)
Our research shows that the common-core standards do not represent a meaningful improvement over existing state standards.
The common core has a greater focus than certain state standards, and a lesser focus than others.
[T]op-performing countries we studied . . . put far less emphasis on higher-order thinking, and far more on basic skills, than does the common core. (Porter, Education Week, August 2011)
Judging the quality of the Common Core standards is of great importance, but it is only partially and tentatively addressed here. (emphasis added, Porter et al., Educational Researcher, April 2011)
Further discussion of the Educational Researcher article is here. In particular, I note that there is evidence that some countries do address higher-order skills, but this evidence is not salient via the methods used by Porter et al. Moreover, its meaning for “focus” differs from the meaning of “focus” in the CCSSM and in Lessons Learned (published by Brookings in 2007). The latter was edited by Tom Loveless, who wrote in Chapter 1 “Coherence addresses when topics are introduced, taught, and then extinguished from the curriculum. Focus refers to the number of topics covered.” Had Loveless, who is the author of the Brookings report, examined the Educational Researcher article, he might have noticed that Porter measured focus in two ways, neither of which is consistent with the meaning of “focus” in Lessons Learned.
Evidence not presented. The Brooking report says:
Bill Gates is right that multiplication is the same in Alabama and New York, but he would have a difficult time showing how those two states—or any other two states—treat multiplication of whole numbers in significantly different ways in their standards documents.
Bill Gates might have a hard time, but he might know Bill Schmidt. (Loveless might know him too, he’s a co-author of a chapter in Lessons Learned.) Schmidt and his colleagues have found large differences in grade level occurrence of topics (including “whole number meaning” and “whole number operation”) in standards for 21 states. (See slide 10 here.)
Another source of variability may be in the meaning of mathematical terms. This may seem quite strange, but it’s well known in the mathematical world. This is not to say that meanings change during a proof, or within a textbook or article (at least not intentionally!), but rather as described in Zalman Usiskin et al.’s study The Classification of Quadrilaterals:
Many terms in mathematics education can be found to have different definitions in mathematics books. Among these are “natural number,” “parallel lines,” “trapezoid” and “isosceles trapezoid,” the formal definitions of the trigonometric functions and absolute value, and implicit definitions of the arithmetic operations addition, subtraction, multiplication and division.
Students and teachers may acquire different meanings for various terms. One example is a recent discussion of the meaning of “fraction” and “ratio” on the Tools for the Common Core blog (see comments beginning on December 22). As I pointed out there, a group of North Carolina researchers (Clark et al., 2003, Journal of Mathematical Behavior) found that within a group of prospective and practicing teachers there were five different views of the relationship between fraction and ratio. For example, some teachers said that fractions were the same as ratios, while some said that fractions and ratios did not overlap. Such differences are not necessarily a sign that one view is wrong. (However, there may be other reasons to favor a particular view.) Such differences can cause difficulty for students because different definitions can have different mathematical implications.
These differences may also cause difficulties in teacher preparation. In the absence of common standards (and tests aligned to them), prospective teachers need to be educated about the possibility that state standards, tests, and/or approved textbooks may use different definitions, or else teacher preparation in different states needs to be attuned to different standards.
The extent of this problem seems almost unstudied. But, the variety in state standards, tests, and textbooks for students together with the variation by state in teacher preparation, certification, and professional development in the U.S. may have created a morass of different definitions. Any set of common standards might help to address this problem. Because of the intense scrutiny that they have received from mathematicians (for whom care with definitions is second nature) and because mathematicians were among their writers, the Common Core State Standards for Mathematics are likely to be particularly useful as a road to common mathematical ground.