Dear BSD Math Adoption Committee, Dr. Cudeiro, BSD School Board,
The BSD curriculum staff's analysis of the BSD Pilot test data does not control for the dramatic differences between BSD schools.
Below is an analysis that does control for the differences between schools. You will see a robust finding in favor of Holt. You can see the key chart at the following URL:
The bars show the differences between course unit tests where both curricula were piloted by the same school. The long red bars indicate tests where Holt did much better than Discovering.
Jock Mackinlay, PhD and Bellevue parent
---------------------------------------------------------------------------------------
To whom it may concern,
A robust finding in favor of Holt can be seen in the test data for the Bellevue School District pilot of Holt and Discovering [http://www.bsd405.org/portals/0/curriculum/mathadoption/Data/MathPilotAssessmentData.pdf]when you focus on the data for schools that piloted both Holt and Discovering for the same course unit. This focus is not done to give Holt an advantage. Rather, it controls for an independent variable: BSD schools have dramatic differences in the math ability of their average student (see chart 3).
As an example, consider Robinswood Middle School and Tyee Middle School [Robinswood grade 8 WASL 2009 pass rate: 29.2%, Tyee grade 8 WASL 2009 pass rate: 78.4%]. The fact that the assessment results for Holt IMT3 were 45% and 58% at Robinswood, compared to Discovering IMT3 results of 89% and 88% at Tyee, is almost certainly due more to the differences in the schools than the differences in the curricula. Unfortunately, the BSD curriculum staff did not report the assessment results from the Spring 2009 pilot of Holt IMT3 at Tyee and Odle, and Discovering IMT3 at Robinswood, Chinook, Tillicum and Highland. (Furthermore, the Fall 2009 data from Tillicum and Highland, which were reported as piloting Holt, is also missing.) If they had, we might be able to control for average school ability as an independent variable by focusing on the IMT3 test results within each middle school. With the current reported data, none of the IMT3 results can validly be considered, and many other datapoints . [Note that another very useful piece of data would be the results of these same assessments as taken by students in our current math courses, namely CMP2 and Core Plus. This would allow further valid comparisons within schools.]
You can see the robust finding in favor of Holt in charts 1 & 2 below and at the following links, which show two views of the school courses that piloted both, one organized by school and the other organized by course unit. In both views, the average grade for Holt and Discovering is plotted in the left pane and the difference between those grades is plotted in the right pane as bars. This difference controls for another independent variable: the difficulty of the test. A given test is equally difficult for both Holt and Discovering, which means that the differences in average grade removes the test difficulty from the bars, allowing them to be compared across the tests (and schools).
In both views, we see many long bars (i.e., large differences) in favor of Holt. Long bars are a robust finding because any error adjustment for these average grades would not change the finding in favor of Holt. When we look at the “by unit” view, we see that the long bars are for the course unit HIAG2-power models (quadratic equations) across multiple schools. Most of the other piloted units have short bars (i.e., small differences), which means that Holt and Discovering are roughly equal. The only data favoring Discovering with any robustness is at SHS in HIAT3-geometric reasoning where there are medium length blue bars (i.e., modest differences).
Chart 1: BSD pilot results by school
Chart 2: BSD pilot results by unit
Good analysis. I also noticed the discrepancy in the schools taking the exam. I would also like to know what test was used and if the questions were more investigation based. The whole process so far has been amazingly biased against the Holt book and I would bet the teacher involved in the pilot were also very partial toward the discovery text book.
ReplyDelete