You hear a lot nowadays about the magic of big data. Getting hold of the right numbers can increase revenue, improve decision-making, or help you find a mate—or so the thinking goes. In 2009, U.S. Education Secretary Arne Duncan told a crowd of education researchers: “I am a deep believer in the power of data to drive our decisions. Data gives us the roadmap to reform. It tells us where we are, where we need to go, and who is most at risk.”
This is a story about what happened when I tried to use big data to help repair my local public schools. I failed. And the reasons why I failed have everything to do with why the American system of standardized testing will never succeed.
A few years ago, I started having trouble helping my son with his first-grade homework. I’m a data-journalism professor at Temple University, and when my son asked me for help on a worksheet one day, I ran into an epistemological dilemma. My own general knowledge (and the Internet) told me there were many possible “correct” answers. However, only one of these answers would get him full credit on the assignment.
“I need to write down natural resources,” he told me.
“Air, water, oil, gas, coal,” I replied.
“I already put down air and water,” he said. “Oil and gas and coal aren’t natural resources.”
“Of course they are,” I said. “They’re non-renewable natural resources, but they’re still natural resources.”
“But they weren’t on the list the teacher gave in class.”
I knew my son would start taking standardized tests in third grade. If the first-grade homework was this confusing, I was really worried about how he—or any kid—was supposed to figure out the tests. I had been spending time with civic hackers, the kind of people who build software and crunch government data for fun, and I decided to see if I could come up with a beat-the-test strategy derived from a popular SAT prep course I used to teach.
In essence, I tried to game the third-grade Pennsylvania System of School Assessment (PSSA), the standardized test for my state. Along with a team of professional developers, I designed artificial-intelligence software to crunch the available data. I talked to teachers. I talked to students. I visited schools and sat through School Reform Commission meetings.
After six months of this, I discovered that the test can be gamed. Not by using a beat-the-test strategy, but by a shockingly low-tech strategy: reading the textbook that contains the answers.
Philadelphia is the eighth-largest school district in the country, and its public students are overwhelmingly poor: 79 percent of them are eligible for free or reduced-price lunch. The high-school graduation rate is only 64 percent and fewer than half of students managed to score proficient or above on the 2013 PSSA.
When a problem exists in Philadelphia schools, it generally exists in other large urban schools across the nation. One of those problems—shared by districts in New York, D.C., Chicago, Los Angeles, and other major cities—is that many schools don’t have enough money to buy books. The School District of Philadelphia recently tweeted a photo of Mayor Michael Nutter handing out 200,000 donated books to K-3 students. Unfortunately, introducing children to classic works of literature won’t raise their abysmal test scores.
This is because standardized tests are not based on general knowledge. As I learned in the course of my investigation, they are based on specific knowledge contained in specific sets of books: the textbooks created by the test makers.
All of this has to do with the economics of testing. Across the nation, standardized tests come from one of three companies: CTB McGraw Hill, Houghton Mifflin Harcourt, or Pearson. These corporations write the tests, grade the tests, and publish the books that students use to prepare for the tests. Houghton Mifflin has a 38 percent market share, according to its press materials. In 2013, the company brought in $1.38 billion in revenue.Pearson came under fire last year for using a passage on a standardized test that was taken verbatim from a Pearson textbook.
Pennsylvania currently has a multi-million-dollar contract with a company called Data Recognition Corporation (DRC) to grade the PSSAs. DRC works with McGraw-Hill as part of a consortium that has a $186 million federal contract to write and grade standardized tests for the rest of the country. McGraw-Hill, meanwhile, also writes the books and curricula schools buy to prepare students for the tests. Everyday Math, the branded curriculum used by most Philadelphia public schools in grades K–5, is published by McGraw Hill.
Put simply, any teacher who wants his or her students to pass the tests has to give out books from the Big Three publishers.
by Meredith Broussard, The Atlantic | Read more:
Image: Matt Stanley
This is a story about what happened when I tried to use big data to help repair my local public schools. I failed. And the reasons why I failed have everything to do with why the American system of standardized testing will never succeed.A few years ago, I started having trouble helping my son with his first-grade homework. I’m a data-journalism professor at Temple University, and when my son asked me for help on a worksheet one day, I ran into an epistemological dilemma. My own general knowledge (and the Internet) told me there were many possible “correct” answers. However, only one of these answers would get him full credit on the assignment.
“I need to write down natural resources,” he told me.
“Air, water, oil, gas, coal,” I replied.
“I already put down air and water,” he said. “Oil and gas and coal aren’t natural resources.”
“Of course they are,” I said. “They’re non-renewable natural resources, but they’re still natural resources.”
“But they weren’t on the list the teacher gave in class.”
I knew my son would start taking standardized tests in third grade. If the first-grade homework was this confusing, I was really worried about how he—or any kid—was supposed to figure out the tests. I had been spending time with civic hackers, the kind of people who build software and crunch government data for fun, and I decided to see if I could come up with a beat-the-test strategy derived from a popular SAT prep course I used to teach.
In essence, I tried to game the third-grade Pennsylvania System of School Assessment (PSSA), the standardized test for my state. Along with a team of professional developers, I designed artificial-intelligence software to crunch the available data. I talked to teachers. I talked to students. I visited schools and sat through School Reform Commission meetings.
After six months of this, I discovered that the test can be gamed. Not by using a beat-the-test strategy, but by a shockingly low-tech strategy: reading the textbook that contains the answers.
Philadelphia is the eighth-largest school district in the country, and its public students are overwhelmingly poor: 79 percent of them are eligible for free or reduced-price lunch. The high-school graduation rate is only 64 percent and fewer than half of students managed to score proficient or above on the 2013 PSSA.
When a problem exists in Philadelphia schools, it generally exists in other large urban schools across the nation. One of those problems—shared by districts in New York, D.C., Chicago, Los Angeles, and other major cities—is that many schools don’t have enough money to buy books. The School District of Philadelphia recently tweeted a photo of Mayor Michael Nutter handing out 200,000 donated books to K-3 students. Unfortunately, introducing children to classic works of literature won’t raise their abysmal test scores.
This is because standardized tests are not based on general knowledge. As I learned in the course of my investigation, they are based on specific knowledge contained in specific sets of books: the textbooks created by the test makers.
All of this has to do with the economics of testing. Across the nation, standardized tests come from one of three companies: CTB McGraw Hill, Houghton Mifflin Harcourt, or Pearson. These corporations write the tests, grade the tests, and publish the books that students use to prepare for the tests. Houghton Mifflin has a 38 percent market share, according to its press materials. In 2013, the company brought in $1.38 billion in revenue.Pearson came under fire last year for using a passage on a standardized test that was taken verbatim from a Pearson textbook.
Pennsylvania currently has a multi-million-dollar contract with a company called Data Recognition Corporation (DRC) to grade the PSSAs. DRC works with McGraw-Hill as part of a consortium that has a $186 million federal contract to write and grade standardized tests for the rest of the country. McGraw-Hill, meanwhile, also writes the books and curricula schools buy to prepare students for the tests. Everyday Math, the branded curriculum used by most Philadelphia public schools in grades K–5, is published by McGraw Hill.
Put simply, any teacher who wants his or her students to pass the tests has to give out books from the Big Three publishers.
by Meredith Broussard, The Atlantic | Read more:
Image: Matt Stanley
