Last week I explored the “ETS Guidelines for Fairness Review of Assessments.” These guidelines were adopted by PARCC, so I decided to take a look at PARCC’s sample items for English language arts. (PARCC is one of the two consortia of states with massive federal grants to create assessments aligned with the Common Core State Standards. Smarter Balanced is the other consortium; ETS developed somewhat different guidelines for it—I’ll take a look at those next week).
The knowledge demands in PARCC’s sample items are very broad, from cougars to Amelia Earhart to DNA testing. While I am happy to see some substantive questions—and hopeful that such test items will reinforce the standards’ call for systematically building knowledge with content-rich curriculum—I worry about the fairness of these assessments given how they are being used.
As I mentioned last week, it would be perfectly fair to have test passages on topics that had been taught. But, since the test developers do not know which topics are taught in each grade, they have to assess “common” knowledge. Due to well-documented differences in opportunities to learn at home and at school, some children know a good bit more common knowledge than others.
Let’s take a look at one of PARCC’s sample items for third grade. Three questions are asked based on the 631-word passage “How Animals Live.” There’s a typical main-idea question paired with a supporting-evidence question, and then a narrower question that assesses the “skills of rereading carefully to find specific information and of applying the understanding of a text.” Here’s the first section of the passage:
What All Animals Need
Almost all animals need water, food, oxygen, and shelter to live.
Animals get water from drinking or eating food. They get food by eating plants or other animals.
Animals get oxygen from air or water. Many land animals breathe with lungs. Many water animals breathe with gills.
Animals need shelter. Some animals find or build shelter. Other animals grow hard shells to protect themselves.
Many words here are undefined: oxygen, shelter, lungs, and gills. Are these words common to all third graders? Probably not, but much of the content is likely familiar to the vast majority of third graders—and perhaps enough content is familiar for most third graders to grasp the section (if not every word). Nonetheless, children who have learned about oxygen, shelter, lungs, and gills start out with a big advantage. They are reading and comprehending more quickly (which is extremely important in a timed test), and they are comfortable as they move into the more difficult content in the rest of the passage.
Here is the second section, and the beginning of the third:
Ways Of Grouping Animals
Animals can be grouped by their traits. A trait is the way an animal looks or acts. Animals get traits from their parents. Traits can be used to group animals.
Animals with Backbones
Animals with backbones belong to one group. A vertebrate is an animal with a backbone. Vertebrates’ backbones grow as they get older. Fish, snakes, and cats are all vertebrates. Vertebrates can look very different.
Let’s ignore the stiff, unengaging style. What really concerns me is the delusion that it is fair for content to be learned and applied during a high-stakes assessment. (As I noted last week, I do not dispute that the assessment is valid and reliable, so my concerns are with accountability policies, not really with this type of assessment.)
Since a definition of trait is given, it’s clear that some significant portion of third graders is not expected to know that word. Now imagine this is the first time you’ve encountered trait and examine the text:
A trait is the way an animal looks or acts…. Traits can be used to group animals…. Vertebrates can look very different.
What is a third grader to make of this? Clearly, vertebrates are not grouped by how they look.
A trait is the way an animal looks or acts…. Fish, snakes, and cats are all vertebrates.
Clearly, vertebrates are not grouped by how they act. How is the backbone (which is not defined) a trait, since it does not seem at all related to how all these animals look or act?
Making matters worse, understanding trait is essential to correctly answering the main-idea and supporting-evidence questions.
Do these vertebrates look or act alike?
(Image courtesy of Shutterstock.)
The fact is, vocabulary is not learned by being given a definition. Definitions can be helpful, but they are always incomplete. Words are learned through multiple exposures in multiple contexts. Even with simple words, multiple contexts are necessary: What, exactly, makes hoagies and gyros and PB&Js all sandwiches? I can’t even attempt a concise answer—I just know a sandwich when I see one.
Third graders who have had a unit on vertebrates and invertebrates will breeze through this passage; its inadequate definition of trait won’t matter. But students relying on this definition will surely be at least a little confused, possibly totally lost. The assessment will accurately tell us that children without knowledge of traits have limited comprehension of this passage—but it will not accurately tell us anything about their teachers or schools, for no one alerted the educators that the test would measure knowledge of traits.