Last week I explored the “ETS Guidelines for Fairness Review of Assessments.” These guidelines were adopted by PARCC, so I decided to take a look at PARCC’s sample items for English language arts. (PARCC is one of the two consortia of states with massive federal grants to create assessments aligned with the Common Core State Standards. Smarter Balanced is the other consortium; ETS developed somewhat different guidelines for it—I’ll take a look at those next week).
The knowledge demands in PARCC’s sample items are very broad, from cougars to Amelia Earhart to DNA testing. While I am happy to see some substantive questions—and hopeful that such test items will reinforce the standards’ call for systematically building knowledge with content-rich curriculum—I worry about the fairness of these assessments given how they are being used.
As I mentioned last week, it would be perfectly fair to have test passages on topics that had been taught. But, since the test developers do not know which topics are taught in each grade, they have to assess “common” knowledge. Due to well-documented differences in opportunities to learn at home and at school, some children know a good bit more common knowledge than others.
Let’s take a look at one of PARCC’s sample items for third grade. Three questions are asked based on the 631-word passage “How Animals Live.” There’s a typical main-idea question paired with a supporting-evidence question, and then a narrower question that assesses the “skills of rereading carefully to find specific information and of applying the understanding of a text.” Here’s the first section of the passage:
What All Animals Need
Almost all animals need water, food, oxygen, and shelter to live.
Animals get water from drinking or eating food. They get food by eating plants or other animals.
Animals get oxygen from air or water. Many land animals breathe with lungs. Many water animals breathe with gills.
Animals need shelter. Some animals find or build shelter. Other animals grow hard shells to protect themselves.
Many words here are undefined: oxygen, shelter, lungs, and gills. Are these words common to all third graders? Probably not, but much of the content is likely familiar to the vast majority of third graders—and perhaps enough content is familiar for most third graders to grasp the section (if not every word). Nonetheless, children who have learned about oxygen, shelter, lungs, and gills start out with a big advantage. They are reading and comprehending more quickly (which is extremely important in a timed test), and they are comfortable as they move into the more difficult content in the rest of the passage.
Here is the second section, and the beginning of the third:
Ways Of Grouping Animals
Animals can be grouped by their traits. A trait is the way an animal looks or acts. Animals get traits from their parents. Traits can be used to group animals.
Animals with Backbones
Animals with backbones belong to one group. A vertebrate is an animal with a backbone. Vertebrates’ backbones grow as they get older. Fish, snakes, and cats are all vertebrates. Vertebrates can look very different.
Let’s ignore the stiff, unengaging style. What really concerns me is the delusion that it is fair for content to be learned and applied during a high-stakes assessment. (As I noted last week, I do not dispute that the assessment is valid and reliable, so my concerns are with accountability policies, not really with this type of assessment.)
Since a definition of trait is given, it’s clear that some significant portion of third graders is not expected to know that word. Now imagine this is the first time you’ve encountered trait and examine the text:
A trait is the way an animal looks or acts…. Traits can be used to group animals…. Vertebrates can look very different.
What is a third grader to make of this? Clearly, vertebrates are not grouped by how they look.
A trait is the way an animal looks or acts…. Fish, snakes, and cats are all vertebrates.
Clearly, vertebrates are not grouped by how they act. How is the backbone (which is not defined) a trait, since it does not seem at all related to how all these animals look or act?
Making matters worse, understanding trait is essential to correctly answering the main-idea and supporting-evidence questions.
Do these vertebrates look or act alike?
(Image courtesy of Shutterstock.)
The fact is, vocabulary is not learned by being given a definition. Definitions can be helpful, but they are always incomplete. Words are learned through multiple exposures in multiple contexts. Even with simple words, multiple contexts are necessary: What, exactly, makes hoagies and gyros and PB&Js all sandwiches? I can’t even attempt a concise answer—I just know a sandwich when I see one.
Third graders who have had a unit on vertebrates and invertebrates will breeze through this passage; its inadequate definition of trait won’t matter. But students relying on this definition will surely be at least a little confused, possibly totally lost. The assessment will accurately tell us that children without knowledge of traits have limited comprehension of this passage—but it will not accurately tell us anything about their teachers or schools, for no one alerted the educators that the test would measure knowledge of traits.
I can hardly wait to find out what knowledge of English History PARCC thinks is essential for American high school students…Holding my breath.
Will Fitzhugh
[email protected]
http://www.tcr.org
“Words are learned through multiple exposures in multiple contexts.”
I would add that a new word is LEARNED when the person begins to use it in conversation.
To Will: I’m holding my breath as well. Were you referring to the development of the language or the background of Britain?
Lisa,
This kind of close reading of test items seems like important work, so thank you.
From what you write here, it sounds as if you think the questions on this passage are assessing background knowledge, at least in part. The PARCC creators would probably object, right? By defining “trait” they’re trying to isolate some variable other than background knowledge. What other variables, if any, do you think the questions on this passage are validly assessing? I gather the creators would say something like “reading skill”, but is this a real thing or a mythological entity –i.e. words with no referent in reality, like “unicorn”? And once this question is answered, the next question is, “Is it possible to teach this?” These seem like very important questions to me, and I have never heard a good answer to them. And yet the validity of the whole high-stakes testing enterprise depends on there being good answers to them.
At my school the teachers that teach music, art, and P.E. create their own assessments as a district. I think this approach would be better for classroom teachers as well. Or at least tests according to their curriculum. The broadness of state tests benefit students that has more exposure than the less exposed students not necessarily what teachers teach.
Hi Ponderosa,
I’m not an expert on test development, but I believe that in defining trait and asking two questions related to traits, they are attempting to measure the ability to comprehend and apply new information. The assumption would be that most of the content of the passage is familiar, so the students should be able to make meaning relatively quickly and then start using their new understanding immediately to grasp why the concept of “trait” was developed and how it is used.
If it were possible to know that all of the third-grade test takers knew the other content and all were just then learning about traits, then I guess that would be fair–but would it be a test of comprehension? Given that these tests are timed, it seems like a measure of learning and applying very quickly. That might be something we want to measure, but let’s call it what it is instead of lumping it in with reading comprehension.
“Assessment of breadth of knowledge and vocabulary, plus ability to quickly learn and apply new knowledge and vocabulary” is a long and clunky name for a test, but it seems to me far more accurate (and thus potentially productive) than “reading comprehension test.”
Lisa,
Is “ability to learn quickly and apply stuff” a skill that can be taught? What if this is a function of the brain’s hard-wiring, with some students genetically predisposed to be slightly better at it than others? Wouldn’t it be perverse to judge teachers on this then? But perhaps it’s not hard-wiring. What else could it be? A function of how many practice sessions (mental workouts) that the student has had? Now it might make a little bit of sense to judge the teacher –did she show enough good judgement to schedule adequate practice time for the kids? Or is “ability to learn quickly and apply stuff” partly a function of background knowledge? This seems plausible to me. If so, and if the teacher knows which domains of knowledge will be on the test, then it seems fair to blame the teacher for kids’ poor performance. What if this “ability to apply quickly…” is a function of all three: hard-wiring, practice and background knowledge? What valid inferences can you make about the teacher then when the kid does poorly on the test? It bothers me that these questions aren’t asked by educators or answered by the test makers. The underlying model of learning and teaching seems so hazy.
Ponderosa,
I am not expert enough to fully answer your very important questions. I am reasonably confident that quick learning and application is not a skill that could be taught. It’s a subset of comprehension and critical thinking, both of which depend on having relevant knowledge already stored in long-term memory.Within a given topic, however, it is an ability that can be increased: the more you know, the faster you learn.
Knowledge speeds comprehension, learning, and thinking, as explained by Daniel Willingham: http://www.aft.org/periodical/american-educator/spring-2006/how-knowledge-helps.
Knowledge also increases IQ, as explained by Richard Nisbett: http://www.aft.org/sites/default/files/periodicals/Nisbett.pdf
I am not familiar with the type of questions on the PARCC assessment, but from what I have read it seems that many are based on background knowledge. I am concerned about the fairness of this assessment because according to Lisa it is designed to test common knowledge.How is common knowledge determined? If this is the case, children from backgrounds with access to more educational resources will fair better. They will have a wider vocabulary and more knowledge of content. As a result, I agree that they will be able to comprehend information faster and therefore finish at a faster rate on a timed test. Also they will agreeably feel more comfortable as they proceed to do more difficult questions. Students, on the other hand, who have less exposure to educational resources besides what they come in contact with at school are at a disadvantage. They may be able to figure out vocabulary using context clues, and may be familiar with enough content to make meaning of text, but will take longer to comprehend information and make application. I therefore agree that when students enter the assessment with some level of prior knowledge it will speed their comprehension, learning and thinking.
Sophia,
As a third grade teacher, I am very concerned about the PARCC assessment and the impact it may have on my effectiveness as a teacher. While PARCC requires students to show their content knowledge, it also requires them to demonstrate their computer skills. My concern is that my students may know the content but because they cannot effectively manipulate the computer, they may not prove to be proficient in the mastery of their content knowledge. For example, this week I had my students taking practice assessments and many had great difficulty manipulating the mouse. This is a timed test and my fear is that this disability will tremendously slow them down and may cause them to fail, even though they may be quite knowledgeable of the content.
PARCC testing not only benefits the children, it also shows the public how well the students are learning. All the other testing done in the schools in the past were able to be modified to benefit the school and the teachers. PARCC is hated by the teachers that are not doing their job. Parents need to know that there children are learning at the top of their potential, PARCC will give you this information. Do not let your teachers convince you other-wise.
While no test is perfect, I think Lisa is implying here that the testing would be more fair if children were exposed to the same CONTENT, not just the same skills. This is a foundational tenet of Core Knowledge. Dr. Hirsch wrote an excellent article contending that comprehension *is* knowledge, a fact easy enough to prove to yourself by picking up a textbook on a subject with which you are unfamiliar (oh, aerodynamics or something.) You can read every word, but it makes no sense. That’s what these kinds of tests are like to kids without the content knowledge and concomitant vocabulary.