You might expect to see a headline like this in the Onion, but you won’t. The Onion can’t run it because it isn’t just ironic—it’s 100% true.

A few years ago, a researcher at one of the big testing companies told me that when developing a reading comprehension test, knowledge is a source of bias. He did not mean the obvious stuff like knowledge of a yacht’s anemometer. He meant typical K–12 subject matter.

Since reading comprehension depends chiefly on knowledge of the topic (including the vocabulary) in the passage, the student with that knowledge has a large advantage over the student without it. And since there have always been great educational inequities in the United States, students’ knowledge—acquired both at home and at school—is very strongly correlated with socioeconomic status.

A logical solution would be to test reading comprehension using only those topics that students have been taught. Teachers can do this, but testing companies can’t—how would they have any idea what topics have been taught in each grade? It’s rare for districts, much less states, to indicate what or when specific books, people, ideas, and events should be taught.

Without a curriculum on which to base their assessments, testing companies have devised their own logic—which is sound given the bind they’re in. They distinguish between common and specialized knowledge, and then they select or write test passages that only have common knowledge. In essence, they’ve defined “reading comprehension skill” as including broad common knowledge. This is perfectly reasonable. When educators, parents, etc. think about reading comprehension ability, they do not think of the ability to read about trains or dolphins or lightning. They expect the ability to read about pretty much anything one encounters in daily life (including the news).

I already had this basic understanding, but still I found the “ETS Guidelines for Fairness Review of Assessments” eye opening. Guideline 1 is to “avoid cognitive sources of construct-irrelevant variance…. If construct-irrelevant knowledge or skill is required to answer an item and the knowledge or skill is not equally distributed across groups, then the fairness of the item is diminished” (p. 8). It continues, growing murkier:

Avoid unnecessarily difficult language. Use the most accessible level of language that is consistent with valid measurement…. Difficult words and language structures may be used if they are important for validity. For example, difficult words may be appropriate if the purpose of the test is to measure depth of general vocabulary or specialized terminology within a subject-matter area. It may be appropriate to use a difficult word if the word is defined in the test or its meaning is made clear by context. Complicated language structures may be appropriate if the purpose of the test is to measure the ability to read challenging material.

Avoid unnecessarily specialized vocabulary unless such vocabulary is important to the construct being assessed. What is considered unnecessarily specialized requires judgment. Take into account the maturity and educational level of the test takers in deciding which words are too specialized.

On page 10, it offers this handy table that “provides examples of common words that are generally acceptable and examples of specialized words that should be avoided…. The words are within several content areas known to be likely sources of construct-irrelevant knowledge”:

ETS table 1

Since having good reading comprehension means being able to read about a wide variety of common topics, table 1 seems just fine. But testing companies’ silence about what their reading comprehension tests actually measure is not. They say they are measuring “reading comprehension skill,” but their guidelines show that they are measuring a vaguely defined body of “common knowledge.”

Common words are not common to all. Even “common” knowledge is knowledge that must be taught, and right now—at home and at school—far too many children from low-income homes don’t have an opportunity to learn that knowledge (which is common to youth from middle-class and wealthy homes). That’s why reading comprehension scores are so strongly and stubbornly correlated with socioeconomic status.

These tests of “common” knowledge are accurate assessments and predictors of reading comprehension ability; but they are not fair or productive tests for holding children (and their teachers) accountable before an opportunity to learn has been provided.

If all testing companies would clearly explain that their reading comprehension tests are tests of knowledge, and if they would explain—as the ACT’s Chrys Dougherty does—that the only way to prepare for them is to build broad knowledge, then we could begin to create a fair and productive assessment and accountability system. Before the end of high school, all students should have broad enough knowledge to perform well on a reading comprehension test. But what about in third, fourth, or even seventh grade? In the early and middle grades, is a test drawn only from topics that have been taught in school the only fair way to test reading comprehension? How many years of systematically teaching “common” knowledge are needed before a reading comprehension test that is not tied to the curriculum is fair, especially for a student whose opportunities to learn outside of school are minimal?

The answer depends not so much on the test as on what is done with the scores. If we accepted the fact that reading comprehension depends on broad knowledge, we would radically alter our accountability policies. Scores on “common knowledge” reading comprehension tests would be recognized as useful indicators of where students are in their journey toward broad knowledge—they would not be mistaken for indicators of teaching quality or children’s capacity. Instead of holding schools accountable for scores on tests with content that is not tied to the curriculum, we would hold them accountable for creating a content-rich, comprehensive, well-sequenced curriculum and delivering it in a manner that ensures equal opportunity to learn. To narrow the inevitable gaps caused by differences in out-of-school experiences, we would dramatically increase free weekend and summer enrichment opportunities (for toddlers to teenagers) in lower-income neighborhoods. (We would also address a range of health-related disparities, but that’s a topic for another day.)

In sum, reading comprehension really does rely on having a great deal of common knowledge, so our current reading comprehension tests really are valid and reliable. To make them fair and productive, children from lower-income families must be given an equal opportunity to learn the knowledge that is “common” to children from higher-income homes.

shutterstock_200841857

Reading is always a test of knowledge (image courtesy of Shutterstock).

9 comments on “Reading Test Developers Call Knowledge a Source of Bias”

  1. 1
    Chrys Dougherty on October 1, 2014

    In this context, I would draw readers’ attention to the description in the ACT Technical Manual (p. 11) of the content areas from which selections are drawn for the ACT Reading Test:

    “a. Prose Fiction. The items in this category are based on short stories or excerpts from short stories or novels.

    “b. Social Studies. The items in this category are based on passages in the content areas of anthropology, archaeology, biography, business, economics, education, geography, history, political science, psychology, and sociology.

    “c. Humanities. The items in this category are based on passages from memoirs and personal essays and in the content areas of architecture, art, dance, ethics, film, language, literary criticism, music, philosophy, radio, television, and theater.

    “d. Natural Sciences. The items in this category are based on passages in the content areas of anatomy, astronomy, biology, botany, chemistry, ecology, geology, medicine, meteorology, microbiology, natural history, physiology, physics, technology, and zoology.”

    These passages reflect the wide range of reading that a college-ready student or an avid adult reader should be able to do. A student who receives a broad, content-rich education in preschool through high school is more likely to have the necessary “common knowledge” from these fields to have an advantage on the ACT, in college, and in life.

  2. 2
    Will Fitzhugh on October 2, 2014

    I knew it! I knew it! Bias Bias Bias, Yes!
    That is probably why people more often eat vegetables rather than mud/dirt and/or sticks! They are biased by their
    knowledge! I hope this news is not arriving too late
    to stop the propagation of knowledge in our Schools!!!

    Take note you Core Knowledge people!!

    Will Fitzhugh
    [email protected]

  3. 3
    Ewaldoh on October 2, 2014

    All old news. Twenty years ago, I was asked to find another term for “density” in a question I had submitted for the SAT test. Actually, I was asked to change the word or to define it. I chose to define it only to be asked to reconsider the term, “mass”.

  4. 4
    Fred Strine on October 2, 2014

    Lisa Hansel’s last sentence ought to be universally accepted as perfectly reasonable.

    However: The entire ETS Guidelines for Fairness Review of Assessments study reduces to “edu-speak” for dumbing down. Within a current culture that elevates victim status, it is incumbent upon all “caring” individuals to prove they share the pain of not knowing. Instead of aspiring to elevate everyone’s knowledge base (common knowledge), many appear all-too-eager to settle for intellectual mediocrity in their goals for public education. Wouldn’t want to be labeled an elitist. Wouldn’t want to embarrass the less fortunate, less advantaged, less privileged.

    Solution: Let nationally-mandated curricula be constructed on an as-needed basis not by the privileged elite, but by those oppressed by lower socio-economic inequality. Standardized test scores will rise dramatically proving the worth of “fairer” curricula. When social engineering via the public school system succeeds in its egalitarian and social justice goals, future generations can take justifiable pride in being truly equal if not truly ignorant.

    Question raised: Whose vision will determine the course of public education?

  5. 5
    Ponderosa on October 4, 2014

    Lisa, what do you make of the new SBAC and PARCC ELA tests? I doubt their makers would say they’re testing something as lowly as “common knowledge”. But perhaps they are, unwittingly. You can try the SBAC here (you can skip the ID etc. boxes; just press the “Sign in” button) :

    https://login4.cloud1.tds.airast.org/student/V42/Pages/LoginShell.aspx?c=SBAC_PT

  6. 6
    Ponderosa on October 4, 2014

    I just started taking the 3rd grade ELA test. I find questions 2, 5, 6 and 7 on the first passage pretty awful.

  7. 7
    Lisa Hansel on October 4, 2014

    Hi Ponderosa,

    I am working on posts on PARCC and Smarter Balanced now. So far, I see them seeming to test content that would be common knowledge if one had the benefit of a rigorous education focused on building academic knowledge and skill. Of course I think there ought to be a core curriculum and curriculum-based tests. But if we accept that politically we are far from that ideal, then my concerns are more with accountability policies than with PARCC and Smarter Balanced. Those tests do assess ELA ability, but I don’t think even the best value added model can fully control for home environment so I have concerns about the high stakes attached to the tests.

    What did you think of the Smarter Balanced sample?

  8. 8
    Brian King on January 24, 2015

    This is a very interesting article, and your thoughts on common knowledge really apply to my situation.

    I teach English at an international school in Malaysia. All curriculum, instruction, and assessment are given in English. Additionally, English proficiency among students varies greatly.

    I teach lower elementary students, so we are working daily on those common knowledge words and sentence structures. Unfortunately, I have seen many content area exams written by colleagues without this in mind. The result is lower scores on content exams, simply because students could not comprehend what was being asked.

    I know you wrote this post with exam companies in mind, but I think this problem persists wherever English Langauge Learners are being instructed in mainstream classrooms. It takes a lot of effort on the part of the content teacher to thoughtfully identify and teach potentially problematic vocabulary. I have found that many of these teachers feel that this basically not there problem. This attitude is something I am passionate about changing at my current school.

    There is a disconnect between what English teachers are focusing on, and what content area teachers believe students can handle in regards to English comprehension. I would like to help educate my colleagues on this problem, and I think you post would be a good starting point.

    Thanks again for the insightful post.

    Sincerely,
    Brian King

  9. 9
    Will Fitzhugh on January 24, 2015

    Knowledge,knowledge,knowledge——that IS the problem. Weren’t Adam and Eve TOLD not to eat of the fruit of the Tree of the Knowledge of Good and Evil? That’s where it all started! And it is the job of our modern educators to keep as much knowledge from children as possible. They can always learn skills, instead, right?

    Will Fitzhugh
    [email protected]
    http://www.tcr.org

Leave a Reply

Your email address will not be published.Required fields are marked *
All comments are held for moderation.