Mindful Measurement: A Generalist Outside View
Mindful Measurement: A Generalist Outsider’s View
The first known instance of the word ‘mindful’ in the English language occurred in a 1382 translation by John Wycliffe of the Bible — specifically, in a phrase from the Book of Proverbs 12:12 where it means ‘full of memories’. The memories that first invoke Mindful Measurement for me commence with my appointment as Chief Learning Officer at Educational Testing Service in 2002. Such a title (which I held until 2017 and then became a Knowledge Broker in Research until retiring this past February) would seem to explain my interest in this topic, but also obliges a disclaimer: my background does not contain proficiency in statistics or assessment. My rank in the educational measurement hierarchy only advanced toward the end of my time at ETS to ‘generalist’ from ‘layperson’: as my friend and former colleague Michael Kane generously put it, someone who has managed to learn a little about a lot of things in the field. CLOs no matter the organization in which they putter and provoke concern themselves with the sharing, generation, and organization of knowledge, the development of individuals and groups, and perhaps most critically, change — its facilitation, transformation, and variations. That is my interest in Mindful Measurement: helping others to change their view of the world of measurement as others helped me.
How did I learn? Anyone who’s going to add value as a CLO needs to know something about the core competencies of the organization in which they are lucky enough to reside. Consequently, I found people who were much smarter than me and yet also very kind and most interesting. They filled my mind with multiple dimensions of measurement. The principles gained from all of that tutelage boiled down to three words — error, claims, and fairness: all things of which to be mindful for anyone involved with measurement, which as either measurer or measured covers just about everyone in this country. If our measurement is to be mindful according to its dictionary definition that is closer to everyday usage — “careful or taking care to do something” — then we should find a way to have fellow Americans remember that measurement always contains error, will be used whether you want it to or not to make claims (often about you and yours), and should be applied in such a way that it advances rather than impedes fairness. Is this quest quixotic? If 350 million people worldwide can play the videogame Fortnite, might not a sizable crowd appreciate the ways in which measurement affects their lives? What would the world look like if the overall mindfulness of the population included a much stronger understanding of error, claims, and fairness even?
Measurement professionals understand that their work always involves error, and in a wider sense occurs in an unavoidable context of uncertainty. But do they always emphasize that in many of our practices we don’t even know exactly what we don’t know? For example, cognition — the workings of the information processes that intervene between input to our brain and behavior that results — while essential to contemporary educational assessments is still described as a ‘black box’, “A device which performs intricate functions but whose internal mechanism may not readily be inspected or understood”. But it’s not just the brain whose contours cannot be determined accurately let alone the workings whirring away inside, it is also the influence of the environment, the social nature of learning itself. Therefore, being mindful about measurement would require us to acknowledge both the uncertainty and error inherent in measurement. In the latter case, such errors are not just simply the difference between a measured quantity such as an observed score for something like a single algebra test and the true score, which we might find if we could measure the student’s knowledge of algebra an endless number of times. The larger sense of error that we should keep in front of us is the realization of imperfection, of the necessary incompletion of any measurement. George Box’s famous dictum that, “All models are false, but some models are useful” serves us well in this pursuit.
Appropriately humbled, we might move to our second key word and emphasize that real-world assessment occurs every time that someone wishes to make a claim about someone else or about a group, whether it is called a claim or not. Picking someone for a job? That’s a claim. Deciding who gets to go to an Ivy League college where the networks accessed can allow someone to join eventually one of the ruling oligarchies? That’s a claim. Allowing someone to hold a piece of paper that enables their employment as everything from an air traffic controller to a zookeeper? That’s a claim. Mindful measurement would make explicit that claims are the conclusion of arguments, arguments rely upon evidence, and evidence can be shoddy or neat, dubious, airtight, or even missing. Mindful measurement would advocate and educate so that our fellow citizens understood that getting rid of testing would not get rid of claims, but only make them less useful.
Working from that foundation, then mindful measurement evangelists could do for others what another friend and colleague, Bob Mislevy, did for me: enlighten how claims are the underpinning of good measurement. If 79 million people will watch a YouTube video on 13 Ways to Survive Wild Animal Attacks, wouldn’t we be able to get a fraction of them interested in understanding how claims are made — the good, the bad, and the ugly? We could draw upon the two sources, to which Bob introduced me, neither of whom were in the field of educational measurement: Stephen Toulmin, a British philosopher, and John Henry Wigmore, an American legal scholar in the first half of the 20th century. The former could help us to articulate that every claim should have an argument. As Toulmin wrote: “A man who makes an assertion puts forward a claim — a claim on our attention and to our belief … just how seriously it will be taken depends, of course, on many circumstances… whatever the nature of the particular assertion, in each case we can challenge the assertion, and demand to have our attention drawn to the grounds (backing, data, facts, evidence, considerations) on which the merits of the assertion are to depend. We can, that is, demand an argument.”
We might persuade those who are the subjects of those arguments to demand more transparency as to the way in which the claims being made about them draw upon what Bob Mislevy and his co-author Michelle Risconcente described as the “evidentiary reasoning, (how) from a handful of things that students say, do, or make, we want to draw inferences about what they know, can do, or have accomplished more broadly.” Mindful measurement would make “explicit the structures of assessment arguments, the elements and processes through which they are instantiated, and the interrelationships among them.”
And then as if we were in the courtroom part of a Law and Order episode, we could inform as to the importance of evidence to the validity of any particular claim. When it comes to the claims we make — or are made about us — in everyday life, evidence often fails to receive appropriate attention. We think much more about the score or outcome (getting into a particular school, snagging an A grade on an exam, being awarded a job) than we do about how it was derived. As Bob Mislevy noted in a 1994 paper discussing Wigmore at length, “there are multiple routes to an outcome, and observing the outcome alone does not indicate the route.” In the most useful sorts of processes that end up producing claims, we should want evidence not only of what someone knows or can do but of how they are able to use that knowledge in order to accomplish the tasks that constitute the assessment.
This educational project of the MM crowd should seek to disabuse the common sense that all that matters is the score; a belief that unfortunately and somewhat cynically is sometimes expressed by educational measurement professionals themselves. A score or outcome is as Charles Wheelan puts it in his book Naked Statistics a “handy tool for collapsing complex information into a single number.” If we are mindful about measurement, we realize that this consolidation may lead to false claims unless we know more about the evidence and how it was gathered. By centering our discussions of measurement on claims, we would shift attention away from the score and back to decisions about what was worth measuring in the first place : the construct. Getting folks to look more closely at a measurement’s construct — the “complex of knowledge, skills, or other attribute (that) should be assessed” — might stimulate rethinking what it is that is being measured, not just how it is being measured.
Thus my third and final point about mindful measurement is that if we are going to measure people then fairness demands that we find a way to do so that allows people to change and develop while avoiding construct-irrelevant barriers to the success of test takers, including those with disabilities, learning differences, and English-language unfamiliarity. Researchers and assessment developers like another friend and former colleague Mark Hakkinen have made remarkable progress in those areas but the next frontier of fairness is making sure that the measurements do not reinforce existing structural inequalities. As other friends and former colleagues Caroline Wylie, Christine Lyon, and Randy Bennett articulated with what is generally called Formative Assessment and specifically ‘Cognitively Based Assessment of, for and as Learning or CBAL™, mindful measurement will figure out how to more universally institute classroom-based processes “in which students and teachers collect evidence of learning in order to understand current learning progress and to make adjustments to learning or to teaching as necessary”. As a generalist, I describe it as weaving measurement into the learning in a continuous cycle. It’s not cheap and still requires further refinement, but fairness requires we move away from assessments that focus more on delivering the latest score than the new opportunity.
There is another dictionary definition of mindful, “fully aware of the moment.” For this kind of mindfulness, we must take apart our own preconceptions and paradigms. That practice is not just for those whom we seek to edify. We should seek to inquire among those with whom we disagree all the time attempting to be agreeable. We should admit that much of what we want to measure that matters still involves a great deal of uncertainty. And we should not let that challenge stop us from promoting this important activity of making claims that allow people to advance, to learn, and to succeed.
