Blog


What’s in a 4? DOK and Proficiency Scales

When it comes to measurement, four is a popular number; rather, a range of 1 to 4 is a common scheme. Two different powerful measurement systems use a range of 1 through 4 scores to clarify levels of quality. The Depth of Knowledge [DOK] framework by Webb, 2005, uses a 1 through 4 scale to rank the cognitive complexity of an assessment task. The Proficiency Scale framework by Marzano and Kendall (2008) uses a 1 through 4 scale to rank students’ performance levels on individual standards.

Both assessment measurement systems have much in common:

  • Both systems range from 1 to 4
  • Each number in the range of both systems represents a degree of sophistication
  • The 1 represents the lowest degree of sophistication and the 4 represents the highest degree, or the “best” number

But that’s where the similarities stop. The DOK scale is used to measure the rigor level of a task (the source), whereas a proficiency scale is used to measure the quality of student work (the result) on an academic standard. A side by side summary of both frameworks highlights the differences in each level:

Depth of Knowledge Levels Proficiency Levels
(The measure of cognitive complexity in a task) (The measure of proficiency found in student work)
4: Extended Thinking—Requires an investigation, time to think and process multiple conditions of the problem (Synthesize, Analyze, Prove, Connect, Design, Apply Concepts). 4: Advancing—The student has met standard expectations and advances the standard requirements with in-depth inferences and/or extensive sophisticated connections, etc.
3: Strategic Thinking—Requires reasoning, developing a plan or a sequence of steps, some complexity, more than one possible answer (Assess, Revise, Critique, Draw Conclusions, Differentiate, Formulate, Hypothesize, Cite Evidence). 3: Achieving—The student is independent and demonstrates accuracy (no major errors or omissions) on standard expectations.
2: Skill/Concept—Use information or conceptual knowledge, two or more steps, etc. (Infer, Identify Patterns, Modify, Predict, Distinguish, Compare). 2: Developing—The student is independent, but demonstrates only partial accuracy on standard expectations
1: Recall—Recall of a fact, information, or procedure (Recite, Recall, Label, Naming, Define, Identify, Match, List, Draw, Calculate). 1: Initiating—The student is dependent on scaffolding and support to demonstrate minimal or inaccurate / incomplete understandings of standard expectations

 

Even though one system measures the input (task) and one measures the output (result), it’s a very common and alluring misstep to equate the two systems as synonymous. The argument could be made, for example, that a student is advancing (proficiency level 4) if he/she can engage in extended thinking (DOK level 4) regarding the standard(s).

However, it’s important to remember that the two 4-point schemes are meant to serve very different purposes. All four levels of proficiency can apply to any task that is not binary (there are no gradations of quality, so evaluation is solely based on right/wrong, yes/no, or present/absent responses); and, a single performance task can be assigned any one level of cognitive complexity. In other words, a task that requires strategic thinking (DOK level 3), can generate all 4 levels of proficiency in student responses.

The 3 Dangers

There are a few dangers in tying the two measures together.

Trapping Students at the Lower Levels

The first involves the limitations such a combination would place on students: Students who struggle to earn high levels of proficiency would become trapped in lower level tasks. Obviously, this is a significant concern. The joy and the big picture of learning often rest in DOK 3 and 4 level tasks which many struggling learners need, but if deemed a 1 or a 2 in proficiency would not be allowed to experience.

Forcing Students to Only Engage at Level 4

The second danger is the reverse of the first: Students wouldn’t be able to earn a 4 level of proficiency unless they were always engaged in extended thinking tasks.

Oversimplification

The third danger is the oversimplification of two different sophisticated measurement systems. Turning complex evaluation processes into streamlined, simple algorithms compromises the accuracy of teacher decision making and exempts educators from needing to think carefully about what constitutes deep learning.

So, what’s in a 4? It turns out there is a lot of important information behind any assigned score … but the information will vary based on the specific measurement system being employed and the items being measured. Let’s use each system as it was intended and avoid the temptation to merge and simplify them.

“I wouldn’t give a nickel for the simplicity on this side of complexity, but I would give my life for the simplicity on the other side of complexity.” —Einstein

 

References:

Marzano R. and Kendall J. (2008) Designing and assessing educational objectives: Applying the new taxonomy. Thousand Oaks, CA: Corwin Press

Webb, N. L., Alt, M., Ely, R., and Vesperman, B. (2005). Web alignment tool (WAT) training manual. Washinton, DC: Council of Chief State School Officers


It’s Okay NOT to Reassess

It’s okay NOT to reassess everything all of the time; there, I said it. Now, before I explain more thoroughly let me say that I am fully aware that this post may not make me the most popular kid at the assessment table and that I’m almost certain that the standards-based purity police will be out in full force; doesn’t matter because what I’m about to say needs to be said. Read more



Can Assessment and Open-Ended Contexts Coexist?

When we consider all of the ways to ensure successful learning outcomes, knowing the criteria for success definitely tops the list. When we know where we are going, our chances of reaching that destination increase dramatically. But what about those times when we are trying to invite open-ended experiences: creativity, play, and imagination? How does criteria-setting fit within that paradigm? Can assessment practices, such as criteria-setting and self-assessment, live in harmony with these open-ended or emergent outcomes? Read more


What a Difference Our Words Make! Asking Questions that Require Understanding

There are so many ways to ask a question! I was reviewing some assessments yesterday and noticed that one or two simple word changes can totally change the sophistication needed for the response. When we consider what we want a student to know, and then create the question, I think we need to look at the question from a different perspective. We need to ask ourselves if the question leads to a response that truly measures understanding. Read more


Is Assessment Having a Moment?

A recent job change has extended my daily commute and as a result I have been listening to audiobooks to pass the time and minimize the frustration with road construction.  I know that I am late to this party, but audiobooks are a pretty great way to both decompress after a challenging day and get excited about a new one.  I recently listened to The Power of Moments by Chip and Dan Heath and it has been rolling around in my head for a while since I finished it.  (This poses a new challenge with audiobooks.  With a traditional book, I would flip through the pages and reread different sections.  I haven’t quite figured out how to do that with the audio version.)  I have been a big fan of the Heath brothers since reading their book Switch, which contained one of my favorite metaphors about the change process and proved incredibly helpful in a variety of settings.

Read more


Comments: 3

What are Common Formative Assessments Anyway?

One thing I’ve learned as I work with schools across the country is that there are a lot of different definitions collaborative teams are using for common formative assessments, and what these teams think common formative assessments are influences how they write and use these assessments with their students. In our book, Collaborating for Success in the Common Core, we offer the following definition to help teams make sure they’re able to use their results to improve student learning Read more


A Beautiful Noise: Productive Student Talk Time

It’s a beautiful noise
And it’s a sound that I love
And it makes me feel good
—Neil Diamond

I’ve been working a lot lately with educators in developing curricular units of study and the corresponding assessments while talking about the learning skills necessary for students to experience success. As an aside, I’ve deliberately not used the label “21st Century” in front of “learning skills” as I think we all understand in 2017 that we are in the 21st century. It’s lost its cache or novelty. Read more


Adult Leadership Skills and Student Dispositions

I was on a plane a few weeks ago and picked up the in-flight magazine. There was an interesting article about leadership and the dispositions needed to succeed written by Gary Kelly, the president of Southwest Airlines. The article concentrated on the following:

  • Leaders Must Care.
  • Leaders Must Communicate.
  • Leaders Must Have Character.
  • Leaders Must Be Competent.
  • Leaders Must Have Courage.

It occurred to me quite quickly that these are the skills we also want our students to have. Read more


Comments: 1

Start Your Year by Building a Learning Partnership with Your Students

It’s here. The start of the school year—that crucial time when educators excitedly “set the stage” with their students and jumpstart their vision for a successful learning experience in their class. It’s an official opportunity to initiate a strong learning partnership with students that empowers them to grow in their independence and empowerment as learners (Popham, 2011). Sounds good, right? Yet, if we think about the typical approach to setting the stage at the beginning of the year, it often falls short of establishing a strong foundation for that partnership. Read more