Feedback for Learning in Science, Technol­ogy, Engineering, and Mathematics

Maria Ruiz-Primo and Min Li (2013) examined formative assessment from the perspective of research in the areas of science, technol­ogy, engineering, and mathematics (STEM) edu­cation. This chapter is based on a meta-analysis where the research papers were examined and found to be appropriate to be included – from 9000 to 238 papers. This focus allowed the researchers to examine the state of research into formative feedback in the classroom context.

This statement, “We argue that feedback that is not used by students to move their learn­ing forward is not formative feedback.” (p. 219) is a helpful one to teachers reflecting on their practice. This is a similar statement to one made by Dylan Wiliam and Paul Black (2009) although they were examining different work. Ruiz-Primo and Li (2013) give a detailed description of what formative feedback does look like in the classroom context elaborating on the following points:

1.    Be seen as a process strongly guided by the learning goal(s) that the teacher and students work toward.

2.    Actively involve students in the feedback process by engaging them in (1) defining the evidence of learning and/or success criteria (or goal or reference level) being targeted, (2) comparing the current or actual level of performance with the evidence or the criteria for success, and (3) using assessment information to improve their own learning to reduce the gap.

3.    Be considered as an instructional scaffold that goes beyond written or oral comments.

4.    Is specifically intended to improve learning outcomes (e.g., deepening conceptual understanding) and processes (e.g., reflecting on one’s learning and learning strategies or making new connections to what has been learned…)

5.    Ensure its usefulness by making feedback accessible and practical. Appropriate feedback helps students develop sufficient insights into their own learning and become self-critical, self-reflective, and self-directed…

6.    Consider different sources of information about students’ learning and understanding from highly formal (e.g., taking tests or quizzes, filling in a handout, or completing an investigation report) to very informal (e.g., questions, comments, observations, and evaluative and interpretative listening)

7.    Demonstrate, over time, alignment with a learning trajectory at least within a unit or module. That is, students will take actions based on feedback aligned to the trajectory used to design and map the instructional and assessment activities…

And, as they do so, they carefully cite the research that supports their statements. They explain that the definition highlights two characteristics:

1.    Students are essential players in the feedback process. They need to be partners in clarifying learning goals or success criteria, comparing current perfor­mance with success criteria, and using the infor­mation to formulate an action plan and as feedback providers and developers of actions to be followed.

2.    Feedback that supports learning is far more than oral or written comments to students’ performance. It involves looking for evidence of students’ learning on ongoing interactions and continually engaging students in discussions about learning and quality.

This chapter is well worth reading. I appreciated the thoughtful proposal of a framework for research in this area along with the frameworks implications from classroom practice. It is important that feedback be studied in the context of classroom learning and framing powerful research questions is an important part of that work.

As I read this chapter, I especially appreciated the emphasis on the role of the student – the learner. For example, “If [formative assessment] is exclusively in the hands of the teachers, it can be argued that students cannot be entrusted with managing their own learning, and self-regulation skills thus are less likely to be developed (Andrade, 2010) (p. 224).

This chapter is a ‘must read’ if you are working with educators and promoting high quality feedback to support student learning. You might also enjoy Dylan Wiliam’s (2007) chapter focused on feedback from an instructional perspective in mathematics classrooms.

Those of you attending Assessment for Learning: Canada in Conversation with the World in Fredericton, NB, will enjoy listening to and learning from Maria Araceli Ruiz-Primo. Keep track of the conversations at: https://tagboard.com/AforLConversation/160953

Or, check in on Twitter @Anne_DaviesPhD or #AforLConversation

References

Andrade, H. (2010). Students as the definitive source of formative assessment: Academic self-assessment and the self-regulation of learning. In H. Andrade & G. Cizek (Eds.), Handbook of Formative Assessment (pp. 90-105). New York: Routledge.

Ruiz-Primo, M. & Li, M. (2013). Examining formative feedback in the classroom context: New research perspectives. In J. H. McMillan (Ed.), SAGE Handbook of Research on Classroom Assessment (pp. 215-232). New York. SAGE Publications.

Black, P. & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1): 5 - 31.

Wiliam, D. (2007). Keeping learning on track: Classroom assessment and the regulation of learning. In F. K. Lester Jr. (Ed.), Second Handbook of Mathematics Teaching and Learning (pp. 1053–1098). Greenwich, CT: Information Age Publishing.

Assess to Learn - Impressive gains in student learning and achievement

Are you seeking more information and support for powerful professional learning using assessment for learning in support of numeracy and literacy?

Do you want to document impressive gains in student learning and achievement, and have teachers and schools report positive sustainable changes in teaching, learning, and assessment processes, practices and systems?

 

The New Zealand Ministry of Education’s professional development Assess to Learn project report documents the impact on teachers, students, and schools in New Zealand. The key outcomes of the project were to:

•      improve student learning and achievement

•      shift teachers’ knowledge and assessment practice

•      develop coherence between assessment processes, practices, and systems in classrooms and in schools so that they promote better learning

•      demonstrate a culture of continuous school improvement

This project has been underway since 2002. Evaluation has been ongoing since 2003 by national evaluators Dr. Jenny Poskitt (Massey University) and Kerry Taylor (Education Group Limited).

They have documented impressive gains in student learning and achievement, and teachers and schools report positive sustainable changes in teaching, learning, and assessment processes, practices, and systems. The full report, published in 2008, describes the project, summarizes their approach to professional learning, and documents the changes that emerged as a result of the focus on assessment for learning.

This report is of interest to those seeking more information and support for powerful professional learning using assessment for learning in support of numeracy and literacy.

Jenny Poskitt is presenting at Assessment for Learning: Canada in Conversation with the World.

Find out more about the conversation at https://tagboard.com/AforLConversation/160953

Exploring Professional Judgement New Zealand Style

There has been alot of interest in the notion of “informed professional judgement” as many Canadian educators experience a change in the balance between large-scale external assessments across a province and classroom assessment. As I’ve been reading the work of the various International experts in the area of assessment for learning, it has been fascinating to see how the notion of professional judgement is emerging differently in different jurisdictions. 

As educators, we can find ourselves “adopting a good idea” without having taken the time to examine the underlying assumptions and the context in which the idea first took root. For example, if one quickly reads the well-written chapter by Susan Brookhart (2013) titled, Grading, you might miss that she clearly notes  the findings are pertinent in the United States. You might assume that means they would be the same elsewhere. You would likely be wrong. The current assessment, teaching, and learning conditions in the United States are unique and not necessarily transferable to other countries, given their context.

In a recent post, I shared the work of some of the Australian team members in the area of moderation of student work. Australia has, during the past decade, begun to implement national testing. 

This work is fascinating and is of immediate interest to educators in Canada where many provincial/territorial jurisdictions also have external examinations. Here in Canada there are many jurisdictions that use the process of moderation with students, with teachers, across schools and across school systems. In the 1990s, British Columbia engaged in moderation processes as part of developing reference sets – descriptions of student learning over time – in a variety of areas such as reading, writing, numeracy, group communication, and technology. In the late 1990s, Ontario, British Columbia, and many individual school districts collected student work samples in subject areas such as writing, numeracy, social responsibility, and so on and engaged in a variety of moderation processes. The process itself varies and yet is typically described as incredibly powerful professional development.

Recently I’ve been reflecting on two papers focused on “Overall professional judgement” submitted by New Zealand team members - Charles Darr and Jenny Poskitt. While the process they describe also involves looking at student work, it is evolving in a very different context.

In the 1990s, New Zealand developed a powerful and extensive set of assessment exemplars that illustrated learning over time in a variety of subject areas and disciplines. During the past decade, New Zealand has moved to articulated national standards - however, there is no national testing. Instead of relying on national testing, they are seeking to continue the work towards  “overall professional judgement.”

Poskitt and Mitchell (2012) in their article titled, New Zealand teachers’ overall teacher judgements (OTJs): Equivocal or unequivocal? describe some of the work being done as teachers develop their understandings of National Standards and work towards common understandings of quality in relation to the standards. They write, “Teacher capacity to judge current and future performance is important. With multiple opportunities to gather pertinent information, teachers are best placed to make valid (unequivocal) judgements on student achievement when they have shared understandings of standards. Because standards are comprised of multiple criteria, not all of which are evident in samples of student achievement, teacher understanding of standards develops through professional conversations and moderation processes.”

In this article, after examining the International context, the history of this work, and the challenges that arise, they detail the research methods – case study – that they used. I appreciated this concluding statement, 

“In essence there is tension between teachers’ tacit knowledge (gut feeling), intra and inter professional judgements, and explicit knowledge. Although having clarity about the composition of OTJs and standards, appropriate evidence to underpin those judgements and using exemplars and verbal descriptors to support the judgements is likely to lead to greater consistency (Sadler, 1998; Gardiner et al., 2000), it is only throughmoderation processes that teachers will reach deeper understanding of standards (Wyatt-Smith et al., 2010). Deep or meaningful change in teacher beliefs and professional practice takes time, generally considerably longer than anticipated (Timperley, Wilson, Barrar, & Fung, 2007).” (Poskitt and Mitchell 2012, p. 73)

A second paper by Charles Darr (2014) describes the development of an online tool to support teacher judgements. Darr (2014) outlines the framework and the challenges of supporting teachers professional judgement with an online tool. As I read Darr’s paper, I was reminded of Malcolm Gladwell’s book, Blink:The Power of Thinking Without Thinking (2005) as well as the Assessment Reform Group’s 2006 report, The role of teachers in the assessment of learning.

The work focused on professional judgement has relevance to Canadian educators as it supports some of the work already being done and offers fresh ways of thinking about it as work in the area of professional judgement moves forward in Canada - in particular, the many projects across Canada exploring alternative ways to communicate to parents and communities about student learning. 

To read more about ‘overall professional judgement’ from a New Zealand perspective, you might want to read the papers by these New Zealand researchers - I've included links in the reference list below.

In closing, we have much to learn from our international colleagues from New Zealand and elsewhere. I am heartened that these ongoing international conversations support all of us to be critical and thoughtful consumers of research and ideas imported from other contexts. 

We will post glimpses of our deliberations at: https://tagboard.com/AforLConversation/160953

References

Assessment Reform Group. (2006). The role of teachers in the assessment of learning. UK: ARG.

Brookhart, S. (2013). Grading. In J. McMillan’s Sage Handbook Research on Classroom Assessment (Chapter 15). Thousand Oaks, CA: Sage Publications.

Darr, C. (2014).  The development of an online tool to support teacher judgements. New Zealand Council for Educational Research. Auckland; NZCER.

Gladwell, M. (2005). Blink: The Power of Thinking Without Thinking. New York: Little, Brown and Co.

Poskitt, J. & Mitchell, K. (2012). New Zealand teachers’ overall teacher judgements (OTJs): Equivocal or unequivocal? Auckland: Assessment Matters 4, 53 – 75.

Find your Assessment Community

Working by ourselves in isolation is hard. When we are the ones tromping a new path in a new direction, it is easy to get discouraged. I was thinking about how hard it is to walk alone when I was walking through the woods after three days of snow.  I looked ahead and saw undisturbed snow as far as I could see – not one footprint. “This is going to be a lot of work,” I thought.

I was right. It was a lot work. Just like it was a lot of work in the 1980s when Rick Stiggins, Terrance Crook, and others were laying the foundation for what would become the field of classroom assessment. Every one of us who have come along and followed in their footsteps has found it easier to make a difference because of the work they did. We benefit from the hard work of those whose steps we follow. We need to find our assessment community and continue to expand the community as much as we are able to do so. And share who we are learning from and what we are learning with others. After all, we benefit from the shoulders upon which we stand.

Right now I’m in the airport on my way to Atlantic Canada. I’m doing this blog because my flight is delayed – the reality of air travel these days. Over the coming week more than 250 people will be travelling to Fredericton from all across North America and as far away as Finland, Scotland, Israel, Australia, Japan, Singapore, and New Zealand. The International and Canadian Symposium delegates and the conference attendees are making the trip because we know that ‘together we are better.’

These events can only happen because every single participant – presenter, delegate, conference attendee – is donating their personal time to engage in conversations that will go beyond what any one of us know and on to that place where together we can create something more on behalf of learners. Everyone shares the hard work they've done and the learning that is hard won!

It is together that we can share what we’ve learned about learning and assessment in classrooms, in faculty groups, and across systems. We will each leave with resources to support our own work in the area of classroom assessment – assessment in the service of learning.

And, most importantly, we will leave with HOPE that this work we’re engaged in – this tough work – is worth it for the learners whom we serve.

We hope to see you there! If you can’t come, watch #AforLconversationfor updates and ‘word bites.’

All my best,

Anne

PS NEWS!!! There is one spot available to a Canadian who wants to be a delegate to the Canadian Symposium. If you want to have that spot you need to be able to get yourself to Fredericton for 5:30 pm Tuesday, April 7th, and stay until Saturday, April 12th, at 4:30 pm. There is a registration fee and accommodation plus related costs. If you are interested, please get in touch with me through Twitter :-)

PPS There is a lot of snow in Fredericton, NB –  100 cm of fresh snow last week. Has the metaphor become reality? Please send along your best wishes that we all make it through the drifts to the conversations.

Reinventing Instructional Sequence: Creating Experts

  

I was with an amazing group of dedicated educators focusing on instructional sequence last week – and I thought I would take a break from the Symposium readings and share this work with you. 

In our recent work, we’ve been talking a lot about instructional sequence. An instructional sequence is a series of learning tasks designed in a logical sequence (progression) with the purpose of accomplishing specific learning related to outcomes and standards. You can think of it as the other side of the assessment and instruction ‘coin.’ Or, picture a mobius strip -
– one moves from assessment to instruction and back without ever leaving the path.

We know it is important to be clear about what we intend our students to learn – to know, do, and articulate – and what quality is expected. It is also important that we know what it looks like along the way – so we can affirm what students are doing that is heading in the right direction, refocus their learning along the way, and provide new direction and instruction. We can then assess student evidence of learning and provide in-the-moment direction that acts as feedback to forward the learning.

Substantial evidence exists that shows that well-aligned instruction – which uses assessment in the service of learning – produces results 2 to 3 times greater than non-aligned instruction. Yet, there is also substantial evidence to show that teachers are often unable to identify the ‘next teaching steps’ based on what students are creating, doing, or saying.

THAT IS THE PROBLEM. 

A two-fold challenge:

1. How can we assist teachers to know the pathways to learning so they can guide students?

2. How can we assist students to know the pathways to learning so they can guide themselves?

The solution?

 

In the classroom this is a process of involving students in examining samples and co-constructing criteria which I describe in detail in Making Classroom Assessment Work (2011). In the school or system this is a process of involving teachers and others in examining samples, co-constructing criteria, and building a learning continuum which Sandra Herbst, Beth Parrott Reynolds, and I examine from a leadership perspective in Leading the Way to Assessment for Learning: A Practical Guide (2012).

 

Many researchers have been documenting the power of moderation at the system and school level to help teachers assess and evaluate quality. See earlier blog.

In some cases researchers have located their work regarding moderation in classroom settings – one recent example is a paper by Royce Sadler looking at the power of moderation in a higher education class – see earlier blog.

When I engage participants in considering assessment in the service of learning from this perspective, I deliberately use a series of video examples to illustrate the key ideas. 

The video clips show teachers (K-12) working with students in every part of the process. After all, as learners we need to see what this can look like with our students - and be part of the story of our colleagues' work.

Where are we going?                                

How will we get there?      

Are we there yet?         

         

These powerful strategies that teachers deliberately use form an instructional sequence that teaches students how to self-monitor their learning. This process is an important part of what is being called ‘co-regulation’ – that is, students and teachers working together to understand how to learn, what quality looks like, and what next steps towards quality can be taken at this point in the learning.

The result? Over time students learn to self-regulate because teachers have deliberately involved students in this instructional sequence over and over again. As Malcolm Gladwell (2008) explained so powerfully in Outliers - practice makes expert. Malcolm Gladwell based much of his book on the work of Anders Ericsson, who famously documented the 10,000 hours needed to become an expert.

This process is also a critical part of making grading and reporting transparent. After all transparency isn’t about being clear about “what counts.” It is about quality and what matters. You can find out more about this important concept in A Fresh Look at Grading and Reporting in High Schools.

 

Think about it – students are in public school almost 10,000 hours over their K-12 education. That is not a lot of time to become expert in many, many things. But what if we asked them to become expert in ONE THING? What might that ONE THING be? I propose we work towards making our students experts at LEARNING and using assessment in the service of learning. And that means helping them become experts at self-regulation of learning.

References:

Davies, A. and S. Herbst (2014). A Fresh Look at Grading and Reporting in High Schools. Courtenay, BC: Connections Publishing and Bloomington, IN: Solution Tree Press.

Davies, A., S. Herbst, and B. Reynolds. (2012). Leading the Way to Assessment for Learning: A Practical Guide. Courtenay, BC: Connections Publishing and Bloomington, IN: Solution Tree Press.

Davies, A. (2011). Making Classroom Assessment Work, 3^rd Ed.Courtenay, BC: Connections Publishing and Bloomington, IN: Solution Tree Press.

Ericsson, Anders K., N. Charness, P. Feltovich, and R.R. Hoffman. (2006). The Cambridge Handbook of Expertise and Expert Performance. Cambridge, UK: Cambridge University Press.

Gladwell, M. (2008). Outliers.New York: Little Brown and Company.

Sadler, D. R. (2013). Opening up feedback: Teaching learners to see. In S. Merry, M. Price, D. Carless & M. Taras (Eds.), Reconceptualising Feedback in Higher Education: Developing Dialogue with Students. (Ch. 5, 54‑63). London: Routledge.