Assessment

Assessing oracy: Is Comparative Judgement the answer?

Oracy skills are vital to success in school and life. And yet, for many children, opportunities to develop them are missed. Educationalists are engaging in a growing debate about where oracy fits into the school system. Labour has put it at the heart of its plans to improve social mobility and an independent commission is looking at how it is taught in the classroom. This renewed focus on oracy means it is more important than ever that teachers have a way to reliably assess and understand their students’ attainment and progression. Amanda Moorghen of oracy education charity Voice 21 explains how Comparative Judgement can help with that and why it may be a game changer.

Date18/04/24

AuthorAmanda Moorghen

Using Comparative Judgement to assess oracy: something to shout about?
Photo: Patrick Fore /Unsplash

Oracy is the ability to articulate ideas, develop understanding and engage with others through spoken language.

In school, it is a powerful learning tool.

Teaching students to become more effective speakers and listeners empowers them to better understand themselves, each other and the world around them.

Oracy is also a route to social mobility, empowering all, not just some, to find their voice for success in school and life.

Around the UK, hundreds of schools see it as a priority, so why is it not more valued in the heart of the education system?

It could be that there are other issues taking precedence such as staffing levels, budgetary demands or timetabling pressures.

Another reason could be that there is no national, standardised test for oracy.

Beyond the Early Years, we have no national statistics for oracy unlike literacy and numeracy.

As a result, teachers cannot reliably understand student attainment and progress in comparison to the national picture.

Those monitoring schools, whether in government, local authorities or MATs, lack the evidence needed to identify areas where support is needed or measure the effect of that support.

A key reason for lack of standardised testing is that historically, oracy has been considered “too subjective” to assess, particularly in high-stakes exams.

We have been looking at a solution for these issues with a pilot project using comparative judgement to assess and reliably score students.

Students are filmed completing a spoken assessment task, then those videos are shared with a nation-wide network of assessors – familiar with the Oracy Framework – who compare multiple videos , multiple times.

Videos are judged in pairs against a simple statement such as “choose the best explanation.”

This human assessment is fed into specialist software which combines them to create a rank order of all the performances

When assessing oracy, comparative judgement offers two significant benefits that help us reliably grade/score student performances.

First, it avoids the challenging task of describing which levels of performance should receive which marks. Instead of being compared to mark schemes, performances are ranked against each other.

Second, it tackles “subjectivity.”

Because each performance is assessed by multiple judges before the final rank is generated, moderation is in effect built in.

Our initial trials in around thirty schools used assessment tasks developed specifically for use with comparative judgement.

This approach yielded more reliable rankings of student work than we could have gained through existing methods.

The trials’ success led to a larger-scale pilot with Year 5 students which generated a national sample of oracy performances and informed our latest Impact Report.

It showed that students in our longest-standing oracy schools returned the higher performances.

There was a 50% increase in scores between students in first year oracy schools and those in fourth year oracy schools.

These findings will not only enable us to provide schools with much clearer guidance on what ‘good’ looks like, but also develop an assessment tool for schools to measure oracy attainment and progress in a national context.

This will raise the bar for oracy education by letting teachers and students see what “great” looks like and work towards that.

We believe this approach could be a game changer.

If comparative judgement is proven reliable and fair for assessing oracy, it enables an evidence-informed approach at all levels.

Teachers will be able to target support to those who need it most and ensure students progress as they move up the school.

Schools, trusts, local authorities and central government can identify and nurture best practice.

And students will be able to understand clearly how their oracy abilities are developing and value them as skills to be honed rather than fixed attributes.

Some major challenges remain to the widespread use of comparative judgement. When done on a large scale, it can be resource intensive, taking up lots of teachers’ valuable time.

However, Voice 21 is committed to working to develop the methodology and technology that is both robust and time efficient.

We think a new oracy assessment, powered by comparative judgement, is worth exploring as a way of ensuring every child receives the high-quality oracy education to which they are entitled.

Amanda Moorghen is Head of Learning, Impact and Policy at Voice 21 UK.
She has an MSc in Social Policy and Social Research from UCL’s Institute of Education.

As part of AQi’s work, we are inviting people from a wide range of viewpoints to engage with us on a wide range of topics. We welcome alternative views to help stimulate discussion and ideas. The views of external contributors do not necessarily represent the views of AQi or AQA.

Assessment

What can this year’s GCSE entries tell us to look for in tomorrow’s results?

With GCSE Results published on 22nd August, Dr Chinwe Njoku looks into the underlying data on what subjects this year's cohort took and how this has changed from previous years.

Assessment

A-level maths students hit six figures – what’s behind its popularity?

On Thursday, more than 100,000 A-level maths students in England will find out their results – 11.4% more than last year. Why the upturn? Dr Chinwe Njoku, AQA Education Insights Lead and former maths teacher, was heartened by the news and keen to look at the story behind the data.

Education Policy

What comes after ‘urgent’ for the new Education Secretary?

After the burning issues are addressed, what should come next for the new Education Secretary?

Education Policy

Labour’s oracy plans: They need clear goals

Sir Keir Starmer has said he wants to boost students’ confidence by raising the importance of speaking skills – oracy. In this previously published blog, Reza Schwitzer, AQA’s director of external affairs, applauds the ambition but warns there needs to be clear goals

Education Policy

Through the looking glass: How polling the public can help policymakers learn about themselves

Public attitude data is key to effective policymaking. Proper polling can reveal what people think about existing policies and what they want for the future. But, if looked at from a different angle, it can also help policymakers question themselves and their assumptions about the public. In this blog, AQA’s Policy and Evidence Manager Adam Steedman-Thake, reveals the lessons he learned about himself while reading a recent public attitude survey.

Assessment

Assessing oracy: Is Comparative Judgement the answer?

Education

TV subtitles as an aid to literacy: What does the research say?

Jack Black is probably best known in educational circles for playing a renegade substitute teacher in School of Rock. But the Hollywood star has made a more conventional foray into education by backing the use of TV subtitles to improve child literacy. Stephen Fry and the World Literacy Foundation also want parents to use their TV remotes to get children reading. So, could this simple click of a button be a solution to boost pupils’ reading skills? AQA’s resident expert on language teaching, Dr Katy Finch, casts her eye over the research to see whether it stacks up.

Data Analysis

What is left behind now education’s Data Wave has receded?

Is data the solution to all education’s issues? About a decade ago the prevailing wisdom said it was. Advocates of this Data Wave argued that harvesting internal statistics would help schools solve issues such as teacher accountability and attainment gaps. As with all waves, after crashing onto the beach they recede, leaving space for another to roll in. In this blog, teacher, author and data analyst Richard Selfridge looks at the legacy of the Data Wave to see what schools can take from it.

International Approaches

Finland & PISA – A fall from grace but still a high performer?

Finland was once recognised as one of the most successful educational systems in the world. At the turn of the millennium, it topped the PISA rankings in reading, maths and science. But by 2012, decline set in. The last set of results showed performances in maths, reading and science were at an all-time low. In this blog Dr Jonathan Doherty of Leeds Trinity University outlines some reasons that may account for the slide.

Briefing

PISA, TIMSS and PIRLS: What actually are they and what do they tell us?

According to the latest PISA results, England’s science scores are still on a downward trajectory that started a decade ago. Yet TIMSS, another respected study, has science performances rising. Which of them is right? Is one more valid than the other? In this blog AQi examines three International Large-Scale Assessments and finds that, although they may look the same from a distance, get up close and you’ll find they are very different beasts.

Download a PDF version.

Download a copy of this content to your device as a PDF file. We generate PDF versions for the convenience of offline reading, but we recommend sharing this link if you'd like to send it to someone else.

Assessing oracy: Is Comparative Judgement the answer?

What can this year’s GCSE entries tell us to look for in tomorrow’s results?

A-level maths students hit six figures – what’s behind its popularity?

What comes after ‘urgent’ for the new Education Secretary?

Labour’s oracy plans: They need clear goals

Through the looking glass: How polling the public can help policymakers learn about themselves

Assessing oracy: Is Comparative Judgement the answer?

TV subtitles as an aid to literacy: What does the research say?

What is left behind now education’s Data Wave has receded?

Finland & PISA – A fall from grace but still a high performer?

PISA, TIMSS and PIRLS: What actually are they and what do they tell us?

Join the conversation on Twitter

Download a PDF version.

Sign up to our newsletter