Bridging the gap between current item Adaptive assessments and our Multi-Stage Assessments

A blog by Gunter Maris: Senior Scientist – Statistical Analysis and Programming (TCS iON).

Education, the birthright of every child, is their primary means to touch success. Examination and education go hand-in-hand. Examinations judge a student’s progress at every stage, thus revealing any existing gaps.

However, data shows that even after 12 years of extensive schooling and exams, several students (data shows more than one in three) do not achieve the socially expected levels. The chief reason behind this failure is not always the learner; the school, our education system, also has a massive role to play in this.

Our current learning conception is almost like a straight line. Placing questions across the 12-year curriculum such that their position reflects their difficulty, allows for placing learners on this line based on how well they perform on assessments. The Rasch model statistically encodes this idea. Being positioned further along the learning curve than another student or the student himself at an earlier stage suggests that they are somewhat better at everything. Similarly, if a question is placed further along the line than another one, we conclude that this question is more difficult for everyone. Following this logic, a learner must get better at everything to progress. It is an admirable aspiration, but it is neither practical nor realistic for any learner.

Examinations should empower educators to predict gaps in learning. They should allow us to estimate the probability of success after compulsory education and to infer how it would vary if a specific (malleable) learning gap is closed. Authors Zwitser, Glaser and Maris (2017) provide a methodology that enables us to make this necessary prediction while guaranteeing comparability among learners and across time. The technique presented by these authors is sufficiently adaptable to detect attainment gaps at the individual level, giving educators knowledge of the areas where particular learners require intervention.

Let’s take a simple example. We’ve all heard of the high school student who claimed to know nothing about linear equations while excelling in trigonometry. When given a test consisting of both this week and another the next week, this student would probably perform poorly on linear equations exams while performing far better on his trigonometry exams. A Rasch model implies that the second test results are equally likely to be reversed, which is not the case. Since the teacher cannot detect the root problem, he cannot fix it.

How can we ensure exams help identify the root cause and fix it? By bringing in a personalised learner/learning support system that

• does justice to the diversity found among learners,
• assisting each one of them to succeed,
• considering scalability in terms of the number of users and questions, and
• reducing carbon footprints

Personalized assessments are praiseworthy if and when they encourage learning and achieving learning objectives. Adaptive testing is seen by many as the pinnacle of personalized assessments. Though you get a minor increase in precision from the same number of item responses, there is little opportunity to diagnose individual strengths and weaknesses and promote learning. In the ideal adaptive test, every item presented to a learner has a ½ probability of being answered correctly.

In such a test, if our learner, who does not understand linear equations, fails all three of them in an adaptive test, this outcome is not surprising (under the Rasch model). We wouldn’t conclude that this specific learner struggles with linear equations. Now, on responses to three linear equation problems, if we test the hypothesis that this particular learner does not understand linear equations, taking into account circumstantial evidence (how did this learner do on trigonometry), the same result would be surprising. It would point strongly to the correct conclusion. Once we detect the root cause, we can deal with it and ensure the student advances in the proper direction.

Our idea here is not to argue that the current assessments are inadequate but rather to show where we can improve by modifying the technical nature of the examinations. The aim is to empower our future generations with all the skills they need, and personalised examinations can be the perfect ally to a teacher.