Beyond Multiple Choice Session 4: Quizzes that tell you something: Guttman analysis in the age of GenAI

The Future-Ready Assessment Ecosystem: Institutions

Assessment organisations are navigating rapid technological change, evolving learner expectations and increasing demands for efficiency, quality and governance. Building a future-ready assessment ecosystem requires institutions to rethink how assessment is designed, delivered and managed.

This week focuses on the organisational challenges and opportunities shaping modern assessment. Sessions will explore AI in assessment operations, strategic planning, data-driven decision making and AI governance, providing practical insights into how institutions can build resilient, responsible and scalable assessment systems.Generative AI (GenAI) can produce the products of thinking with minimal cognitive effort from the student, and emerging evidence shows that students who use it as a substitute for thinking perform worse once access is removed (Bastani et al., 2025; Stromberg et al., 2026). Educators therefore need evidence of where students are in their learning, not just what they can produce (Masters, 2013).

About This Session
Quizzes are widely used but vulnerable to GenAI and need not be abandoned. Repositioned as diagnostic instruments that reveal readiness to learn rather than reward performance, they retain their value, provided the inferences drawn from them are valid (Dawson et al., 2024).
Guttman analysis (Guttman, 1950), adapted for classroom use by Griffin et al. (2017), orders response data by item difficulty and student performance. The resulting chart shows where learning is clustered and what students are ready to learn next, their zone of proximal development (Vygotsky, 1978). Uptake in higher education appears limited, constrained by workload (Datnow & Park, 2018) and by the interpretive demands of reasoning diagnostically from data (Mandinach & Gummer, 2016).
This session addresses both barriers by demonstrating two pedagogy-informed prompts. One generates diagnostic quiz items from a rubric or developmental assessment framework (Griffin et al., 2017) and item-writing guidelines (Haladyna et al., 2002; Scully, 2017). The other generates a Guttman chart from response data. Both are structured through context, epistemic grounding, procedure and reflexivity, on the premise that output quality depends on the frameworks that instruct it. GenAI undertakes the drafting and data reordering; the educator retains judgement over validity and teaching decisions. A worked example models interpretation of the chart, and the prompts and a sample dataset are provided. The approach is equally applicable in K–12 settings.