AI Special Interest Group

Dr Liberty Munson - Microsoft Corporation ChatGPT has captured the imagination of the world, including the credentialing industry. Many of us have already started to experiment with how we can leverage large language models (LLMs) like this to drive efficiencies in our design, development and delivery processes, but are we doing it with the care and diligence that we should be?

This session will examine:

• What have we learned so far?

• What other areas of research are needed?

• What does the future look like?

Integrating ethical AI in high-stakes educational assessment with Dr Cesare Aloisi Dr Cesare Aloisi - AQA Education This presentation explores how to introduce AI technologies progressively and responsibly in high-stakes assessment in a way that could preserve and improve public trust. Artificial intelligence has limitations that could negatively affect the validity of high-stakes assessment such as national exams. Solutions are being developed which will make AI systems more accurate, transparent and trustworthy; however, this may not be sufficient to command public trust. The assessment industry will need to demonstrate it can deploy trustworthy AI responsibly. This is what Shen (2022) called the “human – AI trust’ vis-à-vis the human – AI – human trust”: stakeholders may trust that the AI systems deployed by high-stakes assessment providers work as intended, yet mistrust the assessment providers themselves. A paradigm shift from ‘people assessing people’ to a ‘people and machines assessing people’ will be required to command public trust. Three factors could help high-stakes assessment providers transition to the new model: 1. Developing an assessment-focused ethical AI framework using ‘validity’ as a foundational concept. This will serve as the basis for policy and regulation update. 2. Working with the AI industry to improve the assessment literacy of data scientists and software developers. This will facilitate cultural integration. 3. Identifying and responding to learner and stakeholder needs, to ensure the meaning and value of assessment results (such as exam grades) are preserved.

AO Network Meeting Recording AI in Assessment 06 03 23