AI in Assessment: ChatGPT One year on, an interview with eAA Chief Executive Patrick Coates
On 30th November 2022, Open AI launched Chat GPT, a large language model-based chatbot. With a reported 100m users within weeks, others rushed to follow and launch their own versions. The potential of these advances in technology to change the way we work for ever, both positively and negatively, made it headline news.
As we come to the first anniversary of the launch of ChatGPT, Patrick Coates, Chief Executive Officer of the e-Assessment Association (eAA), reflects on the impact this has had on the assessment industry.
Patrick says, “I have been working with AI, for a number of years now. Firstly, with ACT and the work they were doing with essay marking and assessing 21st century skills. Then with Riiid Labs, who were using AI to accurately predict student exam scores in only a few questions. Whilst both of these in their own way were pretty impressive, they are nothing compared to what Generative AI, and specifically GPT3.5m, can do. It isn’t just about the tech, it is about how cheap it is to use. In my work with Certiverse, Generative AI is being used to create exam content, there isn’t even an extra cost, it is just absorbed into the platform fee. I understand how it does it, but it is still amazing just the same. By using prompts from a job task analysis, job role description and domains, the AI can write any question, including the keys, distractors and the rationale for these. This is on any certification exam subject area. I know, I have tried it, and a year on it almost seems passe. Ok, it’s not perfect and we still have an expectation that subject matter experts will check and review the content. Will the same be true in 12-month’s time? I don’t know for sure, but with new, bigger and better models being created by Open AI and with Google’s launch of Gemini in December to compete with Chat GPT, surely the capabilities of these tools will increase.
“I know that there are a lot more tools out there that are using Generative-AI. In my role as CEO of the eAA, and more so as a regular judge for our annual International e-Assessment Awards, I am constantly impressed by the way technology is being used to improve assessment and learning. With a year of ChatGPT, I suspect there will many new ways that AI has been used to improve assessment. And as you would expect from the eAA, we harnessed the power of AI at our recent Beyond Multiple Choice conference event, when the AI-Avatar, Josie, made the continuity announcements between speaker sessions.
The only caveat to all this is that AI is not perfect, it is just a tool. Like any tool, it is beneficial if used in the right way. There is an inherent bias in these Generative-AI models, which we can’t get away from. The data that these models are trained on is biased therefore the model is biased, which we need to allow for. Incidentally, people aren’t perfect either. My wife is always telling me I am not as funny as I think I am, and maybe ChatGPT can help there too…”