The e-Assesment Association Company Logo
the e-Assessment Association

Seven ChatGPT Takeaways from the 2023 Innovations in Testing Conference

Seven ChatGPT Takeaways from the 2023 Innovations in Testing Conference

A ChatGPT Impact Update by eAA Board member Tim Burnett

The 2023 Innovations in Testing Conference, hosted in Dallas and organised by the Association of Test Publishers, featured ChatGPT as a key focal point of discussion. With a strong emphasis on AI in assessment, I've distilled seven key insights from the event, highlighting the unprecedented disruption in the assessment industry. In this article, I'll delve into these insights and provide an actionable plan for the year ahead.

In recent years, many industry events have grappled with the lingering effects of COVID-19, causing a significant decrease in delegate attendance compared to pre-pandemic levels. The 2019 Innovations in Testing Conference boasted over 1,300 attendees from around the world. With this year's theme of "Better Together," the organisers were thrilled to see nearly 1,000 delegates in attendance, an improvement on 2022 and a clear demonstration that events are slowly recovering. While virtual events have undoubtedly played a role in sustaining idea generation, there's no denying the revitalising impact of in-person gatherings on the industry.

As I prepared for the conference, several questions about the future of assessment in light of ChatGPT's emergence weighed on my mind.

Read on to discover the answers I uncovered during the event.

1. Is the implementation of AI in assessment a novel concept for the industry?

This question invites an intriguing discussion, as numerous psychometric and technology experts have noted that the industry has been working on AI-related projects for years. Indeed, some organisations have already utilised AI for tasks such as essay scoring/marking and Automatic Item Generation (AIG). However, the recent shift lies in the increased accessibility of this technology, extending its reach beyond only the technical experts to creative professionals as well.

In the past, visualising the potential of AI proved challenging for me, as I lacked the opportunity to experiment with and explore its capabilities personally. ChatGPT, however, has transformed my perspective on the matter, offering an interactive experience and unveiling the true potential of AI in assessment

2. Is there a genuine interest in ChatGPT within the industry?

 

Judging by the packed attendance at sessions featuring ChatGPT in their titles, it's evident that people are paying attention. The conference offered diverse sessions, from introductory overviews to in-depth future explorations, showcasing a broad spectrum of opinions. Enthusiastic proponents stood in contrast to those who were more cautious or even sceptical about the technology.

Talking with delegates, I was surprised that some had not tried ChatGPT, and therefore, as the first major event to take place since version 3 was launched, the conference provided an excellent opportunity for people to level up their knowledge.

One aspect that seemed to be underrepresented was the importance of effective prompting. As a crucial element in unlocking ChatGPT's potential, I hope to see more emphasis on this topic at future conferences.

3.  What is the most apparent use of ChatGPT in assessment?

While the ability to save time and enhance accuracy in marking has consistently been a primary motivation for employing AI in assessment, using it to make life-altering decisions and determine pass or fail outcomes has often been considered too risky for some or beyond reach for others. The application of AI in remote proctoring led several organisations to backtrack from granting it full control to assigning it a more supportive role.

For me, the most frequently debated topic surrounding ChatGPT's application in assessment was its potential to develop test items.

4. Will AI replace our jobs?

AI is poised to significantly disrupt the assessment industry, along with many others. As has been previously mentioned, AI itself won't take your job, but the person utilising it might. It is essential for every organisation to establish a working group focused on exploring AI applications within their operations and community. At the upcoming e-Assessment Conference and Awards, we'll delve deeper into this topic and offer practical guidance on how to leverage AI in your assessment programs.

While there's no immediate need to clear your desk, those with considerable time left in their careers, like me, should undoubtedly pay attention and adapt to the evolving landscape. As for my apprentice students, I've been keen to point out that AI will play a huge role in their future careers.

My concern is that many educators will shy away from discussing the topic, worrying that doing so will only invite cheating in exams. That would be a colossal error. We should teach the future workforce to understand and harness AI, and certainly not ignore it.

5. Will AI disrupt the role of Subject Matter Experts (SMEs)?

Building on my earlier points, AI is likely to impact SMEs' role in the assessment process. SMEs typically rely on industry knowledge sources and their own expertise to research and write the items.

Tools like ChatGPT can augment this process in two ways, as demonstrated in a presentation by ASC and Certiverse. The first scenario involves using AI to reduce the number of SMEs required for item development, thereby lowering costs. Since question creation is considered one of the most expensive aspects of test development, this cost reduction could be significant.

The second scenario involves employing AI alongside the same number of SMEs to create a substantially larger volume of test items. This increased item bank can enable smaller organisations to explore approaches such as Linear On the Fly Testing (LOFT) or Computer Adaptive Testing (CAT), both of which require larger item pools and which have traditionally been reserved for the bigger players. Additionally, a more extensive collection of high-quality test items can potentially reduce the impact of isolated incidents of item harvesting or cheating. One common concern with online proctoring is the increased likelihood of cheating; thus, having more test items can help address some of the risks associated with this mode of test delivery.

6. Who holds the copyright for AI-generated items?

While it's important to note that I am not a lawyer and cannot provide legal advice, the general consensus among conference delegates I spoke with was that the ownership of content created by ChatGPT might fall into a legal grey area. Very few people seemed confident to say that creating content in ChatGPT to be used in a test was without risk. It's worth noting that delegates may have been more inclined to play the safer card as an event held in the USA. However, as pointed out on several occasions, it's improbable that such content would be directly incorporated into an assessment without some review and revision. The prevailing opinion was that using ChatGPT to generate content is not significantly different from SMEs basing their items on information and research derived from copyrighted materials.

To adhere to best practices, it's essential to document all research and data sources and to clarify the process of item generation. Don't assume that content created by AI hasn't been authored by a person and thus could already be copyrighted. Personally, I believe organisations should be more transparent about their use of AI technology and the ethical considerations involved. As with matters of equal opportunities and pay transparency, addressing these issues proactively rather than concealing them in lengthy terms and conditions will ultimately benefit organisations. Building trust in your brand by leveraging honesty as a competitive advantage is a valuable approach in the long run.

 

7. What does the future hold for ChatGPT in assessment?

 

It is crucial to recognise the impact of AI on your assessment programme and take action if you haven't already. I recommend forming a working group within your organisation or collaborating with others in your community. Gather your ideas and bring them to the next significant event in the assessment industry so that we can explore and evolve together.

Numerous companies have already begun implementing proof-of-concept projects involving ChatGPT in assessment. It's now apparent that the assessment community is paying attention, and it's time to take action.

If you have a technology partner, enlist their assistance in exploring AI's potential. There's no doubt that going back in time is not an option. We are already moving at pace. I believe that fast-paced and agile organisations are going to benefit. Without action, we may see some of the bigger players struggle

Stay informed about AI developments and join your peers at the 2023 International e-Assessment Conference & Awards in London on June 6th and 7th. As the world's leading digital assessment conference, this event offers a unique opportunity to engage with the best technology in the assessment industry first-hand.

News categories

Upcoming Events

Quicklinks