Seven ChatGPT Takeaways from the 2023 Innovations in Testing Conference

A ChatGPT Impact Update by eAA Board member Tim Burnett

The 2023 Innovations in Testing Conference, hosted in Dallas and organised by the Association of Test Publishers, featured ChatGPT as a key focal point of discussion. With a strong emphasis on AI in assessment, I've distilled seven key insights from the event, highlighting the unprecedented disruption in the assessment industry. In this article, I'll delve into these insights and provide an actionable plan for the year ahead.

In recent years, many industry events have grappled with the lingering effects of COVID-19, causing a significant decrease in delegate attendance compared to pre-pandemic levels. The 2019 Innovations in Testing Conference boasted over 1,300 attendees from around the world. With this year's theme of "Better Together," the organisers were thrilled to see nearly 1,000 delegates in attendance, an improvement on 2022 and a clear demonstration that events are slowly recovering. While virtual events have undoubtedly played a role in sustaining idea generation, there's no denying the revitalising impact of in-person gatherings on the industry.

As I prepared for the conference, several questions about the future of assessment in light of ChatGPT's emergence weighed on my mind.

Read on to discover the answers I uncovered during the event.

1. Is the implementation of AI in assessment a novel concept for the industry?

This question invites an intriguing discussion, as numerous psychometric and technology experts have noted that the industry has been working on AI-related projects for years. Indeed, some organisations have already utilised AI for tasks such as essay scoring/marking and Automatic Item Generation (AIG). However, the recent shift lies in the increased accessibility of this technology, extending its reach beyond only the technical experts to creative professionals as well.

In the past, visualising the potential of AI proved challenging for me, as I lacked the opportunity to experiment with and explore its capabilities personally. ChatGPT, however, has transformed my perspective on the matter, offering an interactive experience and unveiling the true potential of AI in assessment

2. Is there a genuine interest in ChatGPT within the industry?

Judging by the packed attendance at sessions featuring ChatGPT in their titles, it's evident that people are paying attention. The conference offered diverse sessions, from introductory overviews to in-depth future explorations, showcasing a broad spectrum of opinions. Enthusiastic proponents stood in contrast to those who were more cautious or even sceptical about the technology.

Talking with delegates, I was surprised that some had not tried ChatGPT, and therefore, as the first major event to take place since version 3 was launched, the conference provided an excellent opportunity for people to level up their knowledge.

One aspect that seemed to be underrepresented was the importance of effective prompting. As a crucial element in unlocking ChatGPT's potential, I hope to see more emphasis on this topic at future conferences.

3. What is the most apparent use of ChatGPT in assessment?

While the ability to save time and enhance accuracy in marking has consistently been a primary motivation for employing AI in assessment, using it to make life-altering decisions and determine pass or fail outcomes has often been considered too risky for some or beyond reach for others. The application of AI in remote proctoring led several organisations to backtrack from granting it full control to assigning it a more supportive role.

For me, the most frequently debated topic surrounding ChatGPT's application in assessment was its potential to develop test items.

4. Will AI replace our jobs?

AI is poised to significantly disrupt the assessment industry, along with many others. As has been previously mentioned, AI itself won't take your job, but the person utilising it might. It is essential for every organisation to establish a working group focused on exploring AI applications within their operations and community. At the upcoming e-Assessment Conference and Awards, we'll delve deeper into this topic and offer practical guidance on how to leverage AI in your assessment programs.

While there's no immediate need to clear your desk, those with considerable time left in their careers, like me, should undoubtedly pay attention and adapt to the evolving landscape. As for my apprentice students, I've been keen to point out that AI will play a huge role in their future careers.

My concern is that many educators will shy away from discussing the topic, worrying that doing so will only invite cheating in exams. That would be a colossal error. We should teach the future workforce to understand and harness AI, and certainly not ignore it.

5. Will AI disrupt the role of Subject Matter Experts (SMEs)?

Building on my earlier points, AI is likely to impact SMEs' role in the assessment process. SMEs typically rely on industry knowledge sources and their own expertise to research and write the items.

Tools like ChatGPT can augment this process in two ways, as demonstrated in a presentation by ASC and Certiverse. The first scenario involves using AI to reduce the number of SMEs required for item development, thereby lowering costs. Since question creation is considered one of the most expensive aspects of test development, this cost reduction could be significant.

The second scenario involves employing AI alongside the same number of SMEs to create a substantially larger volume of test items. This increased item bank can enable smaller organisations to explore approaches such as Linear On the Fly Testing (LOFT) or Computer Adaptive Testing (CAT), both of which require larger item pools and which have traditionally been reserved for the bigger players. Additionally, a more extensive collection of high-quality test items can potentially reduce the impact of isolated incidents of item harvesting or cheating. One common concern with online proctoring is the increased likelihood of cheating; thus, having more test items can help address some of the risks associated with this mode of test delivery.

6. Who holds the copyright for AI-generated items?

While it's important to note that I am not a lawyer and cannot provide legal advice, the general consensus among conference delegates I spoke with was that the ownership of content created by ChatGPT might fall into a legal grey area. Very few people seemed confident to say that creating content in ChatGPT to be used in a test was without risk. It's worth noting that delegates may have been more inclined to play the safer card as an event held in the USA. However, as pointed out on several occasions, it's improbable that such content would be directly incorporated into an assessment without some review and revision. The prevailing opinion was that using ChatGPT to generate content is not significantly different from SMEs basing their items on information and research derived from copyrighted materials.

To adhere to best practices, it's essential to document all research and data sources and to clarify the process of item generation. Don't assume that content created by AI hasn't been authored by a person and thus could already be copyrighted. Personally, I believe organisations should be more transparent about their use of AI technology and the ethical considerations involved. As with matters of equal opportunities and pay transparency, addressing these issues proactively rather than concealing them in lengthy terms and conditions will ultimately benefit organisations. Building trust in your brand by leveraging honesty as a competitive advantage is a valuable approach in the long run.

7. What does the future hold for ChatGPT in assessment?

It is crucial to recognise the impact of AI on your assessment programme and take action if you haven't already. I recommend forming a working group within your organisation or collaborating with others in your community. Gather your ideas and bring them to the next significant event in the assessment industry so that we can explore and evolve together.

Numerous companies have already begun implementing proof-of-concept projects involving ChatGPT in assessment. It's now apparent that the assessment community is paying attention, and it's time to take action.

If you have a technology partner, enlist their assistance in exploring AI's potential. There's no doubt that going back in time is not an option. We are already moving at pace. I believe that fast-paced and agile organisations are going to benefit. Without action, we may see some of the bigger players struggle

Stay informed about AI developments and join your peers at the 2023 International e-Assessment Conference & Awards in London on June 6th and 7th. As the world's leading digital assessment conference, this event offers a unique opportunity to engage with the best technology in the assessment industry first-hand.

Surpass Assessment announces Co-CEO succession plan following Keith Myers’ retirement

Surpass Assessment, a leading international assessment technology provider and winner of the Queen’s Award for Enterprise in 2020, announce the retirement of its current CEO, Keith Myers.

With effect from 21 March 2023, Sonya Whitworth and Andy McAnulla will assume the role of CoCEOs, bringing their extensive experience and deep understanding of the company to the forefront of its leadership.

Sonya Whitworth, winner of the 2019 Stevie Award for Women in Business, has been with Surpass Assessment for over 20 years and has served on the board for more than sixteen years. In her new role as Co-CEO, she will continue to lead the operational side of the business, focusing on ensuring the consistent delivery and support of the company’s services.

Andy McAnulla, President of the US Surpass operation, has been with Surpass Assessment for over 22 years and was instrumental in establishing the company’s presence in North America. He has successfully grown the business, earning recognition as the ‘Fastest Growing Tech Company of the Year’ in 2020. Andy will also continue to lead the overall technology strategy of the business in his new role as Co-CEO.

Dr Bob Gomersall, owner of Surpass Assessment, expressed his enthusiasm for the new Co-CEO structure, stating, “Sonya and Andy are already very well known to our clients, and I am delighted that they have agreed to take on this joint role and therefore provide the continuity, in-depth knowledge, and strategic vision required both for our company and for the wider Surpass Community.”

The company is positioned for continued growth and success under the leadership of Sonya Whitworth and Andy McAnulla. Their shared commitment to innovation, customer satisfaction, and the advancement of the assessment industry ensures a bright future for Surpass Assessment and its clients worldwide.

For more information about Surpass Assessment, please visit surpass.com, contact [email protected] or call +44 1274 203250.

About Surpass Assessment (previously known as BTL Group Ltd.)

Surpass Assessment is an international business providing innovative and reliable assessment solutions to clients across the globe. With offices in the UK and the US, the company specialises in software development and services for the assessment industry, offering a comprehensive suite of tools and services for the creation, delivery, and analysis of high-quality assessments. Surpass Assessment is dedicated to empowering organisations to make informed decisions and drive success through its robust, secure, and user-friendly platform.

New NEBOSH Chief Executive outlines plans to continue mission for social good

Andy Shenstone stated his intention to continue the charity’s mission for social good as he took up the post of NEBOSH Chief Executive on Monday 20 March.

“NEBOSH’s social and charitable purpose is clear: to make the world a healthier and safer place through education. I’m dedicated to continuing the phenomenal work that the organisation does. Just this financial year, more than 100,000 people from over 170 countries have taken a NEBOSH qualification and I’m determined that our impact will continue to grow, reaching new regions, countries and industries with the learning and knowledge they need to keep people and communities safe.

“I want to nurture our existing culture of collaboration, partnering with organisations who share our vision. By working together, we will further increase the opportunities and quality of learning available in occupational health and safety.”

Andy has 23 years’ experience in education, training and consultancy. His early career was as a civil servant in the then HM Customs and Excise. In the following 16 years Andy established and grew an education consulting practice, culminating in a position as Group Sales Director for Capita PLC

In his most recent role, Andy was a lead executive involved in the creation and development of Advance HE – an internationally active, member-focused UK registered charity dedicated to the development and enhancement of higher education for the benefit of students, staff and society. He was responsible for executive leadership of domestic and international marketing, sales, business development and commercial solutions

Les Philpott, NEBOSH Chair, added: “I am delighted to welcome Andy Shenstone to NEBOSH. Andy has an excellent track-record in realising the potential of educational and charitable organisations. The NEBOSH Board is confident he will bring a strong sense of purpose and fresh direction to NEBOSH’s activities so that, through our globally recognised qualifications, we can make an even greater contribution to the management of health and safety in workplaces at home and internationally.”

Meet the eAA Board

eAA Members – do you have a question you would like to put to eAA board members, perhaps about their strategy for the Association, their plans for 2023 and beyond, or about how to be more involved? Alternatively, you might have ideas about new activities the eAA could consider, or ways in which we could improve the services we already provide to our members?

In April we will be recording a Q&A session with members of the board, which we will share with everyone via the newsletter and our website. If you have questions or comments that you’d like put to the board, please send these to [email protected] by no later than the 5pm GMT on Friday 24th March 2023.

The eAA at the Bett Show – 29th – 31st March 2023

Following on from the success of our involvement in last year’s show, the eAA is contributing to three exciting sessions.

eAA Vice Chair Paul Muir will be taking part in the panel discussion, Generative AI in Education, discussing ChatGPT and the impact generative AI is having in education. This takes place on 29th March 1400 in the Bett Futures Theatre.

The second session takes place in the Bett Academy Live theatre at 3pm on Thursday 30th March and will showcase two 2022 eAssessment Award winners – Steve Dew, winner of the Best Practitioner Award and Nuture, winner of the Best Formative Project Award.

The third session takes the form of a panel discussion focusing on how educators and leaders within schools, colleges, universities and work-based learning can learn from each other to deliver more impactful technology enabled assessments. This session takes place in the BETT Futures theatre at 11.45am on Friday 31st March.

The BETT Show is free to attend (registration is necessary) full details here

Welcome to the eAA’s new Association Sponsors for 2023

The eAA’s annual Association Sponsors enable us to deliver activities throughout the year including this newsletter and our brand new member podcasts.

During January and February we have welcomed a number of new Association Sponsors to whom we’d like to extend a particularly warm welcome. These include: Eintech, GamaLearn, Graide and Secure and Confidential Documents.

We also launched a brand new Assessment Organisation Network for organisations involved in the authoring, delivery and marking of assessments and we’d like to extend a warm welcome to its founding members: The Chartered Insurance Institute (CII), the Chartered Institute for Securities & Investment (CISI), Highfield, Kaplan Assessments, Lazer Learning Awards, London Institute for Banking & Finance (LIBF), NCFE, Nebosh, Pearson School Qualifications, the Scottish Qualifications Agency and VTCT.

Please do take a few minutes to learn a bit more about each of these organisations here

A huge thank you to all of our Association Sponsors and AO Network members for their continued support.