the e-Assessment Association

Decoding the Future: A Guide to Choosing AI for Test Development

Decoding the Future: A Guide to Choosing AI for Test Development

A blog by Steve Shapiro, Prometric’s Senior Vice President of Artificial Intelligence (AI) in Assessments and CEO of Finetune

Artificial Intelligence (AI) is among the latest topics in almost every industry today. Specifically in assessment and education, AI has revolutionized traditional methodologies enabling greater efficiency and accuracy. From automating item generation and grading processes to facilitating adaptive testing and personalized learning experiences, AI is reshaping how tests are developed, administered, and evaluated.

The Types of AI in Testing and Assessment

AI for the Testing and Assessment Industry comes in many different forms. At one end of the spectrum, there are general-purpose or “generalist” AI tools. A generalist tool is designed to be as minimal and flexible as possible so that users can leverage it regardless of their particular purpose, domain, or requirements. Many generalist AI tools are readily available, like ChatGPT and ChatGPT plug-ins. This approach offers some clear benefits, including speed and cost savings. However, because it must appeal to the lowest common denominator of its user base, such a generalist approach can lead to a disconnect from subject matter expertise, a lack of understanding of specific exam requirements, and potential validity and security threats.

On the other end of the spectrum, special-purpose or “specialist” AI tools focus on mastering a particular domain and workflow. A specialist tool is designed to match the workflow of human domain experts by incorporating the contextual knowledge and practices of an expert or team of experts into the AI model. This not only provides a more accurate output but also enables additional review of the security and integrity of content. Let’s take a closer look at both approaches to determine which tool would work best for your program.

When deciding which AI model is best for your program, it’s essential to carefully consider all available options.

Generalist AI Tools: Easy to use but often lack critical security measures

The simplicity of generalist tools lends itself to wider availability, with millions of users interacting with the most popular options. Generalist AI tools also have the advantage of being highly versatile, with numerous applications served by the same interface. With so many daily users, these tools make it easy for application providers to scale up quickly, optimize costs, and allow developers to identify the most frequent problems. However, the emphasis that generalist AI tools place on easy collaboration, fast development, and bare minimum functionality often also means they lack vital security measures, are difficult to customize, add additional overhead to each individual user, and are unable to meet requirements around high accuracy and reliability. These factors are significant drawbacks for the Testing and Assessment Industry when trying to integrate AI technology.

In some cases, by allowing multiple individuals and organizations to use, review, and edit shared content within generalist tools, the material of one user can be exposed or influenced by the materials of other users, causing it to become less secure and less aligned with an exam’s intended subject matter and testing objectives. When members of the general public, such as test candidates, are able to interact with these same tools, that kind of leakage can compromise academic integrity, possibly even enabling test content to be available before test day.

Weaknesses of Generalist AI Tools

  • Unsecure
  • Poor Quality of Generated Output
  • Not Personalized
  • Uses Pre-Existing Model
  • AI Does NOT Permeate the Entire Process

Specialist AI Tools: Customizable and more secure

What makes specialist AI tools truly different from generalist AI tools is that the AI interaction can be personalized to the specific content and designed to support any process. This becomes especially important when building exam content, as the process can be time intensive and costly. However, combining specialist AI technology and human expertise can help power test development teams and allow them to scale more quickly, enhance their creativity, and expand the capabilities of testing programs.

Although generalist AI models can be used for many different tasks, they cannot master domains and workflows like specialist models. With specialist AI models, test developers can create more unique, targeted exam items and do so much faster than humans alone. The support of AI enables subject matter experts (SMEs) to spend less time fact-checking and proofreading and more time prioritizing critical organizational needs.

Specialist AI tools empower test developers to create more unique exam items much faster than humans alone.

 Creating more exam items helps to develop more test forms, allowing for more diverse tests and reduced item exposure. Now, protecting the new content created becomes a critical need for test development teams. Specialist models typically incorporate mandatory credential verification to ensure only those authorized to create, view, and edit content are accessing the item library. Unlike generalist models, the content is secure and not editable by anyone other than the designated SMEs, one of the key benefits of utilizing specialist models in test development.

How to Choose an AI Tool

In today’s dynamic testing and assessment landscape, choosing between generalist and specialist AI tools is a pivotal decision. It is crucial to carefully consider your organizational needs as well as the specific requirements of each test to select the most suitable model for you. If you’re looking to develop exam forms quickly and don’t mind using shared, public, or crowd-sourced content, you might be looking for a generalist AI tool. But if you place a premium on personalization and security and are looking for high-quality content that you can own and control as an organization, a specialist tool might be a better option. The choice between generalist and specialist AI tools in test development hinges on balancing ease of use with security and customization. By carefully weighing these considerations against organizational needs, testing, and assessment professionals can make informed decisions to drive the future of assessment development.

 


About the Author:

Pioneering the Charge of AI

Steve Shapiro is Prometric’s Senior Vice President of Artificial Intelligence (AI) in Assessments and CEO of Finetune. Finetune is a Prometric-owned technology company that specializes in AI-assisted assessment and learning technology across the credentialing, licensure, workforce readiness, and education sectors. Prometric acquired Finetune in 2022.

In his role, Mr. Shapiro develops highly innovative, patent-pending technology involving AI enabled products that can organically and creatively produce new content and intelligently classify and meta-tag content. Along with being a serial entrepreneur and three-time founder in the education technology and workforce training industries, he is also a member of Launchpad Venture Group and Cornell Red Bear Angel Group and a partner with LearnLaunch, a premier EdTech Accelerator program in Boston.

About Prometric:

Prometric is a leading provider of testing and assessment solutions, supporting over 25 million exam hours and serving more than seven million candidates every year. Using Finetune’s AI-powered development tools, robust assessment delivery capabilities, stringent security, and dedicated candidate support services, Prometric ensures the success of testing programs for leading organizations in over 180 countries.

Learn more: AI-Human Test Development (prometric.com)

Prometric

Share this: