Continuing item banking activities remotely? Here’s our top tips from Surpass for getting the most from your Subject Matter Experts

Many organisations have had to transition traditionally in-person test development activities to virtual sessions over the last year, and feedback from the community tells us that going forward, we’re likely to see a hybrid approach combining in-person with virtual meetings.

Remote working has many benefits, such as reduced costs and less need to travel, and fortunately, by leveraging technology, we can ensure that security and integrity of the test remains key during activities such as item writing, review, and standard setting. In this article, we’ll share some considerations for running virtual test development activities, as well as sharing some best practice for transitioning workshops and training sessions from face-to-face to virtual meetings.

Thanks to Amanda Dainis, CEO and Lead Psychometrician of Dainis & Co for providing some of these considerations as part of a ‘peas in a pod’ session at the recent ATP Innovations in Testing conference.

Timing

Remote test development activities can be conducted either synchronously, or asynchronously. The nature of the activity will probably determine which way will work best, or it’s likely that you’ll need to adopt a combination of the two.

A synchronous process, such as a workshop, would be where the entire panel convenes at the same time for a virtual meeting. For a meeting like this, it’s important to consider the length of the meeting, and whether the panel is likely to remain engaged and produce their best work, during what could be a long time in front of a screen.

An asynchronous process may see the organiser give assignments to SME’s, to work on a shared document in their own time.

A hybrid option could be the way to go, bringing all SMEs together for introductory training and assignments, allowing them to then work independently with regular progress check-in calls.

Variables affecting this decision: 

• Time required from each SME
• Discussion level expected / required
• Motivation of SMEs to work independently
• Effects of groupthink (the practice of making decisions as a group where the desire for conformity or presence of dominant personalities could negatively affect the decision-making process).
• Project timeline: Synchronous may get the overall task done a lot sooner than an      asynchronous process.

Maintaining item security

No matter what the method of working, item security should always be at the forefront. One of the limitations of distributing a shared document, or printed document, for reviewers to work on, is that you risk exposing the entire item bank to all members of the panel. A compromised item bank could be costly and resource intensive to rectify.

Platforms such as Surpass allow a subset of items to be assigned to reviewers, meaning the full item bank is never exposed and you have control over which items they see. Additionally, all activity on an item is logged with an audit trail, so you know exactly what changes have been made, and by who. Permissions also control who can log into the system, and the functionality they have access to.

In addition to technology and processes, it’s important to reiterate the legal obligation of team members not to distribute items, and the terms and conditions that they have agreed to.

If required, additional security measures could be implemented for item review or standard setting activities by using a remote proctoring service, or monitoring via Zoom or Teams.

Technology Savviness

Virtual platforms have increased in use over the past year, and so has user comfort. However, considering the comfort level of the SME panel with completing independent tasks in an unfamiliar platform is essential when planning. Simplified tools such as surveys, Excel workbooks, or the task management features in Surpass, can help this process be simple and painless for the SMEs. Zoom still has a habit of tripping people up when screen sharing, but look out for a new feature coming to PowerPoint that means you can share a presentation in Teams directly from PowerPoint, no screen sharing required.

Availability of resources

For in-person item writing and review activities, often, a number of resources may be provided for the panel to refer to when creating and editing items. Shipping resources for SMEs to work remotely could be costly and inconvenient, so consider creating electronic resources such as PDFs. 

In Surpass authoring tasks, where SMEs are assigned a number of items to write, up to 30 documents can be attached to the task, conveniently keeping resources all in one place.

Plan ahead

Whereas with an in-person meeting, you’ve got the SME’s full attention for the duration of the session, when planning a virtual session you may find you’re competing with the distraction of other work or home commitments. Try and get time booked in with your SMEs as far in advance as possible, and set clear expectations of how the virtual sessions will run and their role within those sessions.

If you’re taking a hybrid approach, it can still be useful to have a virtual kick-off meeting to provide introductions, training and set expectations, so that when the group get together in-person the focus is purely on the task at hand.

5 top tips from the Surpass Training Team

The Surpass Training team have transitioned their training and workshop programme to virtual delivery over the last year. Ashleigh Whittle, Surpass Training Manager provided her top tips for moving from in-person to remote delivery based on the team’s recent experiences.

• Engaging a room of people and engaging a virtual room are two very different skillsets. Don’t try and run virtual events exactly how you would face-to-face events – they won’t work!
• You need to master your technology. Just as you would rehearse a presentation over and over, you need to rehearse the technology until you are completely confident with it. Someone will inevitably leave themselves on mute, so you need to know how to help your attendees.
• Short, bitesize sessions are the best. Keep the session focussed on one objective, and if you have multiple objectives, allow for breaks so attendees can rest from their screens.
• For workshops/training sessions, it’s important to keep attendees engaged and interested. Include some type of interaction every 3-4 minutes, from answering a question to typing a chat, or even using the interactive features of whatever technology you are using (which is another reason you need to master it!). Look out for a new feature in PowerPoint called ‘Rehearse with Coach’ on the Slide Show tab, it helpfully analysers your pitch and tone to help you improve your presentation in private.
• Seek advice from the experts! There are lots of great courses out there to help you reconsider how you deliver a virtual workshop.

Hear from Ashleigh in this short interview: https://vimeo.com/548747513/408d0826fd 

Discover how Surpass can facilitate test development activities whether in-person or remotely at  www.surpass.com