Developing and delivering e-assessments: the importance of communication

by Helen Claydon

For much of the past 10 years I have been involved in e-assessment content development across a variety of organisations and assessments. In all instances, my experience of e-assessment has involved the assessment ‘owner’ outsourcing various aspects to third parties.

Handing over responsibility for aspects of your assessment to a third party can be a nerve wracking experience. Establishing a collaborative working relationship is key to successful delivery. In this blog I share some thoughts based upon past experience and hope that they strike a chord with your own experiences.

Aims

We clearly have the shared aim of wanting the assessment to succeed, but to what extent do we agree on how to achieve this? Quite often compromises will need to be made along the journey. Main areas for discord and compromise relate to quality, timescales and costs. In my view quality is very important, but it is important to remember that quality and accuracy are not the same thing. A candidate will probably not be too bothered if a question doesn’t look ‘pretty’.

However, it is a different story if there is an error in a question, which results in the candidate losing a mark and potentially failing the exam. For high stakes examinations, it is no exaggeration to say that one small error in a line of code has the potential to be life altering for a candidate. It is worth agreeing up front who has responsibility for ensuring accuracy and how and when this will be achieved. For example, will it be through:

• up front quality assurance testing prior to launch of the exam, or
• remedial work post-launch to fix issues identified by candidates when they were taking their actual exam?

What are the consequences if the assessments are not ready by the agreed launch date? If the launch of a new high-stakes examination needs to be delayed, particularly if it was due to be administered on a fixed day, the consequences can be significant. All parties need to be clear about the importance of the timeline, with early indication provided of any potential slippages to the critical path. In the event of a slippage it is important that all parties work together in agreeing realistic mitigations. Transparency is key; covering up slippages or downplaying risks has the potential to create bigger problems later on which are more difficult to resolve. However, the launch date isn’t always critical, sometimes it is okay for dates to slip to avoid compromising quality.

The test owner will often have a finite budget, so additional spend to address unexpected issues will not always be a realistic option. As a test owner, we never want to compromise on quality or timescale. However, there will be instances when we have no choice; it is worth deciding up front the location of the line beyond which we are not prepared to cross.

Successful collaboration

Collaborative scoping of requirements at the start of a project is invaluable. Making time to talk before commencement of any system development will help to develop a shared and consistent understanding of high level requirements and reduce the likelihood of time and money being invested in developing a solution that doesn’t meet the test owner’s expectations.

At the start of a project, a common pitfall is failing to realise that you do not have a shared vocabulary. The resulting repercussions for a project can be significant, with solutions developed by the e-assessment provider not meeting the expectations of the assessment owner. If you perceive this to be a problem, it can help to work together to develop a shared glossary of terms and definitions. An alternative is to use definitions from an externally published source such as the ‘e-Assessment Glossary’, commissioned by JISC and QCA and published in 2006.

In conclusion …

To use the old BT adage, ‘It’s good to talk.’ The importance of investing time in up-front collaborative discussion should not be underestimated. It can help to increase the likelihood that our e-assessments succeed, with reduced need for compromise along the way.