the e-Assessment Association

Reality vs. Myth: What you need to know now about Remote Invigilation

Reality vs. Myth: What you need to know now about Remote Invigilation

By Louella Morton, Executive Director, TestReach

I was recently at an assessment event and in one of the sessions the speaker authoritatively told the audience that you need a minimum internet connection speed of 5Mb per second to take a remotely invigilated exam. As the provider of a remote invigilation solution, we run thousands of exams delivered by remote invigilation (RI) and our standard recommendation for candidates is a minimum internet connection speed of 0.5Mb per second. That’s a difference of a factor of 10. We’ve even had candidates taking RI exams on a hot spot on their 3G mobile phone.

It is this kind of mis-information that leads to problems when organisations are trying to figure out what assessment technologies and systems can meet their needs. This is particularly true when you’re dealing with relatively new solutions such as remote invigilation, or remote proctoring, as our US friends call it.

Remote invigilation is the method by which formal exams are supervised over the web, so instead of candidates having to come to a test centre, they can take their exam on a computer located anywhere in the world and they will be remotely invigilated.

But what exactly does “remotely invigilated” mean?

The word “invigilated” can mean different things to different people and one of the biggest issues in the industry at this point is that the terms “remote invigilation” and “remote proctoring” are being used to cover a wide and varied mix of technologies, approaches and solutions. It’s a bit like using the term accounting software to cover anything from a spreadsheet through to an enterprise finance solution – it is quite unclear and leads to confusion.

In addition, as remote invigilation is a hot topic, it’s the main subject in many conferences and blog posts. Often it is positioned almost as a straightforward commodity: simply select your remote invigilation solution, it doesn’t really matter which one, easily bolt it on to an assessment engine and hey presto, in no time you’ll be securely delivering exams anywhere in the world with complete integrity.

Contrary to what the marketeers would have you believe, this is not quite in line with reality.

Navigating the marketing myths and determining what remote invigilation solution will best suit your needs is by no means straightforward, and even as a person working in the industry for many years, I sometimes find it quite tricky to understand available options.

Unfortunately, as is often the case with sales people, if you need a car but happen to chat to a bicycle dealer, you will often be left with the impression that a bicycle is every bit as good as a car. It’s not until you actually use the bicycle that you realise there is a substantial difference. At this point it’s important to have full disclosure, so I’d like to confess that I am actually in sales. However, if someone doesn’t want my car I would much rather they bought another vendor’s car than head off on a bicycle, which I know for a fact will not meet their needs.

The other important factor to bear in mind is that some remote invigilation solutions are just not very good. Reasons range from use of deprecated technologies, inability to cope with firewalls or the user interface is not intuitive and so on. So even though you’ve bought a car, you might not actually make it home from the garage.

I met someone at a trade fair recently who told me that remote invigilation doesn’t work, as she knew an organisation who did a trial and it was a disaster. This is a good example of someone buying a second-hand banger who now has the perception that all cars are useless. This perception couldn’t be further from the truth.

In a recent survey of over 1,000 RI candidates that one of our customers ran, over 95% said that the experience was “above average” or “excellent”.

It is frustrating therefore that some people have a negative perception of remote invigilation as a viable solution. If they’ve heard about a bad experience with one system or vendor, or to use the example from earlier, if they’ve mistakenly bought a bicycle when they were expecting a car, then there is a tendency to tar everyone with the same brush.

The technology that enables remote invigilation is complex, particularly the communications such as video, audio and screen-sharing, so not surprisingly some vendors run into problems. You also have to consider that candidates could be connecting from a myriad of computer types from PCs to Macs, that range in age, capacity, processing power and configuration, to name but a few variables, plus there are many different levels of firewalls and connection speeds. It is complicated, but although there are many vendors who get it wrong, there are also vendors who absolutely get it right and can deliver a high-quality solution that brings very significant benefits to organisations.

Looking at the different types of remote invigilation

Looking at the different types of remote invigilation, the most common option is for real people to supervise your exams. This means that your candidates will connect to an actual person, who will go through steps to authenticate each candidate (ensure the correct person is sitting the exam), secure the environment (e.g. asking them to pan their webcam around the room), and monitor them for the duration of the exam. The big benefit of having real people is that the candidate has someone to talk to who can answer their questions – this cannot be underestimated. When someone is taking an exam, they tend to be nervous and stressed, so it really helps to have an invigilator at the other end of the line who is helpful, reassuring and has a calming influence.

Another benefit of real people is that they can use their judgement to intervene if someone is behaving suspiciously or they hear a noise. Much as there are many cutting-edge technologies currently available, typically when it comes to making a judgement call as to whether or not someone is cheating, human beings are hard to beat, although technologies are continuing to evolve so you should keep an eye on this space.

Often technologies will be used in conjunction with real people to enhance the level of remote invigilation provided. These can vary from facial recognition and biometric scanning (e.g. key-stroke pattern recognition, where the way you type is used like a fingerprint to identify you) through to movement tracking and scanning behavioural patterns during the exam.

At the other end of the scale are fully automated solutions and for the most part these are “record and review”, which means that candidates are recorded during their exam, then the recordings are reviewed, either by people or technology, after the event. Although typically cheaper than having people involved in the process, the upshot is that although suspicious behaviour will be identified, it is typically well after the exam has finished and there is no intervention in real-time. This might be okay for certain styles of lower-stake exams, for example, “answer as many questions as you can in one hour”. However, for higher-stakes exams, where there are a lot of security considerations around questions or case studies, it’s not really a feasible option.

The other factor to consider when evaluating remote invigilation systems is how the remote invigilation is actually provided. By this I mean that often remote invigilation is offered as a “bolt-on”, so a remote invigilation system from one vendor is integrated with an assessment solution from another vendor. Although this is typically positioned as being “seamless”, issues may arise around the integration and having to correlate data from two systems, but obviously if you already have an online assessment solution it can be an attractive option. Alternatively, some assessment systems like TestReach have built in remote invigilation, where you can just turn it on for any exam.

When it comes to determining what kind of remote invigilation will suit your own needs, I have two pieces of advice:

  1. Talk to organisations who have already rolled out these different types of solutions. Ask vendors for references or case studies and find organisations who run similar kinds of examinations to your own. You might also consider your regulator, are any other organisations they currently regulate running remotely invigilated exams? Over the past few years I’ve seen many regulators start to embrace remote invigilation as an approved method to securely deliver exams. At TestReach we now have customers running certification exams via remote invigilation who are regulated by organisations such as OFQUAL, the SQA and the CSCS. By and large regulators welcome positive change, so long as there are clear and documented controls, processes and procedures in place to preserve the integrity of the exam.
  2. Try it out. Nothing replaces actually using a solution first-hand within your own environment. For remote invigilation systems, the most important factor is ensuring you have candidates who are connecting from a variety of locations – home, work, etc., so that you see how the solution performs under various conditions. Even a small trial or pilot can go a long way to showing you whether or not the software works as it should, what the experience of candidates is actually like and whether or not it can meet your needs. It also allows you to see what the vendor is like to work with. Remote invigilation requires a significant degree of change management, so you need to ensure that whoever you are partnering with, can help and support you to effect that change.

The bottom line is that remote invigilation solutions are not all the same and some systems are much more suited to certain applications than others. At least with the above advice, if you need a car you are a lot less likely to walk away with a bicycle.