Reality vs. Myth: What you need to know now about Remote Invigilation

By Louella Morton, Executive Director, TestReach

I was recently at an assessment event and in one of the sessions the speaker authoritatively told the audience that you need a minimum internet connection speed of 5Mb per second to take a remotely invigilated exam. As the provider of a remote invigilation solution, we run thousands of exams delivered by remote invigilation (RI) and our standard recommendation for candidates is a minimum internet connection speed of 0.5Mb per second. That’s a difference of a factor of 10. We’ve even had candidates taking RI exams on a hot spot on their 3G mobile phone.

It is this kind of mis-information that leads to problems when organisations are trying to figure out what assessment technologies and systems can meet their needs. This is particularly true when you’re dealing with relatively new solutions such as remote invigilation, or remote proctoring, as our US friends call it.

Remote invigilation is the method by which formal exams are supervised over the web, so instead of candidates having to come to a test centre, they can take their exam on a computer located anywhere in the world and they will be remotely invigilated.

But what exactly does “remotely invigilated” mean?

The word “invigilated” can mean different things to different people and one of the biggest issues in the industry at this point is that the terms “remote invigilation” and “remote proctoring” are being used to cover a wide and varied mix of technologies, approaches and solutions. It’s a bit like using the term accounting software to cover anything from a spreadsheet through to an enterprise finance solution – it is quite unclear and leads to confusion.

In addition, as remote invigilation is a hot topic, it’s the main subject in many conferences and blog posts. Often it is positioned almost as a straightforward commodity: simply select your remote invigilation solution, it doesn’t really matter which one, easily bolt it on to an assessment engine and hey presto, in no time you’ll be securely delivering exams anywhere in the world with complete integrity.

Contrary to what the marketeers would have you believe, this is not quite in line with reality.

Navigating the marketing myths and determining what remote invigilation solution will best suit your needs is by no means straightforward, and even as a person working in the industry for many years, I sometimes find it quite tricky to understand available options.

Unfortunately, as is often the case with sales people, if you need a car but happen to chat to a bicycle dealer, you will often be left with the impression that a bicycle is every bit as good as a car. It’s not until you actually use the bicycle that you realise there is a substantial difference. At this point it’s important to have full disclosure, so I’d like to confess that I am actually in sales. However, if someone doesn’t want my car I would much rather they bought another vendor’s car than head off on a bicycle, which I know for a fact will not meet their needs.

The other important factor to bear in mind is that some remote invigilation solutions are just not very good. Reasons range from use of deprecated technologies, inability to cope with firewalls or the user interface is not intuitive and so on. So even though you’ve bought a car, you might not actually make it home from the garage.

I met someone at a trade fair recently who told me that remote invigilation doesn’t work, as she knew an organisation who did a trial and it was a disaster. This is a good example of someone buying a second-hand banger who now has the perception that all cars are useless. This perception couldn’t be further from the truth.

In a recent survey of over 1,000 RI candidates that one of our customers ran, over 95% said that the experience was “above average” or “excellent”.

It is frustrating therefore that some people have a negative perception of remote invigilation as a viable solution. If they’ve heard about a bad experience with one system or vendor, or to use the example from earlier, if they’ve mistakenly bought a bicycle when they were expecting a car, then there is a tendency to tar everyone with the same brush.

The technology that enables remote invigilation is complex, particularly the communications such as video, audio and screen-sharing, so not surprisingly some vendors run into problems. You also have to consider that candidates could be connecting from a myriad of computer types from PCs to Macs, that range in age, capacity, processing power and configuration, to name but a few variables, plus there are many different levels of firewalls and connection speeds. It is complicated, but although there are many vendors who get it wrong, there are also vendors who absolutely get it right and can deliver a high-quality solution that brings very significant benefits to organisations.

Looking at the different types of remote invigilation

Looking at the different types of remote invigilation, the most common option is for real people to supervise your exams. This means that your candidates will connect to an actual person, who will go through steps to authenticate each candidate (ensure the correct person is sitting the exam), secure the environment (e.g. asking them to pan their webcam around the room), and monitor them for the duration of the exam. The big benefit of having real people is that the candidate has someone to talk to who can answer their questions – this cannot be underestimated. When someone is taking an exam, they tend to be nervous and stressed, so it really helps to have an invigilator at the other end of the line who is helpful, reassuring and has a calming influence.

Another benefit of real people is that they can use their judgement to intervene if someone is behaving suspiciously or they hear a noise. Much as there are many cutting-edge technologies currently available, typically when it comes to making a judgement call as to whether or not someone is cheating, human beings are hard to beat, although technologies are continuing to evolve so you should keep an eye on this space.

Often technologies will be used in conjunction with real people to enhance the level of remote invigilation provided. These can vary from facial recognition and biometric scanning (e.g. key-stroke pattern recognition, where the way you type is used like a fingerprint to identify you) through to movement tracking and scanning behavioural patterns during the exam.

At the other end of the scale are fully automated solutions and for the most part these are “record and review”, which means that candidates are recorded during their exam, then the recordings are reviewed, either by people or technology, after the event. Although typically cheaper than having people involved in the process, the upshot is that although suspicious behaviour will be identified, it is typically well after the exam has finished and there is no intervention in real-time. This might be okay for certain styles of lower-stake exams, for example, “answer as many questions as you can in one hour”. However, for higher-stakes exams, where there are a lot of security considerations around questions or case studies, it’s not really a feasible option.

The other factor to consider when evaluating remote invigilation systems is how the remote invigilation is actually provided. By this I mean that often remote invigilation is offered as a “bolt-on”, so a remote invigilation system from one vendor is integrated with an assessment solution from another vendor. Although this is typically positioned as being “seamless”, issues may arise around the integration and having to correlate data from two systems, but obviously if you already have an online assessment solution it can be an attractive option. Alternatively, some assessment systems like TestReach have built in remote invigilation, where you can just turn it on for any exam.

When it comes to determining what kind of remote invigilation will suit your own needs, I have two pieces of advice:

1. Talk to organisations who have already rolled out these different types of solutions. Ask vendors for references or case studies and find organisations who run similar kinds of examinations to your own. You might also consider your regulator, are any other organisations they currently regulate running remotely invigilated exams? Over the past few years I’ve seen many regulators start to embrace remote invigilation as an approved method to securely deliver exams. At TestReach we now have customers running certification exams via remote invigilation who are regulated by organisations such as OFQUAL, the SQA and the CSCS. By and large regulators welcome positive change, so long as there are clear and documented controls, processes and procedures in place to preserve the integrity of the exam.
2. Try it out. Nothing replaces actually using a solution first-hand within your own environment. For remote invigilation systems, the most important factor is ensuring you have candidates who are connecting from a variety of locations – home, work, etc., so that you see how the solution performs under various conditions. Even a small trial or pilot can go a long way to showing you whether or not the software works as it should, what the experience of candidates is actually like and whether or not it can meet your needs. It also allows you to see what the vendor is like to work with. Remote invigilation requires a significant degree of change management, so you need to ensure that whoever you are partnering with, can help and support you to effect that change.

The bottom line is that remote invigilation solutions are not all the same and some systems are much more suited to certain applications than others. At least with the above advice, if you need a car you are a lot less likely to walk away with a bicycle.

To read more from TestReach on the subject of remote invigilation, visit https://www.testreach.com/remote-invigilation-online-proctoring.html

New blog: Remote invigilation / Remote proctoring

Patrick Coates, Director of International Skills UK and Board Member of the eAA writes about remote invigilation / remote proctoring and invites you to get in touch if you are involved in using or trialling remote proctoring, particularly with a view to share the challenges, questions and answers across the e-assessment community.

Remote invigilation or proctoring₁ has been around for a while and I have seen evidence of its use in the mainstream for exams in the professional and academic world in the UK, US and Australia for a number of years. However, it has not taken off as quickly as would have been expected or hoped. It is fair to say that there have been a number of technology innovations over the years in e-assessment, from adaptive testing to on-demand testing or machine marking of free text responses, which have not quite made it in the mainstream as yet. Remote invigilation is one of these.  Why is that?

On the one hand, there could be perceived or actual deficiencies in using the technology for a particular assessment, or as is more likely, a lack of understanding of what can be achieved or a general willingness to commit to something new. Having seen it in action, I think the big challenge with remote invigilation is a lack of understanding of what is involved and perceived issues when compared to the traditional test centre model. It is this lack of understanding which leads to a slow uptake of the ‘new’ technology.

 https://en.wikipedia.org/wiki/Technology_adoption_life_cycle

Whilst this is not an academic piece, if we look at the technology adoption model (with the caveat that the early part of the bell curve should be more of an S-shape in my opinion, but is too in depth for discussion here) the real challenge is that on the X-axis there isn’t any indication of time.

So we don’t know whether the early majority are going to be using the technology in a matter of a few years, or decades.

A key thing that will accelerate that adoption is communication. If we can help explain what remote invigilation is, what the opportunity is, then this will hopefully expedite adoption. Hence this blog and if there’s interest perhaps a more in depth piece of research.

So probably, best to start from basics. What is remote invigilation/proctoring?

The truth is it can take on a number of forms. Just as there is still confusion over what is an online test versus and on-screen test, there are different ways in which technology can be involved in the process and the detail is absolutely key. For simplicity, I have categorised the different sorts of remote invigilation under three headings, really simply to show the variety of ways in which this can work. In reality each test sponsor will look for a slightly different blend of solutions.

Online Invigilation

This is where an invigilator is online looking at the candidate and/or their screen in real time, with them checking the process, listening and watching for any untoward activity. The invigilators are not necessarily looking at one candidate but could be looking at a number of candidates on a screen. It is likely that the number of screens being monitored at any one time impacts on the price of the service. The candidates could be based at home, in a formal test centre or any 3rd party venue. The only requirement from a venue perhaps being the meet and greet element. Then again, maybe we should be thinking of the passport style testing booth in a station or post office.

Offline Invigilation

In this instance, the test session is recorded, perhaps where there is intermittent or low bandwidth, or perhaps where all the tests have to take place on the same day and it is not feasible to have them all monitored simultaneously. These session recordings are then transferred to a location where they can be checked after the test has taken place. The results of the test result would only be provisional until they had been checked.

Hybrid model

Advances in technology will see an increase in hybrid models where computer algorithms flag up any untoward activity or issues, only referring to a human grader where need be. Whether running this technology in parallel with live invigilators or by recording all sessions, we will see increasing reach of technology in this sector.

I would certainly argue that any of these methods are in fact more secure than a traditional test centre model as there isn’t an opportunity for local fraud based on a candidate’s relationship with the test administrator – or statistically unlikely anyway.

Biometrics and identification

Whilst the delivery is theoretically straight forward the other key issue with remote invigilation is being as sure as you are in a traditional test centre model, that the candidate is who they say they are.

The easiest way is holding a photo ID up to a screen to have it checked against your image whilst capturing both images for future comparison. Plus, potentially embedding the candidate picture on any results. Of course now there are more sophisticated and rigorous methods with the use of biometrics with passport scanning equipment and the potential new means of retinal scan, face recognition or fingerprinting all of which can be automated and would work certainly in a test venue of some form, but in the future could be used at home. It is just a question of time.

Test Sponsors

So I have said that communication is key to adoption and the gatekeeper to adoption here is the test sponsors – those who actually own the test, exam or assessment, rather than the software companies developing the technology. They are the ones that need to be convinced.

Simply put, test sponsors need to become more informed. What is now possible? What point have we reached with the technology? And furthermore, how does this technology enable us to make a step change to how we test, rather than looking to replicate existing systems simply because we can. Why copy the inadequacies of the previous system when you can do things in different ways?

The problem with the classic test centre model

Remote invigilation opens a door to on-demand testing and more personalised assessment. I would urge current test sponsors to look at the costs, but also the flaws and the advantages of current systems and procedures, and to start by asking what would they like to improve, it needs to be viewed from a strategic perspective.  I am confident that there is technology out there to make the changes you seek.

1 “Proctor” and “Invigilate” are interchangeable words with identical meanings in both the Cambridge Dictionary and the Merriam Webster Dictionary. However, it is worth recognising that the former term is used extensively in the US, whereas the latter is the term more commonly used in the UK.