the e-Assessment Association

Internationalisation of e-Assessment programmes – short overview

Internationalisation of e-Assessment programmes – short overview

As a by-product of the imperialist age(s), exam delivery in a different country to that in which the exam was first delivered or created has been with us for over a century. Countries that were under direct colonial rule (e.g. Ghana[1]) or who had close trade ties, still have exam systems that have familiar structures and outcomes to those of us living in the UK (where the author is based).

The genesis of on-screen assessment is usually placed at the door of the US military and universities such as Illinois and Stanford through the 1960s[2]. What we would today recognise as ‘dumb’ computer terminals, together with the use of punch-cards, enabled knowledge assessment of military trainees and in-service personnel at locations such as US Air Force bases and Naval depots across the world, in a secure environment.

Fast forward to the 1990s: the omnipresence of the PC as the common platform for learner, and worker/ home IT (compared to Macs, Mainframes etc.), a reduction in the price[3] of IT equipment and the development of IT/ ICT as a recognised profession has increased the usage of IT as a means to deliver exams across borders.

Depending on who you talk to, the epoch for international e-Assessment as it is recognised today was established in the mid 1990’s with companies such as Novell launching the Certified NetWare Engineer certification (now Certified Novell Engineer) and licensure organisations such as the National Association of Securities Dealers (NASD, now FINRA) wanting to deliver exams securely outside of the United States of America.

Most IT companies now have a certification programme as part of their sales strategy, to help IT workers prove competence in core technologies, but also to provide an opportunity for students and educators to be ‘first to market’ with a new technology or platform. With a small number of notable exceptions (e.g. CompTIA), IT certification is primarily built to shift IT kit or Operating Systems, and to influencers, decision makers and budget holders. While there is proof that taking IT certification leads to better career prospects and remuneration[4], there is a clear notion that IT is a real social leveller: it is an occupation that appears to be far less class-ridden and more meritocratic than others, such as law. A constant need to refresh knowledge and understanding and prove it, time after time, coupled with increasing cross-border recognition and embracement of non-native qualifications means that it is a cornerstone of international e-Assessment growth.

The growth in internationalisation has not been without problems. Industry specific standards and codes of practice are still ghettoised to the assessment cognoscenti, and are either not well known or understood by those people (and organisations) that they should be serving. While it is encouraging that government agencies and larger organisations are insisting on internationally recognised quality assurance standards (e.g. ISO 9001), little attention is given to either delivery (e.g. ISO 23988) or development standards (e.g. SCORM). Attitudes towards standards differ by industry segment, as well as from country to country.

The increasing role of third party test centres for secure and robust delivery to an acceptable standard is also becoming an issue. In addition to the non-adherence to industry standards, the risks involved with insecure internet connections and on-site hosting, together with weak centre selection procedures[5] compound the issues regarding internationalisation.

There is plenty of evidence of rogue candidate groups hearing about ‘weak’ test centres or ‘fences’[6] where an exam can be compromised and a pass can be obtained more easily than what was anticipated by the programme owner. Countries which are subject to United Nations embargoes and/or restrictions can be subject to techniques such as in-person and virtual candidate tourism. This is where the candidate either travels in person to a recognised country to take their test (which is not permitted by the credentialing authority or programme owner), or the centre in the embargoed country uses a virtual private network to access a ‘friendly’ test centre in a non-embargoed country. Many assessment programme owners take precautions to prevent both scenarios from damaging their exam and international scope[7]. When an e-Assessment is of sufficiently high-stakes that the summative result can be a ‘ticket-to-a-job’ or entry to extended education opportunities, the need for secure and standardised e-Assessment delivery (as insisted upon by programmes such as GMAT, IELTS, TOEFL/TOEIC etc.) is paramount and, typically, non-negotiable.

There is evidence of some Awarding Organisations (AO’s) using their Foundation level exam for capturing as much market share as possible, which often means that candidates are given the opportunity to be invigilated remotely[8]. While this has a benefit of increasing access and availability of the exam and awareness of the AO, there can be an undesirable effect of diluting the perceived value of the assessment, especially as it is delivered in an electronic environment.

e-Assessment creation is also becoming increasingly international in scope. For example, the skill, craft and assurance time that is taken when question writing in a native language, isn’t always replicated when it comes to translating it into a (lucrative) non-English language. The use of ‘Double byte charter sets and languages[9] where text can also be read from top to bottom, and right to left, can have huge and unpredictable ramifications on the intended psychometric values of items and distracters. The ability for programme owners to effectively compare results across cohorts and delivery centres can be compromised if insufficient statistical analysis is planned for within the development of the e-Assessment. Culturally, if the candidate is not used to receiving an instant result with insightful feedback then this can also create issues in e-Assessment design and delivery, such as delaying results for simultaneous global candidate cohorts.

There is still a limited appetite for exam creation, management and deployment in lower cost non-English speaking countries. There is limited evidence to suggest that the larger Awarding Organisations (AOs), IT Certification bodies or medical boards are off-shoring their high-stakes e-Assessment development. While some UK companies have set up subsidiaries to take advantage of international talent pools[10], much of UK Government guidance is for such work to be done within the European Economic Area. e-Assessment developers can inhabit the same pools as animators, banking staff, film workers and government workers with the ability to safely and securely transmit both item banks and software code/ applications across borders. A key part of internationalisation is the ability to tap into available talent pools to work on such projects with software coders and items writers facing the same problems as animators and film workers – protection of IP across the web.

e-Assessment item production of the future may see forms, learning outcomes etc. are all pre-determined by the organisation’s on-shore Chief Examiner or Director of Education. Off-shored item production will be done to control costs, but co-incidentally to boost item bank production and refresh to levels that will help to prevent fraud.

There will be an increase in the trend to break out item production into separate component parts and then to off-shore those parts that can be produced cost-effectively with managed risk.

A comparative analogy can be made with car production: Research & Development done in Turin, Italy or Woking, UK with production done in Curitiba, Brazil. Where quality and modern production techniques can be continually improved (with appropriate security), then production will be near shored (e.g., Sunderland, England for Nissan EMEA; Zwickau, East Germany for Volkswagen). There will be organisations willing to adopt this approach, where the cost of effectively policing test centre fraud is prohibitive or collusion between tutor and candidate is possible, or culturally acceptable.

Future internationalisation of e-Assessment is being driven by a variety of factors, not least the increasing competitive area of education delivery andprofessional post-school licensure bodies owning their industry niche space worldwide. More private/ commercially funded programmes will seek to gain efficiencies and more efficient delivery methods, thus passing on savings to students, or helping to fund shareholder value.

by Geoff Chapman, Calibrand and e-Assessment Association


[2] p.8