Case study from Cirrus: E-Assessment Award winner for Best Transformational Project 2021

Debuting a digital exam platform to the time-honored establishment of Chartered Accountants Ireland was nothing short of a monumental task, but Cirrus did not shy away from the challenge.

Faced with the rising costs of testing centers and the fast-paced demands of a world going digital, Chartered Accountants Ireland knew they needed to make an enterprising decision to transform their institution.  CAI is a prestigious organisation founded in 1888 and is the third oldest accountancy body in the world, dating back to before the Irish republic.  They were traditional in the way they approached assessments, paper was always the way exams were taken.  

Cirrus signed the contract with the Chartered Accountants in January 2019, with the deadline of delivering their first exams at the end of March, with the additional need to develop two bespoke question types for them. Cirrus has never backed down from a challenge and started immediately with gathering a team of experts for CAI. Cirrus understood that there was much at stake for CAI, both reputational and economic, and therefore when successfully implemented it could mean the complete metamorphosis from old tradition to the digital age.

The Challenge

Chartered Accountants Ireland was heavily dependent on paper-based assessments to successfully examine students whilst undergoing the ACA qualification at all levels. 

Chartered Accountants Ireland’s desire was to migrate all of its existing paper based examinations, roughly 25,000 paper based assessments per annum, to an E-Assessment platform on a phased basis over the next three to five years.  A pilot program was to be rolled out covering 10% of the exams but due to COVID-19, lockdown measures had to be implemented in March and pushed the entire project into overdrive. What started as a pilot program covering 10% of exams meant to slowly roll out over the course of five years now needed to be shortened to five months, covering ALL exams.  Testing centers were closing and they needed to move quickly on a solution.

 The challenge was tremendous, but if implemented properly, this new assessment platform will assist in reducing existing examination venue capacity issues, reducing workload relating to GDPR, increasing global reach of the ACA qualification and enhancing student satisfaction 

Method and Steps Taken to Ensure Success

• Identifying and mapping IT competencies
• Organizing a “kick-off” for all users involved
• Training of participants in project teams, users in need of extra IT support, super-users and administrators
• Individual coaching / assistance for project participants via Internet / Phone.
• Frequent meetings of project group / management committee.
• Discussion panels in which participants exchange experiences, show progress and
• achievements, and discuss any challenges.
• Project completion including a final evaluation of the project

A detailed blueprint plan was devised to tackle these challenges and activities proposed to assist with transition:

• Detailed strategy sessions where Cirrus designed exactly the type of high-stakes exam questions CAI needed – including two new bespoke question types.
• Training programmes for staff including initial training and refresher workshops throughout the year. These would focus on each delivery phase (item creation, assessment creation, pilot delivery, marking and moderation).
• Chartered Accountant Ireland and Cirrus Assessment would together outline plans, provide updates, demonstrate functionality and answer questions and concerns in open forums.

The success of this transformation was due, in large part, to Cirrus truly listening to exactly what this accountancy body needed. It not only transformed Chartered Accountants Ireland from traditional paper-based to digital, but in turn also transformed Cirrus from a generic exam platform to one capable of delivering the most complex financial questions in the world. 

Achieved metrics:

• 100% of CAI high-stakes exams online in 2020;
• Full regulatory approval that exams were as secure as paper-based exams in exam halls
• Annualized pass rates at all levels being completely consistent with the prior 3-5 years;
• Quality of responses from candidates was stronger than in prior years as students could type more within the time allowed and the formatting functionality facilitated clearer, more concise and professional report responses. This resulted in higher average marks than prior paper-based outcomes;
• Candidates had the ability to have 5 practice exams to familiarize themselves with the new platform. This would not have been feasible in an exam hall;
• Candidates all had access to a practice “onboarding” session to ensure that their hardware and broadband met requirements one month prior to the actual exam. This helped to facilitate the 100% real exam day onboarding success rates, which are completely unrivalled in the high stakes global professional accounting exams market (worldwide onboarding success rates ranged between 70-95%);
• Streamlined, intuitive end-to-end marking within the Cirrus platform resulted in exam marking times being reduced by up to 50%. This facilitated the earlier release of exam results in 2020, resulting in significant resource, time, and cost savings in 2020 and beyond;
• The very efficient marking process has resulted in greater satisfaction levels within our internal teams
• Quicker exam results feedback for students, heightening satisfaction
• Reduced carbon footprint 
• Cirrus’ flexibility assisted CAI in existing customer retention, and facilitated growth into other markets. 
• Success of the project has given CAI a reputational boost. IFAC published a positive case study stating how CAI is at the forefront of digital exams on the global accounting stage. 
• FRC and IAASA have both taken the CAI/Cirrus journey as a ‘best in class’ experience in 2020, enhancing CAI’s reputation.

About the Project

For CAI, the benefits of digital exams are numerous. They now have the ability to give exams to students all over the world in all circumstances. Tragedies like COVID-19 will never completely derail their students’ ability to take exams and continue prospering in their careers.  Thanks to results being delivered more quickly due to digital grading, students are able to let accountancy firms know their results quicker, thus resulting in them being able to progress to a better payscale faster.  CAI has also saved money on the administration costs of setting up, invigilating, and hosting exams in testing centers. 

Perhaps the most unique aspect of the project was not only the enormous task of making the switch from paper to digital, but the condensed timeframe in which it was done within. This transformation would not have been at all possible had it not been for the communication, support, hard work, and open-mindedness of these two companies. 

Thanks to the success of this cooperation, the E-Assessment project rollout in 2020 has given CAI a significant reputational uplift globally as IFAC published a very positive case study on its website stating how CAI are leading the way in this space on the global accounting stage.

CAI students found the system extremely intuitive to use as evidenced by the increase in average marks and the consistency in annualised pass rates. The student feedback post-exam was extremely positive. Students love the rich text formatting options that they were familiar with, and the ability to have pop up windows for resources, response windows or split screen capability was all welcomed too. For students with reasonable accommodations, Cirrus was able to provide a wide range of solutions to help, such as read speaker, adjustable text size, and the ability to run a much larger second monitor (only allowable for visually impaired students). Where students had poor broadband in almost all cases students were able to sit their exam whilst hotspotting their laptop from their mobile phone, which made CAI exams accessible to absolutely everyone in 2020.

CAI provided administrators, examiners and markers with process documentation and video tutorials on how to use the Cirrus platform. These materials were designed to be as minimalistic as possible. Due to the extremely intuitive user interface for these stakeholders, all constituents were able to be highly functioning on the platform with minimal training. The platform was praised by all users as to how effective, efficient and fit for purpose it was. This was only achieved through huge collaborative work during Summer 2020 between Cirrus and CAI teams. 

Quotes

“Tonight, as a huge combined team, we managed to create a truly momentous milestone in the assessment of professional accountancy globally. Chartered Accountants Ireland’s first-ever remote proctored fully regulated exam completed successfully!”

“The first thing I noticed about Vegard and the team was, when I was first asking questions like ‘can we do this’ or ‘could we do that’ was that there was never the answer of ‘no’.”