Case study from Excelsoft, winner of E-Assessment Award for Best International Implementation 2021

A scalable and secure assessment solution to deliver exams across Mexico in a challenging environment

Problem/Challenge
Mexico’s pension system for workers and employers fund individual retirement accounts which are managed by private companies and regulated by the government entity. To ensure workers receive accurate and trustworthy information from the private companies, an independent third party was to test all “promoting agents” before they are hired, and re-test them every three years. These high-stakes exams were to be held online with a focus on improving the quality of promoting agents and reducing workers’ complaints against them. This was a very complex project involving many stakeholders with different constraints, across two countries, two languages, and four time zones.

Excelsoft- a leading Global EdTech company, a Mexican company with long experience in developing exams, and TAEC- a Mexican company with deep expertise in implementing e-learning solutions joined hands together to deliver this solution.

Some of the challenges faced during project implementation were:

• Verify candidate identity to ensure the genuineness of the candidate appearing for the exam.
• Prevent any malpractice amongst promoting agents during testing.
• Ensure exam content security and prohibit Item harvesting.
• Ensure test can be delivered in low internet bandwidth across remote locations in Mexico.
• Lack of homogeneity of technical infrastructure at private companies’ offices.
• Time zone difference of 13.5 hours between India and Mexico.
• Language Difference –Most client-side teams spoke Spanish.

Solution
The aim was to-

• Comply with the testing mandate, with minimal disruption to existing hiring processes and commercial activities, and keeping the costs at a minimum. 
• Being able to schedule a large number of exams at short notice.
• Minimize exam cancellations due to system downtime.
• Have a high percentage of passing scores. 
• Reduce the number of complaints by workers because of misinformation from their promoting agents

The course of action-

• The contracts had to be signed with each of the eleven large, private financial institutions, with their own, sometimes conflicting, constraints and business cultures. 
• The goals of Regulatory authority were to be met, which did not always align with the financial institutions. 
• The non-profit managing the National Database of candidates had its own set of restrictions, which had to be overcome. 
• A steering committee representing each stakeholder was composed, which met twice weekly. 
• The Regulatory authority was kept informed regularly, through meetings and formal written communications.

Work was soon divided into two teams:

• One in charge of developing the items and tests
• Another one in charge of the technical infrastructure, including
  • installing the software in virtual servers in the cloud
  • connecting to an MPLS network
  • developing the web services required to connect to the database
  • developing new functionality in SarasTM AMS, such as taking pictures at random times during the exam to prevent cheating
  • Providing training to the TAECs team, which in turn translated the training into Spanish and delivered it to each of the private companies

Excelsoft also sent an experienced consultant who spent three weeks in Mexico, which was key to the project’s success. He continued to be the main point of contact with the team in India, through the launch and first months of operation.

Due to time constraints, the testing system was deployed in two phases- 

• phase one was held in over 30 test centers provided by TAEC throughout the country
• while phase 2 involved full functionality

The test results were downloaded from SarasTM AMS & monthly reports were provided to each private company identifying areas where training needed to be improved. Additionally, a statistical analysis of answers to items including techniques from item discrimination theory was provided to improve the item database and keeping it fresh and updated.

Results/Outcomes

• Around 84,000 exams were delivered in 10 months with a concurrency of more than 200 exams at times. 
• There were peaks of 300 exams during critical times, without detriment of response times. 
• Contingency plans were drawn in case of disasters. 
  • • • These plans were effectively tested in September 2017, when a major earthquake hit central Mexico. 
      • The testing operations were not suspended, as the Bogotá and Chihuahua offices took over the operation within 15 minutes of the earthquake striking. 
• After the project implementation, the initial large percentage of candidates failing tests dropped substantially. 
• The wide variation in test results between the different private companies narrowed. 
• The Mexican workers’ complaints were reduced.

About Excelsoft

Founded by Edu-entrepreneurs in the year 2000, Excelsoft has evolved as one of the most sought-after technology solution providers in the education and training space. With a strong focus on pedagogy, our solutions have been able to deliver compelling learning experiences through world-class eLearning applications specific to the business requirements of our customers. We architect, design, and develop learning environments, assessment tools, digital content, and product solutions for some of the best-known companies and academic institutions worldwide. Excelsoft’s mission is to integrate its extensive experience in e-learning technologies with its dedication to the development of key strategic applications. This combination has resulted in a constant quest to raise the bar and meet the growing expectations and business requirements within the e-learning marketplace. We’ve managed to make a positive impact on the lives of over 25 million users worldwide through our innovative products and solutions.

For more information, visit www.excelsoftcorp.com or follow us on LinkedIn