e-Assessment Supports School and System Improvement in Ontario: A Case Study

By Richard Jones, President – RMJ Assessment & Pino Buffone, Director of Education – Renfrew County District School Board

Introduction

Province-wide student assessments have made a strong contribution to education in the province of Ontario for over a quarter century. Through the Education Quality and Accountability Office (EQAO), an arm’s-length agency of the provincial government that contributes to the quality and accountability of Ontario’s publicly funded education system for K-12, paper-based provincial assessments have been offered since the late 1990s, with a transition to online assessments beginning in 2022. The provincial assessments are meant to measure students’ achievement at key stages of their learning as related to the expectations of the curriculum. The assessments include: Primary Division (Grade 3) and Junior Division (Grade 6) Assessments of Reading, Writing and Mathematics; Grade 9 Assessment of Mathematics; and, Ontario Secondary School Literacy Test (typically administered in Grade 10), which measures whether students are meeting the minimum standard for literacy (Reading and Writing) up to the end of Grade 9.

The purpose of this article is to provide an overview of current elements of Ontario’s provincial e-assessment in Grade 9 Mathematics and demonstrate how information and data from the assessment have supported improvement-related initiatives in one educational context, the Renfrew County District School Board (RCDSB).

Context

Academic streaming, the practice of separating students into distinct program pathways based upon their perceived abilities, has been practiced in Ontario for decades. It is a practice that originated in the mid-19^th century, at a time when education was solely for the colonial elite, as opposed to the general working class, more broadly. The process of academic streaming involves Grade 8 students and their parents/guardians selecting the course types (e.g., Academic, Applied, and/or Essentials) to follow in secondary school, based on their perceived academic abilities/performance. The selection of courses is based upon a multitude of factors, including students’ own interests, their perceived readiness on the part of their educators, as well as the perspectives of their parents/guardians.

Many educators and members of the wider school communities̶̶̶-locally, provincially, nationally and internationally-believe that Grade 8 is too early for students to be channeled into distinct pathways that ultimately limit some students’ future scholastic and occupational opportunities. The unintended consequences of academic streaming-a disproportionate number of students from low-income families, racialized students, Indigenous students, and students with special needs enrolled in applied courses-has perpetuated their underrepresentation among students who graduate and transition into post-secondary education. Consequently, the de-streaming of Ontario secondary school courses in Grade 9 began in the fall of 2021, with mathematics, and has continued with a number of other courses during the current school year.

Grade 9 Assessment of Mathematics^[1]

 What is Assessed?

Students enrolled in the Grade 9 de-streamed mathematics course are assessed on the knowledge and skills defined in the Grade 9 Mathematics Curriculum (2021), including the content strands: Number; Algebra; Data; Geometry and Measurement; and Financial Literacy. The curriculum policy document for mathematics also contains sections on social-emotional learning skills, as well as  mathematical thinking and making connections. The blueprint associated with the Grade 9 Assessment of Mathematics can be accessed in the associated Framework document (p. 7).

How is the Assessment Administered?

The assessment comprises three main components: Introductory, Assessment and Questionnaire Sessions. In the Introductory Session, students have an opportunity to experience a 27-item practice test that demonstrates the kinds of questions that will appear on the actual assessment, as well as the tools that are available, such as text-to-speech, zoom in/out, high contrast, draw line, highlighter, eraser and calculator. The sample/practice test can be viewed here. In the Assessment Sessions, students complete two components: Sessions A and B. Each session is designed to be completed in approximately one hour. Students can write the two sessions in a single sitting, back-to-back with a break in between, or on different dates. Once they have completed the two assessment sessions, students respond to a survey that asks them questions about their mathematics-related attitudes, perceptions and behaviours.

[1] Information in this section has been accessed from EQAO’s resource: the Grade 9 Assessment of Mathematics Framework (2022), located here.

How is the Assessment Designed?

The assessment comprises only selected-response items (questions), including drag-and-drop, drop-down menu, ordering and single- and multi-selection item formats. Out of the total 54 items, 50 are “operational,” meaning they count toward students’ final results, and four are field-test items, which do not count toward their outcomes.

The Grade 9 Assessment of Mathematics employs a multi-stage computer adaptive testing model, in which assessment items’ level of difficulty are adjusted based on students’ performance during the two sessions. For example, at the beginning of Session A, students attempt a set of questions (Module 1) that is targeted at the medium level of difficulty. Depending on how students perform on Module 1 they move on to either Module 2 (low/medium difficulty level) or Module 3 (medium/high difficulty level). The same approach is used in assessment Session B: Module 4 begins with items of medium difficulty, and Modules 5 and 6 are characterized by items of low/medium and medium/high levels of difficulty, respectively.

How is the Assessment Scored and Reported?

Once the student’s operational items are machine-scored, Item Response Theory (IRT) is used to calculate a scaled/standardized score, which is then converted to a performance level, consistent with the Ontario Ministry of Education’s four achievement levels, generalized below:^[1]

• Level 1 represents achievement that falls much below the provincial standard.
• Level 2 represents achievement that approaches the standard.
• Level 3 represents the provincial standard for achievement.
• Level 4 identifies achievement that surpasses the provincial standard.

To provide more precise information, in addition to the achievement level attained, the scaled score indicates approximately where, within the general achievement level, student performance lies. The reports not only provide information about students’ performance, but also their behaviours and attitudes as they relate to the given subject area and their learning, generally.

In addition, the agency generates teacher and principal (school) questionnaires to gather information on factors that have the potential to influence student performance. These surveys include questions about classroom and school demographics, the school learning environment, the use of EQAO data and  instructional resources, teaching practices, educator collaboration, parental engagement, educator background and professional development.

Besides the individual student reports, EQAO also generates school, school board and provincial reports that are meant to enable students, parents/guardians and educators to support students’ learning, plan for improvements at both the school and system levels, and provide for accountability to the wider public. School, school board and provincial results can be accessed here. A sample individual student report can be viewed on page 13 of the Grade 9 Assessment of Mathematics Framework document (source provided above).

Since 2019, EQAO has engaged an assessment technology partner to collaborate with the agency to support the design, development and implementation of modernized, large-scale e-assessments, including the Grade 9 Assessment of Mathematics.

[2] Further information about The Ontario Ministry of Education’s achievement levels can be accessed here.

How Does RCDSB Use e-Assessment Data/Information to Support Improvement?

As outlined in the Ontario Ministry of Education’s resource, Multi-Year Strategic Planning: A Guide for School Board Trustees (2017), every district school board in the  province is governed by a Board of Trustees that has the legislated requirement to ensure the creation of a multi-year strategic plan. The multi-year strategic plan serves as a visioning and policy document that sets the direction for the organization. The multitude of operational and improvement plans developed and implemented annually by staff, through the Director of Education, are based upon the objectives of the multi-year strategic plan.

At the RCDSB, annual improvement planning processes at the school and system levels are in coherence and alignment with the strategic priority to ‘achieve excellence in teaching and learning’ of the RCDSB Strategic Plan 2021-2025, as well as the District’s framework for Inspired Learning.

More specifically, the Annual Action Plan for School Effectiveness and Student Success outlines growth targets for improvement planning at the elementary and secondary levels, and specifies tasks, timelines, responsibility centres, and resources required, along with measurable indicators of success. One of the measurable indicators of success is the utilization of e-assessment results provided annually by EQAO.

Moreover, with the interruption to education caused by the global pandemic period, along with the evolution of the agency’s assessments to an online format, the most recent data will serve as a new baseline for the District’s improvement planning processes. The reports provided by EQAO, related to provincial trends in achievement, including strengths and areas for improvement related to curriculum expectations, will be invaluable to the District’s dedicated and diligent staff as they prepare next steps for the plan of action, especially in the context of a post-pandemic reset for education, K-12. Prompts/tasks from the e-assessments deemed to require additional attention may indeed serve as areas of focus in the next iterations of the plan of action.

As it pertains to Grade 9 Mathematics in particular, the de-streamed course (MTH1W), initiated by the Ontario Ministry of Education for the 2021-2022 school year, ensures high expectations for all students in that there is a greater opportunity for every student to choose the academic pathway (as opposed to other pathways) for Grade 10 if/as appropriate. In fact, a key aspect of the measurement of success—beyond the achievement results on the Grade 9 Assessment of Mathematics—will be the system-wide monitoring over time of selection of course type by students and their parents/guardians for Grade 10 Mathematics. It is aspired there will be an increase in the number of students who choose the academic pathway for Grade 10, over time, in order to keep pathways as open as possible, for as long as possible, throughout their educational journey.

Stated explicitly, in the RCDSB’s Annual Action Plan for School Effectiveness and Student Success, key targets at the secondary level are:

• to increase the knowledge and confidence of educators in implementing Grade 9 de-streamed courses through new learning of effective, research- and equity-based pedagogical practices into classroom instruction and assessment as determined by participation rates of professional learning sessions, student achievement data, educator feedback and anecdotal observations; and,
• to establish a baseline and work towards an increase in the percentage of students over time in Grade 9 who are choosing the Academic Pathway vs Applied Pathway as determined by secondary school course selection data, credit accumulation, withdrawals and course drops, as well as attendance data.

Perhaps, most importantly, as part of the post-pandemic reset of education, the provincial assessments will serve as a stable source of data for improvement planning—especially considering these volatile, uncertain, complex and ambiguous times—a veritable value-add in supporting school and system improvement for the education sector, K-12, in the province of Ontario.

For further information, reach out to:

Richard Jones at [email protected] (for e-assessment)

Pino Buffone at [email protected] (for school and system improvement)