Summer Newsletter 2016

Welcome to the latest eAA newsletter!

It’s been a busy Spring. Summer seems to have finally arrived here in the UK, with us having just enjoyed a lovely spell of warm sunny weather, with a few rather impressive thunderstorms thrown in for good measure!

At the same time, the e-Assessment Association has been busy in all directions – from our Annual General Meeting, elections and Speed Learning Event back in March, to some behind the scenes development work, with two new Committees. One of these has been looking at where the Association wants to be in five years time, and how we can deliver the best value to our members. The other, a spin-off from the first, is working on developing our own e-Assessment Awards programme, to reward best practice across all aspects of technology-enabled assessment. Read on for details as to how you can get involved.

Our Board Member, Tim Burnett gave a well received presentation at the Westminster Education Forum event in April on “Embedding digital technologies and innovative teaching practices in the classroom”; and we have been working with JISC and the E-Assessment Advisory Group (EAAG), to publish the findings from a recent survey into the use of e-Assessment in Further Education.  You can read the full report here as well as a blog post on the JISC website from Lisa Gray and Martyn Roads.

We have kicked off a new monthly blog with an article from Bill Foster on the development and use of open-access, open-source in mathematical sciences. Look out for these posts on our website, or by following us on Twitter, but in case you missed it, you can read Bill’s blog post here.

As the summer continues to ‘hot up’ (hopefully) so too will our planning and preparation for our first annual e-Assessment Awards, which of course I’d like to encourage you to get involved with. As a final note, you may also be interested to hear that we are hoping to bring you some joint webinars with Transforming Assessment come the Autumn – watch this space for more news on this.

Best wishes

Matt Wingfield, eAA Chairman 

CONTENTS

• eAssessment news round-up
• Are Standardised Tests a Sleight of Hand?
• Executive Moves
• Speed Learning at the eAssessment Question
• NEW e-Assessment Awards – have your say
• New to e-Assessment adoption?
• The development and use of open-access, open-source material in mathematical sciences
• eAA Vision and Values – key priorities in 2015
• Introducing the Centre for Recording Achievement
• Follow the story with the eAA
• Looking ahead

eAA Announcements

Following our recent elections we welcome to our new Board Members, Tim Downie and Graham Hudson; and welcome back to the Board Phil John, Bill Foster and Pat Coates, who stood for re-election. You can view profiles of our Board members here.

Awards time! The eAA has a new sub-committee looking into establishing a brand new e-Assessment Awards programme. See below for further information.

You can email the eAA at any time at [email protected]

Image credit: © Wikipedia / joadl / Cc-by-sa-3.0-at

eAssessment news round-up

Since our last newsletter we have reported a significant contract win for one of our Bronze tier sponsors, Cirrus Assessment, as CPAA (Certified Public Accountants Association) select their platform. We raised our collective eyebrows over the handling of the UK Driving Test and recent headline education news led to an interesting discussion on our LinkedIn group on the future of e-assessment and e-learning for general quals.

We spotted the British Council on Gov.UK looking for an e-assessment provider to support its paper-based testing (now closed).

Finally, we are proud to announce that we are one of a number of key organisations brought together by Jisc, who will be working together to support the aims brought out by their recent research and report on “Technology-enhanced assessment and feedback in further education and skills”

Watch our website for regular news updates https://www.e-assessment.com/news/

And join the debate on LinkedIn https://www.linkedin.com/groups/1522607

If you have news to share, please email Karen at [email protected] or tag us on twitter @eAssess

Are Standardised Tests a Sleight of Hand?

Excerpt from new blog post from Karim Derrick, Chief Innovation and Education Officer at Digital Assess, who are Silver Sponsors of the Association.

“Is it possible that the Standardised tests that students across the world have faced for decades could, in fact, turn out to be a mathematical sleight of hand? Todd Rose thinks so based on work by Peter Molenaar. And it’s not just testing that is in the firing line — admission tests, mental health tests, personality tests, brain models, IQ tests, depression treatments, hiring policies all suffer the same fate.”

The article goes on to consider the “The flaw in averageism” and what this then means for assessment and testing.

You can read the full article here.

Executive Moves

Learning technologies provider, eCom Scotland, just announced the appointment of a new VP Business Development, Geoff Chapman.

Up until this year’s Board elections when he stepped down, Geoff was the Vice Chair of the eAssessment Association. We wish him all the best in his new role.

Speed Learning at the eAssessment Question                                                     #eAssess16

The 14th annual International eAssessment Question Conference and Exhibition, run by Assessment Tomorrow included an hour of ‘Speed Learning’ for all the delegates. Everyone at the event attended five short and punchy ten-minute debates across a range of topics, in small groups. Feedback was very positive and the full twitter feed will give you a snapshot of the whole event here if you were unable to attend. However, we are transcribing the sessions so we can bring you a series of blog posts over the coming months, starting with ‘The Future of Marking’.

It was a wide reaching session looking at existing marking practice and how technology can and should help as well as gaining some insight into the work taking place with machine learning, the use of ‘digital assistants’ and technology-enabled comparative judgement.

As a taster, here are a few of the key comments from this session hosted by James Redgate, RM Results and Karim Derrick, Digital Assess.

James: 

“Markers are here to stay, there’s definitely room for development of technology on the marking process, but it is about the blend.”

“Increasingly markers are saying they want to develop a valued and viable career in marking. There is a need for more training and development.”

“Last year, 22 million exam scripts were generated through GCSEs and ALevels in England and Wales alone. That equates to something like 150 million sheets of paper; and it’s really really clear that, even to us as a software company, that paper isn’t going away.”

Karim:

“The Future of Marking is, in part, machines. But not to entrench or replicate what we’ve done in the past but to transform it.”

“Marking itself, literally hasn’t changed since 1792 when Cambridge University introduced marking with William Farish, who first allocated marks to students and then totalled them up and ranked the students.  There’s an argument that says not much has changed since then.”

“Everybody knows that feedback is a powerful tool, but as any teacher will know, it can be an onerous piece to deliver. So we’ve been exploring the use of ‘digital assistants’ to help students ask the right questions of themselves.”

New e-Assessment Awards – have your say

The e-Assessment Association is making plans for a brand new awards programme, with a view to celebrating the diverse and innovative field of e-Assessment and the impact that it has across the globe.

We are just finalising the categories and criteria and will soon be able to announce the details and application process. We are looking to celebrate innovation, individuals, institutions, research, software and more. In a world where almost all assessment is, to some degree, enabled and enhanced by technology, we will be looking for the very best examples.

We are also looking for expressions of interest from those of our members who would like to be involved in the Judging process. We are looking for at least three judges for each Award category.

We are hoping to launch the Awards in Spetember this year, with a view to awarding our very first winners and celebrating the best of our industry at a prestigious gala dinner in March 2017.

Email Karen with any expressions of interest to join the panel, media enquiries or with any other questions at [email protected]

New to e-Assessment adoption?

In January this year, the e-Assessment Association presented at the Association of International Accountants (AIA) Examiner’s Conference and we have re-published some of the content from the presentation here.

Thanks to Gavin Busuttil-Reynaud and Geoff Chapman.

The presentation started with a look at who else in the financial sector is using e-Assessment in the UK today, it is not a short list.

• Association of Chartered Certified Accountants_since 1998
• Institute of Financial Services_since 2001
• Institute of Chartered Accountants_since 2001
• Chartered Institute of Bankers in Scotland_since 2003
• Chartered Institute for Securities & Investment_since 2004
• Chartered Institute of Management Accountants_since 2006
• CFA UK Society of Investment Professionals_since 2006
• Chartered Insurance Institute_since 2006
• Chartered Institute of Taxation_since 2007
• Association of Accounting Technicians_since 2008
• Chartered Institute of Public Financial Accountants_since 2008
• Association of Taxation Technicians_since 2009
• Chartered Institute of Credit Management_since 2009
• Institute of Certified Bookkeepers_since 2010
• Institute and Faculty of Actuaries_since 2010
• Chartered Institute of Loss Adjusters_since 2010
• Association of Corporate Treasurers_since 2011
• Lloyds of London_since 2014
• Retail Banking Academy_since 2015

What is e-Assessment?

For an audience who were looking at considering a move to e-Assessment, we gave one definition of e-Assessment,

“The end-to-end electronic assessment processes where ICT is used for the presentation of assessment activity and the recording of responses. This includes the end-to-end assessment process from the perspective of learners, tutors, learning establishments, awarding bodies and regulators, and the general public.”

e-Assessment is e-testing, e-marking, e-portfolios, remote proctoring, peer e-assessment and includes innovations in digitally capturing and assessing soft-skills.

What are the benefits to e-Assessment?

For Learners, Training Providers, Examiners, Community, Learning Centres…

Validity

1. More discriminating assessments
2. Richness and authenticity of the assessment experience
3. Potential for interactive items

Risks & Disadvantages

1. Potential contamination of assessment with IT skills
2. More diverse skills needed to write good items and tests

Reliability

1. Reliable scoring
2. Potential for reduced variation in administration arrangements

Risks & Disadvantages

1. Potential contamination due to tech. failure or poor administration
2. Inter-form comparability issues to the fore
3. Inter-mode comparability for mixed mode systems

Efficiency

1. Savings over the longer term (variable and repeat costs are lower)
2. Centre administration costs may be lower
3. Can distribute new tests instantly

Risks & Disadvantages

1. Capital upfront cost
2. Benefits reduced for extended-response human-marked items
3. Benefits reduced for poorly-implemented systems
4. ICT estate may need updating

Effectiveness

1. Speed of feedback
2. Candidates generally like e-assessment
3. Assessment style aligns with modern learning styles and resources
4. Increased flexibility in delivery administration for assessment (location and timing)
5. Access to more performance data
6. Improved content security
7. Environmental benefits compared to the costs of paper-based exams system

Risks & Disadvantages

1. Custom and practice needs to change for a distributed network of organisations
2. Accessibility and fairness
3. Risk of embarrassing IT system failures

eAssessment Good Practice

• Don’t get blinded by the ‘e’ part – always follow good assessment practice!
• Produce a clear assessment specification, style guide, mark scheme etc.
• Do not make assumptions about what stakeholders do and do not understand about the e-assessment implementation: communication and stakeholder engagement is key.
• Know your learners, know your centres.
• Train your assessment writers and markers.
• Identify skills gaps (Animators? Accessibility specialists?).
• Pilot materials, get feedback and act upon it.
• Consider the assessment lifecycle and do not assume that it will be unchanged by the introduction of electronic systems. e.g. provision of accessibility measures.
• Consider electronic tools to support configuration control throughout the test development process.
• Be prepared to select a mix of forms of assessment to achieve a reliable and valid assessment decision: and not every one has to be ‘e’
• Consider interoperability and system migration at an early stage
• Start small with clear, achievable plans which deliver real user benefits

Finally, one last thing to bear in mind:

Gall’s law

“A complex system that works is invariably
found to have evolved from a simple system that
worked. A complex system designed from scratch
never works and cannot be patched up to make it
work. You have to start over with a working
simple system.” 

 John Gall

We will shortly post the four case studies that Gavin and Geoff presented at this conference on our website and will include a link to all new case studies in our next newsletter.

If you know of, or are working with an organisation that is looking to move to or expand their use of e-assessment, do get in touch.

Blog: The development and use of open-access, open-source material in mathematical sciences

by Bill Foster. e-Learning Team of the School of Maths & Stats, Newcastle University and eAA Board Member.

This blog is on the use of e-assessment in all its forms for mathematics and statistics (mathematical sciences) at HE and was first posted on the blog section of this website as part of a new monthly series of blog posts from the Association.  Click here to open a new window to read the blog.

eAA Vision and Values – key priorities in 2015

In November 2015, the eAA created a new Vision and Values Committee to review the work of the Association and the value it provides. This was set up following a survey of our membership that we undertook last autumn.

As part of this work, we developed a set of 7 key priorities to focus on for 2015.

1. To focus on being the main source of information about e-assessment. As part of this, the Committee is currently working on a new e-Assessment Glossary.
2. Publicise key metrics of success; bring you more case studies on our website.
3. To ensure publication of both quarterly newsletters and bi-weekly newsbite updates.
4. To create a link between our LinkedIn Group activity and the eAA website.
5. Use our social media and our website to retweet and syndicate key online data sources.
6. Set specific targets to grow the sponsors and membership of the Association.
7. Recognise excellence in e-assessment through awards for great practice (see earlier article).

For more information about the work of this Committee, please email Karen at [email protected]

Please note that without the generous support of our Corporate Members, the Association would be unable to carry out many of these activities. You can view our current Sponsors list here.

                                 Introducing the Centre for Recording Achievement

In May this year, we started to look to build stronger links with organisations and partners working in e-assessment worldwide.

Rob Ward, Director of the Centre for Recording Achievement tells their story….

In the beginning…

We started out as a two year MSC funded Project way back in 1991 (the age of the Sinclair Spectrum, the BBC ‘B’ and comment backs for reporting, I recall). Entitled the ‘Recording Achievement and HE Project’ and led by the Wigan Local Education Authority, it involved 11 such authorities and 15 Universities and Colleges of Higher Education across the North of England and North Wales. The focus was very firmly upon progression to HE, and specifically upon the wider achievements and experiences of HE applicants and how these might be appropriately presented in the HE admissions process. This created a context for dialogue and development between HE staff, school and college staff and of course students, with the aim of demystifying the HE recruitment process, and led to thinking as to how such wider experiences might be built upon in HE. Our outcomes were disseminated across the pre HE and HE sectors via the production of Annual Guidance materials disseminated via UCAS.

Subsequently…

At the end of the funded period the Project members decided to continue working together, and – notwithstanding changes in staff – many have done so ever since!  Over time our title has changed, from the ‘RoA and HE Project’ to ‘the Recording Achievement Consortium’ through to ‘the Centre for Recording Achievement’, but our focus upon recording – the process and active verb – as opposed to record has never varied. Over the years we’ve worked with Becta (remember them?) and Jisc, we’ve promoted ‘Progress Files’ in the school and college sector and ‘Personal Development Planning’ and e-portfolios in HE. We’ve worked with employers and professional bodies; on research, evaluation and consultancy – including as a member of the ‘Enhancing Student Employability Co-ordination Team’ (ESECT, 2002-5); we’ve provided leadership for a HEFCE Project to support the use of electronic portfolios in work-based learning and continue to support sector-wide implementation of the Higher Education Achievement Report. We run National Seminars on the HEAR and in support of holistic approaches to Personal Tutoring, amongst others.  But through all of this we have not lost sight of an emphasis upon developmental potential and usefulness by students and possibly others, alongside issues to do with wider achievements and effective use of such information which characterised our early work. So amidst much change there is also much continuity.

And now…

We are a membership organisation, an educational charity (since 2002) and have links – and members – across the world, with a number of reciprocal membership agreements.  We are also intensely interested in collaboration rather than competition, and on ensuring a holistic approach to practice.  For example, our recent National Tutoring Seminar brought together Learning Gain pilot schemes, the interest of the NUS, the potential impact of the TEF, and the emerging work on learner analytics led by Jisc.  And our International ePortfolio seminar in Edinburgh last week was run jointly with US colleagues.  So if this brief introduction inspires you to think about the potential for such collaboration we’d be happy to talk-and listen!

Rob Ward

[email protected]

Follow the story with the eAA

Looking ahead

The e-Assessment Association is constantly looking ahead. In our next newsletter, we will bring you new articles, blog posts, events and news. But please remember, this is your Association. Do send in your articles and keep us up to date with e-Assessment work where you are, so we can share good practice and information across the sector.

Email: [email protected]

Twitter: @eAssess