AI considerations

This guide suggests some assessment design approaches that enhance assessment and mitigate against the more harmful effects.

As the advancements in artificial intelligence (AI) reshape various aspects of education, including assessment design, educators are faced with the challenge of effectively integrating this technology into their practices. There are four distinct approaches that can be adopted to address the advent of AI in assessments.

The first approach involves embracing AI by designing assessments that utilise its capabilities as an integral part of the evaluation process. The second approach focuses on creating more authentic assessments, with the belief that students will find them more engaging and be less inclined to rely on AI for generating their submissions. The third approach entails reviewing program-level assessments to incorporate a diverse range of approaches, aiming to enhance student motivation while minimising the opportunity and desire to rely solely on AI to produce their work. Lastly, an exam-driven approach seeks to reduce the opportunities for students to leverage AI by implementing assessment formats that limit its applicability. By exploring these four approaches, educators can navigate the evolving landscape of AI and shape assessment strategies that align with their educational goals.

In the last section we provide an example of a complex activity for students to undertake that helps them understand the difference between human and AI intelligence and allows them to develop a uniquely human understanding and skill.

Embracing AI

Here are three specific ways in which assessments can be designed to embrace AI:

AI-assisted planning: Students can utilise AI tools to plan their submissions. AI-based writing assistants or content generation tools can help students brainstorm ideas, organise their thoughts, and outline their work. This approach leverages AI to support the initial stages of assessment preparation, enhancing students' ability to structure their work effectively.
AI-generated drafts for review: AI can generate draft versions of written work based on provided prompts or input. Students can then review these AI-generated drafts, focusing on identifying errors, refining arguments, or adding their own insights. This approach combines the creative input of students with AI's ability to generate initial content, fostering a collaborative and iterative writing process.
AI-powered feedback for improvement: AI can analyse students' written work and provide targeted feedback to help them enhance their writing skills. By utilising natural language processing and machine learning algorithms, AI systems can identify grammatical errors, suggest improvements in clarity and coherence, and highlight areas for further development. This approach empowers students to receive immediate and personalised feedback, promoting self-reflection and continuous improvement.

When designing assessments that embrace AI, there are additional challenges to consider:

Referencing and acknowledging AI contributions: It is essential to establish guidelines on how students should reference or acknowledge the specific contributions made by AI tools in their work. This ensures transparency and academic integrity. The problem is, as yet no clear and appropriate method has presented itself. This may change as we respond to these developments.
Managing workload for staff: Implementing AI in assessment design should not create an excessive workload for staff. Educators should consider the time and effort required to integrate AI tools effectively. Streamlining the process by providing standardised templates or guidelines for referencing AI contributions can help minimise the burden on staff while ensuring consistency and fairness in assessment practices.
Addressing potential errors and biases in AI-generated content: It is crucial to educate students about the limitations, potential errors, and biases associated with AI-generated content. Providing guidance and resources on critically evaluating AI-generated work can help students identify and address inaccuracies or biases that may arise. Encouraging students to engage in reflective practices and peer discussions can foster a deeper understanding of the limitations and potential ethical considerations related to AI-generated content.

To mitigate these challenges, educators can consider the following strategies:

Developing clear and concise guidelines: As yet no clear method for doing this has presented itself. However, when approaching this it is necessary to provide students with some instructions on how to reference AI contributions and acknowledge the limitations of AI-generated content. This ensures academic integrity while guiding students in appropriately using and attributing AI tools.
Student voice: Encourage students to engage in discussions, peer feedback, and reflection on AI-generated content. This promotes a collaborative learning environment where students can collectively identify and discuss potential errors or biases, fostering a deeper understanding of the implications of using AI in assessments. Perhaps through discussion you can collectively arrive at an approach.

Authentic assessment

Incorporating authentic assessments or simulations within the assessment portfolio of a program can offer several advantages in the context of AI. Firstly, students often perceive these types of assessments as more engaging, as they provide real-world contexts and practical problem-solving opportunities. The complexity and specificity of tasks in authentic assessments make it challenging for students to rely solely on AI to answer the problems they are addressing, thus encouraging active participation and critical thinking.

However, there are potential challenges associated with implementing authentic assessments in the context of AI:

Workload for staff: Designing and implementing authentic assessments or simulations may require additional time and effort from staff, particularly in creating realistic scenarios, developing rubrics, and providing personalised feedback. Adequate support and resources should be provided to ensure that the workload does not become overwhelming, enabling staff to effectively manage the assessment process.
Varying student motivation and confusion: While many students find authentic assessments engaging, there may be individual differences in motivation levels. Some students may find the complexity and specificity of tasks challenging or overwhelming, leading to potential confusion. Providing clear instructions, scaffolding, and support mechanisms, such as example tasks or training sessions, can help mitigate confusion and ensure a positive learning experience.
Gamification considerations: Gamifying authentic assessments or simulations can enhance student engagement and motivation. However, careful thought must be given to the design of gamified elements to avoid overshadowing the learning objectives or creating superficial motivation. Balancing game elements with the authenticity of the assessment tasks is crucial to ensure meaningful and effective learning outcomes.

By addressing these challenges, educators can harness the benefits of authentic assessments or simulations in the context of AI. This approach provides students with valuable opportunities to apply their knowledge and skills in authentic contexts, fostering deeper understanding and higher-order thinking. It also helps students develop critical thinking abilities, problem-solving skills, and adaptability, which are essential in navigating the complexities of AI-driven environments.

Programme assessment review

Implementing a combination of the three methods mentioned earlier—providing a range of assessment types, eliminating deadline bunching, and enhancing feedback engagement—can yield several benefits in assessment design. By offering diverse assessment formats, such as essays, projects, and presentations, students' varied skills and interests can be catered to, resulting in increased motivation and engagement. Moreover, removing deadline bunching allows students to allocate more time and effort to each assignment, promoting deeper learning and reflection.

Furthermore, incorporating effective feedback mechanisms, such as timely and constructive feedback, peer feedback, or self-assessment opportunities, can enhance students' engagement with feedback, enabling them to identify areas for improvement and make meaningful revisions. This iterative feedback process contributes to the development of critical thinking and self-directed learning skills.

While this approach offers numerous advantages, it is important to acknowledge the potential challenges, particularly the time required for reviewing and planning. Implementing a variety of assessment types and providing detailed and meaningful feedback may demand significant investment of time and effort from instructors. Adequate planning, allocation of resources, and support from educational institutions can help alleviate this challenge and enable educators to effectively implement the approach.

Exam driven approach

Relying predominantly on exams as an assessment approach offers certain advantages, such as assessing students' individual understanding and knowledge without direct reliance on AI. Exams can provide a reliable measure of students' retention and comprehension of course material within a specific time frame. However, it is essential to acknowledge the limitations associated with an exam-driven approach.

Exams, by their nature, tend to primarily measure specific types of intelligence, such as memorisation and regurgitation of information, while potentially overlooking other valuable skills and competencies, such as critical thinking, problem-solving, and creativity. This narrow focus may not fully capture the diverse range of talents, perspectives, and capabilities that students bring to the university setting.

Furthermore, an exam-heavy assessment strategy may not effectively cater to the needs of all students, particularly those with different learning styles, varied backgrounds, or diverse abilities. It may inadvertently lead to inequitable outcomes and hinder inclusive education, as students from marginalised or underrepresented groups may face additional barriers in demonstrating their true potential through this limited assessment approach.

To mitigate these concerns, it is crucial to adopt a balanced assessment strategy that combines different methods, such as authentic assessments, projects, and group work, alongside exams. This approach can provide a more comprehensive and inclusive evaluation of students' knowledge, skills, and abilities. By embracing a broader range of assessment formats, educators can promote student engagement, critical thinking, problem-solving, and collaboration, fostering a more holistic and equitable learning environment.

While exams can be an important component of the assessment portfolio, supplementing them with other forms of assessment allows for a more comprehensive evaluation and better aligns with the diverse needs and strengths of the student body. A thoughtful and inclusive assessment design that goes beyond exams can better support the development and success of all students in their academic journey.

Example of an assessment design

This assessment is by Bearman and Luckin in Preparing University Assessment for a World with AI: Tasks for Human Intelligence, in the book Re-imagining University Assessment in a Digital World 2020. This chapter proposes that assessment should focus on the development of higher order knowledge and skills, and this needs to happen over time rather than one discreet task.

Example: Critically Appraising Scientific Papers in the Health Sciences

Assignment 1: Critically appraise a paper using a checklist

Assessment task is to discuss how knowledge is constituted and what a machine could do and why.

“Examples of checklist items might be: is the trial randomised? if so, is it ‘double blinded’? and so on. The student may work through the checklist and see if they can see these features. In doing so, they are learning to identify scientific knowledge and its markers of study design quality. However, this assignment doesn’t ask a student yet to do anything an AI can’t do. This first part of the assignment is about rules-based processing. However, the second part of the assignment targets building personal epistemologies and evaluative judgement.”

Assignment 2: Critically appraise a paper using a checklist and compare your appraisal with the work of peers and experts

Assessment task is to identify differences between perspectives and reasons for them and consider how you might change your own perspective and why

“In the second half of the assignment, we propose adding a reflective task. The reflection should be framed by a primer which outlines the differences between following the rules and understanding the content. This brief should also outline that, while graduates need a degree of competency with respect to rules, following rules and recognising patterns are areas where AIs should be able to assist. On the other hand, the human intelligence contribution is to be found within a series of reflective prompts on completion of the checklist itself. These prompts could commence with formative reflections: What do I learn from this paper and how? What do I learn from this task and how? These might feed into two gradable reflections: What parts of this task are based on rules? What sort of interpretation is required and why? Another possibility is to ask students: what part of the task you have just done might be performed by machines and why?

Assignment 3: Critically appraise a paper using a checklist and compare your appraisal with peers.

Assessment task is to outline how you would change the checklist and why

“As before, students have a rules-based checklist, and a personal reflection, exploring how they made meaning of the task. This time the task is increased to take account of social intelligence. The students are asked to compare their checklist and their reflections with that of peers and experts. The gradable tasks might be: (a) identifying differences between self, peer and expert work; (b) articulating reasons for these differences; and (c) considering how they would rework their checklist and reflections and why. This type of assessment deliberately builds evaluative judgement, which necessarily develops a deeper understanding of both the content and of the foundations of disciplinary knowledge. It is also possible to include AIs as a reference point; for example, in the future, the students could compare their checklists with those of an AI and peers and experts. In the third year, the task might look quite different again. Students might be given a range of papers and a checklist. Tasks now should ask students to critique and compare but then subsequently generate work. For example, students may be asked to critique the checklist itself; to propose alternative items and any doubts or ambiguity associated with these changes. In this way, they come both to understand what the standards are and to think about the concept of standards in general.”