What issues do you need to consider when choosing a tool to administer during an assessment?

MAPP, CAAP, C-BASE, CLA, NSSE, to name just a few. With so many published assessment instruments available, how do you know the best one for assessing student learning at your school?

“In order to choose the right assessment instrument, you have to ask the right questions,” says Linda Suskie, vice president of the Middle States Commission on Higher Education.

In the recent seminar, Choosing a Published Instrument to Assess Student Learning, Suskie discussed the benefits of published instruments and outlined the important questions to ask when evaluating assessment tools for your school. The important thing to remember, she says, is there is no one instrument that will match the goals of every higher education institution so you have to do your homework to find the best one for you.

Questions to Ask When Evaluating Published Assessment Instruments

Does it match our needs? Review the assessment instrument’s stated purpose and objectives and make sure they align with your philosophy and goals as well as what you teach. Also make sure that the results of the test include actionable information.

Is it practical to administer? Contact the publisher to find out how much the test costs and exactly how much time is required for planning and administering the assessment. Tests that must be taken in one two-hour sitting or require a computer lab can create logistical complications for campuses where most courses are 50-minutes long.

Is there any cultural or gender bias? Although any reputable publisher goes to great lengths to ensure a bias-free test, it’s always a good idea to examine sample questions and past editions through the lens of your students’ experiences for offensive material or words that might not have common meaning.

Does it truly measure what it purports to measure? This is where you dig into test validity. Do test scores have appropriate correlations with grades as well as other tests? Do subscores have appropriate intercorrelations?

Are scores consistent? Score consistency in test/re-test scenarios as well as between forms will tell you how reliable this particular assessment tool is.
[report_ofie=1471]
Are norms adequate and appropriate? Because the quality of your scores will depend on the quality of the norm groups, you need find out all you can about the norm groups. How were the colleges and students selected? How many colleges and students participated? Do the norms represent your institution and the group of students who will take the test?

What do the score reports look like? Test results, and what you do with them, are at the heart of any initiative to assess student learning. Find out what kind of reports you will receive from the publisher. How useful is the feedback? Is it easy to understand and share with faculty? Will they tell you what you are doing right as well as what you’re doing wrong?

A good source for identifying potential instruments to review is the Buros Institute of Mental Measurements Yearbooks and Tests in Print.

Post Views: 2,403

Several attempts to define good assessment have been made. There is a general agreement that good assessment (especially summative) should be:

• Valid: measures what it is supposed to measure, at the appropriate level, in the appropriate domains (constructive alignment).
• Fair: is non-discriminatory and matches expectations.
• Transparent: processes and documentation, including assessment briefing and marking criteria, are clear.
• Reliable: assessment is accurate, consistent and repeatable.
• Feasible: assessment is practicable in terms of time, resources and student numbers.
• Educational impact: assessment results in learning what is important and is authentic and worthwhile.

The aspect of authenticity is an important one. Authentic assessment can be defined as:

'An assessment requiring students to use the same competencies, or combinations of knowledge, skills, and attitudes that they need to apply in the criterion situation in professional life.'

Gulikers, Bastiaens, and Kirschner, (2004, p. 69)

Hence it puts emphasis on being assessed on real life skills through real life tasks that will be or could be performed by students once they leave university. Some examples of how this can be achieved in practical terms can be found in Assessment methods.

Assessment principles

The good assessment principles below were created as part of the REAP Reengineering Assessment Practices Project which looked into re-evaluating and reforming assessment and feedback practice. This set of principles in particular is referred to here as it serves as the basis for many assessment strategies across UK HE institutions. For each of the principles a number of practical strategies are provided which give a more pragmatic indication of how to put them in practice.

The principles below outline good assessment

• Provide clear definitions of academic requirements before each learning task
• Provide explicit marking criteria and performance level definitions. This is an example of marking criteria used on the MEd ULT programme [pdf] and by the Department of Life Sciences [pdf]
• Provide opportunities for discussion and reflection about criteria and standards before learners engage in a learning task
• Ask learners to reformulate in their own words the documented criteria before they begin the task. This could be submitted with the assessment.
• Model in class how you would think through and solve exemplar problems
• Provide learners with model answers for assessment tasks and opportunities to make comparisons against their own work. Imperial College policy is to provide model answers to summative exam questions [pdf]
• Explain to learners the rationale of assessment and feedback techniques
• Before an assessment, let learners examine selected examples of completed assessments to identify which are superior and why (individually or in groups)
• Organise a workshop where learners devise, in collaboration with you, some of their own assessment criteria for a piece of work
• Ask learners to add their own specific criteria to the general criteria provided by you
• Work with your learners to develop an agreement, contract or charter where roles and responsibilities in assessment and learning re defined

• Reduce the size (e.g. by limiting the word count) and increase the number of learning tasks (or assessments). Distribute these across the module
• Make such tasks compulsory and/or carry minimal marks (5/10%) to ensure learners engage but staff workload doesn’t become excessive
• Break up a large assessment into smaller parts. Monitor performance and provide feedback in a staged way over the timeline of your module
• Empower learners by asking them to draw up their own work plan for a complex learning task. Let them define their own milestones and deliverables before they begin. Assign some marks if they deliver as planned and on time
• Provide homework activities that build on/link in-class activities to out-of-class activities
• Ask learners to present and work through their solutions in class supported by peer comments
• Align learning tasks so that students have opportunities to practise the skills required before the work is marked
• Give learners online multiple-choice tests to do before a class and then focus the class teaching on areas of identified weakness based on the results of these tests
• Use a ‘patchwork text’ – a series of small, distributed, written assignments of different types. Each of these are complete in themselves but can also be stitched together through a final integrative commentary
• Award fewer marks for early assessments to allocate all marks for the final synthesis. This format gives learners some choice by allowing them to select which patches to include in the final reflective account
• Have learners undertake regular small tasks that carry minimal marks, with regular feedback
• Provide learners with mock exams so they have opportunities to experience what is required for summative assessment in a safe environment

• Provide opportunities for learners to work through problem sets in tutorials, where feedback from you is available. This ensures that the feedback is timely and is received when learners get ‘stuck’
• Ensure feedback turnaround time is prompt, ideally within 2 weeks
• Give plenty of documented feedback in advance of learners attempting an assessment, e.g. a ‘frequently occurring problems’ list
• Give plenty of feedback to learners at the point at which they submit their work for assessment. This feedback might include a handout outlining suggestions in relation to known difficulties shown by previous learner cohorts supplemented by in-class explanations. Learners are most receptive to feedback when they have just worked through their assessment
• Ensure that feedback is provided in relation to previously stated criteria, as this helps to link the feedback to the expected learning outcomes. An example of a feedback form that helps you achieve that is the Diploma feedback sheet [pdf] used on the MEd in University Learning and Teaching
• Limit the number of criteria for complex tasks; especially extended writing tasks, where good performance is not just ticking off each criterion but is more about producing a holistic response
• Instead of providing the correct answer, point learners to where they can find the correct answer
• Ask learners to attach three questions that they would like to know about an assessment, or what aspects they would like to improve. See this Formative assignment cover sheet [pdf] for an example from Imperial's PG Diploma in University Learning and Teaching
• Ask learners to self-assess their own work before submission and provide feedback on this self-assessment as well as on the assessment itself

• Structure learning tasks so that they have a progressive level of difficulty
• Align learning tasks so that learners have opportunities to practice skills before work is marked
• Encourage a climate of mutual respect and accountability
• Provide objective tests where learners individually assess their understanding and make comparisons against their own learning goals, rather than against the performance of other learners
• Use real-life scenarios and dynamic feedback
• Avoid releasing marks on written work until after learners have responded to feedback comments
• Redesign and align formative and summative assessments to enhance learner skills and independence
• Adjust assessment to develop learners’ responsibility for their learning
• Give learners opportunities to select the topics for extended essays of project work
• Provide learners with some choice in timing with regard to when they hand in assessments
• Involve learners in decision-making about assessment policy and practice
• Provide lots of opportunities for self-assessment
• Encourage the formation of supportive learning environments
• Have learner representation on committees that discuss assessment policies and practices

• Review feedback in tutorials. Ask learners to read the written feedback comments on an assessment and discuss this with peers
• Encourage learners to give each other feedback in an assessment in relation to published criteria before submission
• Create natural peer dialogue by group projects. Structure tasks so that the learners are encouraged to discuss the criteria and standards expected beforehand, and return to discuss progress in relation to the criteria during the project
• Use learner response systems to make lectures more interactive
• Facilitate teacher-learner feedback in class through the use of in-class feedback techniques
• Ask learners to answer short questions on paper at the end of class. Use the results to provide feedback and stimulate discussion at the next class
• Support the development of learning groups and learning communities
• Construct group work to help learners to make connections
• Ask learners to set tasks for each other
• Encourage the formation of peer study or create opportunities for learners from later years to support or mentor learners in early years
• Link modules together as a pathway so that the same learners work in the same groups across a number of modules
• Require learners in groups to generate the criteria used to assess their projects
• Ask learners, in pairs, to produce multiple-choice tests, with feedback for the correct and incorrect answers

• Create a series of online objective tests and quizzes that learners can use to assess their own understanding of a topic or rea of study
• Ask learners to request the kind of feedback that they would like when they hand in their work - example worksheet
• Structure opportunities for peers to assess and provide feedback on each other’s work using set criteria
• Use confidence-based marking (CBM). Learners must rate their confidence that their answer is correct. The higher the confidence the higher the penalty if the answer is wrong
• Use an assessment cover sheet with questions to encourage reflection and self-assessment. Ask learners to make a judgement about whether they have met he stated criteria and estimate the mark they expect
• Directly involve learners in monitoring and reflecting on their own learning, through portfolios
• Ask learners to write a reflective essay or keep a reflective journal in relation to their learning
• Help learners to understand and record their own learning achievements through portfolios. Encourage learners to link these achievements to the knowledge, skills and attitudes required in future employment
• Ask learners, in pairs, to produce multiple-choice tests over the duration of the module, with feedback for the correct and incorrect answers

• Give learners opportunities to select the topics for extended essays or project work, encouraging ownership and increasing motivation
• Give learners choice in timing with regard to when they hand in assessments – managing learner and teacher workloads. Particularly appropriate where students have many assignments and the timings and submissions can be negotiated
• Require learner groups to generate criteria that could be used to assess their projects
• Ask learners to add their own specific criteria to the general criteria provided by the teacher. Take these into account in the final assessment
• Ask learners, in pairs, to produce multiple-choice tests with feedback for correct and incorrect answers, which reference the learning objectives. Let the rest of the class take these tests and evaluate them. These could be used in final assessment

• Request feedback from one minute papers (see formative assessment strategies) where learners carry out a small assessment task and hand it in anonymously at the end of a class. Use the information from these to inform teaching in the next class
• Have students request the feedback they would like when they make an assignment submission
• Provide opportunities for frequent low-stakes assessment tasks with regular outputs to help you gauge progress
• Use online tools with built-in functionality fir individual recording and reporting – providing information about levels of learner engagement with resources, online tests and discussions
• Use learner response system to provide dynamic feedback in class. The stored data provides information about responses, which can be analysed
• Provide opportunities for learners to self-assess and reflect on their learning. A record of these reflections provides information about the learners ability to evaluate their own learning
• Request feedback from learners on their assessment experiences in order to make improvements
• Carry out a brief survey mid-term or mid-semester while there is time to address major concerns

What considerations must be made for choosing an appropriate assessment tool?

What are two things that you need to consider when conducting administering assessments?

What are the major issues in assessment?

What determines the choice of assessment tool to be used in assessing learning outcome?