Multiple Evaluators and assignment feedback control

The Assignments tool now features the ability to assign specific graders to students (or groups of students) and options for creating a shared grading workflow incorporating a final instructor approval stage.

These options can be beneficial when managing classes with large teaching teams and as a method by which an instructor can review any/all feedback before its release to students.

Instructors can allocate grading tasks to specific teaching assistants or marker-graders via the Evaluators options available when editing an individual Assignment. These must be configured for each Assignment.

Default settings

By default, all Instructors and Teaching Assistants can evaluate submissions and publish feedback in an Assignment. The features described below can expand, control, and/or restrict the grading workflow, but are entirely optional.

Assign grading responsibility using Evaluators

Edit an Assignment and find the Evaluation & Feedback settings.

Remove all Evaluator settings

Evaluator options can be removed from an Assignment (even if submissions have been received) by returning to Edit Evaluators and removing any selected Evaluators. This will reset to the default grading method, where all members of the teaching team can evaluate submissions and publish feedback.

Manage Publishers to control feedback release

Instructors can designate who can publish any grades or feedback written by the Evaluators. This is helpful for those who want to review marking or have the final say in grades before they are released. Manage Publishers presents three options:

• All evaluators can publish any evaluations (for shared grading with an approval step).

• Evaluators can only publish their evaluation (for restricted grading and release).

• Manually select publishers (to select only those who can review and publish already evaluated work).

Create a grading, review, and release workflow using Publishers

Anyone not designated as a Publisher can provide feedback (per their status as an Evaluator) and will have the Ready for Review button on their grading screen in place of the standard draft and publish options. Click Ready for Review to make the initial feedback available for review by any Publishers.

Those in the Publisher role can view, edit, and then publish the prepared feedback.

Manage Allocations to assign students, Sections, or Groups to specific Evaluators

All Evaluators can see and grade all submissions with the default settings, but Evaluator grading responsibility can be restricted via Manage Allocations. This may be useful in large classes where TAs are responsible for specific Sections or Groups.

Additionally, more than one Evaluator can be allocated to each student for shared or aggregate grading.

The Allocation Options provide both manual and automatic creation methods. It is possible to make changes after random allocation, but applying a new random allocation will override previous selections.

Automatically allocate Evaluators to students

Automatically allocate by choosing from the dropdown menu and clicking Apply Allocation Method. The options are:

• Allocate all evaluators to every learner.

• Randomly allocate 1…2…3 evaluator(s) to every learner evenly (further options become available if there is more than one Evaluator in the Assignment).

Assign one or more Evaluators to each student

Manage Allocations can be used to assign teaching assistants or marker-graders to specific students. This can be done in place of managing TA Enrollment options within the Classlist tool, although restricting TA access to a specific Section is more easily accomplished in that way (and only needs to be set up once).

1. Choose Randomly allocate 1…2…3 evaluator(s) to every learner evenly and then click Apply Allocation Method. The table of Student and Evaluator associations will populate but can be edited.

2. Click Done when satisfied.

Manually match students with Evaluators

1. Do not use the Apply Allocation Method options.

2. Use the checkboxes in the table of student names to associate students to Evaluators.

   • Filters can help organizing students by Sections or Groups, which can be used when assigning TAs or marker-graders to their respective labs/seminars/tutorials.

Remove Allocations

Any allocation method can be undone by returning to the Manage Allocations menu and switching to the Allocate all evaluators to every learner option in the dropdown

Coordinate Multiple Evaluators

If multiple Evaluators have been allocated to each student (which is the default behaviour for both standard Assignment grading and what will happen if Evaluators have been enabled), then Instructors can choose between One shared evaluation and Multiple individual evaluations.

The selection depends on the preferred workflow and the number of Evaluators in a site:

• One shared evaluation provides the standard (and potentially collaborative) grading experience where all Evaluators access a single shared grading and feedback area and then, if applicable, Publishers release this to students.

• Multiple individual evaluations gives a separate grading and feedback section to each Evaluator, who can save their feedback using the Ready to Aggregate button. Publishers then view, combine, and release a single instance of combined feedback. This aggregation is done manually and does not need to be an average of other scores.

Grade collaboratively with aggregation

1. [Evaluators] View a submission, grade, and save feedback with the Ready to Aggregate button.

1. [Publishers] Each Evaluator with completed feedback will get a checkmark [2]. All Evaluators must submit their grading before Publishers can aggregate [3] and there is no way to override this.

2. [Publishers] Go to the Aggregated tab [1] and manually combine grades into a final evaluation. Click Publish to release to students. Only the information from the Aggregated tab is shared with students.

Section membership and responsibility

Notes on the shared grading experience