Multiple Evaluators and assignment feedback control

The Assignments tool now features the ability to assign specific graders to students (or groups of students) and options for creating a shared grading workflow incorporating a final instructor approval stage.

These options can be beneficial when managing classes with large teaching teams and as a method by which an instructor can review any or all feedback before its release to students.

Instructors can allocate grading tasks to specific teaching assistants or marker-graders via the Evaluators options available when editing an individual Assignment. These must be configured for each Assignment.

Default settings

By default, all Instructors and Teaching Assistants can evaluate submissions and publish feedback in an Assignment. The features described below can expand, control, and/or restrict the grading workflow, but are entirely optional.

Edit Evaluators to enable advanced assessment

Instructors can choose who can evaluate a particular Assignment from a list of all Instructors and Teaching Assistants in a site. This can be helpful if grading duties are separated by course section or are to be otherwise exclusively marked by specific individuals.

Anyone not selected as an Evaluator for a specific Assignment will still be able to view all student submissions and details but will not be able to grade or provide feedback.

Evaluators can utilize the standard Submission grading options, including the annotation tools, Rubrics, and Overall Feedback.

Subsequent options will not display unless Edit Evaluators is enabled by clicking and saving, even if no changes are made to the selection.

Open

The Edit Evaluators option within the Evaluation & Feedback menu.

Assign grading responsibility using Evaluators

Edit an Assignment and find the Evaluation & Feedback settings.

1. Click Evaluators, which will initially list all Instructors and Teaching Assistants in the course site. Everyone in these roles can grade any submission to the Assignment.

2. Select Edit Evaluators to choose who will be allowed to grade the Assignment. This can be a subset of the teaching team tasked with marking or the whole team to allow for shared grading.

3. Choose the Evaluators and select Save. All designated Evaluators will now be listed below.

Open

All Evaluators are listed once configured.

Remove all Evaluator settings

Evaluator options can be removed from an Assignment (even if submissions have been received) by returning to Edit Evaluators and removing any selected Evaluators. This will reset to the default grading method, where all members of the teaching team can evaluate submissions and publish feedback.

Manage Publishers to control the release of feedback

Instructors can designate who can publish any grades or feedback written by the Evaluators. This is helpful for those who want to review marking or have the final say in grades before they are released. Manage Publishers presents three options:

• All evaluators can publish any evaluations (for shared grading).

• Evaluators can only publish their evaluation (for restricted grading and release).

• Manually select publishers (to select only those who can review and publish already evaluated work).

Create a grading, review, and release workflow using Publishers

Anyone not designated as a Publisher can provide feedback (per their status as an Evaluator) and will have the Ready for Review button on their grading screen in place of the standard draft and publish options. Click Ready for Review to make the initial feedback available for review by any Publishers.

Those in the Publisher role can view, edit, and then publish the prepared feedback.

Manage Allocations to assign students, Sections, or Groups to specific Evaluators

All Evaluators can see and grade all submissions with the default settings, but Evaluator grading responsibility can be restricted via Manage Allocations. This may be useful in large classes where TAs are responsible for specific Sections or Groups.

Additionally, more than one Evaluator can be allocated to each student for shared or aggregate grading.

The Allocation Options provide both manual and automatic creation methods. It is possible to make changes after random allocation, but applying a new random allocation will override previous selections.

Automatically allocate Evaluators to students

Automatically allocate by choosing from the dropdown menu and clicking Apply Allocation Method. The options are:

• Allocate all evaluators to every learner.

• Randomly allocate 1…2…3 evaluator(s) to every learner evenly (further options become available if there is more than one Evaluator in the Assignment).

Assign one or more Evaluators to each student

Manage Allocations can be used to assign teaching assistants or marker-graders to specific students. This can be done in place of managing TA Enrollment options within the Classlist tool, although restricting TA access to a specific Section is more easily accomplished in that way (and only needs to be set up once).

1. Choose Randomly allocate 1…2…3 evaluator(s) to every learner evenly and then click Apply Allocation Method. The table of Student and Evaluator associations will populate but can be edited.

2. Click Done when satisfied.

Manually match students with Evaluators

1. Do not use the Apply Allocation Method options.

2. Use the checkboxes in the table of student names to associate students to Evaluators.

   • Filters can help organizing students by Sections or Groups, which can be used when assigning TAs or marker-graders to their respective labs/seminars/tutorials.

Remove Allocations

Any allocation method can be undone by returning to the Manage Allocations menu and switching to the Allocate all evaluators to every learner option in the dropdown.

Coordinate Multiple Evaluators

If multiple Evaluators have been allocated to each student (which is the default behaviour for both standard Assignment grading and what will happen if Evaluators have been enabled), then Instructors can choose between One shared evaluation and Multiple individual evaluations.

The selection depends on the preferred workflow and the number of Evaluators in a site:

• One shared evaluation provides the standard (and potentially collaborative) grading experience where all Evaluators access a single shared grading and feedback area and then, if applicable, Publishers release this to students.

• Multiple individual evaluations gives a separate grading and feedback section to each Evaluator, who can save their feedback using the Ready to Aggregate button. Publishers then view, combine, and release a single instance of combined feedback. This aggregation is done manually and does not need to be an average of other scores.

Collaborative grading with aggregation workflow

1. Evaluators view a submission, grade, and save feedback as normal, then click Ready to Aggregate.

2. Publishers will see each piece of completed feedback flagged with a checkmark [2]. All Evaluators must submit their grading before Publishers can aggregate [3] and there is no way to override this.

3. Publishers use the Aggregated tab [1] and manually combine grades into a final evaluation. Click Publish to release to students. Only the information from the Aggregated tab is shared with students.

Notes on the shared grading experience