How to proceed in OLAT

The course element "Peer Review" is always used in combination with the course element "Task".
The table below shows an overview of how to use the course element "Peer review" in combination with the course element "Task".

Step	Course element	Course owner	Course participants
Step 1: Configuration	Course element "Task"	Configure task (incl. setting an absolute deadline) (course editor: Tab "Workflow")
	Course element "Task"	Create task (course editor and/or course run)
	Course element "Peer Review"	Connect with a course element "Task" (course editor: Tab "Workflow)
		Activate the workflow step "Calibration" (incl. setting a deadline) (course editor: Tab "Workflow) ⁽²⁾
		Activate the workflow step "Peer Review" (incl. setting a deadline) (course editor: Tab "Workflow) ⁽¹⁾
		Activate the workflow step "Expert Review" (incl. setting a deadline) (course editor: Tab "Workflow) ⁽³⁾
		Activate grading (course editor: Tab "Workflow) ⁽⁴⁾
		Create review questions (course editor: Tab "Review criteria") ^(1)(2)(4)
		Configure calibration (course editor: Tab "Calibration") ⁽²⁾
		Create calibration tasks (Course run: Tab "Administration") ⁽²⁾
		Configure grading (course editor: Tab "Grading") ⁽⁴⁾
Step 2: Task	Course element "Task"		Complete task by the deadline
Step 3: Review	Course element "Peer Review"		Definitively complete the calibration tasks by the deadline ⁽²⁾
		Carry out the peer review assignment (Course run: Tab "Administration") ⁽¹⁾
			Definitively complete the peer reviews by the deadline ⁽¹⁾
		Definitively complete the expert reviews by the deadline ⁽³⁾
Step 4: Grading	Course element "Peer Review"		View feedback on one's own essay ⁽¹⁾⁽³⁾
		Finalize assessment (Course run: Tab "Grading") ⁽⁴⁾
		View assessment per course participant (Course run: Tab "All participants") ⁽⁴⁾	View assessment ⁽⁴⁾

(1) If you want to activate the "Peer Review" workflow step. / If "Peer Review" is activated.
(2) If you want to activate the "Calibration" workflow step. / If "Calibration" is activated.
(3) If you want to activate the "Expert Review" workflow step. / If "Expert Review" is activated.
(4) If you want to activate the calculations. / If the calculations are activated.

Step 1: Configuration

If you as the course owner want to use the course element "Peer Review" in your course, you always need a course element "Task".

Configuring the course element "Task"

In the course element "Task", you specify the desired settings. Here, it is important to set an absolute deadline in the future.
It is also possible to add several tasks to the course element "Task".

Configuring the course element "Peer Review"

In the course element "Peer Review" (course editor), you select the appropriate course element "Task" (Tab "Workflow", Section "Task").
In the tab "Workflow", section "Peer Review workflow steps" (course editor), you activate or deactivate the appropriate workflow steps and you configure the corresponding deadlines:
- Workflow step "Calibration"
- Workflow step "Peer Review"
- Workflow step "Expert Review"
- "Calibration"
In the tab "Review criteria" (course editor), you create one or more review questions. Each review question consists of the question itself and at least two answer options. If grading is enabled, each answer option needs a score. In addition, you can demand the reviewers to justify their response. The review criteria are used in the different workflow steps ("Calibration," "Peer Review," and "Expert Review").
In the tab "Calibration" (course editor), you set the number of calibrations tasks, the number of questions that have to be correctly answered for each task, and the number of allowed repetitions of failed tasks.
In the tab "Administration" (course run), you add the calibration tasks. A calibration task always consists of the task itself (one or more files) and the correct answer for each review question.
In the tab "Grading" (course editor), you configure the assessment for each activated workflow step. In the section "Evaluation strategy of the reviews", you define how large the share of each activated workflow step is with regard to the calculation of the final grade.

Step 2: Task

After the configuration and the successful publishing of both course elements, course participants can take over the task in the course element "Task", complete it, and submit it within the absolute deadline.

Step 3: Review

Step "Calibration"

Before the "Calibration" workflow step can be started, the course owner has to create the calibration tasks (see "Configuration").

→ Completing the calibration tasks

The calibration tasks are displayed.
Each calibration task consists of one or more files and the review questions.
All review questions have to be answered, only then the submission of the calibration tasks can be finally confirmed.
Course participants receive a message whether they have completed the calibration tasks within the allowed deviation.
If a repeat has been configured, the calibration tasks with a non-allowed deviation can be repeated by the deadline.

Step "Peer Review"

Before the "Peer Review" workflow step can be started, the course owner has to execute the peer review assignment.
The assignment happens randomly and is limited to the essays submitted on time. After the assignment is finished, course participants can begin the peer reviews. If a start date exists for the peer review, they cannot begin prior to that date.

→ Peer review assignment

Peer review assignment can only take place if in the associated course element "Task"

the configured absolute submission deadline is now in the past, and
there are submitted essays.

Depending on the number of tasks and submitted papers in the linked course element "Task" one or more assignment options become available.

Random assignment
Here the participants are randomly get the defined number of papers to review.
This option needs at least one submitted paper more than the number of peer reviews as configured in the tab "Reviews", section "Peer Review" (course editor mode).
Always assign the same task as the own task
Here the participants get randomly the defined number of tasks to be reviewed, but the tasks to be reviewed have the same task as the own task.
At least two tasks must be defined in the linked course element "Task". In addition, each task needs at least the number of submitted papers as configured in the tab "Reviews", section "Peer Review" (course editor mode).
Always assign the same task but not the own task
Here, participants randomly get the defined number of papers to be reviewed, but the papers to be reviewed do not have the same task as their own task.
At least two tasks in the linked course element "Task" must be defined. In addition, each task needs at least the number of submitted papers as configured in the tab "Reviews", section "Peer Review" (course editor mode). The largest task group must not be larger than the sum of the other task groups.

As soon as these two conditions are met, the course owner can execute the peer review assignment. This assignment cannot be changed afterwards.

→ Completing the peer review

Immediately after the peer review assignment is finished, the course participants know the essays they have to review. If there is a start date for the peer review, the papers will not be displayed until then.
The 3 essays to be reviewed are displayed anonymously. Based on the review criteria, the course participants assess the different essays.
At any time, the course participants can see how far they have progressed in the review process.
At the end, the completion of the peer review has to be finally confirmed.

Step "Expert Review"

All course members having the OLAT roles "Course owner" and "Coach" can carry out expert reviews. They can register themselves as experts and assess the submitted pieces of work. If there is a start date for the expert review, it is only possible to begin with the assessment after this start date. A deadline must be configured for the expert review. This deadline will be announced to the course participants. From this date on, the expert reviews will be accessible to them.

→ Register (and sign out) as an expert

Course owners and coaches can register as an expert for one or more course participants. They can then assess the submitted work within the deadline.
As long as the experts have not finally assessed a paper, they are able to sign out as an expert again.

→ Completing the expert review

After having registered as an expert, course owners and coaches can start to assess submitted pieces of work according to the review criteria.
At the end, the completion of the expert review is finally confirmed.

Overview for course owners and coaches

In the overview and on the detail pages, course owners and coaches can also check at any time where a course participant is in the review process.
The course owners and coaches can also have a look at the individual reviews.

Peer review tasks are assigned anonymously and randomly, but be aware that file names cannot be anonymized.
For example, if course participants include their name in the file name, this will also be visible to their peers.

Step 4: Grading

After the completion of the peer reviews and expert reviews, the course participants can take a look at the feedback on their own essays (formative).
If the grading is activated, the course element "Peer Review" can also be used summatively.

In the "Calibration" workflow step, the quality of the assessment of the calibration tasks is rated.

In the "Peer Review" workflow step, 2 aspects are assessed:

Quality of the peer review:
How accurate is the given assessment of the essays? How did the peers assess the essays, i.e., did the reviewers assess the essays more or less similarly or do large discrepancies exist?
Quality of the work:
How did the peers assess the submitted essay?

In the workflow step "Expert Review", the submitted work is assessed by the expert.

Each activated workflow step is part of the assessment.
The share of each activated workflow step is configured in the evaluation strategy of the reviews (tab "Grading" in the course editor).

Assessment of the calibration

Here, the quality of the assessment of the calibration tasks is evaluated.

→ At the level of the calibration task

As each correct answer is worth one point, the number of questions in the review criteria is equal to the maximum number of points.
For each correct answer, one point is earned. Thus, we can calculate the achieved score for each calibration task.

In the calibration configuration ("Calibration" tab in the course editor), we defined how many correct answers must be achieved for each calibration task. If the course participant has achieved as many or more correct answers it is considered a allowed deviation and he/she will receive points. If the deviation is not allowed , i.e., if the participant has fewer correct answers than the configured number, then he/she does not receive any points.

The RSI (Reviewer Skill Index) is responsible for the weighting of the result and defines the quality of the participants' assessment skills. This weighting is needed to evaluate the subsequent reviews or the quality of the essay. The RSI is the achieved score in percent divided by 100, so we always obtain a number between 0 and 1.

For every time the participant needs a repeat for completing the calibration, a deduction in percent can be configured.

To calculate the final score, we need the Reviewer Skill Index (RSI) as well as the number of repeats including the corresponding deduction. Only in case of an allowed deviation the user will receive points. If the deviation is not allowed, the final score is always 0. In case of a allowed deviation, the final score is equal to the achieved score, reduced by the deduction for repeats.

Finally, we can define the total score. For this, we need the final score and the minimum score needed to pass the calibration (see tab "Grading", section "Calibration" in the course editor).

→ Final RSI, total score and assessment

Now, we can assess the whole calibration.

The final RSI (Reviewer Skill Index) is the average of the received different RSI scores.
The Total Score is the sum of the different final scores.
And the Total Assessment is based on the scores of the different final assessments.

Peer review assessment (assessment of finished reviews)

Here, the quality of the peer review is assessed.

→ At the peer review level

In the review criteria, each answer option available for a review question has a score. For each peer review conducted, the sum of the answer options is calculated. This is the given score.
The maximum score is the sum of the maximum score for each review question.

The average score is the average of all different assessments of a essay, i.e., the score given by the other reviewers for the same essay.

Crucial to this calculation is the deviation. The deviation is the difference between the given score and the average score.
There are 3 possibilities for dealing with the deviation, they were previously defined (tab "Grading", section "Quality of peer Review" in the course editor):

allowed deviation (full credit): in case of an allowed deviation (full credit), the reviewer will receive the full score for each peer review (as previously defined).
allowed deviation (partial credit): in case of an allowed deviation (partial credit), the reviewer will receive the full score for each peer review, reduced by a deduction (as previously defined)
non-allowed deviation: if the deviation is too large, the reviewer will not receive any points

The final score is based on the deviation.

At last, we can define the score. For this purpose, we need the final score and the minimum score necessary to finally pass the peer review (see tab "Grading", section "Quality of Peer Review" in the course editor).

→ Overall score and assessment

Now, we can assess the overall quality of the peer review.

The overall score is the sum of the different final scores.
And the overall assessment is based on the scores of the different final assessments.

Assessment of the essay (assessment based on the reviews received)

Here, the quality of the submitted essay is assessed.

→ At the level of the review received

The score received is the score that I received from my peer for my submitted essay.
The maximum score is the sum of the maximum scores for each review question.

If the calibration was activated, the reviewers' RSI is also visible. It is important for the calculation of the final score.

→ Overall score and overall assessment

For the overall score we differentiate between the calculations with and without calibration.

Without calibration:
- The RSI is not available.
- The total score is the average of all the different final scores received.
With calibration:
- The RSI is available and is taken into account for the calculations.
- A review by a reviewer who has a high RSI is worth more than a review by a reviewer who has a low RSI.

At last, we can define the overall assessment. For this purpose, we need the overall score and the minimum score needed to finally pass the assessment of the quality of the essay (see tab " Grading," section "Quality of the essay" in the course editor).

Expert review assessment

Here, the submitted work is assessed by the expert.

→ At the expert review level.

In the review criteria, each answer option available for a review question has a score. For each expert review conducted, the sum of the answer options is calculated. This is the given score.
The maximum score is the sum of the maximum score for each review question.

At last, we can define the score. For this purpose, we need the final score and the minimum score necessary to finally pass the expert review (see tab "Grading", section "Expert Review" in the course editor).

→ Overall score and assessment.

Now we can evaluate the overall quality of the expert review.

The overall score is the sum of the individual final scores.
And the overall assessment is based on the scores of the individual final assessments.

Overall evaluation

The evaluation strategy defines how large the share of each activated workflow step is.
The total scores of the activated steps are calculated. The sum is equal to the final grade.

Last update: 2024-02-20