• Section 1 - Introduction
• Section 2 - Compliance
• Section 3 - Intent
• Section 4 - Relevant Definitions
• Section 5 - Procedures
• Number of Units Selected
• Eligible Units
• Unit Selection
• Assessment Selection
• Seeking Calibrators
• Calibrator Selection
• Calibration Package
• Calibration Timeframes
• Review of Reports
• Report Dispute
• Report Implementation
• Reports Relationship to Course Accreditation, and Professional Accreditation
• Calibration Payment
• Recruitment of Calibrators
• Eligibility of Calibrators
• Expectations of Reviewers
• Payment to Calibrators
• Records

Section 1 - Introduction

(1) Academic Calibration is an external peer review process developed by Innovative Research Universities (IRU) that is undertaken in collaboration with participating Universities to support meeting the Higher Education Standards Framework (Thresholds Standards) 2015 requirement that, the academic standards intended to be achieved by students and the standards actually achieved by students in the course of study are benchmarked against similar accredited courses of study offered by other higher education providers.

(2) Academic Calibration was developed by academics and learning and teaching professionals as an efficient and effective process that verifies academic judgements around the assessment standards across the sector.

Section 2 - Compliance

(3) This is a compliance requirement under the Higher Education Standards Framework (Threshold Standards) 2015.

Section 3 - Intent

(4) This document provides instructions to ensure the academic calibration process at the University is comprehensive, consistent, and distributed across the University.

Section 4 - Relevant Definitions

(5) In the context of this document:

1. Academic calibration or calibration means a form of moderation. It is a defined quality assurance process designed by the Innovative Research Universities group, where an appropriately qualified independent individual from another university reviews selected materials and student samples from a single assessment task within a unit to verify grades awarded, and evaluate whether the quality of the unit is comparable to that of similar institutions;
2. Academic nomination means evidence of registered interest from an academic at another University, who has the appropriate qualifications to evaluate or ‘calibrate’ a unit;
3. Assessment task means a component of assessment that is normally based on learning undertaken during the semester. Assessment tasks can be formative or summative and can take many forms such as essays, short answer questions, computer programs;
4. Calibrator means an independent, appropriately qualified, current or former academic staff member of an Australian university who has substantial expertise in the relevant discipline area and has no conflict of interest in accordance with the Conflicts of Interest Policy;
5. Calibration package means the curated set of documents that are sent to a Calibrator to support their evaluation of a unit.  It is a mixture of student samples, and supporting information that relate to the selected assessment task, unit, and related course/s;
6. Calibration report means a standard report designed by the IRU, which captures the Calibrators evaluation of the select unit and assessment task;
7. Capstone unit means a final year unit that purposefully integrates, synthesizes and applies the knowledge and skills acquired throughout the course in a consolidated assessment. This may be a work placement, project, thesis, portfolio or test. Normally, a degree course would culminate in a capstone unit (for a Bachelor this would be at the 400 unit level; for Masters this would be at 600). Vocational and Educational Training (VET) and enabling courses may also finish with a capstone unit;
8. Common unit means a group of units intended to provide an introduction to higher education study and to assist students to develop a satisfactory standard of academic literacy and the capacity to critically explore social and cultural issues of local, national and international relevance;
9. Core unit means those Units that are compulsory for, and directly address the learning outcomes of, the award course;
10. Course means a formally approved/accredited program of learning that leads to the award of a qualification;
11. Coursework means a method of teaching and learning that leads to the acquisition of skills and knowledge that does not include a major research component;
12. De-identify means the process of removing information from student samples provided.  This includes removing student names, marker names, and any information that may identify persons or organisations other than the University;
13. Feedback means in the context of assessment, information returned to students on their progress towards unit/ program of learning outcomes. The information can be quantified in the form of marks or grades for assessment tasks, and/or in qualitative form, for example, comments, model answers and suggestions for reading. All assessment should incorporate both formative and summative feedback for students to use as life-long learners (assessment for learning) as well as assessment of learning to date (assessment of learning);
14. Grade (for example Pass, Credit, Distinction) means a symbol that indicates the level of student performance in a unit against specified standards. Grades are awarded for the purposes of summative assessment, to enable the University to provide a final statement for the student and the outside community of the student's achievement of the learning outcomes in that unit;
15. Innovative Research Universities (IRU) means a policy group consisting of seven (7) Australian universities; Charles Darwin University, James Cook University, Griffith University, La Trobe University, Flinders University, Murdoch University, and Western Sydney University;
16. Learning outcomes means the expression of the set of knowledge, skills and the application of the knowledge and skills a person has acquired and is able to demonstrate as a result of learning;
17. Mark means the points awarded for an individual item of assessment;
18. Rubric means a type of matrix that provides scaled levels of achievement or understanding for a set of criteria or dimensions of quality for a given type of performance, for example, a paper, an oral presentation, or use of teamwork skills;
19. Specialist Elective Units means those Units that are, in addition to the core Units, compulsory for, and directly address, the Major/s and/or Minor/s within an award course;
20. Tertiary Education Quality and Standards Agency (TEQSA) means the established national regulator for the Australian Higher Education System;
21. Unit means a single component of a qualification, or a stand-alone unit, that has been approved/ accredited. A unit may also be called a module, subject, unit of competency or accredited unit;
22. Unit Coordinator means the academic staff member responsible for the unit for the semester it will be reviewed;
23. Unit outline means an official statement describing:
    1. the nature of the unit;
    2. the learning outcomes of the unit;
    3. how the unit is delivered and assessed;
    4. the specific requirements students have to meet in order to complete the unit successfully;
    5. information specific to the unit; and
    6. resources required; and
24. Work-integrated learning (WIL) means a range of approaches that integrate theory with the practice of work within a purposefully designed curriculum. The most common form of WIL involves a student placement or project within a workplace.

Section 5 - Procedures

Number of Units Selected

(6) The required number of units to be calibrated will be determined annually by the University Learning and Teaching Committee, as the amount of units being calibrated will be subject to budget availability.

(7) Additional units can be calibrated by request to the role/s responsible for academic calibration, however the honorarium payment to the Calibrator will be paid by the School or Faculty.

(8) It is recommended that at least one unit from every Undergraduate and Postgraduate coursework degree be calibrated each year.

Eligible Units

(9) Academic Calibration focuses on the selection of units that are indicative of the key elements of a courses’ structure, and have defined learning outcomes and assessment structure. This includes:

1. capstone units – these units take preference in the selection process;
2. core units;
3. specialist electives; and
4. work integrated learning units – these units may be calibrated, but will be assessed for eligibility on a per unit basis to identify complications from commercial in confidence or client-based information. It is the responsibility of the Unit Coordinator to inform any third party about their potential involvement in the calibration process.

(10) The selection of common units will be at the discretion of the University Learning and Teaching Committee.

Unit Selection

(11) Unit selection will be determined through a cyclic communications process that is facilitated by the role/s responsible for academic calibration.

1. Annual reporting will be provided to the University Learning and Teaching;
2. the University Learning and Teaching Committee will review, and make any determinations on any particular focus or distribution method for the upcoming calibration schedule, if any;
3. the Faculty Learning and Teaching Committee will disseminate any direction expressed by University Learning and Teaching Committee. If no focus has been expressed, the Faculty Learning and Teaching Committee may choose determine any focus or methods of distribution for the following calibration schedule;
4. if there is no direct determination from Faculty Learning and Teaching Committee, the Heads of School will make a determination on how units may be selected, or the method of distribution; and
5. the Heads of School will return the listing of selected units to the role/s responsible for academic calibration.

(12) It is the responsibility of the School to inform Unit Coordinators their units have been selected for academic calibration.

Assessment Selection

(13) A single assessment task, or combination of smaller tasks will be selected for review for each unit. When selecting an assessment task for review, priority should be given to assessment tasks that cover a high number of unit learning outcomes, and have a considerable assessment weighting.

(14) It is preferable that assessment tasks can be de-identified without obscuring the ability to evaluate them. Where this is not possible, the Academic Calibration Declaration Form ensures that Calibrators abide by the confidentiality and conflict of interest measures stipulated.

(15) Academic Calibration is designed so that twelve (12) student samples with supporting information can be reviewed and assessed within an eight (8) hour period. Where possible, student samples are to be provided that are considered assessable within this time frame, or an equivalent sample to assessment size ratio of comparable volume. Where there are less then twelve (12) student samples, and minimum of six (6) is allowable.

(16) If a Unit Coordinator requires assistance in the assessment selection process, the role/s responsible for academic calibration will provide a recommendation for approval.

Seeking Calibrators

(17) The role/s responsible for academic calibration will gather information from the Unit Coordinator that can be sent to participating Universities to look for suitable Calibrators. This information includes:

1. Unit Code
2. Unit Name
3. Related Course/s – Unit Coordinators can list all related courses and their information via their URL, or select a ‘dominant course’ based on unit placement in the course, and the amount of participating students
4. School/Faculty
5. Unit URL
6. Assessment task selected
7. Assessment task information, as a URL or summary statement
8. Materials availability date – the date that marked student samples will be available for review. This date may be specific 24/11/2017, or broad ‘Late November 2017’

(18) The role/s responsible for academic calibration will supply the information to corresponding academic calibration representatives at participating Universities to seek appropriate Calibrator nominations.

Calibrator Selection

(19) Calibrators are nominated from multiple institutions, and it is the final decision of the University to make the selection. The role/s responsible for academic calibration will send all relevant nominations to the Unit Coordinator, whose unit is being calibrated, and the relevant Course Coordinator and Head of School to make a selection.

(20) Calibrators are not expected to have currently or previously taught a similar unit, and it is sufficient that the nominated Calibrator is experienced in the discipline and has a well-developed sense of academic standards.

(21) Where there are no suitable academics within the participating Universities, the role/s responsible for academic calibration may look for suitable academics from other universities, which have expertise in the same area as the unit seeking Calibrators.

Calibration Package

(22) Unit Coordinators in consultation with role/s responsible for academic calibration will collate student samples and supporting materials relating to the selected assessment task and unit, which includes:

1. information about the course structure/s of which the unit is a part of;
2. course level learning outcomes, and their relationship to the unit level learning outcomes;
3. the unit outline provided to students;
4. unit learning outcomes;
5. grade distribution of the selected assessment task;
6. grade distribution of the unit for the calibrating semester (optional);
7. selected assessment task detail;
8. marking Rubric for the selected assessment task;
9. context Statement (optional), that describes important features of the unit, or the context of the particular teaching period that may not be immediately obvious through standard documentation; and
10. twelve (12) marked student samples with feedback in the following grade ranges, or as close as possible.

(23) The role/s responsible for academic calibration, is responsible for the overall curation of the calibration package. They will de-identify the student samples prior to submission to the Calibrator. Assistance can be sought from the role/s responsible for academic calibration in the materials collation process.

(24) The inclusion of additional information to support the calibration of a unit can be raised with role/s responsible for academic calibration, who will negotiate with the corresponding University.

Calibration Timeframes

(25) Once the Calibrator receives the calibration package, they have to evaluate all items provided and fill out the academic calibration forms and report within the negotiated time frame. If the calibration package is not made available to the Calibrator within the agreed time frame, the agreed date for the return of the calibration report may be re-negotiated.

Review of Reports

(26) Once the role/s responsible for academic calibration has received the report this will be sent onto the Unit Coordinator, Course Coordinator, and Head of School for review.

(27) All parties have a two-week window to review the report, and supply any feedback or clarifying questions relating to the report. Requests for further information or clarification are to be submitted to the role/s responsible for academic calibration to send to the corresponding university. If no feedback is received within two weeks, it is assumed the report is accepted.

(28) At the end of semester, all reports for that period will be sent for review to the relevant Heads of School and the Faculty Learning and Teaching Committee.

(29) Statistics that do not identify any specific reports will also be made available to the University Learning and Teaching Committee annually.

(30) Calibration Reports also form part of the TEQSA Renewing Registration process, and may be available for TEQSA to review during this period.

Report Dispute

(31) If after questions or clarification the report is still not agreed with, the role/s responsible for academic calibration, the Unit Coordinator, Course Coordinator, and Head of School will review the dispute against the report and internal data that may be available to determine if further action is required.

(32) Disputes against a report should be raised with the role/s responsible for academic calibration. Where the dispute claim is accepted by the calibrating university, a report amendment will be requested by the role/s responsible for academic calibration. Where a dispute continues either:

1. a mediated discussion with the Calibrator will take place;
2. the Calibrator will reassess the unit for another semester or year; or
3. a second Calibrator will be engaged to assess the unit.

Report Implementation

(33) While it is critical that a Calibrator’s feedback is constructive so that it imparts a level of usefulness to the Unit Coordinator, the Calibrator is not expected to remedy any faults apparent in the unit, nor is expected to provide future support or assistance unless agreed between the Calibrator and the Unit Coordinator through their respective academic calibration representatives.

(34) As the calibration report is an additional piece of support data for a unit, it is at the discretion of the Unit Coordinator, Course Coordinator, and Head of School as to how they implement any changes as a result of calibration.

(35) It is recommended that desired unit changes be incorporated within the existing accreditation processes, unless the report indicates that immediate attention to the unit is required.

(36) For any unit changes, the standard Higher Education Minor and Major Unit Amendment Procedures should be consulted.

Reports Relationship to Course Accreditation, and Professional Accreditation

(37) The Course Coordinator and Head of School are responsible for determining whether they are required to submit calibration reports as part of their professional accreditation responsibilities.

(38) It is expected that as part of existing unit and course accreditation processes, any calibration reports that have been submitted in that period would be supplied as documentation to support the process.

Calibration Payment

(39) The Education Strategy will make payment to Calibrators who have reviewed units on behalf of the University.

(40) Additional units that have been calibrated outside of the amount determined by the University Learning and Teaching Committee will be paid by the School or Faculty where the unit is situated.

Recruitment of Calibrators

(41) The University encourages eligible academic staff members to express their interest in becoming a Calibrator for participating Universities by contacting the role/s responsible for academic calibration. As unit selection is not scheduled around discipline there is no guarantee of having a unit to review, but selection will be fast tracked should a relevant unit be nominated by a participating University.

(42) Academic staff members who nominate as Calibrators will be required to provide access to an up-to-date online academic staff profile from the University’s or other website, or a document based Curriculum Vitae.

Eligibility of Calibrators

(43) Staff members must have experience in the selected discipline of the unit seeking Calibrators, as well as extensive teaching experience, including the grading of student assessment work.

(44) Staff members must be at least an Academic Level C or higher, however, an Academic Level B with extensive experience in a specialised area will also be considered.

Expectations of Reviewers

(45) The standard expectations of peer review which often applies to research are still applicable and expected in the teaching and learning context. These include:

1. being fair, and timely;
2. providing feedback that is comprehensive and constructive;
3. acting in confidence and not disclosing the content or outcome of the calibration;
4. being informed about, and comply with, the criteria being applied;
5. declaring all conflicts of interest, including personal prejudice, which may influence the calibration process;
6. giving proper consideration to teaching and assessment that challenges or changes accepted ways of thinking; and
7. not participating in calibrations where the unit is outside the Calibrators area or level of expertise, or where this occurs, declaring the limitations.

(46) The Calibrator will be required to sign the Academic Calibration Declaration Form, which acknowledges the above requirements and acceptance that once reports are returned to the external institution, the Calibrator retains no rights, including copyright and moral rights, in connection with the materials produced for the review.

Payment to Calibrators

(47) The selected method for payment will be determined between the corresponding University and the role/s responsible for academic calibration.

Records

(48) It is expected that when the calibration forms and report is returned, the calibration package supplied to the corresponding University is destroyed.

(49) It is expected that when the calibration is completed, all documents and student samples provided by the corresponding institution are destroyed.

(50) Reports will be retained and stored in the relevant records management system in accordance with the University’s Records Management - Retention and Disposal Procedure.