Skip to Main Content

Open Educational Resources Collective Publishing Workflow

Conduct Peer Review Using Rubric


Peer Review and OER

During peer review, subject experts read through your textbook and provide critical feedback and suggestions for improvement. Peer review ensures your content is accurate, adequately covers the material and is suitable for classroom use.

Because open textbooks are low-cost and easy to produce, they’re sometimes perceived as low-quality compared to commercial textbooks. Peer review allows you to dispel these notions by ensuring a high-quality product.

Not only does peer review signal to potential adopters that your textbook has passed through a rigorous quality control process, but reviewers often end up adopting texts they’ve reviewed themselves.

Designing a Peer Review Process

When designing your peer review process, you’ll need to consider:

  • who’ll coordinate peer review – the author or project manager
  • your goals for the peer review process
  • when you’ll conduct peer review – chapter-by-chapter, once all the content has been written and edited (pre-publication – most common) or post-publication (less common)
  • whether or not reviews will be anonymous
  • your criteria for selecting reviewers (e.g. expertise, cultural background, etc.)
  • how you’ll recruit reviewers (e.g. personal invitations, expressions of interest, etc.)
  • whether you’ll offer incentives to reviewers (e.g. an honorarium) your review criteria (e.g. peer review rubric)
  • what tools you’ll use for review (e.g. Word, Google Docs, Hypothesis, etc.) the time line for returning reviews.

You’ll need this information for your invitation or call for applications.

Conducting a Peer Review Process

Below is a summary of a the steps involved in conducting a typical peer review process:

  • Prepare a review guide with –
    • project summary links to the textbook or textbook chapters
    • reviewer guidelines
    • questions to guide reviewers’ feedback
    • review deadlines
    • review tools
    • reviewer etiquette
    • compensation (if any)
    • ways reviewers will be credited.
  • Create a call for reviewers that includes –
  • Share the call via: websites and social media (e.g. Twitter, ResearchGate and LinkedIn) professional networks listservs cold calls.
  • Write a peer review workflow so everyone involved understands the onboarding process for reviewers.
  • Keep an updated tracking sheet showing reviewers’ progress and when check-ins have been conducted.
  • Respond to peer review feedback.

Example open textbook peer review process:

Peer Review Roles

Some of the people you’ll need to include in your peer review process:

  • Review coordinator – prepares the review guide, writes calls for reviewers, manages recruitment, tracks progress and relays information between reviewers and the authoring and publishing team
  • Reviewers – (minimum of two) subject-matter experts who provide critical input and suggestions to improve the resource
  • Authors – incorporate reviewers’ feedback, communicate with editors and reviewers directly or through the review coordinator
  • Editors – coordinate with the authors and reviewers or review coordinator about revisions to the book, implement these revisions in some cases
  • Project manager – communicates with or acts as the review coordinator, shares calls for reviewers and project updates.

Peer Review Models for Open Textbooks

There are a number of different peer review models you can adopt for your open textbook:

  • By author invitation – Authors invite and coordinate peers to review their work before publication. This review can be private or public (for example, through review documents that are published alongside the textbook). Reviewers may be compensated for their time if funding is available.
  • Via publisher (e.g. library) – Project managers send the textbook or portions of the textbook to reviewers. Reviewers may be compensated for their time if funding is available. Common turnaround times range from two weeks to one month. The process may or may not be anonymous (blind).
  • Student tested – Some authors test their textbook in the classroom and incorporate student feedback. This allows authors to hear directly from their key audience about what’s working and what isn’t.
  • Open Textbook Library – Instructors who teach at Open Education Network member institutions are invited to review published textbooks using a rubric. The reviews are public and unedited.

Peer Review Rubrics for Open Textbooks

The following example peer review rubric for open textbooks is based on The Open Textbook Library’s Open Textbooks Review Criteria:

Comprehensiveness The text covers all areas and ideas of the subject appropriately and provides an effective index and/or glossary.
Content accuracy Content is accurate, error-free and unbiased.
Relevance/longevity

Content is up-to-date, but not in a way that will quickly make the text obsolete within a short period of time.

The text is written and/or arranged in such a way that necessary updates will be relatively easy and straightforward to implement.

Clarity The text is written in lucid, accessible prose and provides adequate context for any jargon/technical terminology used.
Consistency The text is internally consistent in terms of terminology and framework.
Modularity

The text is easily and readily divisible into smaller reading sections that can be assigned at different points within the course (i.e. enormous blocks of text without subheadings should be avoided).

The text should not be overly self-referential and should be easily reorganised and realigned with various subunits of a course without presenting much disruption to the reader.

Organisation/structure/flow The topics in the text are presented in a logical, clear fashion.
Interface The text is free of significant interface issues, including:
  • navigation problems
  • distortion of images/charts
  • any other display features that may distract or confuse the reader.
Grammatical errors The text contains no grammatical errors.
Cultural relevance The text is not culturally insensitive or offensive in any way. It should make use of examples that are inclusive of a variety of races, ethnicities and backgrounds.

Additional example peer review rubrics for open textbooks:

Attributions

Adapted from:

Peer Review Process Summary’ and ‘Peer Review Process Guide’ in The Rebus Guide to Publishing Open Textbooks (So Far) by Apurva Ashok and Zoe Wake Hyde, licensed under a CC BY 4.0 licence.

Peer Review’ in Authoring Open Textbooks by Melissa Falldin and Karen Lauritsen, licensed under a CC BY 4.0 licence.

Open Textbook Library Open Textbooks Review Criteria’ developed by BCcampus, licensed under a CC BY 3.0 licence.

Review an Open Textbook’ by BCcampus, licensed under a CC BY 4.0 licence.

Assisting Academics with the Peer Review Process

Peer review is an important part of the quality control process for open textbooks. Sitting down with authors and working through the information in the 'For Authors' section of this step is a helpful strategy for ensuring everyone understands why and how peer review is being conducted, who is responsible for coordinating this process and what they should do with the resultant feedback.

Encourage authors to take an active role in selecting, recruiting and acknowledging peer reviewers – even if the library is coordinating the peer review process – and to record reviewers’ feedback along with their own responses and share this documentation with everyone on the authoring and publishing teams.

Planning for the Library's Contribution

Authors will require input from library staff at key points in the production process, across all stages of the workflow. Read through the guides in the workflow to see where library staff will need to work with authors on tasks such as copyright checking, peer review and publishing. Review the authors' time line and make a note of when you might expect to be called on to provide assistance.