Professional Skills: Open Research

debruine.github.io/talks/prof-skills-open-res/

Lisa DeBruine

Key Concepts

UofG Open Research Video

Replication

From A manifesto for reproducible science 10.1038/s41562-016-0021

Reproducibility

via the Turing Way

• Reproducible: A result is reproducible when the same analysis steps performed on the same dataset consistently produces the same answer.

• Replicable: A result is replicable when the same analysis performed on different datasets produces qualitatively similar answers.

• Robust: A result is robust when the same dataset is subjected to different analysis workflows to answer the same research question (for example one pipeline written in R and another written in Python) and a qualitatively similar or identical answer is produced. Robust results show that the work is not dependent on the specificities of the programming language chosen to perform the analysis.

• Generalisable: Combining replicable and robust findings allow us to form generalisable results. Note that running an analysis on a different software implementation and with a different dataset does not provide generalised results. There will be many more steps to know how well the work applies to all the different aspects of the research question. Generalisation is an important step towards understanding that the result is not dependent on a particular dataset nor a particular version of the analysis pipeline.

Error Detection

An analysis by Nuijten et al. (2016) of over 250K p-values reported in 8 major psych journals from 1985 to 2013 found that:

• half the papers had at least one inconsistent p-value
• 1/8 of papers had errors that could affect conclusions
• errors more likely to be erroneously significant than not

Analysis Reproducibility

Of 35 articles published in Cognition with usable data (but no code, Hardwicke et al. (2018) found:

• only 11 could be reproduced independently
• 11 were reproducible with the original authors’ help
• 13 were not reproducible even by the original authors

Code Reproducibility

Of 62 Registered Reports in psychology published from 2014–2018, 36 had data and analysis code, 31 could be run, and 21 reproduced all the main results (Obels et al, 2020)

Key Practices

Open Access

Open access is promoting the early and wide sharing of research.

For research articles this means making the full text freely available on the web. This facilitates discovery and sharing of your work.

We recommend that you consider open access options for journal articles and conference proceedings in advance of submission. The Research Excellence Framework (REF) and many funders have open access requirements and policies.

Can be funded via Grant Funding or Publisher Agreements (contact research-openaccess@glasgow.ac.uk)

Preregistration

Haroz (2022) Comparison of Preregistration Platforms

Registered Reports

osf.io/rr/

Ten simple rules for writing a Registered Report

Open Materials/Data/Code

Protocols.io (webinar)

Open Science Framework (OSF)

GitHub

Ideally, give open materials a permanent reference, like a DOI.

Code Review

The process of methodically and systematically checking over code–your own or someone else’s–after it has been written.

• Is the code is legible and clear?
• Is the analysis reproducible?
• Are other outputs reproducible?
• Does the code do what was intended?
• Does the code follows best practices?

Code Review Slides/Video

Preprints

List of 62 preprint servers

Contributorship

• Conceptualization
• Data curation
• Formal analysis
• Funding acquisition
• Investigation
• Methodology
• Project administration

• Resources
• Software
• Supervision
• Validation
• Visualization
• Writing (original draft)
• Writing (review & editing)

Tenzing: App/Paper

ORCiD

• ORCID iD: a unique, persistent identifier
• ORCID record: organise outputs and grants
• ORCID APIs: login and connection

Resources

Resources at UofG

• Code of Good Practice
• Open Research
  • Research Data Management (research-datamanagement@glasgow.ac.uk)
  • Open Access (research-openaccess@glasgow.ac.uk)
• Moodle Training

External Resources

UK Reproducibility Network

Framework for Open and Reproducible Research Training

ReproducibiliTea

Next Steps

• Read Easing into Open Science
• Get an ORCiD
• Ask your supervisor about open research practices and tools in your areas
• Think about open research (especially open data) in the planning stages

Open Science research practices across the research cycle

Thank You!

debruine.github.io/talks/prof-skills-open-res/