What is Open Research?

debruine.github.io/talks/UofG-OR-PGR-2022/

Lisa DeBruine

Key Concepts

Sketch of a 'map' to open research. There is a rainbow over the Glasgow Uni Building. Each stripe is labelled: Open research has many benefits, Visibilty, Transparency & reproducibility, Collaboration, Efficient use of funds, Credit for ideas, Public confidence.

UofG Open Research Video

Replication

An idealized version of the hypothetico-deductive model of the scientific method. Various potential threats to this model exist (indicated in red), including lack of replication, hypothesizing after the results are known (HARKing), poor study design, low statistical power, analytical flexibility, P-hacking, publication bias and lack of data sharing.

From A manifesto for reproducible science 10.1038/s41562-016-0021

Reproducibility

Reproducible: same data, same analysis; Replicable: different data, smae analysis; Robust: same data, different analysis; Generalisable: different data, different analysis

via the Turing Way

Reproducible: A result is reproducible when the same analysis steps performed on the same dataset consistently produces the same answer.
Replicable: A result is replicable when the same analysis performed on different datasets produces qualitatively similar answers.
Robust: A result is robust when the same dataset is subjected to different analysis workflows to answer the same research question (for example one pipeline written in R and another written in Python) and a qualitatively similar or identical answer is produced. Robust results show that the work is not dependent on the specificities of the programming language chosen to perform the analysis.
Generalisable: Combining replicable and robust findings allow us to form generalisable results. Note that running an analysis on a different software implementation and with a different dataset does not provide generalised results. There will be many more steps to know how well the work applies to all the different aspects of the research question. Generalisation is an important step towards understanding that the result is not dependent on a particular dataset nor a particular version of the analysis pipeline.

Reproducibile Code

Of 35 articles published in Cognition with usable data, Hardwicke et al. (2018) found:

only 11 could be reproduced independently
11 were reproducible with the original authors’ help
13 were not reproducible even by the original authors

An analysis by Nuijten et al. (2016) of over 250K p-values reported in 8 major psych journals from 1985 to 2013 found that:

half the papers had at least one inconsistent p-value
1/8 of papers had errors that could affect conclusions
errors more likely to be erroneously significant than not

Hexagon with rainbow borders; text inside reads Applied Data Skills

Why Code? talk expanding on this

Applied Data Skills Microcredential starts 23 January

Open Access

Can be funded via Grant Funding or Publisher Agreements (contact research-openaccess@glasgow.ac.uk)

Key Practices

Preregistration

Haroz (2022) Comparison of Preregistration Platforms

Registered Reports

LEGO minifigs measuring each other, next to a flow chart of the RR process: Develop idea, design study, stage 1 review, collect and analyse data, write report, stage 2 review, publish report.

osf.io/rr/

Ten simple rules for writing a Registered Report

Open Materials/Data/Code

Raccoon face logo

Protocols.io (webinar)

OSF logo

Open Science Framework (OSF)

Shadow logo of a cat head with an octopus tentacle tail

GitHub

Preprints

arXiv logo (word with the X like a chain link)

AfricArXiv logo with Africa made of colourful stripes

bioRxiv logo (word with red R)

edArXiv logo (word with a quill and fountain for the i)

MetaArXiv logo (white Meta on a multicoloured background)

medRxiv logo (word with blue R)

psyArXiv logo (Psi, A, and Chi in red boxes)

earthRxiv logo (padlock where half the round part is a globe)

SocArXiv logo (white word on a black background)

zenodo logo (word on a blue background)

List of 62 preprint servers

Contributorship

Conceptualization
Data curation
Formal analysis
Funding acquisition
Investigation
Methodology
Project administration

Resources
Software
Supervision
Validation
Visualization
Writing (original draft)
Writing (review & editing)

Tenzing: App/Paper

ORCiD

Flowchart showing steps: Authenticate, Collect ID, Display in funding base, Connect to award, synchronize with ORCiD, and synchronize with other systems

ORCID iD: a unique, persistent identifier
ORCID record: organise outputs and grants
ORCID APIs: login and connection

Resources

Resources at UofG

Code of Good Practice
Open Research
- Research Data Management (research-datamanagement@glasgow.ac.uk)
- Open Access (research-openaccess@glasgow.ac.uk)
Moodle Training
Methods & MetaScience

External Resources

Purple globe with UK in green and letters UKRN — UK Reproducibility Network

Grey hexagon with a castle tower insite and the word FORRT below — Framework for Open and Reproducible Research Training

Blue hexagon with a teacup inside and the word ReproducibiliTea below — ReproducibiliTea / @GlasgowTea

Open Research Across Disciplines

ukrn.org/disciplines/

Next Steps

Read Easing into Open Science
Get an ORCiD
Ask your supervisor about open research practices and tools in your areas
Think about open research (especially open data) in the planning stages

Flowchart of main steps and practices. Conceptualisation: journal club, project workflow; Design: Preregistration, Registered reports, data sharing planning; Analysis: Reproducible code; Reporting: transparent writing; Dissemination: Preprints, data sharing — Open Science research practices across the research cycle

Thank You!

QR code for this link — debruine.github.io/talks/UofG-OR-PGR-2022/