14 Session 14: Reproducibility and Provenance

14.1 Reproducibility and Provenance

14.1.1 Learning Objectives

In this lesson, you will:

  • Learn how to build a reproducible paper in RMarkdown
  • Review the importance of computational reproducibility
  • Review the of provenance metadata
  • Review tools and techniques for reproducibility supported by the NCEAS and DataONE

14.1.1.1 Reproducible Papers

A great overview of this approach to reproducible papers comes from:

Ben Marwick, Carl Boettiger & Lincoln Mullen (2018) Packaging Data Analytical Work Reproducibly Using R (and Friends), The American Statistician, 72:1, 80-88, doi:10.1080/00031305.2017.1375986

This lesson will draw from existing materials:

The key idea in Marwick et al. (2018) is that of the “research compendium”: A single container for not just the journal article associated with your research but also the underlying analysis, data, and even the required software environment required to reproduce your work. Research compendia make it easy for researchers to do their work but also for others to inspect or even reproduce the work because all necessary materials are readily at hand due to being kept in one place. Rather than a constrained set of rules, the research compendium is a scaffold upon which to conduct reproducible research using open science tools such as:

Fortunately for us, Ben Marwick (and others) have written an R package called rrtools that helps us create a research compendium from scratch.

To start a reproducible paper with rrtools, run:

remotes::install_github("benmarwick/rrtools")
rrtools::use_compendium("mypaper")

You should see output similar to the below:

> rrtools::use_compendium("mypaper")
The directory mypaper has been created.
✓ Setting active project to '/Users/bryce/mypaper'
✓ Creating 'R/'
✓ Writing 'DESCRIPTION'
Package: mypaper
Title: What the Package Does (One Line, Title Case)
Version: 0.0.0.9000
Authors@R (parsed):
    * First Last <first.last@example.com> [aut, cre]
Description: What the package does (one paragraph).
License: MIT + file LICENSE
ByteCompile: true
Encoding: UTF-8
LazyData: true
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.1.1
✓ Writing 'NAMESPACE'
✓ Writing 'mypaper.Rproj'
✓ Adding '.Rproj.user' to '.gitignore'
✓ Adding '^mypaper\\.Rproj$', '^\\.Rproj\\.user$' to '.Rbuildignore'
✓ Setting active project to '<no active project>'
✓ The package mypaper has been created
✓ Now opening the new compendium...
✓ Done. The working directory is currently /Users/bryce

Next, you need to:  ↓ ↓ ↓ 
● Edit the DESCRIPTION file
● Use other 'rrtools' functions to add components to the compendium

rrtools has created the beginnings of a research compendium for us. At this point, it looks mostly the same as an R package. That’s because it uses the same underlying folder structure and metadata and therefore it technically is an R package. And this means our research compendium will be easy to install, just like an R package.

Before we get to writing our reproducible paper, let’s fill in some more structure. Let’s:

  1. Add a license (always a good idea)
  2. Set up a README file in the RMarkdown format
  3. Create an analysis folder to hold our reproducible paper
usethis::use_apl2_license(name="Your Name") # Change this
rrtools::use_readme_rmd()
rrtools::use_analysis()

At this point, we’re ready to start writing the paper. To follow the structure rrtools has put in place for us, here are some pointers:

  • Edit ./analysis/paper/paper.Rmd to begin writing your paper and your analysis in the same document
  • Add any citations to ./analysis/paper/references.bib
  • Add any longer R scripts that don’t fit in your paper in an R folder at the top level
  • Add raw data to ./data/raw_data
  • Write out any derived data (generated in paper.Rmd) to ./data/derived_data
  • Write out any figures in ./analysis/figures

It would also be a good idea to initialize this folder as a git repo for maximum reproducibility:

usethis::use_git()

After that, push a copy up to GitHub.

Hopefully, now that you’ve created a research compendium with rrtools, you can imagine how a pre-defined structure like the one rrtools creates might help you organize your reproducible research and also make it easier for others to understand your work.

For a more complete example than the one we built above, take a look at benmarwick/teaching-replication-in-archaeology.

14.1.1.2 Feedback!

Survey request

Please provide us with anonymous feedback via the Survey below.

Reproducible Research for Synthesis Short Course Feedback

We are also collecting participant demographics, but separate from the feedback survey to preserve anonymity.

Reproducible Research for Synthesis Short Course Participant Demographics

Reproducible Research: Recap

Working in a reproducible manner:

  • increases research efficiency, accelerating the pace of your research and collaborations
  • provides transparency by capturing and communicating scientific workflows
  • enables research to stand on the shoulders of giants (build on work that came before)
  • allows credit for secondary usage and supports easy attribution
  • increases trust in science

To enable others to fully interpret, reproduce or build upon our research, we need to provide more comprehensive information than is typically included in a figure or publication.

What data were used in this study? What methods applied? What were the parameter settings? What documentation or code are available to us to evaluate the results? Can we trust these data and methods?

Are the results reproducible?

Computational reproducibility is the ability to document data, analyses, and models sufficiently for other researchers to be able to understand and ideally re-execute the computations that led to scientific results and conclusions.

Practically speaking, reproducibility includes:

  • Preserving the data
  • Preserving the software workflow
  • Documenting what you did
  • Describing how to interpret it all

Computational Provenance

Computational provenance refers to the origin and processing history of data including:

  • Input data
  • Workflow/scripts
  • Output data
  • Figures
  • Methods, dataflow, and dependencies

From Provenance to Reproducibility

At DataONE we facilitate reproducible science through provenance by:

  • Tracking data derivation history
  • Tracking data inputs and outputs of analyses
  • Tracking analysis and model executions
  • Preserving and documenting software workflows
  • Linking all of these to publications

Introducing Prov-ONE, an extension of W3C PROV

ProvONE provides the fundamental information required to understand and analyze scientific workflow-based computational experiments. It covers the main aspects that have been identified as relevant in the provenance literature including data structure. This addresses the most relevant aspects of how the data, both used and produced by a computational process, is organized and represented. For scientific workflows this implies the inputs and outputs of the various tasks that form part of the workflow.

You will recall the example presented of data from Mark Carls: Mark Carls. Analysis of hydrocarbons following the Exxon Valdez oil spill, Gulf of Alaska, 1989 - 2014. Gulf of Alaska Data Portal. urn:uuid:3249ada0-afe3-4dd6-875e-0f7928a4c171., where we showed the six step workflow comprising four source data files and two output visualizations.

This screenshot of the dataset page shows how DataONE renders the Prov-ONE model information as part of our interactive user interface.

Data Citation

Data citation best practices are focused on providing credit where credit is due and indexing and exposing data citations across international repository networks. In 2014, Force 11 established a Joint Declaration of Data Citation Principles that includes:

  • Importance of data citation
  • Credit and Attribution
  • Evidence
  • Unique Identification
  • Access
  • Persistence
  • Specificity and Verifiability
  • Interoperability and Flexibility

Transitive Credit

We want to move towards a model such that when a user cites a research publication we will also know:

  • Which data produced it
  • What software produced it
  • What was derived from it
  • Who to credit down the attribution stack

This is transitive credit. And it changes the way in which we think about science communication and traditional publications.

The 5th Generation

Whole Tale simplifies computational reproducibility. It enables researchers to easily package and share ‘tales’. Tales are executable research objects captured in a standards-based tale format complete with metadata. They can contain:

  • data (references)
  • code (computational methods)
  • narrative (traditional science story)
  • compute environment (e.g. RStudio, Jupyter)

By combining data, code and the compute environment, tales allow researchers to:

  • re-create the computational results from a scientific study
  • achieve computational reproducibility
  • “set the default to reproducible.”

They also empower users to verify and extend results with different data, methods, and environments. You can browse existing tales, run and interact with published tales and create new tales via the Whole Tale Dashboard.

By integrating with DataONE and Dataverse, Whole Tale includes over 90 major research repositories from which a user can select datasets to make those datasets the starting point of an interactive data exploration and analysis inside of one of the Whole Tale environments. Within DataONE, we are adding functionality to work with data in the Whole Tale environment directly from the dataset landing page.

Full circle reproducibility can be achieved by publishing data, code AND the environment.