Reproducible Analytical Pipelines (RAP): bronze-silver-gold framework

Modern tools such as R or Python are preferred over tools such as SPSS, Stata or SAS because they are better at reading a range of different data formats, support modularised code, and can be used to automate the creation of outputs in a range of different formats (for example, markdown, HTML, Word, Excel and PowerPoint files, which will become important for meeting the silver standard principles later on). They also reduce (or eliminate) the number of times where data needs to be moved from one programme into another (for example, from SPSS into Excel into Word).

Open source software, such as R and Python, are also better for transparency compared to other proprietary software where licenses are required to run the code. Those licenses can also be expensive, whereas there are no costs involved for most applications of R and Python.

The use of open source software is also encouraged by the Technology Code of Practice.

UKHSA and other health-sector colleagues might want to refer to this handy guide: R for applied epidemiology and public health.

Minimising the number of manual steps through automation helps reduce the risk of human error and improves efficiency.

Data extraction and analysis should be automated as much as possible, recognising that some manual steps may be required in some situations (for example, where direct connections to databases are not possible).

Automation should ideally be done in R or Python (see the “use open source analytical software” principle) and various other principles in the framework help with how to write good code (for example, including good documentation, using functions, applying good code style).

While automation is encouraged, this does not mean that the entire pipeline needs to be run in one single step. For example, trying to pull and shape data and produce outputs in the same line(s) of code might make it difficult to debug errors. It can also make it difficult for other analysts to follow what you are doing or reuse parts of your code for similar projects. Your project should therefore instead be structured in a clear and sensible way.

Simply being reproducible does not mean that you are doing your analysis correctly. It is important to quality assure each stage of the process, not just the end product (though this of course also remains important!). Quality assurance should align with the principles outlined in the Aqua Book, and the Code of Practice for Statistics for those who have adopted the code.

Please refer to our more detailed guidance on quality assurance for more information.

Checks should be built in at various points throughout the code to flag anything unexpected for further (human) exploration. These might print warning messages into the console for example, or output more detailed markdown files flagging where potential issues might exist. Note that while many quality assurance checks can be automated, some human input will always be needed, such as for final proof readings and checking that automated text still matches the figures.

Broader, more manual checks should also take place, such as checking that the analysis and outputs meet user needs and checking that any assumptions about the model or method are likely to be true. Code reviews are also advised.

To formalise your quality assurance processes and to keep an audit trail, it is a good idea to have defined roles (particularly the “assurer” and “analyst” roles), and keep detailed QA logs. Both of these are described within our guidance on quality assurance.

Having good quality documentation accompanying your code is important for reproducibility. The simplest form of documentation is code comments. These can be supplemented with additional documentation such as a “README” file. Standard Operating Procedures (SOPs), QA logs, assumptions logs are other good methods of documentation.

Good documentation will help improve the reuse of code, either by other analysts using it for the first time or by the same analyst using it in the future. It should explain what the code does and how it can be used, including what options or parameters are available, examples of how to run it, and an explanation of any software or environment dependencies. It should ideally be embedded as part of the project, rather than being saved elsewhere, so that it can be easily found. If it is saved elsewhere, it should be clearly linked from within the project so that analysts are aware of it and where to find it.

There are no set “requirements” for what documentation you should have in place as it will be very dependent upon the needs of each project. A good test, however, is to think about whether someone else would be able to confidently pick up, understand and run your project if you were suddenly absent from work (for example, due to sickness). If you think it would be difficult for someone to do that, then you likely need to improve your documentation.

It is a good idea to make code available to others because this means that the pipeline can be reproduced should the usual analyst go on annual or sick leave, or leave the organisation. It should be stored in a place that is accessible to others (in other words, not just on a local drive that nobody else can access).

Keeping your project in a repository such as GitHub (ensuring others know where to find, and have permissions to access this!) can make it easy for others to pick up your work and also provides other benefits outlined in the “use version control” principle to follow. But otherwise the project should be kept on a shared network drive.

Ideally, the code should be made publicly available for transparency, but this may not always be appropriate. Please adhere to any organisational policy on this.

Version control is important for documenting what changes have been made when, why, and by whom, as well as helping to ensure that people running your code know that they are using the right version. It can act as a safety net, allowing you to roll back to earlier versions of code in the event of an issue, or to review how an earlier report was produced.

GitHub (or GitLab) can be used as a central repository to store your code and version history, allowing it to be easily accessed by others. Sharing code via GitHub can also help avoid duplication of work when other analysts are looking to perform similar tasks. It can also be used as a place for analysts to be open about areas they have identified for improvement via an issues log, also inviting ideas from others.

Private repositories can be used for internal use only, or public repositories can be used to make your code more openly available. Please adhere to any organisational policy on creating open repositories.

It is important not to commit anything sensitive to version control repositories, such as datasets, secret keys and credentials. Datasets should be excluded to avoid disclosing any personal information. It is recommended that keys and credentials should be saved in a separate file that can still be used within the project, but which is excluded from the repository. Pre-commit hooks, such as those included within the cross-government “govcookiecutter” tool, or .gitignore exclusions can help mitigate this risk.

Remember that even when sensitive information is included deep within the version history, these can be checked out and viewed! So you should adhere to these security principles right from the start of the project, or otherwise create a new repository with a clear version history.

While datasets are usually not committed to repositories, it is still important to ensure that some form of version control is applied to the source data, so that others know what version of the data was used for the analysis. If your project relies on snapshot CSV files, rather than connecting to a database directly, it is good idea to store those files in a read-only folder, to prevent them being modified or deleted; any changes like this could make it difficult to reproduce the same outputs in the future. For these files, it is important to ensure that they are saved prior to any data cleaning taking place, so that that data cleaning process can be documented within the production pipeline.

Even where direct connections to databases are made, it may be a good idea to save a copy of a tidy dataset (the minimum dataset needed to produce an output, rather than the full underlying dataset) at some point in the pipeline to keep a record of the data used, and so that it can also be reused in other projects.

Please adhere to proper information governance procedures when saving datasets, particularly avoiding saving copies of anything that might be considered sensitive.

The purpose of the peer review principle here is to ensure that the analytical pipeline is fully reproducible and that it does indeed appropriately meet the principles outlined the framework.

(Note that you should also do some peer review for quality assurance as part of some of the other principles; the principle here is concerned only with checking that RAP principles have been met.)

It can, for example, involve asking a peer to test whether they can successfully run your code on their machine, checking that documentation is clear and complete, and checking that version control has been appropriately implemented.

A peer review can also provide a useful opportunity to more thoroughly question your pipeline and may help identify areas for improvement that have otherwise not been identified previously (for example, including questions on methodology). This is likely best done by someone not already very familiar with the existing pipeline.

The peer review should not just focus on the bronze standard but should also review silver and gold standard principles where they have been applied. Having this principle in the bronze standard is just about ensuring that a peer review process is in place for all pipelines.

Automating the production of outputs, such as reports, spreadsheets or dashboards, further reduces the risk of human error that might be associated with manual copy-pasting or other transfers of numbers and charts.

Note that this does not mean your outputs should necessarily be entirely automated. In some cases, this might be appropriate; in other cases, manual input may be needed for providing narrative in a report, for example.

There are packages available to help produce reports automatically, such as RMarkdown, Python-Markdown and Quarto.

Spreadsheet workbooks can similarly be produced using the a11ytables package for R or gptables for Python, for example.

Producing outputs in these ways means that you know all elements of the output are in sync, as everything is produced from that single pipeline. With more manual approaches, forgetting to update the charts, for example, may mean that they are no longer in sync with what is presented in the text or tables.

A good test for how well automated your end-to-end pipeline is might be to think whether or not you would be happy to delete your outputs. If you would not want to delete your outputs, because it would take time and energy to re-make them, then your end-to-end pipeline is not fully automated.

Functions are particularly useful when you are repeating the same operations at multiple points in your code. Rather than writing out the same code each time you need to carry out an operation, you can create a function and call that each time instead.

This lessens the risk of error because if you need to change that operation, you only need to change the function definition, not each place in the code where it is run (some of which may otherwise be missed, causing inconsistencies).

Functions also lend themselves nicely to sharing snippets of functional code with others, removing duplication of effort in other projects.

Working to a common code style makes your code neater and makes it easier for other analysts to pick up your code as they will be able to interpret and understand it more quickly.

For R users, the tidyverse style guide is the most commonly used style in the global community; “PEP 8” is an alternative for Python users.

You should also consult the Duck Book for guidance on coding standards, which also includes some handy checklists for quality assuring your code.

Validating your input data is important as the quality of your outputs will necessarily depend on the quality of the inputs. Remember: “garbage in, garbage out”.

This principle is intended to build upon the bronze standard quality assurance principle, introducing a more thorough interrogation of your input data in addition to quality assuring your analytical steps.

Validating inputs in an automated way at the very start of the pipeline allows you to quickly identify and address any quality issues early. You might produce code to check, for example, for missing data, incorrect data types, extreme or outlier values, and whether there have been any unexpectedly large changes in the latest iteration of a longitudinal dataset. Other sense checks can also be performed, depending on the nature of the project. You could have your code produce a markdown document which flags any rows that may warrant further exploration - this may be particularly useful for any reports which are run on a regular basis.

Unit testing is where functions are tested with controlled inputs, to check that expected outputs are returned. This can be a useful method to raise alarms if changes made to code elsewhere in the project, or some other environment change, adversely affects the function’s operation.

The principle behind unit testing is that where the functionality of the smallest units of the code can be guaranteed, it is more likely that the overall project is running as expected. They also encourage analysts to focus their thinking more on quality assurance, and what the purpose of each piece of code is. They also act as another layer of documentation: it tells users what the intention behind a certain piece of code was and what the expected inputs and outputs are.

You should also consider including assertive programming methods into your code, as a way of checking that your functions are operating as expected.

Error handling for functions involves stating how the function should behave when it encounters something unexpected, such as a missing or incorrectly formatted parameter passed through the function call.

Common actions might include stopping the code with an error message, or printing a warning message but allowing the code to continue.

It is sensible to provide documentation for your functions so that others know how to use them.

Aspects of functions to document include its purpose, the parameters required in function calls along with any default values, and expected inputs and outputs. The preferred place to include this documentation would be within package help files and vignettes.

A package more formally captures all the code and documentation required for a project or collection of functions in one place. It also allows for easier sharing of all the functions needed for a pipeline, or indeed the whole pipeline itself, as well as enhanced version control for the project (via package version numbers).

Packages are particularly useful when you have a set of general functions, as converting these into a package can facilitate their use across other projects.

Producing logs of data and analysis checks can help analysts, users and quality assurers to understand whether anything is going wrong with the pipeline, and to understand how and where to respond to any issues. They also act as a useful audit trail, to help diagnose how errors might have been missed, when the do occur.

These logs might include, for example, the outputs of any input data validation, unit tests, other checks, messages and warnings, as well as more manual checks (for example, via QA logs).

Please see our guidance on quality assurance for more information on the kinds of things you might want to check and log.

Continuous Integration (CI) describes the managed process of accepting and integrating changes made by individual analysts into a definitive main version of a project. For example, it includes the management of conflicts and the running of regular (automated) tests to ensure that no bugs have been introduced by recent changes. CI helps project leads to identify and resolve any conflicts and bugs early, and helps keep individual collaborators up to date with developments.

Pull requests can be used in GitHub as part of this purpose, to require a sign-off step before any new code is accepted into the main pipeline. Branches can also be used to work on changes without the affecting the main pipeline until you are ready, for example to build in a new feature or when fixing a bug. GitHub Actions can be used to run automated checks before merging code.

Dependency management is important as your code is ultimately dependent on the specific state of the software used at the time the analysis is run. If the environment differs for another analyst, or for the same analyst in the future (for example, following software and package updates), then outputs may differ.

A simple illustration of this kind of issue is provided by the Turing Way Community: A basic division of 1 by 5 can either return 0 or 0.2, depending on which version of Python is being used (Python 2 defaults to integer division; Python 3 does not).

To combat risks such as these, details of the environment can be stored, such as via the renv package for R or via a Docker container image. These tools save a record of the environment in which the analysis was originally performed, allowing others to restore that same environment.

Workflow management packages such as Targets and Orderly in R can also help manage internal dependencies within the code, handling external data dependencies (data, code, templates, outputs from other modules) and the sequence in which modules of your code need to be run.