HTA in R Manifesto

This manifesto is to foster a common understanding and expression of health economic decision analysis models in R. Many of these principles apply to general R code but we focus, at least in the first instance, on models designed to address specific questions. By using common terminology, data structures and procedural approaches we hope to enhance comprehension and transparency of models. In turn, this will support greater reliability, efficiency and trust. These ideas are in the main generalisable to other programming implementations. This idea has been touched on in e.g. [2]. In the same way for standardising the building of model using the CHEERS checklist [4] we would like to standardise the implementation of these models.

This is a work in progress, living document. Any suggestions or comments please post an issue.

Often, on receipt of some new model code, the steps taken to interrogate it are something like this

What does it look like? -> Does it run? -> Is it correct?

In order to facilitate this process we are essentially looking for a sensible consistency in three areas:

1. Consistent code
2. Data and metadata
   • Standardised input and output data formats
   • Standardised records of where data and other modelling information can be found
3. Testing

Consistent code

An agreed specific submission style, e.g. for all models written in R submitted to NICE, would make things easier to interrogate and combine. This is common across code bases and there are lots of examples eg. for the tidyverse.

An agreed upon list of dependent packages could provide some assurance of the underlying code. Simply this could be only base R packages or packages submitted to CRAN or Bioconductor; Or this could be packages that have near 100% code coverage e.g. indicated via CI.

A list of health economics packages that with their 'trustworthiness' can be compiled e.g. as started here. These could be more formally triaged or vetted in some way, e.g. like the open-review carried-out by rOpenSci. Custom badges could be made, e.g. using badger. D. Spiegelhalter talk about trust in numbers and we could adopt some of these ideas. In particular, he uses a 'star rating' for data to indicate its reliability.

The package versions used in a model can be listed explicitly in a DESCRIPTION file and collected in packrat or renv.

There are other readability and clarity of code issues than just the style mentioned above. This can include how to comment code and structure folders. All functions should be documented. The model folders should contain a README. One option is that a template README is provided to be filled-in. This would detail how to run the model and other key pieces of information.

Data

Standardised input and output data formats

A data package is a simple container format used to describe and package a collection of data (a dataset). Some example can be found here.

An example schema for cost-effectiveness model output could be something like this.

Raw data should not be modified. Changes to the data should be documented and saved as new objects.

Testing

You don't have to understand all the code to know if works so you don't need to be an expert R programmer. Just like in industry, the quality of the code is determined by the testing. I think that all you need is to be sure that it passed a defined set of tests. This ideas has been started here [1].

They make suggestions like:

• Life expectancy test sets
  • the discount rate for QALYs to zero
  • all dis-utilities to zero
  • disease-specific mortality rates to the all-cause mortality rates.
• Quality-Adjusted Life Expectancy
• Total undiscounted intervention costs
• Changes in intervention cost
• Cohort size
  • total remains constant
  • number of patients in each health state in all cycles >=0.
• Sample PSA input means

Testing validity has been discussed in [3].

The collection of test are recorded in a document similar to this example.

Then these are translated to actual tests in the target language, such as the testthat packages examples here.

For tests using random numbers the same seed must be used.

References

[1]: Dasbach, E.J., Elbasha, E.H. Verification of Decision-Analytic Models for Health Economic Evaluations: An Overview. PharmacoEconomics 35, 673–683 (2017). https://doi.org/10.1007/s40273-017-0508-2

[2]: Alarid-Escudero, F., Krijkamp, E. M., Pechlivanoglou, P., Jalal, H., Kao, S. Y. Z., Yang, A., & Enns, E. A. (2019). A Need for Change! A Coding Framework for Improving Transparency in Decision Modeling. PharmacoEconomics, 37(11), 1329–1339. https://doi.org/10.1007/s40273-019-00837-x

[3]: McCabe, C., & Dixon, S. (2000). Testing the validity of cost-effectiveness models. PharmacoEconomics, 17(5), 501–513. https://doi.org/10.2165/00019053-200017050-00007

[4]: Husereau, D., Drummond, M., Petrou, S., Carswell, C., Moher, D., Greenberg, D., … Loder, E. (2013). Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. European Journal of Health Economics, 14(3), 367–372. https://doi.org/10.1007/s10198-013-0471-6