Reproducibility problems: what reproducibility problems??

Francis Bacon pointed out in the 17th Century that a lot of ‘science’ was performed by simply reciting the statements of authorities as a way to establish truths. He argued for a robust methodology for observing and recording phenomena and then carefully analysing that to then accept or reject the original hypothesis (Novum Organum, 1620). You’ll recognise this, as it forms the basis for all scientific investigations today. However, where many modern experiments come undone, is when you read a fascinating paper and think ‘that’s worth trying with my particular area of biology’. Then when you try to reproduce the methods you discover that you either have incomplete documented methods and/ or the output you obtain is different, even for something as simple as the positive and negative controls. Both happen far too frequently in science, and especially when it comes to light microscopy, where we find major inadequacies in accurately recording and/or reporting the image acquisition and analysis parameters used. Adding to, and aggravating, this problem is that standardisation in light microscopy is practically non-existent, so even if you have an ‘identical’ microscope to the one reported in the original paper you read, there is absolutely no guarantee that it will function at the same level of parameter specifications.

Bacon’s treatise coincided with the development of microscopes, which have since transitioned from basic, qualitative image-collecting tools to sophisticated instruments capable of automatically acquiring information-rich images that are further processed via advanced image processing and analysis steps to extract quantitative information about the underlying science. This has dramatically increased the potential quantifiability of the data obtained from them, yet we have found ourselves in a position where we have little to no globally defined and accepted standards for how the instruments are calibrated and maintained. I suspect this is mostly due to the length of time they have been around as qualitative instruments, meaning that they are not seen as a new and novel technology and because of this, the question of standardisation has been overlooked. In comparison, the development of other analytical tools (DNA and RNA sequencing, for example) have been accompanied with robust quality checks.

Yet efforts have been underway to standardise manufacture since as early as 1858 (eg Royal Microscopical Society (RMS) standards for objective thread size) and more recently with standards issued by ISO for measurement of various parts of the optical light path in confocal microscopes (ISO 21073:2019). This latter example underlines a push towards helping us achieve a level of defined standards and Quality Control (QC) of a system which will improve the rigour of published results if you know that the data were captured on a system that had conformed to such standards.

Re-usability and Repeatability

In an ideal world, the experiments performed in lab A on microscope 1 would be recorded in such a way that the same data could be captured in lab B on microscope 2. In reality, we are currently very far from this. The majority of papers do not report enough methodology to reproduce even the hardware configuration, and hardly any include enough information for some of the imaging parameters themselves. The obvious answer is to capture all of the relevant metadata with the images, and also to ensure that microscope 1 and microscope 2 are both performing within a defined specification limit, and that this operating specification is also recorded with the imaging data. The benefit to science is huge- far more reproducibility, far less time wasted trying to ascertain how to repeat something, far more trust in reported observations if they can be reproduced, and far more openness in terms of the methodology. Public trust in our work would be vastly improved too.

Anyone that maintains microscopes (or uses them and doesn’t maintain them!) will be thinking ‘that sounds like a lot of extra work just to show my images in a paper!’. From discussions with other scientists and/ or microscopists, there is a vast spectrum of belief in how much of this should be done. However, if we continue with the current state of affairs it is of benefit to no-one, other than those who do not pay much attention to the scientific method and the need to correctly describe your work. The major obstacle from almost all antagonists is that it is too much work and will take too long. Therefore, the key to gaining broader acceptance of such standardisation is to develop defined standards and methods for measuring them as well as the automation of the whole process to reduce work load and avoid human error.

QUAREP-LiMi: An Open, International Initiative

Based upon discussions already started as early as 2005 at the session “Standardisation and Quantification in Microscopy” of the Focus on Microscopy (FOM) conference and again arising at the European Light Microscopy Initiative (ELMI) conferences in 2018 and 2019, combined with initial work on a methods paper outlining how to implement the Confocal ISO, in April 2020, the QUality Assessment and REProducibility for instruments and images in Light Microscopy (QUAREP-LiMi) initiative was formed (https://quarep.org). This initiative comprises experts from academia and industry who share a common interest in achieving a better understanding of the performance and limitations of microscopes and improved QC in light microscopy. The ultimate goal of the QUAREP-LiMi initiative is to establish a set of common QC standards, guidelines, metadata models, and tools, including detailed protocols, with the aim of improving reproducible advances in scientific research. The broad aims of QUAREP-LiMi are outlined in a recent Nature manuscript and a more detailed description of the work being performed in the Journal of Microscopy.

Whilst QUAREP-LiMi was originally initiated around the Confocal Microscope ISO 21073 standard, the scope of QUAREP-LiMi has since increased. It is now devoted to establishing a comprehensive set of shared QC guidelines, tools for their capture, and microscopy metadata specifications for their storage and automated reporting. All of these factors are essential to enable reusable and reproducible imaging data. Such a large range of work has been broken up into Working Groups tasked with defining and implementing methods that conform with any current ISO standards, are easy to implement, and are as open as possible in terms of tools (hardware and software). Each Working Group (WG) proceeds at their own pace, and we hope to highlight their work in future blog posts here as well as on the QUAREP-LiMi website. Examples that are quite advanced include the WG 7 Metadata group (manuscripts submitted for methods to capture and store microscope metadata) and the WG 1 Illumination Power Working Groups. All output from the Groups will be published open access to allow uptake by all microscopists, and the work advertised to encourage wider community involvement and uptake of the finalised QC methods.

A further important push within QUAREP-LiMi is to engage from the outset with publishers and microscope manufacturers to help drive such QC uptake from both an automation and built-in hardware point of view and also from a publishing point of view, encouraging journals to ask for detailed methods and associated metadata to be provided. Both aspects will help broaden uptake of such Standards within the community. Less practical for a global community, but potentially more powerful, would be encouraging the primary funding agencies to insist on these data being captured with raw data that is stored from experiments. Since most funding bodies are national, this is a limiting step, but is something QUAREP-LiMi aspires to.

The output from QUAREP-LiMi has to be discussed and modified together with the major microscopy using communities (MMUC), such as cell-, plant-, developmental-, neuro-biology, physiology and many more. It will need consensus, acceptance, education and training and will take time to become embedded within all MMUC, and will likely require various further refinements along the way. To go back to the RMS thread example, whilst set up in 1858, even in 1896 there were complaints that it was impractical due to the production and dissemination of the adjustable threading taps and dies to all manufacturers. Introduction of solid taps and dies was cheaper, but it was found that they didn’t necessarily conform to the standard. The RMS instead ordered new fixed taps and dies which were given the Society’s stamp if they passed their internal QC. These were then made available for purchase by microscope manufacturers (https://www.sizes.com/library/technology/RMS_thread.htm). In a very similar manner, QUAREP-LiMi engages with software and hardware manufacturers to replicate this level of standard production, allowing easy access to affordable tools and with increased automation facilitating simple use.

Your Community Needs You!

Direct and open engagement of the global imaging community, and public and effective dissemination of QUAREP-LiMi advances, are essential to foster consensus-building and global acceptance of QUAREP-LiMi proposals. Consistent with this goal and its foundational principles, QUAREP-LiMi actively seeks participation from any interested party to join the group and work towards establishing a global QC consensus. Join QUAREP-LiMi as a member.

(5 votes, average: 1.00 out of 1)