Iain Hrynaszkiewicz & Cameron Neylon
The PLOS OSI dashboard
In March 2026, working with Cameron Neylon, PLOS released a prototype dashboard enabling exploration of its Open Science Indicators (OSI) dataset. The dashboard prototype is itself an experiment. PLOS has been producing OSI in partnership with DataSeer since 2022, regularly updating and releasing the OSI data and methods in static formats. The reuse of PLOS OSI materials by metaresearchers, funders, and other stakeholders in the research community demonstrates that the appetite for open science measurement solutions extends far beyond PLOS. Now that there is a growing sense of the importance of open science monitoring in global policy – driven in part by the efforts of OSMI – PLOS increasingly saw a need for a format for OSI that enables more people to use and understand the trends in the data, a dashboard. PLOS is tracking quantitative measures of usage of the dashboard in the short term, but success for the dashboard and data will ultimately be whether we observe an increase in the use of the PLOS OSI data, results or methods in a way that supports useful open science policy or practice discussions by other organisations.
Self assessment against the OSMI principles
As initiators and contributors to the Principles of Open Science Monitoring – a framework for creating open science monitoring solutions that themselves align with open science principles – creating the dashboard has been an opportunity to use the Principles in practice. We assessed the PLOS OSI and the dashboard according to both the OSMI Principles, and against PLOS’ own OSI principles (a simpler internal framework that was established at the beginnings of the PLOS OSI initiative). The full assessment led by Cameron Neylon is available in Zenodo and important themes are discussed below.
What’s going well? What could be better?
Openness
Openness should be a tenet of open science monitoring. All the data underlying the PLOS OSI dashboard are openly available, along with the queries one needs to reproduce it. The OSI data that came before the dashboard are supported by open data and detailed methods information to understand and scrutinize how the data and tools are created, and their accuracy.
The OSI dashboard and data are compatible with both sets of principles but do not achieve all the OSMI Principles’ – intentionally aspirational – goals, such as for completely open source software. PLOS OSI was created in the first instance due to a time-bound strategic need of one organisation, PLOS, with limited resources and so, like the OSMI principles, open source is, pragmatically, a strong preference rather than a hard requirement. Nevertheless, DataSeer has made a good proportion of the code and software used to generate the data underlying the OSI dashboard openly licensed for reuse. DataSeer started as a grant-funded project that shared some open source tools and since 2025 the current pipeline used to calculate data and code generation, and data and code sharing, in the PLOS OSI data have additionally been made openly available.
Responsible use
Producing novel indicators and following the principles means thinking upfront about what we really want to know (the INORMS-SCOPE framework is a useful tool for this), and what could go wrong from measurement (or what we often call “unintended consequences”). Inadvertently recreating global rankings based on OSIs was a prime concern – PLOS does not provide institutional and funder information with the OSI article-level data in part to support the responsible use principle. But, it was abundantly clear in testing of earlier prototypes of the dashboard that without a way for a user to see results for a specific research organization, funder, or country, use of the dashboard would be limited. To support one of the most important use cases the dashboard enables filtering on these characteristics – to the extent allowable by openly available metadata in OpenAlex – but comparisons are limited to five entities to promote responsible use.
Scalability
While the assessment notes that the approach to the dashboard scales well with this current technology – and has been previously demonstrated at initiatives such as at Curtin Open Knowledge Initiative – the production of the underlying data is currently limited by the resources of one organization, PLOS, and the capabilities of its partners. PLOS, and no doubt others, would love to see a much larger open corpus of information on open science practices available for reuse. For open science monitoring to be sustainable at a larger scale, we need other publishers to also share similar indicators for content they publish. This requires greater collaboration including shared definitions and standards for measurement, which OSMI working group 3 seems well placed to help enable.
The increasing use of Large Language Models (LLMs) to conduct automated analysis of research outputs presents challenges and opportunities for scale, sustainability, and openness. This is evident in the adoption, in 2025, by DataSeer and PLOS of an LLM-based approach to measure preprint detection. The LLM-based approach outperforms previous methods in terms of completeness and accuracy, but these models can come with increased costs in terms of their operation, and in how easy it is to make them openly available.
Measuring what matters
The research article remains the unit of analysis to detect open science practices in PLOS OSI, and the journal article is increasingly recognised as inhibiting better ways of understanding the quality, production, and impact of research. Second, the dashboard shows prevalence of open science practices but not, so far, their potential impact, or effects. But in line with the PLOS OSI and OSMI principles, PLOS, DataSeer, and researchers have undertaken exploratory work to explore the association of open science practices in OSI with measures of research impact, including citations (here and here), and data reuse (here).
Assessing the assessment
Our self-assessment is among the first using the OSMI principles (see Grypari, I., & Binchy, A. (2026). National Open Access Monitor – OSMI Principles Self-Assessment – 2025. Zenodo. https://doi.org/10.5281/zenodo.18981620 for another example that takes a different, more narrative approach). In common with many self-assessment processes it can be challenging to judge performance, both because the principles are qualitative in nature but also because we have limited examples to compare to. It will be valuable to gather assessments together and encourage conversations and critical reflections and we have shared this initial attempt in that spirit. Self-assessment without community critique will have a limited impact on improving practice in general, even if it helps refine our individual practice.
The OSMI Principles were a useful set of general points for a qualitative assessment, guiding our attention to the full life-cycle of a monitoring system. They helped to ensure that our self assessment covered all the relevant aspects and in this sense they were very helpful. Some challenges arose as the principles applied to different extents to the dashboard and to the data used as an input to produce it. As noted above, some aspects of the production of the PLOS OSI data is not fully open, even though the final product is. In producing the dashboard we focused on the openness of that process. Future self assessments might benefit from considering the principles across multiple stages of monitoring systems, even where some of the inputs may come from outside sources.
What was particularly valuable was having an external rubric to compare against and consider. The OSMI principles, alongside the PLOS OSI Principles, were a touchstone throughout the process. Many times in the design process we asked ourselves “what do the principles tell us about this graph or functionality” and in doing so we were consistently guided towards better practice and reminded of what spaces there are for future improvement.
Disclosures
CN is an Independent Consultant and was paid by PLOS to help produce the OSI dashboard. IH is employed by PLOS and is a member of the OSMI Steering Board. CN and IH are contributors to the OSMI principles.