LJMU Statement on Responsible Research Assessment
This Statement was approved by the University Research and Knowledge Exchange Committee in February 2025.
Opening statement
Together, the Code of Practice for Research and Knowledge Exchange and the Research and Knowledge Exchange Plan 2030 govern and celebrate our inclusive research culture at Liverpool John Moores University. All staff are empowered to develop their research careers and to generate high-quality work of which we can all be proud. It is important for us to be able to monitor the University’s progress at all levels in accurate, equitable, and well-understood ways. In certain situations, and shaped by practice in different disciplines, we recognise that research metrics can be one of several helpful tools for monitoring our progress.
Situations where we anticipate research metrics being used include the annual Research Workload Allocation (RWA) procedure; research group, centre, and institute development and biennial reporting to Faculty- and University-level Research and Knowledge Exchange Committees; Personal Development and Performance Reviews for academic staff; hiring, promotion, and internal funding decisions; and applications and reporting to research funders and other external assessors of proposed or achieved research activity.
This statement is based on the underlying principle that we do not use journal-based metrics, or other metrics related to output venue prominence, as surrogate measures of the research quality. It encompasses the expectations of the San Francisco Declaration on Research Assessment, to which we are an institutional signatory. To generate it, we have drawn on current leading practice, and it is based on the INORMS SCOPE Framework, which offers a clear approach to responsible research assessment. As such, it outlines a set of principles for thoughtful, situation-based assessment design, and not a set of ‘approved metrics’.
SCOPE
Start with what you value
To evaluate, we must know what we value. We expect that all research assessments will be performed in line with our values of being Courageous, caring about our Community, being Inclusive, and being Student Focused. Therefore, research assessments should:
Measure performance against agreed goals and purposes
Shaped by our values, our Research and Knowledge Exchange plan underlines the importance of empowering diverse and inclusive voices in research, including the communities we partner with outside out institution. We understand there can be a need for monitoring the diversity of populations involved in research or the accessibility or visibility of our research outputs. We expect that research assessments will occur within agreed frameworks that accommodate the goals and purposes required and that acknowledge the variation of effectiveness between indicators. Researchers and research groups, centres, and institutes will be supported to develop and report on research plans through the established periodic review processes.
Recognise the diversity of research and knowledge exchange activity
Liverpool John Moores University is a signatory to the Concordat to Support the Career Development of Researchers, holds the HR Excellence in Research Award, and holds an institutional Athena Swan Silver accreditation and Race Equality Charter Bronze accreditation. We are committed to researcher development and celebrating the diversity of research and knowledge exchange activity at our University. Our academic promotions process recognises a variety of career pathways: Research and Scholarship; Academic Leadership; Engaged Scholar; Learning, Teaching, and the Student Experience. Where a pathway assesses research and knowledge exchange activity, the assessment will give due consideration to the diversity of agreed goals and purposes of that activity to then be conducted in accordance with our values.
Consider the context
We recognise that research assessments occur in different contexts and shape appropriate practice. We will be clear on the context of what we are evaluating and why we are evaluating it. Our assessments will:
Consider their purpose prior to commencement
INORMS outlines six main reasons for assessing research, which may happen at any level from individual researchers to institutions to entire countries or beyond:
- Analysis: to understand the research
- Advocacy: to promote the research
- Accountability: to regulate the research
- Acclaim: to benchmark the research
- Adaptation: to incentivise the research
- Allocation: to reward the research
The reason for conducting the assessment and the subject of the assessment will both influence what is an appropriate way to conduct the assessment. Any assessment that may lead to some sort of priority or reward ought to be designed with care so that everyone has a fair chance to succeed.
Consider field-specific research and knowledge exchange practices
Research practices vary widely between disciplines. Indicators may serve some disciplines better than others. Given the extent of our University’s cross-disciplinary research infrastructure, including the three pan-University research institutes, it is important to develop assessment practices that respond to and accommodate disciplinary differences. For instance, a humanities discipline may favour single-author publishing more so a natural sciences discipline, this will return lower scores on domestic or international collaboration indicators. Numbers of research grant applications submitted and the size of awards for research programs will differ between fields. To mitigate against issues arising thus, we encourage normalisation processes, for example, the use of percentile distributions where possible, noting that averages can be skewed by a few large outliers.
Options for evaluating
We seek the best, fairest, research assessment possible. Quantitative research indicators can complement the expert judgment involved in qualitative peer review, including reducing bias or highlighting aspects of research outputs for additional consideration. Our assessment design will consider its options to develop combined qualitative and quantitative approaches to achieve equitable research assessment. We will:
Promote combined methods assessment
We will use quantitative indicators to support expert peer review, not to replace it. We encourage research assessment to use quantitative indicators where they are well-considered and appropriate, but not in isolation. This principle applies to research assessments at all levels, from the individual researcher to larger organisational entities. Examples include assessing research for professional development and performance review processes for an individual or assessing the quality of collective research outputs and research grant applications from a research centre or institute.
Assess researchers on a qualitative judgment of their research portfolio
Not all research activities can be easily benchmarked or measured. Nor is it appropriate to attempt to measure some activities, where there may be prohibitor contracts or where certain approaches to measurement may be seen as extractive or psychologically threatening to research and knowledge exchange participants. We also recognise that a researcher’s personal characteristics and career profile may affect the relevance of some research assessment indicators. For instance, a person who has had a career break may have a smaller set of research outputs compared to others in their School. Where possible, it is important to use an overarching understanding of the research portfolio, including through researcher-supplied narratives. The assessment should consider as wide a view of the researcher’s or entity’s full portfolio of research activity and expertise as possible and in alignment with the relevant agreed research goals and purposes.
Emphasise transparency in Research Workload Allocation processes
In the annual Research Workload Allocation, criteria will be reviewed relative to each academic discipline. At the start of data collection for the RWA process, each School Director and the relevant Unit of Assessment Leads will develop the School’s criteria for differentiation between RWA statuses. Where multiple Schools administer a discipline, the School Directors and Unit of Assessment Leads will collaborate to standardise their consistent approach to RWA assessment.
School Directors and Unit of Assessment Leads will share their baseline criteria for visibility between Schools within their Faculty, also keeping a record for potential audit of the decision-making.
Avoid false precision and over-interpretation
When it is possible, we will use multiple indicators in research assessment. We want to avoid over-reliance on single indicators, which can induce assessors to take indicators out of context or draw unsupported inferences. We will avoid false precision in the representation of data, for instance giving values to unnecessarily deep decimal places or withholding any mathematical transformations we have used to reproduce the data, and we will avoid over-valuing large percentage changes over small bases or one-off unusual annual averages (for example if in one year a researcher has published 50% of the articles they published in each other year under review).
Recognise the potential for differences in data quality
We will recognise that data sources can have differing coverage of things we want to assess. Therefore, when we conduct a research assessment, we will consider the quality of the data we are using and commit to checking it. This point is especially important for assessments that use or compare multiple indicators with the data coming from different sources. For example, an evaluation of a research group will recognise the reporting databases we use (including Scopus and SciVal) may not represent all the journals or outputs the group has produced. Evaluators are encouraged to check multiple databases to ensure the data used is representative of the entity they are evaluating.
Probe assessments for unintended consequences
We are aware that research assessments can use indicators in ways which have inadvertent, unwelcome effects, such as unintentional discrimination or misaligned incentivisation. We will:
Review the effects of our assessments and their indicators
Any measurement can incentivise behaviour in unintended or undesired ways. We do not want this to happen. We will use suites of indicators and consider their weighting, mitigating negative side effects from any single indicator. When we conduct research assessments that use a few quantitative indicators in their process, we will reassess the indicators that we are using in the context of our values, our research goals and purposes, and any potential unintended effects of the indicators. We will update indicators to ensure incentivisation is aligned with our ambitions. We will also use responsible assessment as a lens to review representation within outcomes from research assessments, including academic promotion routes. Staff who lead promotion decision-making are responsible for probing disparities between populations represented in conferment routes. When we do this, we will not do it silently. We will use internal staff email and other like methods to share news of the update to all research staff who fall within the assessment’s purview.
Avoid known inappropriate metrics
The evaluation should consider the relevance of the metrics to our institutional values. We will not use the ‘H-Index’ in research assessments at any level. We will not use the ranking of a research organisation in the selection of researchers for recruitment, promotion, external examination and evaluation, or for any other purpose.
Conduct an Equality Impact Assessment on institutional approaches to research and knowledge exchange evaluation
When we develop any new form of institutional research assessment to represent the institution, we will conduct an Equality Impact Assessment (EIA). A key example is our periodic development of a Code of Practice for the Research Excellence Framework submission or equivalent codes for any cognate selection processes. We will use the Assessment to inform the ways we design, implement, and use the research assessment.
Evaluate the evaluation
Fairness is an ongoing and developing concept in evaluation. Liverpool John Moores University is committed to ensuring our research assessment remains appropriate, transparent, and fair. We will:
Share evaluation with the evaluated
As much as possible, we will design research assessments and how we interpret them with the communities under assessment. We will ensure researchers are able to see and verify any data and analyses that relate to them, whether as individuals or as members of a collective or organisation. We make the data collection tools and systems that we use to monitor research publication data available to staff, and our professional services provide training on how to use these tools and systems. When data relate to specific staff, for instance data for Research Workload Allocations or professional development and performance reviews, these data will be made available to the staff before the assessment is conducted; the purpose of this is to give staff the opportunity to inspect the data and to recommend corrections where needed.
Keep records so we can review informal weightings
It is important to use evidence-bound evaluation of our evaluations. When we use quantitative indicators as part of assessments held in recorded meetings, we encourage retention of the records, typically as written minutes, so it is possible to check if specific indicators attracted undue weight in the decision-making. Where possible, we will look to develop evidence of how we use quantitative indicators to inform our qualitative, expert peer-review. We will review the evidence as part of our evaluation.
Scrutinise indicators from a senior level on a regular basis
The University Research and Knowledge Exchange Committee will revisit the use of indicators in research assessment annually, receiving a report developed by staff in Research and Innovation Services and the Library. The Head of Research Excellence and Research Strategy will lead the reporting for Research and Innovation Services. The report will be on a template submitted to the Committee in September annually, running ahead of the customary schedule for the University’s Research Workload Allocation process. The Committee will recommend revisions to the use of indicators where necessary. The Pro-Vice-Chancellor Research and Knowledge Exchange will be responsible for this annual process.
Ensure everyone understands Liverpool John Moores University’s responsible research assessment statement
Library Services and Research and Innovation Services staff will share the statement with all staff who are known to be designing research assessments. When we update the statement, we will share it with all staff, and flag training opportunities associated with it. The Library researcher support team provides ongoing training in responsible research assessment that is open for all staff, both those who are evaluated and those who conduct evaluations. We make all our responsible research assessment training available to the professional staff who manage research activity or data management systems, empowering them to record data with maximal fairness and robustness. Indicatively, this training will include how to use research output databases. The activity and data could include those related to research workload allocation processes, academic promotions, or research centre and institute development).
Implementation and training
A transparent statement for all
This statement applies to all who are involved in research assessment at our University. We encourage all Liverpool John Moores University staff and research students to draw upon this statement when they design, experience, or challenge research assessments.
Empower staff and research students with access to assessment training
All Liverpool John Moores University staff and research students can access responsible assessment and research metrics training sessions through the Library.
Research Workload Assessment planning
We have updated our Research Workload Assessment guidance to require all staff with RWA decision-making responsibility have read this Statement. Copies of the Statement are distributed to all decision-making staff in their papers ahead of RWA decision meetings. It is the responsibility of the Head of Research Excellence and Research Strategy to ensure this distribution occurs.
Research centres and institutes
Every two years, Research Centres and Institutes report to the University Research and Knowledge Exchange Committee on their agreed progress. This Statement is appended to every reporting template issued to Research Centre and Institute Directors and to the reports returned to the University Research and Knowledge Exchange Committee. It is the responsibility of the Head of Research Excellence and Research Strategy to ensure the Statement is appended to the documents.