CoARA Network: a threat to science

One of the most important moments in science is the assessment of academic merit, which is essential for the allocation of resources, hiring of new academics or career progression. There is no perfect method for conducting this exercise, but in Portugal a methodology has been applied that does not value merit and is therefore detrimental to science and the investment made by taxpayers in the sector. At a time when the quality of Portuguese universities is in “alarming decline” , the issue of the assessment of academics deserves deep reflection.

Currently, the methodology for assessing the merit of academics used in Portugal is based on an alleged assessment of the quality of CVs, as stipulated in paragraph 4 of article 5 of Decree-Law no. 57 of 29 August, which approved a regime for hiring PhD holders. More recently, this approach has been promoted and expanded in its application by CoARA – Coalition for Advancing Research Assessment, an international collective formed in 2022, which aims to “reformulate the methods and processes by which research, researchers and research organisations are assessed”. CoARA includes several national institutions , including the Foundation for Science and Technology (FCT).

In CoARA’s view (pp. 4–5), academic evaluation criteria should focus on a nebulous notion of quality, rejecting the exclusive use of bibliographic metrics (number of articles, chapters, books, etc.). CoARA also argues that the assessment of this alleged quality should be carried out by authorities in the respective scientific fields (the peer review process).

Herein lies the first weakness of this methodology, which is fatally wounded if the evaluators are not qualified enough for the role. As I have already mentioned , this is what happened in one of the last competitions organized by the FCT. On the other hand, to work effectively, peer review must be doubly anonymous, that is, neither the evaluator knows who the person being evaluated is nor vice versa. Obviously, in a curriculum evaluation, it is impossible to meet the first of these conditions.

Furthermore, even if the evaluators are true authorities on the subject, a qualitative assessment is necessarily subjective, especially when we look at the criteria suggested by CoARA (p. 3): diversity of activities, originality of ideas, professional conduct, transparency of processes, development of knowledge and potential and societal impact. These criteria grant evaluators a very wide latitude and allow them to justify any decision. In fact, any challenge is doomed to failure, since opinions and not facts will be discussed.

For these reasons, I state that the CoARA proposal has nothing scientific or academic about it, but rather is political, and aims only to maintain in the hands of career academics (normally those responsible for assessment exercises) the power to decide who enters, who leaves and who stays in academia. The enormous subjectivity and latitude underlying the CoARA methodology can promote and legitimize cronyism, nepotism and cronyism to the detriment of merit, work and excellence, with serious harm to science, research and the taxpayers' money that funds research.

It is therefore difficult to understand how this methodology promotes transparency, as CoARA dishonestly argues (p. 5). CoARA also argues that this methodology will help institutions regain their autonomy (p. 6), but autonomy should not mean using taxpayers' money without rules or with very lax rules.

Furthermore, the CoARA methodology has two harmful consequences for academic activity, namely promoting sexual and moral harassment and further suffocating the democracy of institutions, especially in internally organized assessment exercises.

If the future of an academic (whether hiring, contract renewal or promotion) depends on a subjective assessment by a superior, the latter gains enormous leverage over the former. We are all aware of the events in Coimbra and how some victims were harmed for rejecting sexual advances from their supervisor.

Likewise, the freedom of expression and opinion of academics both inside and outside academia is seriously restricted, as they will hesitate to intervene civically or institutionally knowing that they may be harmed in the future. This is particularly serious because if there is a space where criticism, debate and the confrontation of ideas should be practiced and encouraged, that space is academia.

For these reasons, I believe that the assessment of the merit of academics should be based on purely and strictly metric and quantitative bases, taking into account all activities that are reasonably required of them: publications (in leading journals and publishers), teaching and supervision activities, patents, communications at conferences, organization of events, scientific dissemination, editorial activity, executive functions, etc. Each of these activities should be assigned a coefficient, resulting in a value, which would be the only evaluation or ranking criterion. The selection of activities and their respective coefficients may be applied differently according to the scientific area, but should be determined at national level, by law.

Regarding the use of metrics, CoARA disputes that their use is prone to multiple biases (p. 10), an indefensible and somewhat shameless argument coming from an organization that proposes subjective evaluations, incomparably more permeable to biases of all kinds.

Paradoxically, the use of metrics is in line with some principles defended by CoARA, namely the appreciation of the diversity of research activities (p. 4), the recognition that academics do not have to be excellent in all activities reasonably required of them and the possibility given to academics to define their objectives and aspirations (p. 4).

Furthermore, a quantitative assessment puts the careers of academics in their own hands. Their assessment and future depend solely on their work, regardless of the friendships they make, regardless of whether they fail to see the intellectual sex appeal of their septuagenarian supervisors, regardless of the opinions they express inside and outside academia.

And, most importantly, taxpayers would see their tax money being used effectively to promote excellent research in quantity and quality and not to respond to more or less veiled agendas and practices within academia.