São Paulo State universities establish new metrics for academic performance and international comparisons

The state-run higher education institutions USP, UNICAMP and UNESP create an office to manage indicators. Cooperation and data interoperability could be extended to federal universities.

By Maria Fernanda Ziegler  |  Agência FAPESP – The three public universities maintained by São Paulo State, Brazil have joined forces to develop new metrics for assessing performance and making international comparisons. They will establish a shared digital system that will be run by their indicator management offices. The system will be capable of more precise evaluation not only of performance but also of socioeconomic, cultural and environmental impacts.

The University of São Paulo (USP), the University of Campinas (UNICAMP) and São Paulo State University (UNESP) have agreed to implement interoperable performance metrics and to audit the indicators submitted to international ranking systems.

The initiative is supported by FAPESP via the project “Performance indicators in São Paulo State universities”, which is linked to the FAPESP Public Policy Research Program and has been renewed until 2022.

Led by Jacques Marcovitch, a professor at USP, and by the Council of Rectors of the Universities of São Paulo State (CRUESP), the project also involves a partnership with the São Paulo State Department of Economic Development, Science and Technology.

USP has already set up a new indicator management office, and it is implementing changes to its technological systems for data capture and dissemination. UNICAMP is reformulating its internal structures with the aim of generating data for use in strategic planning. UNESP is executing a long-term multidisciplinary plan to analyze the possible relationships between academic performance and the tendencies evidenced by the rankings.

“University rankings have become extremely popular not only among academics, journalists and education administrators but also in public opinion. However, they’re seen in a highly distorted way, as a sort of global Olympics in which universities compete with each other and win or lose positions,” said Marco Antônio Zago, President of FAPESP, in his opening address to the Second Forum on Academic Performance Indicators and International Comparisons: Impacts for Society, held on October 18, 2019, at FAPESP in São Paulo.

For Zago, the profusion of new ratings and indicators has an undesirable effect. “It’s our responsibility, therefore, as important universities, to respond in a well-grounded manner to the challenge of identifying relevant indicators and assuring their quality, while taking into account the heterogeneity of our universities and their influence on the city, the state and the country,” he said.

Rethinking the university

The meeting also featured a book launch, Repensar a Universidade II: Impactos para a Sociedade (“Rethinking the University II: Impacts for Society”), which is the project’s second publication, with articles on university performance assessment and new performance metrics for an enhanced presence in international comparisons, with further developments to be implemented by 2022.

“The project is fundamental, for our universities are under attack more than ever before in this country. It’s important to understand the reasons for this assault and to demonstrate the socioeconomic impact of universities on the society as a whole. This can only be done by communicating data and showing results,” said Marcelo Knobel, chair of CRUESP and Rector of UNICAMP.

The state-run universities in São Paulo, which account for 33.8% of Brazil’s scientific production, plan to add data referring to federal higher education institutions across the country to the project. The federal HEIs are also setting up bureaus to manage e-data, metrics and indicators. The forum was attended by rectors and other representatives of the Federal University of Itajubá (UNIFEI), the Federal University of Minas Gerais (UFMG), the Federal University of Ceará (UFC) and the Federal University of São Paulo (UNIFESP).

“No university can genuinely evolve without being compared to others to measure its progress. We have to be collective and promote a movement in favor of quality in Brazil’s higher education institutions. The contemporary university makes alterity one of its foundational principles,” said Marcovitch, stressing that the project is a groundbreaking initiative and includes among its aims the construction of a public policy to pursue excellence and better positions in global rankings.

Bridge to society

To this end, a course on indicators and metrics associated with academic performance monitoring and international comparisons will be held in March 2020.

Initially, the project focuses on digitization. “Since its inception, we’ve seen the impact of content digitization and the role of universities in both generating and curating knowledge,” said Marcovitch.

The curatorial role of universities was also highlighted by Priscila Cruz, Executive President of Todos Pela Educação, a Brazilian education advocacy organization. Cruz argued that universities should both train teachers and produce the science necessary for the formulation of evidence-based public policies.

“No one says high-quality public education isn’t important. In contrast, we haven’t given it the proper priority,” Cruz said. “We must go beyond rhetoric and strengthen the bridges between the universities and society, using academic production to help governments produce the solutions our educational system so urgently needs. In this way, we will be able to construct a just society in the field of education but with a significant impact on the economy, income distribution and guarantees of other rights.”

The agricultural sector affords a historical example of a bridge between the university and a part of society. In one of the articles in the book, Solange Santos and Rogerio Mugnaini, researchers affiliated with USP’s School of Communication and Arts (ECA-USP), analyze the production of São Paulo’s three state-run universities between 2007 and 2016. Internationalization in agrarian sciences increased from 44% to 64% during the period in terms of articles indexed by the Scielo and Web of Science platforms.

According to Santos, human capital formation and knowledge creation take place above all at universities. “Academia is a priority for the nation because of its social relevance, economic impact and environmental importance,” she said. “Until the 1980s, Brazil was a major food importer but since then it has become a leading producer. Our results show that Brazil achieved this transformation due to research and human capital formation – factors that are closely linked to universities – in an area with significant social, economic and environmental impacts.”

The book also highlights the impact of USP’s graduate course on the quality of other universities. Analyzing data for more than 50,000 alumni who earned degrees between 1970 and 2014, the book shows that 52% are faculty members of universities in Brazil or abroad. “In the case of UFABC [the Federal University of the ABC, in metropolitan São Paulo], 52% of its faculty are graduates of USP. At UNESP, the proportion is 40% and at UNICAMP, 34%,” said Aluísio Segurado, coordinator of USP’s Office for Management of Academic Performance Indicators (EGIDA).

According to Teresa Atvars, the General Coordinator of UNICAMP, the new indicators and metrics will offer the university a tool to discipline decision making. When comparing institutions, she stressed that it is also necessary to take into account their differences in terms of context and modus operandi.

“In the case of UNICAMP, the institution performs activities of enormous scope, from education to research and extension, and has a huge impact in the area of health. For this reason, the analysis cannot be based only on objective data. Qualitative information must also play a central role,” she said.

For Sabine Righetti, Academic Coordinator of Ranking Universitário Folha (RUF, an initiative of Folha de S. Paulo, one of Brazil’s leading daily newspapers), universities work with different metrics. “Universities are very different from each other, and the rankings look at them all as if they were the same. This shouldn’t be the case if we’re to avoid risking mistaken public policies,” she said.