The Capes Evaluation Records Progress on Quality Indicators for Master’s and Doctoral Courses in Brazil: Pesquisa Fapesp Journal

The Coordinating Body for the Improvement of Higher Education Personnel (Capes), an agency linked to the Ministry of Education (MEC), which released in December, one year later than expected, used the results of the four-year evaluation of postgraduate studies in Brazil, since 1990. 1970 to assess the quality of master’s courses And doctoral, academic and professional, directing the distribution of scholarships and funds. Even in a scenario of underfunding for science, with cuts in research resources and erosion of grant value, many programs have improved their indicators. Of the 4,512 courses assessed between 2017 and 2020, 34% have increased their score. The number of international-level programs, which receive the highest scores on the Capes scale (6 and 7), increased by 37%, jumping from 490 to 671.

Most of the courses are concentrated in the southern and southeastern states, especially São Paulo, Rio de Janeiro, Rio Grande do Sul and Minas Gerais. But other areas have also gained momentum. In the Northeast, the number of Excellence programs increased from 37 to 60, an increase of 62%. The North Region, which had no 7-degree program until its previous evaluation, now has three at this level, two at the Federal University of Pará (UFPA) and one at the National Research Institute of the Amazon (Inpa). “We expected the university to get better grades, but the results exceeded expectations, which will have an important impact on our ability to raise funds for the development of our postgraduate studies,” said psychologist Emmanuel Zagury Torreno, dean of the university. , in a video posted by the UFPA on social networks.

The results reaffirm trends already achieved in previous assessments. The University of São Paulo (USP) remains the Brazilian institution with the most sixth- and seventh-grade programs. There are a total of 114 universities, mainly in the fields of biological sciences, engineering and medicine. Created just 17 years ago, the Federal University of the ABC (UFABC) also celebrated the results. Of the 29 programs evaluated, 10 improved their grades, and two achieved a 6. Nanoscience, Advanced Materials, and Chemical Science and Technology, “an outstanding performance for a relatively young institution,” assesses Charles Murphy, the institution’s dean of graduate degrees. After graduating from São Paulo State University (Unesp) he now has 10 seventh grade programs, including Physics, Chemistry and Dentistry. “This is due to the goal-based graduate policy and program monitoring throughout a four-year period,” says chemist Maria Valenice Boldrin, associate graduate student at Unesp.

At Campinas State University (Unicamp), 30 courses have increased their score in relation to the previous course, with 37 of them achieving a level of distinction, focusing on the areas of humanities, food, engineering, medicine and dentistry. The university’s dean of graduate studies reports that it will meet with third- and fourth-grade program coordinators in order to identify strategies for improving performance in the upcoming assessment.

In addition to academic standing, Excellence programs have more autonomy and can receive funding directly from the Capes through the Academic Excellence Program (ProEx) – those with grades between 3 and 5 also receive federal agency resources through the Graduate Support Program (ProAP) , but the transfer is smaller. Projects associated with degree programs with high scores are also more likely to be selected at calls from the Capes and the National Council for Scientific and Technological Development (CNPq).

It is true that in recent years, difficulties in financing grants and projects have affected courses with good or bad ratings. “Programs in recent years have had to manage a series of constraints imposed not only by the pandemic but also by scarcity of science and education resources,” says political scientist Rachel Menigello, dean of graduate studies at Unicamp. “The outdated and insufficient value of federal grants illustrates this scenario well.” For years, students have suffered from the erosion of the value of the aid, which was most recently revised in 2013, while the accumulated inflation since then, measured by the IGP-M, has been 125.1% – currently, a master’s student receives R$1,500 per month and Ph.D. R $2,200.

Initiatives such as the Institutional Program for the Internationalization of Brazilian Universities (PrInt), launched by Capes in 2017 with the aim of encouraging international partnerships in master’s and doctoral programmes, have also been affected. In 2020, due to the pandemic, no mobility grants, which provide student exchange, were awarded, and in 2021 they were suspended. Boldrin explains that the program, which will end in 2021, has been extended through 2024 and the grants, which were supposed to be paid out in 2020 and 2021, have been moved to 2023 and 2024.

There is no consensus on what factors contributed to the programs’ good performance, but there are some hypotheses. One relates to changes in the Capes classification model, which in the latter process has begun to give greater weight to certain qualitative aspects of the training offered by the courses and their intellectual output. Until then, those in charge of the programs filled out a questionnaire with various information about the program proposal, faculty qualifications, student profile, number of works published in the highest Qualis strata, and the Capes classification system for scholarly purposes. Journals. “In the latest evaluation, they were also able to point to the best quality work of their students and researchers,” highlights epidemiologist Rita Paradas-Barata, former director of KIPES, who has been involved in analyzing graduate professional programs in the field of public health.

The programs recorded 4.7 million intellectual productions between 2017 and 2020. Of this total, the curators themselves highlighted 280 thousand, and they were screened for their impact, originality and innovation. “This may have contributed to the programs performing better than they would have been if the evaluation had focused on quantitative aspects,” Barata says.

Experts heard by the report suggest there is a trend within the Capes Scientific Technical Council for Higher Education (CTC-ES), a collegial body that deliberates on assessment, to give more autonomy to the work done by district coordinators, following their own grading recommendations – in assessments From previous years, Capes created a special committee to reanalyze sixth and seventh grade scholarships.

It is also possible that the evolution of program quality indicators reflects the investments made in the previous four-year period and was not affected by the recent difficulties. “Course performance in each assessment usually reflects some portion of what has been accomplished in the past,” says Murphy, of the UFABC. “The first half of the 2000s was marked by a policy of postgraduate assessment and expansion, and it is likely that current results still reflect this.”

For Rita Bharata, Capes will have to adapt its criteria for the next assessment, which will cover the period between 2021 and 2024. “Increasing percentage of withdrawal from master’s and doctoral courses, interruption of projects, late conclusion of dissertations and dissertations, and publication of scientific articles,” she adds. “All of this will be reflected in the performance and production of programs in the coming years, and the agency must be aware of this.”

According to her, heads will also need to re-examine the effectiveness of their rating model. “The logic of the four-year evaluation is to identify nuances and differentiate programs so that, based on this, Capes itself and CNPq and other development agencies can determine investment priorities and strategies for postgraduate development,” he explains. “By the time many programs are able to rise to the highest score, the scale used by the agency loses its ability to recognize excellence.”

The scientific community expects program performance to be calculated on the basis of a multidimensional model, with various scores attributed to five dimensions of performance: teaching and learning, knowledge production, international and regional inclusion, and innovation and transfer. knowledge and its impact on societyto see FAPESP Research No. 286). “This proposal has been discussed for years and has broad community support, but we still don’t know if it will be implemented,” says agronomist Marcio de Castro Silva Filho, University Chair of Graduate Studies at USP. “The new model will play an important role in optimizing the system, as it will promote a more accurate and detailed assessment of programmes, taking into account specific occupations,” adds Menegello.

The last stage of the Capes assessment process was marked by a series of limitations. The main event took place in September 2021. In response to a public civil action by the Federal Public Prosecution Service (MPF), Judge Andrea de Araujo Peixoto, of the 32nd Federal Court of Rio de Janeiro, issued an injunction specifying the suspension of the evaluation, which was to announce results in December. According to the lawsuit filed by the MPF, Capes had changed standards over the course of the four years of operation, creating legal uncertainty for the programs. At the time, the MPF suggested that the agency adopt standards similar to those used in 2016 to measure the quality of courses in 2021 (to see FAPESP Research No. 309).

In December, Judge Antonio Henrique Correa da Silva, also of the 32nd Federal Court in Rio de Janeiro, responded to a request from the prosecutor’s office and annulled the injunction ordering the halt to the process. could be concluded, but publication of the results remained pending. The permission to appeal came amid the resignation of about 80 researchers who worked as coordinators and consultants in the fields of chemistry, mathematics and physics. They complained about the agency’s lack of commitment to trying to reverse the court’s decision and to press for approval of new courses.

In September 2022, Capes signed an agreement with the MPF to end the general civil procedure and allow the results to be published. Initially, scores were only reported to program coordinators and deans of graduate studies, who could request a reconsideration—it’s normal for universities to move on when they’re not happy with initial scores. However, the agreement has been criticized by the Brazilian Association for the Advancement of Science (SBPC) for authorizing courses to maintain the score of the previous assessment for four years if their concept is downgraded in the current assessment. Capes did not tell programs that requested reconsideration to use this device to avoid reducing the concept of courses. The agency said in a note that it had received 975 reconsideration requests, of which 707 maintained its 2017 rating. In all, only 189 received lower scores than in 2017.

Such setbacks disrupted the courses. “We went through two years of great instability, with changes in chairmen, interruptions in evaluation and delays in publication of results, causing program coordinators to delay their planning,” Boldrin highlights. “We are already in the middle of assessing the next four years, and only now will it be possible to assess strengths and weaknesses and set new goals.” Also due to the pandemic, the discussion of the Postgraduate National Plan (PNPG) for the decade 2021 to 2030 has just begun and should set goals to be achieved in the coming years. “It is up to the new management, now headed by biologist Mercedes Bustamante, to continue these discussions with the community,” concludes Menegello, of Unicamp.

repost