Abstract

In recent decades, changes in science have been characterized by internationalization and the quest for impact. This article introduces the impact assessment process of 23 research projects of the Capes prInt Program aimed at internationalizing Brazilian science. This work developed an impact assessment model, applying it experimentally. The development was based on scientific texts on modes of production of scientific knowledge and impact assessment. The assessment revealed that, despite the COVID-19 pandemic, the projects have advanced in their internationalization goals. As for the impacts, those related to science and education predominated, with a significant number of projects also indicating impacts on public policies and organizational practices. The article adds to understanding modes of knowledge production and research impact measurement. Furthermore, it can be useful to directors of research institutions, research managers, and researchers interested in measuring the research impact on organizations and society.

Keywords:
Social impact of the research; Research relevance; Knowledge production mode; Research impact assessment; Internationalization of research

Resumo

Nas últimas décadas, as mudanças na ciência foram caracterizadas pela internacionalização e pela busca do impacto. Este artigo apresenta o processo de avaliação de impacto de 23 projetos de pesquisa do Programa Capes prInt, voltado para a internacionalização da ciência brasileira. O objetivo deste trabalho foi desenvolver um modelo de avaliação de impacto, aplicando-o experimentalmente. O desenvolvimento fundamentou-se nos textos científicos sobre modos de produção de conhecimento científico e avaliação de impacto. A avaliação revelou que, mesmo diante da pandemia da COVID-19, os projetos avançaram em suas metas de internacionalização. Os resultados indicaram a predominância de impactos relacionados à ciência e à educação, com um número significativo de projetos também indicando impactos sobre políticas públicas e práticas organizacionais. O artigo contribui para o conhecimento sobre modos de produção de conhecimento e sobre avaliação de impacto da pesquisa. Além disso, pode ser útil a diretores de instituições de pesquisa, gestores de pesquisa e pesquisadores interessados na questão da mensuração do impacto da pesquisa nas organizações e na sociedade.

Palavras-chave:
Impacto social da pesquisa; Relevância da pesquisa; Modos de produção de conhecimento; Avaliação de impacto da pesquisa; Internacionalização da pesquisa

Resumen

En las últimas décadas, los cambios en la ciencia se han caracterizado por la internacionalización y la búsqueda de impacto. El presente artículo presenta el proceso de evaluación de impacto de 23 proyectos de investigación del Programa Capes prInt destinados a la internacionalización de la ciencia brasileña. El objetivo de este trabajo fue desarrollar un modelo de evaluación de impacto aplicándolo experimentalmente. El desarrollo se basó en textos científicos sobre los modos de producción de conocimiento científico y evaluación de impacto. La evaluación reveló que, a pesar de la interferencia de la pandemia de COVID-19, los proyectos han avanzado en sus objetivos de internacionalización. En cuanto a los impactos, predominaron los relacionados con la ciencia y la educación, con un número significativo de proyectos que también indicaron impactos en las políticas públicas y las prácticas organizativas. El artículo contribuye a la comprensión de los modos de producción de conocimiento y a la medición del impacto de la investigación. Además, puede ser útil para directores de instituciones de investigación, gestores de investigación e investigadores interesados en medir el impacto de la investigación en las organizaciones y la sociedad.

Palabras clave:
Impacto social de la investigación; Relevancia de la investigación; Modo de producción de conocimiento; Evaluación del impacto de la investigación; Internacionalización de la investigación

INTRODUCTION

Since the 1990s, a process of change is underway. There is a quest for a more assertive alignment of university research with social demands. In the field of Administration, the issue of research impact beyond the academy has been the subject of editorials in international journals, such as the Journal of International Business Studies (Bello & Kostova, 2012Bello, D. C., & Kostova, T. (2012). From the Editors: Conducting high impact international business research: The role of theory. Journal of International Business Studies, 43(6), 537-543. Recuperado dehttps://10.1057/jibs.2012.14
https://doi.org/10.1057/jibs.2012.14... ), The Academy of Management Journal (George, 2016George, G., Howard-Grenville, J., Aparna, J., & Tihanyi, L. (2016). Understanding and tackling societal grand challenges through management research. Academy of Management Journal, 59(6), 1869-1877. Recuperado dehttps://doi.org/10.5465/amj.2016.4007
https://doi.org/10.5465/amj.2016.4007... ), and the Journal of Management Studies (Wickert, 2021Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that makes a difference: broadening the meaning of impact. Journal of Management Studies, 58(2), 297-320. Recuperado dehttps://doi.org/10.1111/joms.12666
https://doi.org/10.1111/joms.12666... ). In Brazil, the same issue has been addressed by several position papers, in Cadernos EBAPE.BR (Costa, Machado, & Câmara, 2022Costa, F. J., Machado, M. A. V., & Câmara, S. F. (2022). Por uma orientação ao impacto societal da pós-graduação em administração no Brasil. Cadernos EBAPE.BR, 20(6), 823-835. Recuperado dehttps://doi.org/10.1590/1679-395120210222
https://doi.org/10.1590/1679-39512021022... ), Revista de Administração de Empresas [Journal of Business Administration] (Lazzarini, 2017Lazzarini, S. (2017). Pesquisa em Administração: em busca de impacto social e outros impactos. Revista de Administração de Empresas, 57(6), 620-625. Recuperado dehttps://doi.org/10.1590/S0034-759020170608
https://doi.org/10.1590/S0034-7590201706... ), Revista de Administração Contemporânea [Contemporary Management Journal] (Mendes-da-Silva, 2019Mendes-da-Silva, W. (2019). Convergência, comunicação, e impacto da pesquisa em negócios. Revista de Administração Contemporânea, 23(1), 1-7. Recuperado dehttps://doi.org/10.1590/1982-7849rac2019180346
https://doi.org/10.1590/1982-7849rac2019... ), RAUSP Management Journal (Sandes-Guimarães & Hourneaux Junior, 2020Sandes-Guimarães, L. V. D., & Hourneaux, F. Jr. (2020). Research impact - what is it, after all? RAUSP Management Journal, 55(3), 283-287. Recuperado dehttps://doi.org/10.1108/RAUSP-07-2020-202
https://doi.org/10.1108/RAUSP-07-2020-20... ), and Organizações & Sociedade [Organizations & Society] (Ventura & Davel, 2021Ventura, A. C., Davel, E. P. B. (2021). Impacto socioambiental da pesquisa. Organizações & Sociedade, 28(99), 710-721. Recuperado dehttps://doi.org/10.1590/1984-92302021v28n9900PT
https://doi.org/10.1590/1984-92302021v28... ). The challenges brought by climate change and the COVID-19 pandemic, together with traditional social issues, such as inequality, urban violence, and health and education, made changes more urgent.

This article is embedded in this context. It shows the process of impact assessment of 23 projects carried out at Fundação Getulio Vargas (FGV), under the Capes prInt Program. Although the program’s primary objective was internationalization, the projects analyzed also had objectives related to generating social impact and benefits to society. Moreover, the institution itself expresses interest in improving the alignment of its research activities with social demands.

This text was grounded in the body of theory on modes of scientific knowledge production, which has strongly evolved since the 1990s (Eisenhardt, Graebner, & Sonenstein, 2016Eisenhardt, K., Graebner, M. E., & Sonenstein, S. (2016). From the editors: grand challenges and inductive methods: rigor without rigor mortis. Academy of Management Journal, 59(4), 1113-1123. Recuperado dehttps://doi.org/10.5465/amj.2016.4004
https://doi.org/10.5465/amj.2016.4004... ; Etzkowitz & Leydesdorff, 1995Etzkowitz, H., & Leydesdorff, L. (1995). The triple helix university-industry-government relations: a laboratory for knowledge based economic development. EASST Review, 14(1), 14-19. Recuperado de https://ssrn.com/abstract=2480085
https://ssrn.com/abstract=2480085... ; Etzkowitz, Webster, Gebhardt, & Terra, 2000; George, Howard-Grenville, Aparna, & Tihanyi, 2016George, G., Howard-Grenville, J., Aparna, J., & Tihanyi, L. (2016). Understanding and tackling societal grand challenges through management research. Academy of Management Journal, 59(6), 1869-1877. Recuperado dehttps://doi.org/10.5465/amj.2016.4007
https://doi.org/10.5465/amj.2016.4007... ; Gibbons et al., 1994Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: the dynamics of science and research in contemporary societies. London, UK: Sage Publishing House.), and recent developments on impact assessment (Aguinis, Shapiro, Antonacopoulou, & Cummings, 2014Aguinis, H., Shapiro, D. L., Antonacopoulou, E. P., & Cummings, T. G. (2014). Scholarly impact: a pluralist conceptualization. Academy of Management Learning and Education, 13(4), 623-639. Recuperado dehttp://dx.doi.org/10.5465/amle.2014.0121
https://doi.org/10.5465/amle.2014.0121... ; Reale et al., 2018Reale, E., Avramov, D., Canhial, K., Donovan, C., Flecha, R., Holm … Van Horik, R. (2018). A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation, 27(4), 298-308. Recuperado dehttps://doi.org/10.1093/reseval/rvx025
https://doi.org/10.1093/reseval/rvx025... ; Wickert, Bartunek, & Daft, 2021Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that makes a difference: broadening the meaning of impact. Journal of Management Studies, 58(2), 297-320. Recuperado dehttps://doi.org/10.1111/joms.12666
https://doi.org/10.1111/joms.12666... ).

The objective of this paper was to design and apply an impact assessment model, using it experimentally in 23 projects. To do that, we developed a specific method, applied initially to a pilot case, and then, with some adaptations, to the 23 projects. We collected information through interviews with project leaders, in 2021.

This study aims to contribute to expertise on alternative modes of knowledge production and on assessment of research impact. In practical terms, we expect that the method will be extended to other groups of applied research projects. In addition to the introduction, the paper has six sections: theory, methods, results, lessons learned, discussion, and conclusion.

THEORY

This section presents, in a synthetic and targeted way, the two theoretical bodies that support the work done on impact assessment, first including the modes of knowledge production, and second, the research impact evaluation.

In the sub-section on modes of knowledge production, we took as a source the study by Wood, Souza, and Caldas (2022Wood, T. Jr., Souza, R., & Caldas, M. O. (2022), The relevance of management research debate: a historical view - 1876-2018. Journal of Management History, 28(3), 409-427. Recuperado de https://doi.org/10.1007/s11213-022-09596-x
https://doi.org/10.1007/s11213-022-09596... ), based on the bibliometric approach of citation network analysis, which provided significant contributions to the topic. In this subsection, we highlight the paper by Gibbons et al. (1994Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: the dynamics of science and research in contemporary societies. London, UK: Sage Publishing House.), which accumulated, in December 2022, about 22 thousand citations in Google Scholar. In fact, his characterization of modes 1 and 2 of knowledge production is considered a true “Columbus’ egg”, and keeps its relevance and applicability, even after decades of debates and criticisms.

The sub-section on evaluation of research impact was built by doing specific research in Ebsco, Web of Science, and Google Scholar databases, focusing on the most recent developments on the topic. In that section, the paper by Wickert et al. (2021Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that makes a difference: broadening the meaning of impact. Journal of Management Studies, 58(2), 297-320. Recuperado dehttps://doi.org/10.1111/joms.12666
https://doi.org/10.1111/joms.12666... ) stands out, by presenting a model that has been cited in different international academic forums. Such model stems from several previous contributions, representing both a synthesis of the “state of the art” and a system with high potential for practical application.

Hence, we sought to form a theoretical basis for developing and proposing a model for evaluating research projects. Such model results from the combination of previous developments, in addition to a specific mode of operationalization, and is presented in the following sections.

Modes of knowledge production

In the 1990s, questioning the traditional way of producing knowledge gained prominence, which emphasizes the publication of scientific articles, thus circumscribed to science itself. In 1993Funtowicz, S. O., & Ravetz, J. R. (1993). Science for the post-normal age. Futures, 25(7), 739-755. Recuperado dehttps://doi.org/10.1016/0016-3287(93)90022-L
https://doi.org/10.1016/0016-3287(93)900... , Funtowicz and Ravetz presented the concept of ‘post-normal science’, in which academics should involve all those affected by the research problem in searching for solutions. This would take place in a broader community, comprising not only traditional peers.

Gibbons et al. (1994Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: the dynamics of science and research in contemporary societies. London, UK: Sage Publishing House.) identified the emergence of a new mode of knowledge production, called mode 2, in contrast to the traditional mode, called mode 1, which barely connects the production of knowledge with its applications, and is restricted to a specific academic community and controlled by peers of the disciplinary circle itself. Mode 2, on the other hand, values knowledge production together with practitioners, including, since the beginning, the applications; its character is multidisciplinary and research quality is associated with the work impact, not only among academic peers, since it must reach far beyond them.

Still in the 1990 decade and in the next, several models emerged: the triple helix - university, company, and government (Etzkowitz & Leydesdorff, 1995Etzkowitz, H., & Leydesdorff, L. (1995). The triple helix university-industry-government relations: a laboratory for knowledge based economic development. EASST Review, 14(1), 14-19. Recuperado de https://ssrn.com/abstract=2480085
https://ssrn.com/abstract=2480085... ); post-academic science (Ziman, 1996Ziman, J. (1996). ‘Postacademic science’: constructing knowledge with networks and norms. Science Studies, 9(1), 67-80. Recuperado de https://doi.org/10.23987/sts.550954
https://doi.org/10.23987/sts.550954... ); and entrepreneurial university (Etzkowitz et al., 2000Etzkowitz, H., Webster, A., Gebhardt, C., & Terra, B. R. (2000). The future of the university and the university of the future: evolution of ivory tower to entrepreneurial paradigm. Research Policy, 29(2), 313-330. Recuperado dehttps://doi.org/10.1016/S0048-7333(99)00069-4
https://doi.org/10.1016/S0048-7333(99)00... ). These models sought to foster the approximation between rigor and relevance, by proposing to academics to increase interaction with external players.

In the area of Administration, Baldridge, Floyd, and Markóczy (2004Baldridge, D. C., Floyd, S. W., & Markóczy, L. (2004). Are managers from Mars and academicians from Venus? Toward an understanding of the relationship between academic quality and practical relevance. Strategic Management Journal, 25(11), 1063-1074. Recuperado dehttp://doi.org/10.1002/smj.406
https://doi.org/10.1002/smj.406... ), in a sample of 120 articles published in high impact journals, showed that the most cited were considered more relevant to practice by a panel of firm managers. This means that knowledge production in the traditional way has great impact potential outside the academic area, but often does not reach the actors interested in the research, due to inefficient connection and dissemination. Hence, many authors began to emphasize dialogue with practitioners (Amabile et al., 2001Amabile, T. M., Patterson, C., Mueller, J., Wojcik, T., Odomirok, P., Marsh, M … Kramer, S.. (2001). Academic-practitioner collaboration in management research: a case of cross-profession collaboration. Academy of Management Journal, 44(2), 418-431. Recuperado dehttp://doi.org/10.5465/3069464
https://doi.org/10.5465/3069464... ; Anderson, 2007Anderson, N. (2007). The practitioner-researcher divide revisited: strategic-level bridges and the roles of IWO psychologists. Journal of Occupational and Organizational Psychology, 80(2), 175-183. Recuperado dehttp://doi.org/10.1348/096317907X187237
https://doi.org/10.1348/096317907X187237... ; Eisenhardt et al., 2016Eisenhardt, K., Graebner, M. E., & Sonenstein, S. (2016). From the editors: grand challenges and inductive methods: rigor without rigor mortis. Academy of Management Journal, 59(4), 1113-1123. Recuperado dehttps://doi.org/10.5465/amj.2016.4004
https://doi.org/10.5465/amj.2016.4004... ; George, 2016George, G., Howard-Grenville, J., Aparna, J., & Tihanyi, L. (2016). Understanding and tackling societal grand challenges through management research. Academy of Management Journal, 59(6), 1869-1877. Recuperado dehttps://doi.org/10.5465/amj.2016.4007
https://doi.org/10.5465/amj.2016.4007... ; George et al., 2016George, G., Howard-Grenville, J., Aparna, J., & Tihanyi, L. (2016). Understanding and tackling societal grand challenges through management research. Academy of Management Journal, 59(6), 1869-1877. Recuperado dehttps://doi.org/10.5465/amj.2016.4007
https://doi.org/10.5465/amj.2016.4007... ; Hodgkinson, 2006Hodgkinson, G. P. (2006). The role of JOOP (and other scientific journals) in bridging the practitioner-researcher divide in industrial, work and organizational (IWO) psychology. Journal of Occupational and Organizational Psychology, 79(2), 173-178. Recuperado dehttps://doi.org/10.1348/096317906X104013.
https://doi.org/10.1348/096317906X104013... ; Kreiling & Paunov, 2021Kreiling, L., & Paunov, C. (2021). Knowledge co-creation in the 21st century: a cross-country experience-based policy report (OECD Science, Technology and Industry Policy Papers, n. 115). Paris, France: OECD Publishing. Recuperado de https://www.oecd.org/innovation/knowledge-co-creation-in-the-21st-century-c067606f-en.htm
https://www.oecd.org/innovation/knowledg... ; Rynes, 2007Rynes, S. L. (2007). Let’s create a tipping point: what academics and practitioners can do, alone and together. Academy of Management Journal, 50(5), 987-1008. Recuperado dehttps://doi.org/10.5465/AMJ.2007.27156169
https://doi.org/10.5465/AMJ.2007.2715616... ).

Assessment of research impact

Research impact evaluation is a complex subject. However, considering the purpose of this article, this section presents a summary view of the topic, grounded on four components: the reason for changes in research impact assessment; the characteristics of the traditional assessment model; the UK “experiment” with the Research Excellence Framework (REF) cycles; and the trend toward multidimensional evaluation models.

Why assess the impact of research? Pioneers of modern science, such as Friedrich von Humbolt and Vannevar Bush, believed that scientists should work with freedom to choose what and how to research. They assumed that autonomy would be sufficient for academics to contribute to the common good with their work (Wood & Caldas, 2020Wood, T. Jr., & Caldas, M. P. (2020). Posfácio: o desafio da transformação na UNESP. In S. R. Valentini, & S. R. Nobre (Orgs.), Universidade em transformação: lições das crises(pp. 445-453). São Paulo, SP: Editora UNESP.).

However, the idea of free choice that marked the traditional way of doing science faced limits since the end of the twentieth century. First, there were no resources for all research fronts, so choices had to be made. Therefore, governments began to direct funding toward research that was supposed to benefit society. Second, society’s problems became more complex, demanding stronger connections between different players to find solutions. More recently, the United Nations Sustainable Development Goals (SDG) brought the concept of “big challenges,” requiring researchers to address problems of a complex and non-linear nature (George et al., 2016George, G., Howard-Grenville, J., Aparna, J., & Tihanyi, L. (2016). Understanding and tackling societal grand challenges through management research. Academy of Management Journal, 59(6), 1869-1877. Recuperado dehttps://doi.org/10.5465/amj.2016.4007
https://doi.org/10.5465/amj.2016.4007... ), usually in a transdisciplinary way and with participants from outside the academy.

This change required changes in the way of assessing scientific research. At first, the idea of total autonomy meant that academics would not be evaluated. Later, with resource rationalization, the first control models focused on scientific publications and on patents emerged, as well as metrics such as citation and publication indicators, and impact factors. In addition, altmetrics came to use, such as indicators of paper downloads.

Over time, it became clear that these metrics only assessed the research impact among academics, and did not consider wider audiences. The United Kingdom was one of the pioneers in creating a more comprehensive evaluation system, by implementing the REF evaluation cycles in 2008. That model focused on impact beyond the academy, with assessment based on a qualitative case study methodology, showing the effects on research beneficiaries. Despite criticisms to the model, studies have pointed out that the fact that academics prepare cases to submit to REF already provides a better understanding and engagement regarding impact, in addition to encouraging interdisciplinary collaboration and relationships with research users (Manville et al., 2015Manville, C., Jones, M. M., Henham, M. L., Castle-Clarke, S., Frearson, M., Gunashekar, S … Grant, J.. (2015). Preparing impact submissions for REF 2014: an evaluation approach and evidence. Santa Monica, CA: Rand Europe. Recuperado de https://www.rand.org/pubs/research_reports/RR726.html
https://www.rand.org/pubs/research_repor... ).

The idea that research evaluation models should consider different stakeholders and different types of impact, using both qualitative and quantitative indicators, was strengthened. A survey commissioned by the Academy of Management and addressed to its members revealed that, while peers are still seen as the most important audience for research papers, most participants considered relevant the impact on both public and private practice, as well as the need for dialoguing with managers and government members (Haley, Page, Pitsis, Rivas, & Yu, 2017Haley, U. C. V., Page, M.C., Pitsis, T.S., Rivas, J. L., & Yu. K. F. (2017). Measuring and achieving scholarly impact: a report from the Academy of Mangement’s Practice Theme Committee. Recuperado dehttps://doi.org/10.13140/RG.2.2.11747.86561
https://doi.org/10.13140/RG.2.2.11747.86... ).

Conceptually, impact evaluation models can involve different interest groups (Aguinis et al., 2014Aguinis, H., Shapiro, D. L., Antonacopoulou, E. P., & Cummings, T. G. (2014). Scholarly impact: a pluralist conceptualization. Academy of Management Learning and Education, 13(4), 623-639. Recuperado dehttp://dx.doi.org/10.5465/amle.2014.0121
https://doi.org/10.5465/amle.2014.0121... ). Hence, there are several types of impact: organizational; economic; environmental; social; academic; educational; public policies; health; and innovation (Reale et al., 2018Reale, E., Avramov, D., Canhial, K., Donovan, C., Flecha, R., Holm … Van Horik, R. (2018). A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation, 27(4), 298-308. Recuperado dehttps://doi.org/10.1093/reseval/rvx025
https://doi.org/10.1093/reseval/rvx025... ; Wickert et al., 2021Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that makes a difference: broadening the meaning of impact. Journal of Management Studies, 58(2), 297-320. Recuperado dehttps://doi.org/10.1111/joms.12666
https://doi.org/10.1111/joms.12666... ).

Evaluation models can also vary, since they consider, first, different perspectives (evaluating the whole process of impact generation or just the results); second, different pathways (from dissemination of contributions to co-creation, or even to a leading role in social change) (Muhonen, Benneworth, & Olmos-Penuela, 2020Muhonen, R., Benneworth, P., & Olmos-Penuela, J. (2020). From productive interactions to impact pathways: understanding the key dimensions in developing SSH research societal impact. Research Evaluation, 29(1), 34-47. Recuperado dehttps://doi.org/10.1093/reseval/rvz003
https://doi.org/10.1093/reseval/rvz003... ); third, different tools (Reale et al., 2018Reale, E., Avramov, D., Canhial, K., Donovan, C., Flecha, R., Holm … Van Horik, R. (2018). A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation, 27(4), 298-308. Recuperado dehttps://doi.org/10.1093/reseval/rvx025
https://doi.org/10.1093/reseval/rvx025... ); and fourth, different indicators (qualitative and quantitative).

In a literature review on how to assess research impact, Reed et al. (2021Reed, M. S., Ferre, M., Martin-Ortega, J., Blanche, R., Lawford-Rolfe, R., Dallimer, M … Holden, J.. (2021). Evaluating impact from research: A methodological framework. Research Policy, 50(4), 104147. Recuperado dehttps://doi.org/10.1016/j.respol.2020.104147
https://doi.org/10.1016/j.respol.2020.10... ) mention a typology of systems and recommend that institutions create an impact plan and model based on their goals (formative feedback or summative evaluation), and on their context (types of impact that matter, available resources, etc.).

These new assessment models are more complex in their design and operationalization, compared to traditional ones. Moreover, it is difficult to isolate what are actually research results and what are external factors that can affect the results.

In Brazil, the Coordination for the Improvement of Higher Education Personnel (Capes), a foundation linked to the Ministry of Education and Culture and responsible for graduate courses in the country, adopted in the 2021 evaluation cycle a multidimensional model that covers, in addition to training, research, and internationalization, the dimensions of knowledge transfer, innovation, regional insertion, and impact on society (MEC, 2019).

Considering this context and its challenges, we present below the path taken to develop the impact assessment model. As mentioned earlier, such a model (or tool) results from the combination of previous developments, plus a specific mode of operationalization.

METHODS

This section presents the methodological approach used in this paper. Initially, we describe the Capes prInt Program and the Capes prInt FGV Program, relating them to the paper goals. Next, we describe the methods of data collection and analysis.

FGV’s projects were analyzed for the following reasons: first, the interest and openness of the institution in carrying out the analysis, seeking to increase the potential impact of its research activities; second, the fact that the program was carried out in partnership with the United Kingdom, whose research policy is oriented towards practical impact; third, projects’ diversity in terms of areas and topics; fourth, the relevance to society of the subjects addressed in the projects; and fifth, the high potential for learning generation.

Object

The Capes prInt Program, or Institutional Program for Internationalization (Print), is an initiative of Capes. According to the institution (MEC, 2019), the objective of the program is to foster the internationalization of the selected institutions. It mainly involves building international research networks and researchers’ mobility.

The main focus of the program is internationalization. However, despite the incentive to academic impact, the program covered projects that addressed relevant topics to society, with potential impacts on different areas. Barbosa et al. (2022Barbosa, J. C., Ferreira, M. F., Paiva, C. C., Patrício, K. P., Silva, D. H. S., & Portela, J. C. (2022). A experiência da UNESP com a Agenda 2030: a governança universitária como indutora de ações e articulações para o enfrentamento dos desafios locais e globais. Revista Latinoamericana de Ciencias de la Comunicación, 21(41), 132-146. Recuperado de http://revista.pubalaic.org/index.php/alaic/article/view/945
http://revista.pubalaic.org/index.php/al... ) show how the program helped Unesp restructure its internationalization academic projects, according with the UN 2030 Agenda. Similarly, the program carried out at FGV sought to advance internationalization and generate benefits to society.

The Capes prInt FGV Project comprised 23 projects, carried out in seven schools of the institution, involving around 200 researchers. The total budget was approximately R$ 15 million. Among several topics, the projects covered assessment of public policies; improvement of economic regulation; enhancement of management in education, health, and environment; promotion of entrepreneurship, urban management, innovation in management, and digitalization. The total execution period was five years.

Data collection and analysis

The assessment was carried out between March and December 2021, and some projects were not fully completed. The impact potential of most of them had not been satisfactorily accomplished yet. Researchers considered this condition as positive, as an opportunity for additional learning.

The assessment was done in six stages, by three evaluators:

The assessment comprised three blocks of information. The first block concerns general information for identifying the project, complemented by indications of the impact - as perceived by each interviewee - and the “intermediate” (or internal) goals of each project, where internationalization activities were addressed.

The second block consists of the mode of knowledge production (Gibbons et al., 1994Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: the dynamics of science and research in contemporary societies. London, UK: Sage Publishing House.). The objective was to measure the proximity of project management to the more traditional approach - academy-oriented, disciplinary, and for publication in top journals - or, conversely, closeness to alternative modes - multidisciplinary or transdisciplinary and practice-oriented.

The use of Gibbons et al.’s (1994Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: the dynamics of science and research in contemporary societies. London, UK: Sage Publishing House.) model (mode 1 and mode 2) is justified by its relevance and recognition. The model stands out in articles on the subject of knowledge production, and gave rise to extensive debates in the early 2000s and the following decade (Hodgkinson, Herriot, & Anderson, 2001Hodgkinson, G. P., Herriot, P., & Anderson, N. (2001). Re-aligning the stakeholders in management research: lessons from industrial, work and organizational psychology. British journal of Management, 12(s1), S41-S48. Recuperado dehttps://doi.org/10.1111/1467-8551.12.s1.5
https://doi.org/10.1111/1467-8551.12.s1.... ; Hodgkinson & Starkey, 2011Hodgkinson, G. P., & Starkey, K. (2011). Not simply returning to the same answer over and over again: reframing relevance. British Journal of Management, 22(3), 355-369. Recuperado dehttps://doi.org/10.1111/j.1467-8551.2011.00757.x
https://doi.org/10.1111/j.1467-8551.2011... ; Pettigrew, 2001Pettigrew, A. (2001). Management research after modernism. British Journal of Management, 12(s1), s61-s70. Recuperado dehttps://doi.org/10.1111/1467-8551.12.s1.8
https://doi.org/10.1111/1467-8551.12.s1.... ; Starkey & Madan, 2001Starkey, K. & Madan, P. (2001), Bridging the relevance gap: aligning stakeholders in the future of management research. British Journal of Management, 12(1), s3-s26. Recuperado dehttps://doi.org/10.1111/1467-8551.12.s1.2
https://doi.org/10.1111/1467-8551.12.s1.... ; Willmott, 2012Willmott, H. (2012). Reframing relevance as “social usefulness”: a comment on Hodgkinson and Starkey’s “Not simply returning to the same answer over and over again”. British Journal of Management, 23(4), 598-604. Recuperado dehttps://doi.org/10.1111/j.1467-8551.2012.00839.x
https://doi.org/10.1111/j.1467-8551.2012... ). A relevant part of criticisms has focused on the alleged dichotomous nature of the model, leading to the defense, based on this argument, of a possible hybrid mode, or 1.5 mode (Huff, 2000Huff, A. (2000). 1999 presidential address: changes in organizational knowledge production. Academy of Management Review, 25(2), 288-293. Recuperado dehttps://doi.org/10.5465/amr.2000.3312916
https://doi.org/10.5465/amr.2000.3312916... ). In fact, as we explain ahead, the findings of the present study are in line with this proposal, which does not seem to limit the usefulness of the original model.

The third block is directly associated with impact in five dimensions: scientific, practical, social, public policies, and educational (Wickert et al., 2021Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that makes a difference: broadening the meaning of impact. Journal of Management Studies, 58(2), 297-320. Recuperado dehttps://doi.org/10.1111/joms.12666
https://doi.org/10.1111/joms.12666... ). In addition to identifying the type(s) of impact generated, this block also sought to identify the level of achievement, coverage, and intensity of the various types of impact.

The use of Wickert et al.’s (2021Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that makes a difference: broadening the meaning of impact. Journal of Management Studies, 58(2), 297-320. Recuperado dehttps://doi.org/10.1111/joms.12666
https://doi.org/10.1111/joms.12666... ) dimension model is supported by its growing recognition among academics, its simplicity, and the adequacy of these dimensions to the study object, which consists of applied social science projects. Moreover, it seems to represent the apex of a process of theoretical development of the topic, focusing on the multidimensionality of impact. Indeed, in our bibliographic research, we found texts addressing the multidimensionality of research impact (Razmgir, Panahi, Ghalichi, Mousavi, & Sedghi, 2021Razmgir, M., Panahi, S., Ghalichi, L., Mousavi, S. A. J., & Sedghi, S. (2021). Exploring research impact models: a systematic scoping review. Research Evaluation, 30(4), 443-457. Recuperado dehttps://doi.org/10.1093/reseval/rvab009
https://doi.org/10.1093/reseval/rvab009... ; Reale et al., 2018Reale, E., Avramov, D., Canhial, K., Donovan, C., Flecha, R., Holm … Van Horik, R. (2018). A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation, 27(4), 298-308. Recuperado dehttps://doi.org/10.1093/reseval/rvx025
https://doi.org/10.1093/reseval/rvx025... ; Reed et al., 2021Reed, M. S., Ferre, M., Martin-Ortega, J., Blanche, R., Lawford-Rolfe, R., Dallimer, M … Holden, J.. (2021). Evaluating impact from research: A methodological framework. Research Policy, 50(4), 104147. Recuperado dehttps://doi.org/10.1016/j.respol.2020.104147
https://doi.org/10.1016/j.respol.2020.10... ). Viewed together, these texts support Wickert et al.’s (2021Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that makes a difference: broadening the meaning of impact. Journal of Management Studies, 58(2), 297-320. Recuperado dehttps://doi.org/10.1111/joms.12666
https://doi.org/10.1111/joms.12666... ) model, as they present similar logic, differing in the dimensions indicated. In brief, the model by Wickert et al. (2021Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that makes a difference: broadening the meaning of impact. Journal of Management Studies, 58(2), 297-320. Recuperado dehttps://doi.org/10.1111/joms.12666
https://doi.org/10.1111/joms.12666... ) has advantages because of its dimensions, and because it has elements that facilitate its operationalization in the context of this study. Obviously, studies carried out in different contexts may review or add impact dimensions.

For the questions in the last two blocks, we created “templates” with parameters and examples of answers. The interviewer would ask an open-ended question and, according to the respondent’s answer, would allocate the project characteristic in one of the pre-defined levels or classifications of the “template”. Such parts of these interviews were based on a method developed by Bloom and Van Reenen (2006Bloom, N., & Van Reenen, J. (2006). Measuring and explaining management practices across firms and countries (Discussion Paper n. 716). London, UK: London School of Economics and Political Science.).

To evaluate the impact of the projects, we used a multidimensional model (Figure 1), adapted from Wickert et al. (2021Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that makes a difference: broadening the meaning of impact. Journal of Management Studies, 58(2), 297-320. Recuperado dehttps://doi.org/10.1111/joms.12666
https://doi.org/10.1111/joms.12666... ). In the case of scientific impact, we considered the publication of a scientific article in a well-qualified journal or with a high number of citations. Organizational impact considers changes in procedures, processes, and practices, which generate benefits for organizations. Social impact refers to changes in procedures, processes, and practices that generate benefits to society. Public policy impact regards changes in public policies that address relevant social issues (or their creation). Educational impact is usually associated with the production of teaching materials or content that support and innovate teaching and learning processes.

The main adaption from the original model refers to the change of name: from practical impact to organizational impact. The original model uses the term ‘practical impact’ to refer to the impact on organizations. The change was made to make the term clearer, since “practical impact” could also be understood as related to other dimensions, such as social. In addition, the term ‘scholarly’ was translated as scientific; social (more comprehensive and of common use) was adopted, instead of ‘societal’ (which could be associated with social relations, although it also refers to society in general). Box 1 provides a summary of the “template”, showing, as examples, two pieces of evidence for each type of impact.

A research project can generate simultaneous impacts on several dimensions. As a support to the model, in order to facilitate evaluators’ task, we created a box with examples for each dimension of impact, and examples of evidence also for each dimension, adapted from the REF model (2020).

As indicated at the beginning of this section, for each project we created a dashboard with overview of the information. Dashboards were sent to project leaders for approval, with minor changes after feedback from some of them. The purpose was to provide a concise view of projects’ characteristics and impact. Figure 2 shows an unfilled dashboard.

In addition to the dashboard, we also created a project portfolio, to provide an overview of all of them. To facilitate visualization and analysis, we chose the traditional 2x2 matrix format, where the Y-axis represents the potential versus actual impact, and the X-axis shows academic versus practical impact. The academic impact comprised two dimensions: scientific and educational. The practical impact comprised three dimensions: practical, social, and public policies.

RESULTS

Meeting internationalization goals

An analysis of internationalization activities indicates that specific goals were partially compromised, since researcher’s mobility was heavily affected by the COVID-19 pandemic. Missions, fellowships, workshops, faculty visits, and participation in conferences were mostly cancelled. According to the perception of the 23 interviewees, 14 projects had a low level of goal achievement; 7 projects showed had a medium level; and only 2 projects showed a high level of accomplishment. However, most of the research projects managed to strengthen international exchange through virtual interaction. Therefore, we conclude that, even in an exceptional context, there was significant progress in internationalization, allowing the establishment of bases for deepening cooperative processes.

Mode of knowledge production

The compilation of answers to the questions of the block related to the mode of knowledge production enabled creating a general overview of the 23 projects, presented in Box 2.

Most of the projects had a scientific focus, were carried out only by academics, and addressed a single discipline. A significant number of projects used knowledge from other disciplines. For dissemination, they used scientific journals, practitioner-oriented media, and direct communication to practitioners.

This result reflects the practice of hybrid models of knowledge production, which combine elements of mode 1 and mode 2. Therefore, it is close to Huff’s (2000Huff, A. (2000). 1999 presidential address: changes in organizational knowledge production. Academy of Management Review, 25(2), 288-293. Recuperado dehttps://doi.org/10.5465/amr.2000.3312916
https://doi.org/10.5465/amr.2000.3312916... ) proposal, since projects often take, as a starting point, relevant topics for practice; research takes place within the academy, with little participation of practitioners; there are constant contributions from other disciplines; and results are spread by multiple channels, both in the academy and externally. In other words, project execution has moved away from mode 1 in some practices, without fully adopting mode 2 practices.

Project impact

The compilation of answers to the questions of the block related to impact enabled creating a general overview of the 23 projects. Boxes 3A, 3B, and 3C show this summary.

Based on boxes’ analysis, some results stand out, by type of impact:

The results reveal significant academic impacts (scientific and educational) and relatively few practical impacts (on public, organizational, and social policies). This finding is consistent with mentions found in theory on the issue of temporality: studies that intend or have the potential to generate “real-world” change may require a long maturation time, depending on the action of multiple agents (Bornmann, Haunschild, & Adams, 2019Bornmann, L., Haunschild, R., & Adams, J. (2019). Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on 8 International Journal of Qualitative Methods data from the UK Research Excellence Framework (REF). Journal of Informetrics, 13(1), 325-340. Recuperado dehttps://doi.org/10.1016/j.joi.2019.01.008
https://doi.org/10.1016/j.joi.2019.01.00... ; Lauronen, 2020Lauronen, J. P. (2020). The dilemmas and uncertainties in assessing the societal impact of research. Science and Public Policy, 47(2), 207-218. Recuperado dehttps://doi.org/10.1093/scipol/scz059
https://doi.org/10.1093/scipol/scz059... ). At the time of assessment, most of the project had been completed a few months before.

Examples of impact

Interviews also allowed identifying several remarkable cases of impact, comprising impacts on public policies, organizational practices, public debates, and educational practices. Box 4 shows some of them.

The results show that these projects had a significant practical impact, especially if we consider that this type of impact usually has a long maturation time. However, we should consider the institutional weight of FGV, as well as its influence on the debate on public policies, mainly in the area of Economics, which obviously favors practical impacts. Considering the institution’s history, this condition may be difficult to replicate.

Nevertheless, other institutions, especially state and federal public universities, also have (real or potential) power of influence on their communities, whether regions or states. Others, with a more specific scope, such as Fiocruz, Cebrap, Ipea, Embrapa, and Embrapii, also have the reputation and means to leverage (even more) the influence of their research and generate impacts on their practice.

Project portfolio

As previously mentioned, the preparation of the project portfolio aimed to provide an overview of the 23 initiatives covered by the Capes prInt FGV project. Figure 3a shows the distribution of projects (represented by numbers) during the analysis done in the second half of 2021. Figure 3b shows a projection made by project leaders for the following three years.

Figure 3a indicates a strong concentration of projects in the real academic impact cell, showing the predominance of academic orientation and the achievement of academic impact (specifically, publications); and also a significant number of projects with potential for practical impact.

The comparison between the two figures reveals an optimistic perspective of project leaders regarding practical impacts. Only three projects were restricted to academic impact. In the leaders’ view, the remaining twenty projects, within three years, will generate some kind of practical impact.

This vision should be celebrated. However, moving from potential impact to real impact will still require, from leaders and their teams, an additional effort of communication and influence from external agents. In fact, this seems to be a key issue for generating impact; therefore, we address it in the next section.

LESSONS LEARNED

This section deals with four topics that emerged from the analysis of the impact evaluation results of the 23 projects. The first refers to the timing of assessment. The second and third regard issues related to the mode of knowledge production. The last refers to differences between fields of knowledge.

Assessment took place before the peak of impact in several projects

Research projects can have long cycles, from conception to publication of scientific articles and observation of impacts, whether academic or practical. Most of the projects assessed in this study were either completed at the time of analysis, or were close to completion, but still in the preliminary stage of dissemination.

This condition limits the impact and its evaluation, but does not prevent them. This happens for two reasons: first, because some of the projects analyzed had shorter cycles; second, because certain impacts can occur throughout the execution of the project, even in early stages.

The characteristics of the dimensions can define different times for achieving impact. While educational impact can occur quickly, through the creation of courses and disciplines, social impact usually takes time to happen. Evaluation models focused on results in a given period, as in this case, cannot capture the whole impact trajectory. A solution to this problem would be to combine result evaluation with assessment of the impact production process (Reale et al., 2018Reale, E., Avramov, D., Canhial, K., Donovan, C., Flecha, R., Holm … Van Horik, R. (2018). A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation, 27(4), 298-308. Recuperado dehttps://doi.org/10.1093/reseval/rvx025
https://doi.org/10.1093/reseval/rvx025... ).

The traditional mode of knowledge production is dominant

The analysis of all projects showed that the traditional way of knowledge production is prevalent. Among the projects analyzed, 16 had a scientific focus, while only five had an applicable approach, and two had an applied focus. While 21 projects were developed just by academics, only two had the participation of academics supported by practitioners. There were no projects carried out in a knowledge co-creation regime. Twenty projects were developed within a discipline or in a discipline with contributions from others, while only three showed a multidisciplinary or transdisciplinary character.

It was possible to notice signs of change regarding the dissemination of knowledge. Although all projects focus on scientific journals, as expected, given the nature of the Capes prInt project, some of them also focus on media and journals for practitioners (10 projects) and direct communication to practitioners (eight projects).

As some studies have shown (Haley et al., 2017Haley, U. C. V., Page, M.C., Pitsis, T.S., Rivas, J. L., & Yu. K. F. (2017). Measuring and achieving scholarly impact: a report from the Academy of Mangement’s Practice Theme Committee. Recuperado dehttps://doi.org/10.13140/RG.2.2.11747.86561
https://doi.org/10.13140/RG.2.2.11747.86... ), although some academics have awakened to the importance of other audiences, they still prioritize their peers and traditional evaluation metrics. To a large extent, this occurs because the institutions they work for - and often also the government bodies that fund research - encourage the effort for publishing in high-impact journals more than the dimensions of practical impact.

Dissemination and influence on practice are still scarce

Despite the numbers mentioned in the previous paragraph, in general dissemination and influence are still limited. The interviews revealed that a significant number of researchers still do not perceive their activity as a “venture”, since the achievement of social goals depends on doing complementary activities to research. The pattern is still academic isolation, with little time for activities of connection with external agents.

Baldridge et al. (2004Baldridge, D. C., Floyd, S. W., & Markóczy, L. (2004). Are managers from Mars and academicians from Venus? Toward an understanding of the relationship between academic quality and practical relevance. Strategic Management Journal, 25(11), 1063-1074. Recuperado dehttp://doi.org/10.1002/smj.406
https://doi.org/10.1002/smj.406... ) showed that research carried out through the traditional approach can bring contributions to practice, but it is still a gap. In Brazil, Lazzarini (2017Lazzarini, S. (2017). Pesquisa em Administração: em busca de impacto social e outros impactos. Revista de Administração de Empresas, 57(6), 620-625. Recuperado dehttps://doi.org/10.1590/S0034-759020170608
https://doi.org/10.1590/S0034-7590201706... ) and Mendes-da-Silva (2019Mendes-da-Silva, W. (2019). Convergência, comunicação, e impacto da pesquisa em negócios. Revista de Administração Contemporânea, 23(1), 1-7. Recuperado dehttps://doi.org/10.1590/1982-7849rac2019180346
https://doi.org/10.1590/1982-7849rac2019... ) drew attention to the need for opening more communication channels between academic and organizations’ audiences, thus echoing the proposals of their foreign colleagues.

Of course, a shift towards a greater connection with practice is not trivial. It depends on acquiring new competencies - in communication, negotiation, and influence, among others - and time allocation (always a scarce resource) in tasks that, for many researchers, do not make up their core activities.

In this case, it is important to create a culture of impact. In the United Kingdom, after more than a decade of implementing REF, this is already happening (Manville et al., 2015Manville, C., Jones, M. M., Henham, M. L., Castle-Clarke, S., Frearson, M., Gunashekar, S … Grant, J.. (2015). Preparing impact submissions for REF 2014: an evaluation approach and evidence. Santa Monica, CA: Rand Europe. Recuperado de https://www.rand.org/pubs/research_reports/RR726.html
https://www.rand.org/pubs/research_repor... ). However, basic difficulties remain, such as understanding what impact really is. A study by Samuel and Derrick (2015Samuel, G. N., & Derrick, G. E. (2015). Societal impact evaluation: exploring evaluator perceptions of the characterization of impact under the REF2014. Research Evaluation, 24(3), 229-241. Recuperado dehttps://doi.org/10.1093/reseval/rvv007
https://doi.org/10.1093/reseval/rvv007... ) with REF evaluators showed different perceptions of impact. In the assessment carried out in the present study, respondents also showed very different understandings on this matter. For many, impact would be missions and international exchange scholarships, factors that the assessment model considers intermediate goals that can lead to impact.

Knowledge fields are different in terms of connection with practice

The interviews made us notice differences in the processes of knowledge creation and communication among fields of knowledge. This condition can relate to several factors, more present in some fields than in others, such as: the existence of a “critical mass” of PhDs working in companies, thus facilitating contact with academics; a culture of valuing science among practitioners; and associations, events, and publications that connect academics and practitioners.

The knowledge fields that show these characteristics can exploit them to their advantage by using such channels. First, to facilitate collaborative work, also involving the co-creation of knowledge; and, second, to communicate their findings more easily, thus enhancing their impact.

DISCUSSION

This section presents the contributions to theory and practice. We expect that this paper has contributed to building models that meet the demands of research institutions, funders, and regulators.

Contributions to theory

This paper dialogues with the literature on alternative modes of knowledge production (Eisenhardt et al., 2016Eisenhardt, K., Graebner, M. E., & Sonenstein, S. (2016). From the editors: grand challenges and inductive methods: rigor without rigor mortis. Academy of Management Journal, 59(4), 1113-1123. Recuperado dehttps://doi.org/10.5465/amj.2016.4004
https://doi.org/10.5465/amj.2016.4004... ; Etzkowitz & Leydesdorff, 1995Etzkowitz, H., & Leydesdorff, L. (1995). The triple helix university-industry-government relations: a laboratory for knowledge based economic development. EASST Review, 14(1), 14-19. Recuperado de https://ssrn.com/abstract=2480085
https://ssrn.com/abstract=2480085... ; Etzkowitz et al., 2000; George et al., 2016George, G., Howard-Grenville, J., Aparna, J., & Tihanyi, L. (2016). Understanding and tackling societal grand challenges through management research. Academy of Management Journal, 59(6), 1869-1877. Recuperado dehttps://doi.org/10.5465/amj.2016.4007
https://doi.org/10.5465/amj.2016.4007... ; Gibbons et al., 1994Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: the dynamics of science and research in contemporary societies. London, UK: Sage Publishing House.). More specifically, it contributes with a real case, which shows a hybrid condition that could be described metaphorically as bricolage, by combining features of traditional and new modes. Thus, it shows in practice what theorists like Huff (2000Huff, A. (2000). 1999 presidential address: changes in organizational knowledge production. Academy of Management Review, 25(2), 288-293. Recuperado dehttps://doi.org/10.5465/amr.2000.3312916
https://doi.org/10.5465/amr.2000.3312916... ) have conceptually outlined.

This text also dialogues with the literature on research impact assessment, by presenting a method developed for a particular purpose and setting, and its application. Despite its specificity, the case adds knowledge, especially to the body of theory addressing multidimensional approaches for impact assessment (Razmgir et al., 2021Razmgir, M., Panahi, S., Ghalichi, L., Mousavi, S. A. J., & Sedghi, S. (2021). Exploring research impact models: a systematic scoping review. Research Evaluation, 30(4), 443-457. Recuperado dehttps://doi.org/10.1093/reseval/rvab009
https://doi.org/10.1093/reseval/rvab009... ; Reale et al., 2018Reale, E., Avramov, D., Canhial, K., Donovan, C., Flecha, R., Holm … Van Horik, R. (2018). A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation, 27(4), 298-308. Recuperado dehttps://doi.org/10.1093/reseval/rvx025
https://doi.org/10.1093/reseval/rvx025... ; Reed et al., 2021Reed, M. S., Ferre, M., Martin-Ortega, J., Blanche, R., Lawford-Rolfe, R., Dallimer, M … Holden, J.. (2021). Evaluating impact from research: A methodological framework. Research Policy, 50(4), 104147. Recuperado dehttps://doi.org/10.1016/j.respol.2020.104147
https://doi.org/10.1016/j.respol.2020.10... ). Evaluating research impact is a major challenge, given the characteristics of the activity, which occurs in an environment separate from the academy, and the process has a long maturation period, being influenced by multiple agents.

This study shows the use of an impact evaluation model that can be a pattern for different knowledge areas in the social sciences. The system is also cost-effective and allows considering maturation times according to different impact dimensions.

In addition, it helps elucidate the alleged contradiction between rigor and relevance or, specifically in this case, potential conflicts between the purpose of internationalization and goals related to practical impacts and benefits to society and organizations.

Such goals - internationalization and impact generation - may eventually be seen as contradictory. The first supposedly leads to prioritizing one type of impact, the scientific, and seeking publications in top international journals. The second allegedly leads to emphasizing topics linked to issues of organizations, economics, culture, public policies, technology, and other social dimensions. We can say that the first objective focuses on rigor, and the second, on relevance.

Indeed, this alleged dichotomy has been extensively discussed (for a historical review, see Wood et al., 2022Wood, T. Jr., Souza, R., & Caldas, M. O. (2022), The relevance of management research debate: a historical view - 1876-2018. Journal of Management History, 28(3), 409-427. Recuperado de https://doi.org/10.1007/s11213-022-09596-x
https://doi.org/10.1007/s11213-022-09596... ). Although the debate has not converged to a definitive conclusion, it has shown a trend, for some time, towards an integrative perspective (Pettigrew, 2001Pettigrew, A. (2001). Management research after modernism. British Journal of Management, 12(s1), s61-s70. Recuperado dehttps://doi.org/10.1111/1467-8551.12.s1.8
https://doi.org/10.1111/1467-8551.12.s1.... , 2011Pettigrew, A. (2011). Scholarship with impact. British Journal of Management, 22(3), 347-354l. Recuperado dehttps://doi.org/10.1111/j.1467-8551.2011.00769.x
https://doi.org/10.1111/j.1467-8551.2011... ; Starkey & Madan, 2001Starkey, K. & Madan, P. (2001), Bridging the relevance gap: aligning stakeholders in the future of management research. British Journal of Management, 12(1), s3-s26. Recuperado dehttps://doi.org/10.1111/1467-8551.12.s1.2
https://doi.org/10.1111/1467-8551.12.s1.... ). This is explained, first, by the fact that relevance and rigor characterize high quality, and their quest presents challenges for researchers. For a long time, they have argued that scientific rigor and practical relevance are different paradigms or systems of knowledge production (Astley & Zammuto, 1992Astley W. G., & Zammuto, R. F. (1992). Organization science, managers, and language games. Organization Science, 3(4), 443-460. Recuperado dehttps://doi.org/10.1287/orsc.3.4.443
https://doi.org/10.1287/orsc.3.4.443... ; Kieser & Leiner, 2009Kieser A., & Leiner L. (2009). Why the rigor-relevance gap in management research is unbridgeable. Journal of Management Studies, 46(3), 516-533. Recuperado dehttps://doiorg/101111/j1467-6486200900831x
https://doi.org/101111/j1467-64862009008... ; Rynes, Bartunek, & Daft, 2001Rynes, S. L., Bartunek, J. M., & Daft, R. L. (2001). Across the great divide: knowledge creation and transfer between practitioners and academics. Academy of management Journal, 44(2), 340-355. Recuperado dehttps://doi.org/10.5465/3069460
https://doi.org/10.5465/3069460... ). As noted by Wood, Holz, and Souza (2022Wood, T. Jr., Souza, R., & Caldas, M. O. (2022), The relevance of management research debate: a historical view - 1876-2018. Journal of Management History, 28(3), 409-427. Recuperado de https://doi.org/10.1007/s11213-022-09596-x
https://doi.org/10.1007/s11213-022-09596... ), this perspective has led to the identification and analysis of mechanisms that connect both systems (Rolfsen, Johnsen, & Knutstad, 2007Rolfsen, M., Johnsen, A., & Knutstad, G. (2007). Action engagement: improving researchers’ involvement in action research projects. Systemic Practice and Action Research, 20(1), 53-63. Recuperado dehttps://doi.org/10.1007/s11213-006-9049-x
https://doi.org/10.1007/s11213-006-9049-... ; Tenkasi & Hay, 2004Tenkasi R. V., & Hay, G. W. (2004). Actionable knowledge and scholar-practitioners: a process model of theory-practice linkages. Systemic Practice and Action Research, 17(3), 177-206. Recuperado dehttps://doi.org/10.1023/B:SPAA.0000031697.76777.ac
https://doi.org/10.1023/B:SPAA.000003169... ). Second, because, as the most recent impact evaluation models indicate (Wickert et al., 2021Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that makes a difference: broadening the meaning of impact. Journal of Management Studies, 58(2), 297-320. Recuperado dehttps://doi.org/10.1111/joms.12666
https://doi.org/10.1111/joms.12666... ), studies often achieve different types of impact - scientific and organizational, or educational and social. Therefore, we could argue that this is a researcher or research teams’ decision, considering, of course, the institutional and environmental constraints, as well as the challenges inherent to the task.

Contributions to practice

We learned several lessons by analyzing the results of the 23 projects, which are highlighted and commented below. Some of them refer to the institution that carried out the projects, FGV, while others refer to the funding agency, Capes. Taken together, these lessons represent a potential contribution to research funding agencies, research managers, and researchers.

I. Disseminate and foster alternative modes of knowledge creation, focused on solving real challenges, characterized by multidisciplinarity or transdisciplinarity, by co-creation of knowledge with practitioners, and disseminated to practitioners.

The impact evaluation study checked the predominance of the traditional mode of knowledge creation, that is, with topics that “are born” within the academy, are developed within specific disciplines, and complete their cycle by publishing an article in a scientific journal for researchers.

The use of alternative modes of knowledge creation could guide research toward topics of greater practical and social relevance, and enhance the impact on organizations, public policies, and society.

II. Fostering practical impacts - organizational, social, and on public policies - in parallel with academic impacts - scientific and educational.

The interviews revealed that researchers’ main target is science, that is, their main motivation is to write articles that will be published in top international journals. In addition, researchers focus on developing teaching content, courses, and programs. Obviously, these goals are noble. Moreover, they are relatively easier to achieve, since they align with the agents’ own activities: research and teaching.

However, the need to make science more responsive to society demands leads to another class of impacts. Putting efforts on generating organizational, social, and public policy impacts becomes a priority. This means valuing this class of impacts, guiding researchers to pursue them, and creating the institutional conditions to support them.

III. Encourage post-project activities that enhance practical impact: dissemination, mobilization of social actors, etc.

This study found that a significant number of projects are in the “applicable project” stage, that is, they addressed a relevant topic and generated knowledge and products with potential to change organizational and social practices, or public policies. What separates the potential impact from the achieved impact is an additional effort, for example, of dissemination and mobilization of social players.

We should consider that these are not “natural activities” for researchers. For many of them, doing this means leaving their comfort zone, which requires communication and negotiation skills that can be challenging.

IV. Foster practices and the culture of multidimensional evaluation of the impact of research projects, based on evidence and at predetermined time intervals.

Academic culture involves basic values, beliefs, and practices that characterize the university environment, and traditionally assume a monotonic perspective on impact. This culture, by all means, conditions researchers to pursue impact in science, which occurs through innovative and rigorous research that contributes to advancing the frontier of knowledge.

Research with practical impact, aimed at solving real and urgent problems, demands the incorporation of other dimensions related to impact - educational and those that occur in organizational and social practices, and on public policies. In addition, it is necessary to introduce systems for measuring such impacts. For many decades, scientific impact has been measured through many forms and indicators. It is necessary to create the equivalent for the other dimensions of impact. Obviously, their characteristic will lead to adopting qualitative indicators and accepting a certain level of subjectivity in measurement.

However, creating indicators and making them operational is not a trivial task. A report published by the Association to Advance Collegiate Schools of Business reveals dozens of indicators (AACSB, 2012The Association to Advance Collegiate Schools of Business. (2012). Impact of research: a guide for business schools. Tampa, FL: AACSB International. Recuperado de https://www.aacsb.edu/insights/reports/impact-of-research-a-guide-for-business-schools
https://www.aacsb.edu/insights/reports/i... ). In fact, research impact is a multidimensional construct, and it can occur over time, as a result of multiple interactions with different agents. Measuring can also involve considerable time and resources. However, measuring the social impact of research is difficult but necessary (Smith, 2001Smith, R. (2001). Measuring the social impact of research: difficult but necessary. BMJ, 323(7312), 528. Recuperado dehttps://doi.org/10.1136/bmj.323.7312.528
https://doi.org/10.1136/bmj.323.7312.528... ). The UK’s REF (2020) asks universities for cases that show practical impact. Its assessment is considered to be one of the most comprehensive in the world, which grants it a wide and recognized expertise. This study was built on this approach, adding a parameterization of answers. The procedure proved feasible and applicable to different fields. Future studies should face the challenge of unfolding the assessments and indicators.

V. Recognize the differences between fields of knowledge, enhancing their specific characteristics that constitute advantages for generating impact.

As mentioned in the discussion section, project evaluation led to observing differences in the processes of generation and communication between fields of knowledge. More specifically, we found a greater proximity between academics and practitioners in the field of economics, compared to the field of administration. This condition facilitates, for economists, the “connection with practice”, enabling academics to communicate their ideas and theories more easily to the world outside the academy.

This condition can be used in their favor, in processes associated with applied research, enhancing and speeding up the practical impact. In addition, administrators, lawyers, and other researchers can be inspired by such contexts and practices, and seek to replicate them in their fields. It is not a matter of emulating institutional conditions, which may not be feasible, but of seeking to adapt certain practices creatively.

VI. Encourage the use of the Theory of Change in public notices, inducing bidders to relate activities, products, results, and impacts.

This study found a fragile and hazy relationship between the different outputs of a research project. We suggest the adoption of the Theory of Change for better structuring project proposals, as well as facilitating their analysis. According to Vogel (2012Vogel, I. (2012). Review of the use of ‘theory of change’ in international development. London, UK: Department for International Development. Recuperado de https://gsdrc.org/document-library/review-of-the-use-of-theory-of-change-in-international-development/
https://gsdrc.org/document-library/revie... ), the Theory of Change is a process that describes a sequence of events that leads to intended results, considering certain contexts and assumptions. We specifically recommend the model used by the University of Warwick, which summarizes this sequence in a figure based on activities that generate products, which in turn generate outcomes that lead to broader impacts (Warwick Institute for Global Sustainable Development, 2021Warwick Institute for Global Sustainable Development. (2021). URBE Latam theory of change at project proposal stage (UKRI GCRF). London, UK: Warwick University. Recuperado de https://warwick.ac.uk/fac/arts/schoolforcross-facultystudies/igsd/resources/hp-contents/urbe_latam_theory_of_change_-_proposal_stage.pdf
https://warwick.ac.uk/fac/arts/schoolfor... ). We believe that incorporating this approach will enable project leaders to communicate more clearly what they intend to achieve in terms of impact, providing evaluators with a better-informed idea of the potential and scope of a project’s impact.

CONCLUSION

Overview

This study established a method and sought to characterize the mode of knowledge creation and the impacts of 23 research projects carried out under the Capes prInt program. We found that the projects, as a whole, have advanced toward their internationalization goals. We also found that many projects have already generated academic impact, and most have potential to add practical impact. The text presented a framework on the mode of knowledge production used, and a structured analysis of the impacts achieved, including examples.

Limitations and future research

This article was based on a single case from a specific institution. The authors believe that the richness of the case generated interesting and valuable learning. However, this is a unique case, which requires further studies.

In order to add knowledge based on specific characteristics of other research fields, other studies could be carried out in different institutions, possibly involving projects more strongly characterized as hard science.

To track the impact of research projects over time, longitudinal studies should be conducted.

Possibly, through in-depth case studies, the success factors for generating practical impact could be investigated; that is, which activities and competencies should be incorporated by researchers in order to maximize the potential of their projects and bring changes and benefits for organizations, public policies, and society in general.

Future studies could include, besides the perception of project leaders, the perceptions of stakeholders, and/or “customers” related to the projects.

Final comment

There are increasing expectations and demands from society for academics to participate in solutions to complex problems. Programs such as Capes prInt are carried out with public resources, which should be reverted to the benefit of organizations and society.

In this case, collaboration between academics and practitioners is essential. However, these two groups belong to separate and distinct systems. They are driven by different interests, have different working paces, and often present different views on what, to whom, and how research resources should be directed. Solving this conflict is not a trivial task. The authors of this article expect that the learning shared here will stimulate colleagues to deepen their understanding of new modes of knowledge production and, above all, to incorporate mechanisms of connection with practice into their own research.

ACKNOWLEDGMENTS

The authors thank FGV Research Network for the financial support to Adriana Wilner, and to research assistants Amanda Gross and Vanessa dos Santos, for carrying out the field work. The authors also thank Livio Amaral for his comments on an initial version of this paper, and the evaluators and reviewers of Cadernos EBAPE.BR, for their detailed and constructive work.

REFERÊNCIAS

Publication Dates

History