ABSTRACT

Purpose

- This research aims to recognize the performance indicators relating to the management of species reported in the literature.

Design/methodology/approach

- It is an exploratory and descriptive research. It assumes the qualitative characteristics. We presented a theoretical proposal of indicators and sufficient dimensions to manage an IES. The board was selected by convenience. The expert group was composed of people who have expertise in managing performance.

Findings

- We found indicators that were added according to the suggestion of the respondents, resulting in 65 indicators, divided into 9 dimensions. The model contemplates the dimensions: Economic; Internationalization and research; Student/Client; Market; Academia; Operational; Human Resources; Society/extension and Sustainability.

Research limitations/implications

As limitations of the research, there is the possibility of some work on indicators that have not been identified at this stage of research, and also the choice of specialists.

Practical implications -

This study presents the determinants of business performance. The indicators that should be used for each type of organization are of utmost importance and were highlighted as one of the challenges of future research in the management of organizational performance.

Social implications

In the Brazilian literature, 75% of the surveys do not address performance management holistically, thus reinforcing this study’s importance and originality.

Originality/value

- Previous literature were identified 57 performance indicators, distributed in six 6 dimensions based on the concepts of multidimensionality. The work found indicators were added according to the suggestion of the interviewees, resulting in 65 indicators divided into 9 dimensions.

Keywords:
Educational Institutions; Performance indicators; Performance framework

RESUMO

Objetivos

- Esta pesquisa tem como objetivos reconhecer os indicadores de desempenho relacionados à gestão de IES relatados na literatura.

Metodologia

- É uma pesquisa exploratória e descritiva, que assume características qualitativas. Apresentamos uma proposta teórica de indicadores e dimensões suficientes para ogestão de uma IES. O quadro foi selecionado por conveniência. O grupo de especialistas foi composto por pessoas com experiência em gestão de desempenho.

Resultados

- Encontramos indicadores que foram adicionados de acordo com a sugestão dos entrevistados, resultando em 65 indicadores, divididos em 9 dimensões. O modelo contemplando dimensões: Econômicas; Internacionalização e pesquisa; Aluno/Cliente; Mercado; Academia; Operacional; Recursos Humanos; Sociedade/extensão e Sustentabilidade.

Limitações-

Como limitações da pesquisa, existe a possibilidade de alguns trabalhos sobre indicadores que não foram identificados nesta fase da pesquisa, e também a escolha de especialistas.

Implicações práticas -

Este estudo apresenta os determinantes do desempenho empresarial. Os indicadores que devem ser utilizados para cada tipo de organização são de extrema importância e foram destacados como um dos desafios de pesquisas futuras na gestão do desempenho organizacional.

Implicações sociais

Na literatura brasileira, 75% das pesquisas não abordam gestão de desempenho de forma holística, reforçando a importância e originalidade deste estudo..

Originalidade/valor

- Na literatura anterior foram identificados 57 indicadores de desempenho, distribuídos em seis 6 dimensões baseadas nos conceitos de multidimensionalidade. Os indicadores de trabalho encontrados foram agregados de acordo com a sugestão dos entrevistados, resultando em 65 indicadores divididos em 9 dimensões.

Palavras-chave:
Instituições de Ensino; Indicadores de desempenho; Desempenho organizacional

1 INTRODUCTION

The pressure for results is increasingly present in the daily life of organizations. It occurs due to the interests of various interested parties such as shareholders, customers, suppliers, employees, among others. Identifying the best ways to evaluate the performance has challenged for many years (Kennerley & Albany, 2002Kennerley, M. P., & Neely, A. D. (2002). A framework of the factors affecting the evolution of performance measurement systems. International Journal of Operations & Production Management , 22(11), 1222-1245.). Kaplan & Norton (1992Kaplan, R. S., & Norton, D. P. (1992). The Balanced Scorecard - measures that drive performance. Harvard Business Review, 70(1), 71-79.; 1993Kaplan, R. S., & Norton, D. P. (1993). Putting the Balanced Scorecard to work. Harvard Business Review , 66, 134-142. ; 2000Kaplan, R. S., & Norton, D. P. (2000). Organização orientada para a estratégia: como as empresas que adotam o balanced scorecard prosperam no novo ambiente de negócios. Rio de Janeiro: Campus. & 2006Kaplan, R. S., & Norton, D. P. (2006). Alinhamento. Rio de Janeiro: Campus .) have studied the relationship between performance and strategy. In this sense, as reported by Suwignjo, Bititci, and Carrie (2000Suwignjo, P., Bititci, U. S., & Carrie, A. S. (2000) Quantitative models for performance measurement system. International Journal Production Economics, 64, 231-241.), it has created a series of models for the management of organizational performance. Among them, we can mention the Strategic Measurement and Reporting Technique (SMART) (Lynch & Cross,1990Lynch, R. L., & Cross, K. F. (1990). Managing the corporate warriors. Quality Progress, 23(4), 54-59.), the Balanced Scorecard (BSC) (Kaplan & Norton, 1992Kaplan, R. S., & Norton, D. P. (1992). The Balanced Scorecard - measures that drive performance. Harvard Business Review, 70(1), 71-79.), the Skandia Navigator (Edvinsson & Malone, 1998Edvinsson, L., & Malone, M. S. (1998). Capital intelectual: descobrindo o valor real de sua empresa pela identificação de seus valores internos. São Paulo: Makron Books do Brasil.), the prism of Performance (Albany, Adams & Crowe, 2001), among others.

Currently, the management of organizational performance and its indicators have become critical instruments in the management of organizations as compared to the dynamic competitive environment in which organizations are inserted (Birth, Bortoluzzi, Dutra, & Enssilin, 2011). The importance of the use of performance indicators is evident for organizations and academics, and their results can be understood and interpreted by all stakeholders (Kagioglou et al., 2001Kagioglou, M., Cooper, R., & Aouad, G. (2001). Performance management in construction: a conceptual framework. Construction Management and Economics, 19(1), 85-95.; Kennerleye & Albany, 2002Kennerley, M. P., & Neely, A. D. (2002). A framework of the factors affecting the evolution of performance measurement systems. International Journal of Operations & Production Management , 22(11), 1222-1245.). Bourne et al. (2003Bourne, M., Neely, A., Mills, J., & Platts, K. (2003). Implementing performance measurement systems: a literature review. Int. Journal of Business Performance Management, 5(1), 1-24. ) also claim that the management of performance should be used to assess the impact of actions related to the stakeholders of the organization and check their expectations (Ferreira & Otley, 2009Ferreira, A., & Otley, D. (2009). The design and use of performance management systems: an extended framework for analysis. Management Accounting Research, 20, 263-282.).

New criteria for indicators were developed, among which indicators related to the mission and goals of the organization, which reflect the external environment and the purposes of the organization (Kaplan & Norton, 1993Kaplan, R. S., & Norton, D. P. (1993). Putting the Balanced Scorecard to work. Harvard Business Review , 66, 134-142. ). Other authors also argue that there should be a reflection on the indicators used (Bititci et al., 2000Bititci, U. S., Turner, T., & Begemann, C. (2000). Dynamics of performance measurement systems. International Journal of Operations & Production Management, 20(6), 692-704.; Ghalayini & Noble, 1996Ghalayini, A. M., & Noble, J. S. (1996). The changing basis of performance measurement. International Journal of Operations & Production Management , 16(8), 63-80.).

Birth et al. (2011) sought to map the research on indicators of organizational performance in the period from 2000 to 2008 in publications of Brazilian scientific journals A, B, and C in the field of Administration, Accounting, and Tourism. Their conclusions point out that 42.98% of the researches have focused on financial-economic aspects and these studies have are appliable to a single dimension of the organization, while only 25% have holistic form.

The objective of this research is to recognize the performance indicators relating to the management of species reported in the literature. Also, the study seeks to validate these indicators along with the specialists in the sector of private education in Brazil and propose a list of indicators considered the most suitable for management/performance evaluation of Higher Education Institutions (HEI) in Brazil.

Thus, the study makes Proposition 1 - the composition of the model contemplating economic dimensions; Internationalization and Research; Student/Client; Market; Academia; Operational; Human Resources; Society/extension and Sustainability, meets the balance of financial and non-financial measures of an organization. And, besides, presents Proposition 2 - there is an ideal composition of aspects that should be measured in each dimension proposal.

In this study, the educational segment deserves to be highlighted. Each semester becomes more competitive and has forced the organizations to seek ever more tools that enable them to improve their performance. Researchers from various parts of the world, as China (Chen, Wang, & Yang, 2009Chen, S. H., Wang, H. H., & Yang, K. J. (2009). Establishment and application of performance measure indicators for universities. The TQM Journal, 21(3), 220-235.), Holland and Tanzania (Waal & Chachage, 2011Waal, A., & Chachage, B. (2011). Applicability of the high-performance organization framework at an east African university: the case of Iringa University College. International Journal of Emerging Markets, 6(2), 148-167.), among others, have issued their research on performance in education in international journals. To Chen et al. (2009Chen, S. H., Wang, H. H., & Yang, K. J. (2009). Establishment and application of performance measure indicators for universities. The TQM Journal, 21(3), 220-235.), the promotion of quality education is directly linked to the presence of performance indicators. As there is a severe criticism of the management models of organizational performance taken as ready and adaptable for all organizations (Nørreklit, 2000Nørreklit, H. (2000). The balance on the Balanced Scorecard: a critical analysis of some of its assumptions. Management Accounting Research , 11(1), 65-88.), it justifies the need for the development of management models of sectoral performance that are not exclusive (i.e., restricted to a single sector).

On this basis, defining the determinants of business performance and the indicators that should be used for each type of organization is of utmost importance and was highlighted as one of the challenges of future research in management of organizational performance by Albany (1999). Another aspect that justifies this research is a theoretical gap in the literature. Although the essay on management of organizational performance is abundant, there is a scarcity of studies that relate the way that organizations use indicators of performance and the results of the organization. This shortage is even more significant as it discusses the segment of education in Brazil. In the Brazilian literature, as commented by Birth et al. (2011), 75% of the surveys do not address performance management holistically, thus reinforcing this study’s importance and originality.

This paper presents this introduction, a theoretical section that approaches Management systems by indicators, multidimensionality of management systems by indicators, organizational objectives and performance indicators, management of performance of the IES, a methodology section, results and conclusion.

2 THEORETICAL BACKGROUND

2.1 Management by performance indicators

Carneiro da Cunha and Corrêa (2013Carneiro da Cunha, Julio A., & Corrêa, H. L. (2013). Avaliação de desempenho organizacional: um estudo aplicado em hospitais filantrópicos. Rae - Revista de administração de empresas, 53(5), 485-499.) affirm that the first initiatives to assess organizational performance date back to the 19th century. However, it was only after the decade of 1980 that the focus of the management changed when it created the need to develop new models that do not consider only those financial measures (Albany, 1999; Bititci et al., 2000Bititci, U. S., Turner, T., & Begemann, C. (2000). Dynamics of performance measurement systems. International Journal of Operations & Production Management, 20(6), 692-704.; Kennerley & Albany, 2002Kennerley, M. P., & Neely, A. D. (2002). A framework of the factors affecting the evolution of performance measurement systems. International Journal of Operations & Production Management , 22(11), 1222-1245.; Halachmi, 2005Halachmi, A. (2005). Performance measurement is only one way of managing performance. International Journal of Productivity and Performance Management, 54(7), 502-516.). These performance measures, for many authors (Albany, 1999; Kennerley & Albany, 2002Kennerley, M. P., & Neely, A. D. (2002). A framework of the factors affecting the evolution of performance measurement systems. International Journal of Operations & Production Management , 22(11), 1222-1245.; Bourne et al., 2003Bourne, M., Neely, A., Mills, J., & Platts, K. (2003). Implementing performance measurement systems: a literature review. Int. Journal of Business Performance Management, 5(1), 1-24. ), should have multiple dimensions.

The concern with the difficulty of relating to performance management and the organizational strategy affects many companies and intrigued many researchers. This process meant that managers had to rethink the architecture and implementation of their management models of organizational performance (Lynch & Cross, 1990Lynch, R. L., & Cross, K. F. (1990). Managing the corporate warriors. Quality Progress, 23(4), 54-59.; Albany, 1999; Kaplan & Norton, 1992Kaplan, R. S., & Norton, D. P. (1992). The Balanced Scorecard - measures that drive performance. Harvard Business Review, 70(1), 71-79.). Then, it is understood that the management of organizational performance allows the organization to propose improvements to their business. It involves the set of people, processes, methods, tools, and indicators structured with the objective of control related to all stakeholders, at all levels and processes, thereby keeping high standards of performance and approaching the company of its strategic goals. It allows managers to periodically check the health of the organization generating results for high performance (inspired in Albany, 1999; Halachmi, 2005Halachmi, A. (2005). Performance measurement is only one way of managing performance. International Journal of Productivity and Performance Management, 54(7), 502-516.).

Due to the high competitiveness among the companies, the management of organizational performance and its indicators have become instruments of extreme importance. The dynamism that companies are facing requires them tools that enable quick decision-making (Nascimento et al., 2011Nascimento, S., Bortoluzzi, S. C., Dutra, A., & Ensslin, S. R. (2011). Mapeamento dos indicadores de desempenho organizacional em pesquisas da área de administração, ciências contábeis e turismo no período de 2000 a 2008. R. Adm, 46(4), 373-391.). A management system for effective performance depends on a large part of performance indicators used to measure the performance of the organization. These metrics need to be built through the dimensions that make sense for the organization (Kagioglou et al., 2001Kagioglou, M., Cooper, R., & Aouad, G. (2001). Performance management in construction: a conceptual framework. Construction Management and Economics, 19(1), 85-95.).

The definition of performance indicators to be used is part of a logical sequence of procedures for the development and implementation of a system of management of organizational performance. Such measures should be taken with thought for the future, seeking to set goals that reflect the purposes of the organization (Callado, Callado, & Almeida, 2007Callado, A. L. C., Callado, A. A. C., & Almeida, M. A. (2007). A utilização de indicadores gerenciais de desempenho industrial no âmbito de agroindústrias. Revista Eletrônica Sistemas & Gestão, 2(2), 102-118.). Also, Ghalayini and Noble (1996Ghalayini, A. M., & Noble, J. S. (1996). The changing basis of performance measurement. International Journal of Operations & Production Management , 16(8), 63-80.) emphasize the importance of planning the number of indicators. According to the authors, a large number of performance indicators can generate costs that do not involve the benefits that the controlled information can bring. It will not be possible to have efficient management of organizational performance if the indicators used are not directly related to the strategic objectives of the organization (Kagioglou et al., 2001Kagioglou, M., Cooper, R., & Aouad, G. (2001). Performance management in construction: a conceptual framework. Construction Management and Economics, 19(1), 85-95.).

To Ferreira and Otley (2009Ferreira, A., & Otley, D. (2009). The design and use of performance management systems: an extended framework for analysis. Management Accounting Research, 20, 263-282.), the use of measures of financial or non-financial performance is of extreme importance at different levels of organization, because it allows us to understand if the company is reaching its goals. The key performance indicators will enable the approximation of business strategies and plans, thus fulfilling the expectations of stakeholders.

For Albany, Richards, Mills, Platts, and Bourne (1997Bourne, M., Neely, A., Mills, J., & Platts, K. (2003). Implementing performance measurement systems: a literature review. Int. Journal of Business Performance Management, 5(1), 1-24. ), an indicator must contain the following details: (a) Title; b) Purpose; (c) references; (d) Target; and Formula; f) Frequency; (g) information on those measures; h) data source; (i) Information about who acts on the data; (j) function.

2.2 Multidimensionality of management systems by indicators

This grouping of indicators was named as each author developed its management model of organizational performance. To Corrêa and Hourneaux (2008Corrêa, H. L., & Hourneaux Jr, F. (2008). Sistemas de mensuração e avaliação de desempenho organizacional: estudo de casos no setor químico no Brasil. Revista Contabilidade & Finanças, (19), 50-64.), the most appropriate name for the group of indicators is a module; for Kaplan and Norton (1992Kaplan, R. S., & Norton, D. P. (1992). The Balanced Scorecard - measures that drive performance. Harvard Business Review, 70(1), 71-79., 1993Kaplan, R. S., & Norton, D. P. (1993). Putting the Balanced Scorecard to work. Harvard Business Review , 66, 134-142. , 2000Kaplan, R. S., & Norton, D. P. (2000). Organização orientada para a estratégia: como as empresas que adotam o balanced scorecard prosperam no novo ambiente de negócios. Rio de Janeiro: Campus., 2001, 2004 and 2006Kaplan, R. S., & Norton, D. P. (2006). Alinhamento. Rio de Janeiro: Campus .) appointed these groups as perspectives, Crispim and Lugoboni (2012Crispim, S., & Lugoboni, L. F. (2012). Avaliação de desempenho organizacional: análise comparativa dos modelos teóricos e pesquisa de aplicação nas instituições de ensino superior da região metropolitana de São Paulo. Revista Portuguesa e Brasileira de Gestão, 11(1), 41-54.) called them “aspects.” In this study, we will adopt the terminology “Dimensions”.

The relationship between the main management models of organizational performance, and the various topics addressed by the dimensions of management models of performance is shown in Table 1, taken from the study of Hourneaux Jr (2005Hourneaux Jr., F. (2005). Avaliação de desempenho organizacional: estudo de casos de empresas do setor químico. (Dissertação de Mestrado em Administração de Empresas). Faculdade de Economia, Administração e Contabilidade da Universidade de São Paulo (FEA-USP). Universidade de São Paulo, São Paulo.) and Crispim and Lugoboni (2012Crispim, S., & Lugoboni, L. F. (2012). Avaliação de desempenho organizacional: análise comparativa dos modelos teóricos e pesquisa de aplicação nas instituições de ensino superior da região metropolitana de São Paulo. Revista Portuguesa e Brasileira de Gestão, 11(1), 41-54.) based on a review of the literature:

By analyzing Table 1, it is possible to observe some models assume a focus as the Skandia and intellectual capital. Although some dimensions are addressed by several authors, each one covers the indicators according to your needs. Some models do not have a set number of aspects (such as the control panel board), which lets us know that the modules are adaptable to each organization. The VCS model does not precisely yield how many dimensions must be kept.

2.3 Organizational Objectives and performance indicators

The management of the objectives and performance indicators is closely linked. The authors Ganesan and Elberton (2009Ganesan, E., & Paturi, R. (2009). Key performance indicators framework: a method to track business objectives, link business strategy to processes and detail importance of key performance indicators in enterprise business architecture. AMCIS 2009 Proceedings, 736.) comment that the definition and use of performance indicators should help to communicate the strategy to everyone in the organization and engage them. The measures have to be defined and associated with business objectives.

According to Richmond (1997Richmond, B. (1997). The Strategic Forum: aligning objectives, strategy and process. System Dynamics Review, 13(2), 131-148.), the majority of organizations do an excellent job on defining strategic goals and in the drawing up of plans to achieve them. But there is great difficulty in making the objectives comprehensible, even if they are implemented with success (Richmond, 1997Richmond, B. (1997). The Strategic Forum: aligning objectives, strategy and process. System Dynamics Review, 13(2), 131-148.). The key performance indicators (KPIs) should reflect and derive organizational objectives (Shahin & Mahbod, 2007Shahin, A., & Mahbod, M. A. (2007). Prioritization of key performance indicators: an integration of analytical hierarchy process and goal setting. International Journal of Productivity and Performance Management , 56(3), 226-240.). To Shahin and Mahbod (2007Shahin, A., & Mahbod, M. A. (2007). Prioritization of key performance indicators: an integration of analytical hierarchy process and goal setting. International Journal of Productivity and Performance Management , 56(3), 226-240.), the goals guide the organization’s efforts and support the distribution of resources. There are many benefits in the definition of objectives, especially the establishment of goals and indicators that ensure the completion of the work that needs to be done.

Strategies and goals need to be tracked and have their performance monitored so that managers can make decisions. The challenge is to create a framework for identifying, defining, associate, and monitor key performance indicators (Ganesan & Elberton, 2009Ganesan, E., & Paturi, R. (2009). Key performance indicators framework: a method to track business objectives, link business strategy to processes and detail importance of key performance indicators in enterprise business architecture. AMCIS 2009 Proceedings, 736.). Companies should strive to quantify the business goals using a measure of performance, a metric. When the parameter is associated with the business objectives, it is possible to understand the progress of activities. The indicators show us whether the business objectives are being met (Ganesan & Elberton, 2009Ganesan, E., & Paturi, R. (2009). Key performance indicators framework: a method to track business objectives, link business strategy to processes and detail importance of key performance indicators in enterprise business architecture. AMCIS 2009 Proceedings, 736.).

2.4 Management of performance of the HEIs

Chen, Yang and Shiau (2006Chen, S. H., Yang, C.-C., & Shiau, J.-Y. (2006). The application of Balanced Scorecard in the performance evaluation of higher education. The Tqm Magazine, 18(2), 190-205.) emphasize that it is important to think about the management of an educational institution in a specific way. It is necessary to structure its core competencies, mission, vision, objectives and performance indicators in a customized way for HEIs.

The discussion about the construction of indicators has been presented as a relevant factor for the Social Sciences, especially for the educational field (Passion et al. 2014). Since the decade of 1980, it is essential to have as practice the use of performance indicators to compare the performance of higher education institutions or departments of these (Ball & Halwachi, 1987Ball, R., & Halwachi, J. (1987). Performance indicators in higher education. Higher Education, 16(4), 393-405.). Ball and Wilkinson (1994Ball, R., & Wilkinson, R. (1994). The use and abuse of performance indicators in UK higher education. Higher Education , 27(4), 417-427.) also stated that the use of performance indicators makes it possible to compare the teaching institutions and analyze the performance of the others.

Management is expected to be based not only on the efficient use of resources (inputs), but also on the effectiveness of the results achieved (outputs). In this way it is possible to have a balance between financial and non-financial results (Ball & Halwachi, 1987Ball, R., & Halwachi, J. (1987). Performance indicators in higher education. Higher Education, 16(4), 393-405.). For Passion et al. (2014), the use of indicators is complicated, because it is based on theories with different degrees of generality, the abstraction of categories, and propositions. In the educational assessment, complexity is evidenced by the necessity of the construction of models for assessing policies, programs, and projects. Therefore, it is necessary special care, in epistemological and methodological terms, in the process of construction of indicators. The adoption and use of inadequate indicators may result in decisions that conflict with institutional policies (Ball & Wilkinson, 1994Ball, R., & Wilkinson, R. (1994). The use and abuse of performance indicators in UK higher education. Higher Education , 27(4), 417-427.).

Chen et al. (2009Chen, S. H., Wang, H. H., & Yang, K. J. (2009). Establishment and application of performance measure indicators for universities. The TQM Journal, 21(3), 220-235.) affirm that indicators in higher education institutions shall have the following functions: (a) Monitor and measure the quality of education; (b) Provide information to decision-makers of educational policies; (c) Provide references to the resource management education; (d) Provide each department indicators for performance management.

3 METHODOLOGY

The research is defined by Gil (2010Gil, A. C. (2010). Como elaborar projetos de pesquisa (5a ed.). São Paulo: Atlas., p. 1) as a systematic process that aims to provide answers to the problems that are proposed by the researcher. This research is classified as exploratory and descriptive and assumes the qualitative characteristics. It is classified as exploratory because, as discussed by Albany (1999), there is no clear definition of what would be the most appropriate indicators.

After a detailed review of the literature regarding performance management, strategic objectives, and performance indicators, a theoretical proposal of indicators and of sufficient size to manage a HEI is presented. The dimensions and indicators were assessed by a jury of experts in the field of organizational performance management and, in particular, managers of colleges with practical experience and academic, to validate them. The entire jury contained eleven experts in the area. The board was selected by convenience. The expert group is composed of people who have great expertise in managing performance (scholars and consultants in management of performance) or have in the management of colleges (professionals that comprise or composed the Rectorate, the directorship, or other areas involved in the strategic planning of colleges). Each interviewee judged if the information were consistent with the reality of educational institutions, i.e., if, in the daily life of institutions, the data are used or should be used.

For data collection, contact by phone or by electronic mail was made. In this contact, the aims of the research were presented, and a meeting was requested. After scheduling the meeting, the interviewer met the respondents and recorded all interviews with permission from them. Only one interview was carried out via Skype, and the others were face-to-face. On average, the meetings took between 45 minutes and an hour and a half. For the analysis of the data, we used Content Analysis.

At first, the experts were approached about the application of dimensions. The dimensions identified in the literature were presented to the respondents who discussed the applicability of these dimensions. In addition, respondents were asked about dimensions that were not presented. In a second step, the respondents discussed the indicators that made up each dimension. Firstly, they addressed the indicators presented by the literature and in a second moment, they freely discussed which indicators should compose each dimension. After data collection, the interviews were transcribed, and their contents analyzed. For content analysis, we used the mixed grid model proposed by Vergara (2012Vergara, S. C. (2012). Métodos de pesquisa em administração. São Paulo: Atlas .). According to Vergara (2012Vergara, S. C. (2012). Métodos de pesquisa em administração. São Paulo: Atlas .), in a mixed grid content analysis survey, a set of predefined categories is departed, and there is a need for subdivision, inclusion, or exclusion of categories. For the analysis, the dimensions and indicators that will be presented in the following chapters were used as categories.

4 RESULTS

4.1 Analysis of the Dimensions

We found the list of dimensions and indicators based on the bibliographic survey on the adjustment of the indicators for the Brazilian reality and was requested to assess the dimensions. The list of dimensions presented to experts is: (a) Financial; b) Scholar; c) operational; (d) the environment and people; e) Society / Extension; f) Socioenvironmental. After this, we discuss the positions of the respondents for each of the dimensions and their indicators.

4.1.2 Financial Dimension

Respondents F and A partially disagree about the importance of the financial dimension, although they agree that it should be kept in the model. According to respondent A (2016), “Today the financial part is super-representative, primarily for the large groups. For everyone it is important you have a legal financial health. But, for those large groups, it is even more important.” Respondent A is ahead of an institution with open capital. The second respondent works in a nonprofit institution.

The relationship of the strategy with the presence and importance of the financial dimension is ratified by respondent A, and that focuses on the hierarchy of dimensions and objectives of the HEI. Respondent J also cites the mission, goals, and financial sustainability of the HEI. The relationship of the strategy with the presence and importance of the financial dimension is also ratified by respondent I.

For respondent F (2016), “In some companies, until everyone realizes that the financial is not fundamental [...] But, in a HEI, I would say that the financial is important.”

Respondent J complements the economic dimension is something important even for a nonprofit institution. He says: “The company needs to keep healthy. Monitor the number of scholarships, their delinquent revenues, costs” (Respondent J, 2016). Therefore, the financial dimension was kept in the model.

4.1.3 Operational Dimension

The respondents also ratified the operational dimension. Respondents B, D, E, H, J, and K, understand that dimension is of extreme importance. For respondent D, the operational aspect should not concentrate most of the attention HEIs managers, although this happens often. For respondent B, as well as conceived by the Balanced Scorecard, both the financial dimension and the operational dimension are of great importance. On the operational issue he adds: “the operational dimension deals with the implementation of the activity-end of the enterprise, i.e. the teaching. A teaching institution to its operation is the implementation of classes” (Respondent B, 2016).

Yet for respondent B, in an educational institution should be treated separately from the operational dimension of the academic aspect, because the operational dimension is related to the “production” of the institution. In contrast, the academic dimension deals with the quality of service provided.

4.1.4 Academic Dimensions, customer and market

Respondent E and believes that the academic dimension should address issues related to the pedagogical section. A similar opinion was presented by respondent D, who stated (2016) that the educational aspect is “how happy you are in accomplishing this promise made to society”, and the quality with which it is done. Respondents A, B, D, E, F, J, and K understand that the academic dimension should be treated as one of the priorities of the organization and, therefore, the dimension was kept in the model.

However, while discussing the presence of the academic dimension, several respondents stated that it would be interesting to separate the “academic” aspect of the “customer” dimension. They have also commented on the lack of a dimension that looked out of the walls of the organization, i.e., to the market. Respondent J (2016) highlights factors such as the long-term relationship with the customer, the complexity of educational business, customer experience, and similarities between the academic and operational. Respondents B, D, E, F, I, and K also highlighted the need for the creation of dimensions related to “Customer” and “market” and the addition of “scholar.”

The academic dimension was kept and customer and market characteristics were added to the model.

4.1.5 Internationalization Dimension

In addition to the dimensions of customer and market, respondents highlighted the lack of a dimension which evaluates the internationalization of the institution. Respondent D (2016) said: “The internationalization dimension is missing. It is an important issue and much discussed in the institution in which work [...] I know that business schools speak much in internationalization, at least at a global level.” Respondent F (2016) raised the same point: “We have to measure: the internationalization. How many students are you getting that are international? How many you sent to the outside?”. Respondents G, J and K also mentioned this need. Because of these contributions, the internationalization dimension has been added.

4.1.6 Environment and People and Society/Extension Dimensions

The dimensions of the environment and people and society have been ratified by all the interviewees as necessary. Respondent D emphasized that training should be treated with caution because, for him, the practice of teachers is a matter of academic dimension, and the preparation of the technical/administrative staff is a matter of people management. It was also prompted by respondents C, E, and J that the name of the scale was changed from ‘environment and people” to “human resources.”

The respondent B focused on the importance of the HEI to society and that this would be treated as a client and receive follow-up. In the dimension of society/extension, respondent I commented that, even in the private institution in which it operates, there are projects that impact society, in partnership with the municipal management. As all the respondents have ratified the importance of dimensions of human resources and society/extension, the two were kept in the model.

4.1.7 Environmental Dimension

Concerning the environmental dimension, all the respondents also ratified its importance. However, some respondents (B, D, E, and J) considered that the name of the dimension limits its activity. It has been suggested by respondent D that the name was changed to sustainability dimension because it could encompass issues from the use of natural resources, such as paper, energy, and water until the effects of risks, diversity, and ethics. Respondent E adds that, in addition to defining the dimensions, it is of utmost importance to define what makes these dimensions and how to work the relationship between them. Thus, the environmental dimension was kept in the model, but now with the name of sustainability.

4.2 Discussion of results - Dimensions

After the ratings of respondents, the final model was established with the following dimensions: (a) Financial; b) Internationalization and research; c) Student/Client; (d) the market; and academic); f) operational; g) Human Resources; h) Society/Extension; (i) Sustainability. According to Kagioglou, Cooper, and Aouad (2001Kagioglou, M., Cooper, R., & Aouad, G. (2001). Performance management in construction: a conceptual framework. Construction Management and Economics, 19(1), 85-95.), it is essential to understand how each metric or indicator is directly related to the different dimensions that an organization decides to adopt. The proposition of aspects to manage a higher education institution was seen with good eyes by managers. Although many have viewed the composition of the dimensions based on the Balanced Scorecard, the final result is very different from any model proposed in the literature.

It was discussed by the respondents the importance of understanding the cause-and-effect relationship between dimensions and indicators. This data ratifies what has been proposed by Corrêa And Houneaux (2008Corrêa, H. L., & Hourneaux Jr, F. (2008). Sistemas de mensuração e avaliação de desempenho organizacional: estudo de casos no setor químico no Brasil. Revista Contabilidade & Finanças, (19), 50-64.) on no cause-and-effect relationship between modules.

It is possible to also realize with interviews that the importance of the dimension or the position that it assumes in cause-and-effect relationships are directly affected by the organizational goals and strategy established. This data ratifies what has been placed by Ferreira and Otley (2009Ferreira, A., & Otley, D. (2009). The design and use of performance management systems: an extended framework for analysis. Management Accounting Research, 20, 263-282.) when they emphasize that the use of performance measures is of extreme importance at different levels of organization because it allows to understand if the organization is reaching its goals.

Ghalayini and Noble (1996Ghalayini, A. M., & Noble, J. S. (1996). The changing basis of performance measurement. International Journal of Operations & Production Management , 16(8), 63-80.) claim that a large number of indicators of performance in an organization implies significant expenses with the management. For Passion et al. (2014), the use of indicators is a complex construction, and, in educational assessment, the complexity is evidenced by the necessity of development of models for assessing policies, programs, and projects.

There´s no consensus among the respondents since their various functions and experiences, as well as the different profiles of the institutions in which they participate, have had different foci. The analysis of the interviews focused on capturing significant parts of the content presented and thus constitute a model composed of the main dimensions addressed by the interviewees. This approach if ratified by the same authors (Passion et al., 2014) about the special care in the process of construction of indicators. The social reality is multidimensional, allowing only partial views of the phenomena on the basis of indicators.

With the composition of the model contemplating the dimensions chosen meets the assumption made by Kennerley and Albany (2002Kennerley, M. P., & Neely, A. D. (2002). A framework of the factors affecting the evolution of performance measurement systems. International Journal of Operations & Production Management , 22(11), 1222-1245.), according to which the management of the performance of an organization should be distributed in several areas (multidimensionality) in order to balance financial and non-financial measures of an organization. Similarly, the assumptions made by Ferreira and Otley (2009Ferreira, A., & Otley, D. (2009). The design and use of performance management systems: an extended framework for analysis. Management Accounting Research, 20, 263-282.). They say that management by indicators must be directly related to organizational strategy, as well as should encompass financial and non-financial issues at different levels.

4.3 Analysis of indicators for each dimension

For each dimension analyzed, based on the literature presented to each respondent, some information that could be evaluated by the educational institution through performance indicators.

Even before judging the relevance of each indicator, respondent K commented on the degree of specificity of indicators. The more detailed, more accurate information is, however, the more difficult it will be to get the data and, consequently, more costly will the process become.

Respondent E opened the same debate, and, when questioned at that moment, the indicators would be detailed. Respondent G commented on the importance of creating a culture of high performance. In this model, all managers have clear objectives and would accompany their results, rendering the results outside the expected. Still, according to the respondent G, there is even a variable remuneration linked to the management of performance: “The directors and coordinators receive an award for performance. We are taking a reward for performance for the teacher, now also” (Respondent G, 2016).

Furthermore, the interviews emphasized the importance of strategic alignment between the business units and performance management. Respondent J puts it is necessary to take parsimony in the decision of the number of indicators. To analyze each of the dimensions, the respondents requested some changes in the set of information presented.

4.3.1 Indicators of the extent of internationalization

The internationalization and research dimension had not been presented in the initial model. But, as has already been mentioned, it was cited by several respondents (D, F, G, J, and K).

About its composition, respondent F puts at your institution there is a center of research in which the quantity of students and faculty involved with the research is closely monitored. Similarly, respondent J accompanies the number of students and teachers engaged with scientific initiation. Concerning the link with international institutions, respondent F says that it is essential to measure the internationalization of students and whether the institution in question offers dual-modality titration, and whether it accompanies the number of students involved in the program or not.

About the volume of publications, the respondent J (2016) highlights the need to measure the contribution of the institution and the lack of monitoring in the institution where it operates. Respondent D says that, depending on the institution, it is necessary to talk about the generation of knowledge. In addition, he affirms that accompanying the institution’s participation in international conferences and journals is very important for the strategy of some institutions.

4.3.2 Indicators of the financial dimension

Concerning the economic dimension, respondents identified some information which, according to them, are evaluated by their institutions or should be. Respondents I, F, and G (2016) highlight the importance of controlling the participation of third-party capital, such as the student financing and risks brought by this practice.

Respondent E (2016) agrees that “the revenue, liquidity, participation of third-party capital and delinquency are fundamental,” and add to financial indicators “administrative costs and administrative expenses.” Several respondents approached the delinquency, as A, G, and I. Respondent J says that delinquency is fundamental when you work in a large group.

However, also, to ratify the importance of the presence of such information, the interviewees identified information that should be in the model and were not considered: economic value added (EVA), margin/profitability, and EBITDA (Earnings before interest, taxes, depreciation, and amortization). The information related to administrative expenses was addressed only as “expenses,” and the costs were deployed in teaching “cost,” “administrative cost,” and “academic” cost. The information “investment in and use of technology” became the operational dimension to the financial aspect, and was changed to just “investment,” becoming more comprehensive. Except for respondent I, everyone understood that the item of investment should be addressed in the financial dimension.

Respondent K (2016) believes that it is essential to monitor costs and expenses. He says that “put academic fees, costs with teachers and administrative costs. Respondent B (2016) reinforces that the composition of costs is one of the most critical aspects. About the contribution margin and the EBITDA, respondent G (2016) affirms that “here, all courses are already structured by contribution margin, as are some units, we have a limit of each unit that everyone looks to make the management of the whole process. Then look at the EBITDA”.

4.3.3 Indicators of dimensions of society/extension and sustainability

The dimension Company/extension has been kept, but suffered changes in its content. Information about new projects, the insertion in the labor market skills and social activities in the community were changed to “social projects of the institution (number of projects/participants)” and “social action in the community,” because, according to the interviewees, would be sufficient indicators to monitor this dimension.

Since the environmental dimension, which was composed of three pieces of information related to the consumption of natural resources (energy, water, and paper), was expanded to sustainability, as has already been reported. The three information initially present in the environmental dimension were consolidated into single details, to which were added information about risks, ethics, and diversity.

The information “insertion in the labor market expertise” was rejected. According to respondent B, you must measure the performances of colleges in society. He cites, for example, projects such as “open the library on Sundays for the population” (Respondent B, 2016).

Concerning social projects, respondent F says that they are related to the application of knowledge in projects involving the company. An essential item for the institution in which he works is the incubator. For respondent J (2016), “acting on social projects” is related to bringing to society practical issues in the classroom. Concerning the control of information related to energy, water, and paper, respondent B (2016) affirms that are metrics used in literature and practice. For respondent I (2016), it should measure the environmental issue, but also worry about measuring the performance of society in these projects.

Respondent F says that the environmental dimension should be much wider than it was considered. There are several other factors, such as diversity and ethics. Respondent J (2016) cites the risks inherent to the contact between the teacher and the student ( i.e., the divergence of classroom content). The need for indicators of classroom control and use of the single assessment for the respondent G (2016), the issue of diversity, especially the inclusion of people with disabilities, should be measured in some way. Also, concerning variety, access for people with disabilities even approached by respondent C.

4.3.4 Indicators of dimensions of customer and market

As already mentioned, the respondents identified that the dimensions of customer and market lacked in the model. In the model presented to the respondents, there was the indicator “student satisfaction.” The vast majority of respondents stated that this item was not clear because the satisfaction of the student could be linked to several factors. It also commented that the student is not the only stakeholder whose achievement should be accompanied by the institution.

For the respondent B (2016), the satisfaction of the student, unlike what was presented in the initial model, should not be part of the academic dimension, but of the student or client dimension. Respondent G (2016) describes ways in which your institution seeks to monitor the student’s satisfaction, the evaluation committee meetings, and ombudsman

As to the feedbacks to be considered, the respondents D and F discuss that it should go beyond the student’s satisfaction. Respondent E (2016) also observes that evaluating only the student’s achievement in general, as was initially proposed, is insufficient. It should also be taken in consideration the levels of satisfaction of parents, businesses, and even the schools of secondary education.

Respondents C and J affirm that it is essential to measure the satisfaction of the student and the administrative processes of the institution. Respondent I discuss the loyalty of the student, and says that it is essential to create a group for monitoring of alumni, create a relationship, and retain the student to be able to have information about it, thus being able to offer new courses, especially of post-graduation. Respondent I (2016) discusses the importance of institutional image. He comments about the segregation of some institutions by the labor market.

The reputation of the institution is also addressed by the respondent J.

Respondents A and B state that it is necessary to monitor what happens with the market and its alumni. It must measure since the visibility that his institution has on the market” (Respondent A, 2016).

4.3.5 Indicators of academic and operational dimensions

The respondents heavily modified academic and operational dimensions. Respondent D (2016) states: “I understand that everything that is linked to the mission is a scholar.” Respondent E (2016) agrees with him: “Here, their indicators, in the academic perspective, should be directly linked to how the institution is dealing with the students.” For respondent D, the academic dimension involves activities directly related to the mission of the organization. In general, the respondents identified that the operational aspect is essential, and that, generally, is related to the support activities to the activity-end of the institution.

For the respondents, the operational dimension has the function to support the academic area and, thus, is closely linked to activities to support the teaching. Respondent D believes that, on the academic side, it is essential that the institution evaluate the lesson plan, the form as the classes are being carried out, and the satisfaction of the student.

For respondent I (2016), the academic dimension goes beyond the satisfaction of the student or the quality of education, being that the market (such as the acceptance of students into the labor market) should also be taken into account. Respondent C also believes that the academic issue is related to how the institution delivers results to the client. For respondent J (2016), the “rate of approval, disapproval, is something important in terms of monitoring academic indicators, as a success rate.”

4.3.6 Indicators of Human Resources Dimension

The dimension of human resources was highlighted by many respondents. According to respondents C, I, J, and K, the basis of the strategy of a teaching institution becomes how the organization deals with its staff and professors. For respondents C and E, in the BSC, the prospect of learning and growth creates subsidies for the implementation of the strategy. Respondent H (2016) makes a lengthy explanation of the aspects of human resources, highlighting terms such as training for the technical-administrative positions, faculty, and benefits for employees and workers’ affairs.

4.4 Composition of Dimensions

Table 3 shows the complete results of all dimensions and Indicators validated by managers.

It is evident in interviews that none of the dimensions had the consensus of all the respondents, and although all respondents have mentioned some indicators, some other were suggested by only a few.

Concerning the composition of the indicators presented in Table 3, it is aligned with the demands placed by Chen, Wang, and Yang (2009Chen, S. H., Wang, H. H., & Yang, K. J. (2009). Establishment and application of performance measure indicators for universities. The TQM Journal, 21(3), 220-235.), for which the indicators in higher education institutions shall have the following functions. a) Monitor and measure the quality of education; (b) Provide information to decision-makers of educational policies; (c) Provide references for the management of education resources and allocation; (d) provide each department indicators for performance management.

Several questions may have influenced the differences in responses. Such as the formation of the respondents or the profile of the institutions for which they work/worked. As there is not a unanimous list, it is understood that these interviews were the most common indicators and feasible for the sector. Albany (1999) affirms that the validation of the indicators proposed by the literature is a significant gap, and that should be better exploited.

Authors such as Bititci et al. (2000Bititci, U. S., Turner, T., & Begemann, C. (2000). Dynamics of performance measurement systems. International Journal of Operations & Production Management, 20(6), 692-704.); Kagioglou et al. (2001Kagioglou, M., Cooper, R., & Aouad, G. (2001). Performance management in construction: a conceptual framework. Construction Management and Economics, 19(1), 85-95.) reported that how the organization uses the indicators is directly affected by the construction and use of organizational goals. New criteria for indicators were developed to include indicators related to the mission and goals of the organization that reflect the external environment and the internal objectives of the organization (Kaplan & Norton, 1993Kaplan, R. S., & Norton, D. P. (1993). Putting the Balanced Scorecard to work. Harvard Business Review , 66, 134-142. ).

5 CONCLUSIONS

Based on the work collected in the literature, 57 performance indicators were identified, distributed in 6 dimensions according to the understanding of the author based on the concepts of multidimensionality. They focused on national writing, because it is more aligned with the legislation and the objectives of the institutions surveyed later.

At this stage, we identified all possible indicators and essential aspects that could be part of the management model of the performance of a teaching institution. In the validation and refinement of the list by experts, it was possible to achieve a higher degree of refinement in the correct indicators and dimensions of use.

The main changes have occurred regarding the inclusion of the Dimensions market, customer, and internationalization, which were not initially proposed. Some of the other dimensions also suffered changes in the names of “environment and people” to “human resources,” and “socioenvironmental” changed to “sustainability.” The result of this research differs significantly from any pre-formatted model proposed by the literature.

The work found indicators were added according to the suggestion of the interviewees, resulting in 65 indicators divided into 9 dimensions. With the composition of the model contemplating economic dimensions; Internationalization and research; Student/Client; Market; Academia; Operational; Human Resources; Society/extension and Sustainability, complied with Proposition 1 because it was perceived as the balance of financial and non-financial measures of an organization.

We also observed Proposition 2, as the assumptions made by Ferreira and Otley (2009Ferreira, A., & Otley, D. (2009). The design and use of performance management systems: an extended framework for analysis. Management Accounting Research, 20, 263-282.), as also noted that there is an ideal composition of aspects that should be measured in each dimension proposal.

As to the indicators that form each dimension, each respondent dismissed if the information were consistent with the reality of educational institutions, i.e., if in the daily life of institutions, the data are used or should be used. The respondents also requested some changes in the set of information presented. It is understood that this composition meets the requirements placed by Chen, Wang, and Yang (2009Chen, S. H., Wang, H. H., & Yang, K. J. (2009). Establishment and application of performance measure indicators for universities. The TQM Journal, 21(3), 220-235.), considering the changes highlighted by Albany (1999). As there is not a unanimous list, it is understood that these interviews were the most common indicators and feasible for the sector.

As limitations of the research, there is the possibility of some work on indicators that have not been identified at this stage of research, and also the choice of specialists. For future work, we suggest identifying how the indicators will bind to the strategic objectives of the educational institution, as is the relationship between these indicators and goals. Another line of research would seek to understand how each indicator ends up being responsible for the results of the educational institution like, for example, the IE or ENADE or even financial results.

Publication Dates

History