Abstract
The purpose of the research is to classify academic organizations in Kazakhstan and analyze overall higher education competitiveness using the hierarchical cluster analysis method. The results of the study can be used as an effective source of information in development of institutional policies and benchmarking. The lack of a classification scheme for academic organizations in Kazakhstan imposes restrictions on policy research as well as academic research. Without a classification scheme for academic organizations, researchers and policy makers have to work with relatively random comparable groups. Such studies may lead to inappropriate and inconsistent results. Regarding this need, the main purpose of this study is to classify the academic organizations in Kazakhstan in terms of their institutional size and performance.
As a result of the clustering analysis, the academic organizations were divided in 4 groups forming in the dendogram. Academic organizations in cluster A are academic organizations with a focus on growth and research. The cluster consists mainly of high quality education and high performing academic organizations. Academic organizations in cluster B include the oldest mediumsized academic organizations. Cluster C is composed of academic organizations located in the Western region of Kazakhstan, while Cluster D contains small and medium-sized and relatively old academic organizations.
Introduction
In recent years, performance measurement in academic organizations has become an important issue. Since academic organizations are competing for resources such as students, researchers, state budget and fund raising, performance measurement has gained importance. Academic organizations strive to improve performance measures and develop effective management strategies (Ibanez, Larrahaga and Bielza, 2013). Objective, reliable and accurate measurement of organizational performance can help to develop institution policies for assessment and improvement, allocate resources, prioritize research and education strategic goals, attract students and retain employees (Ioannidis et al., 2007). The state sees it as the first step in measuring performance to ensure that the resources of the academic organization are appropriately allocated (Raponi, Martella and Maruotti, 2016).
Classification is considered an effective strategy for the development of the higher education system as it provides transparency and a basis for inter- organizational cooperation, methodological and analytical tools for research (Shin, 2009). Initially, academic organizations were classified in terms of law (legal classification), similarities and differences (Carnegie classification). However, classification on performance has become the main method for assessment and evaluation.
Researchers have used various criteria in the development of academic organizations typologies (Shin, 2009) including the vast spread of clustering and ranking tools of performance measurement in academic organizations (McCormick, 2008). A standard for definition of the best classification is closely linked to its intended use, as there are numerous factors that can be assessed. According to McCormick and Zhao (2005), a useful classification system considers multiple aspects such as the purpose, the nature of the objects, the criteria, the available data and the degree of differentiation.
In recent years, the development of global rankings among higher- education institutions has gained popularity among the public due to easily interpretable information and competitiveness between universities. Besides encouraging competition, rankings help to differentiate and characterize institution and serve as a framework for quality assessment (Harvey, 2008). According to Thakur (2007) ranking systems demonstrated effectiveness in academic field. Moreover, stakeholders also show great interest in rankings such as government for budget allocation and evaluation; prospective students and their families for enrollment in the best instituion; employers for assessment of candidates and their education.
However, rankings are also criticized for the selection of index-forming indicators. The basic criticism is that ranking systems use measurable data rather than quality (Stella & Woodhouse, 2007), vast selection of calculation formulas (Ioannidis et al., 2007), specialization and focus of the universities, failure to measure teaching method and diversity (Van Dyke, 2005; Carey, 2006). Raponi et al. (2016) came to conclusion that purely explanatory approaches used by rankings may not capture the complex structure of academic organizations. According to Erdogmus and Esen (2016), clustering techniques and assessment of university specification are advised to use to compare academic organizations.
Most analyzes initially focused on the institutional classification of academic organizations, and now the focus has shifted to research performance and disciplines (Valadkhani and Ville, 2009). Academic organizations often express certain strengths in one area and weaknesses in others, thus, overall institution analysis can be problematic. The decision to accept research performance as a classification criterion depends on two factors:
(1) The availability of data;
(2) Research outputs (results) that are easy to quantify and qualify (Chu Ng & Li, 2000).
Typical research performance indicators include the number of publications, the number of citations of publications, journal impact factors, and reputation ranking. For example, Ibahezet al.'s (2013) focuses on research activities such as productivity, visibility, quality, prestige and universalization as performance indicators. For better construct validity, research should be preferred over the total number of quotations in performance studies (Ioannidis et al., 2007).
Academic performance indicators vary from state to state. According to Bartelse and Vught (2007), the researchers aim to classify the whole institution and education, research and innovation, student and staff profiles and institutional variables have been identified as the main Componentsin Europe. While, five indicators are commonly used in Australia: education and learning, student profile, research participation, information exchange and international orientation, the Australian Department of Education, Training and Youth Affairs (1998) classified academic organizations according to six criteria of cluster analysis: size, international orientation, diversity, full-time orientation, financial research orientation, and staff research orientation (Valadkhani & Worthington, 2006). The South Korean government has initiated a classification of academic organizations at least three times since the mid-1990s.
However, none of these initiatives is not widely accepted as classification standard. The lack of a classification scheme for academic organizations increases the limitations in academic research and policy development (Shin, 2009).
Higher Education in Kazakhstan
Kazakhstan's higher education system can be defined as a centralized higher education administration. All higher education institutions are under the supervision of the Ministry of Education and Science (MES). It is responsible for the planning, coordination and management of the whole education system.
Universities offer associate degree, bachelor's, master's and master's programs. Academic staff are classified as faculty members with professors (professors, associate professors and assistant professors), faculty members (faculty members, instructors), research assistants or support staff (experts, translators, educational planners). There are state and foundation academic organizations. The operating income of state academic organizations comes from the government. Funding of the foundation's academic organizations comes from the founding, tuition fees and other sources. Most of the higher education institutions are state universities that are largely funded by the government. Government funding up to date was provided without performing a specific performance assessment. However, every university each year sends information on performance indicators to the Ministry of Education and Sciences and significant amount of budget is allocated according to number of students enrolled.
Higher education in Kazakhstan has increased significantly in the past 15 years. With the growing young population and the demand for higher education, the government aimed to increase the number and capacity of higher education institutions. The number of academic organizations increased from 108 in 1995 to 170 in 2001. Since 2001, the government identified main aspects of the strategic development of the higher education system and adopted Western system by joining the Bologna process. Since 2018, the number of academic organizations has risen to 125, 9 institutions are national, 31 are state and 13 are private academic organizations.
The number of teaching staff in 2018 was 38200 people (among them, doctors of science - 3,5 thousand people, candidates of science - 14 thousand, PhD doctors - 1562, 53% of the teaching staff have degrees). Like many countries after the collapse of the Soviet Union, Kazakhstan faces the need for more effective higher education. It is clear that existing resources should be used to meet the growing demand for research, education and services. In order to reach sustainability and continue operation, universities are faced by the necessity to increase their competitive advantage and performance. Unfortunately, there are few quantitative studies that guide ongoing strategy development and policy changes in higher education in Kazakhstan.
The lack of a classification scheme for academic organizations in Kazakhstan imposes restrictions on policy development as well as academic research. Without a classification scheme for their academic organizations, researchers and policy makers have to work with relatively random comparable groups. Such studies may lead to inappropriate and inconsistent results. In relation to this need, the first aim of this study is to investigate the variables of classification of higher education institutions in Kazakhstan based on institutional dimension and performance. The second objective is to classify academic organizations using a cluster methodology. Since the focus of the research was on institutional performance, we approached the issue from the perspective of management and organization. In this study, five research questions are discussed:
- What variables can be used to classify academic organizations in Kazakhstan?
- How can academic organizations be classified in terms of quantitative variables?
- How can academic organizations be classified in terms of ranking scores?
- How can academic organizations be classified in terms of teaching and research performance?
- What are the similarities and differences based on quantitative variables, ranking and performance?
Methodology
Data collection
The main purpose of this study is to classify the academic organizations in Kazakhstan in terms of their institutional size and performance. Clustering analysis requires reliable data sets. For this analysis, taking into account the existing literature, we have developed a framework for the classification of academic organizations. The framework consists of quantitative variables related to academic organizations, ranking scores and the quality of teaching and research for each academic organization. Then, we collected data from IQAA, Web of Science, Webometrics statistics, and collected data on the strategic plans, annual reports, and other relevant information of higher education institutions. During the data collection process, the problem of access to data for some variables such as research publications, finance and research funds was encountered; thus, data sets were created on available and reliable quantitative indicators, such as size of academic organizations, ranking data and performance data.
For quantitative data collection, statistics from 29 academic organizations, 5 national and 24 state, were collected. Date consists of the information on the foundation year, student ratio, number of staff, number of programs offered, number of scientific articles listed on the Science Web page, and the number of academic units. In addition, we reviewed the strategic plans and annual reports of academic organizations to collect institutional data. For a second data set the ranking data was used, in particular 29 academic organizations ranked in IQAA and Webometric ranking in 2017.
Cluster analysis
For this research, hierarchical clustering analysis was performed using IBM SPSS Statistics Base 22.0 program.
Clustering analysis, which is one of the multivariate statistical techniques, is used to classify the number of ungrouped data. Clustering analysis is a technique that allows data to be collected in discrete clusters in terms of similarities between units or variables. Hierarchical Cluster Analysis tries to minimize within- group variance while also maximizing between-group variance (Hair et al., 1998, p. 470).The assumption of normality of data, which is important in other multivariate statistical analyzes, is not very important in clustering analysis, but the normality of distance values is considered sufficient (Tatlidil, 1992: p.252). Clustering is done by looking at the similarity (distance) or distance measure of two observations or two variables according to the purpose determined as described above.
The main assumptions of cluster analysis are that data matrices do not divide рге-analysis prediction and criterion variables into sub-matrices and that the data are partly homogeneous and partly heterogeneous (Atamer, 1992).
Hierarchical clustering analysis is the process of combining clusters one after the other, and once a group has been combined with the other one, it cannot be separated in subsequent steps (Firat, 1995). The results of hierarchical techniques shown with tree diagrams are called dendograms (Lorr, 1983).
Hierarchical clustering analysis is grouped into two groups as agglomerative and divisive techniques.
Agglomerative Techniques. In agglomerative or bottom-up clustering method each observation assigned to its own cluster. Then, compute the similarity (distance) between each of the clusters {1/2 [n (n-l)]}and two closest clusters are combined according to the similarity or distance matrix. The analysis is repeated until there is only a single cluster left and n units are gradually placed in the cluster n, (n-1), (n-2), ... (nr), ... 3,2,1 (Everitt, 1971). Distance measurement differs depending on what method is used: single linkage technique, complete linkage technique, or average linkage technique.
Single linkagemethod. This technique was first applied by Florek et al. (1951) and then by Sneath (1957) and Johnson (1967), respectively. 1995). Using the distance or similarity matrix, the two closest observations or clusters are combined and the linkage process is repeated (Senturk, 1995; Firat, 1995).
In the merging process used criteria are:
- Similarity type measurements
Sk (i, j) = max (Ski, S4)
- Distance type measurements;
dk (i, j) = min (dki, dig)
Completelinkage method. This technique is the exact opposite of a single linkage method. In this technique, the two closest clusters or observations are combined using the obtained distance or similarity matrix.
In the merging process used criteria are:
- Similarity type measurements;
Sk (i, j) = min (Ski, S4)
- Distance type measurements;
dk(i, j) = max (dki, d4).
The complete linkage technique cannot guarantee that all clusters can be formed in a right way if the distances of the observations in the same cluster are smaller than a certain value (Tatlidil, 1992).
Average linkage method. Sokal and Michener proposed the average linkage technique. In this technique, the difference between two sets is taken as the average difference between element pairs between one set and element pairs in another set (Everitt, 1981). There are modified types of this technique. In the most widely used type, the arithmetic mean of the distance between the observation pairs is calculated. The average linkage technique is widely used in biology, but its use in social sciences is increasing. Similar dendrograms usually occur in full linkage and average linkage techniques. However, since the distance is defined differently in each method, the joins can occur at different levels (Firat, 1995).
Since we prefer a data-driven approach to academic organizations in this study, we have classified them on the basis of objective data rather than predetermined criteria. All state academic organizations, when included in the relevant data sets, were included in each cluster analysis.
This analysis showed that we need to perform a separate cluster analysis for each variable (quantitative, ranking, and performance). Based on the findings of a trial cluster analysis, we decided to conduct a three-stage cluster analysis for state academic organizations. The steps are:
- Clustering analysis of academic organizations based on quantitative indicators
- Clustering analysis by ranking criteria of performance measures of academic organizations
Using data sets and the findings of a pilot cluster analysis, we have developed a framework for cluster analysis. The main variables and sub-variables for classification are:
- Quantitative variables
- Foundation year
- Number of academic units
- Number of students
- Number of academic staff
- Number of academic programs offered
- Performance variables
- IQAA score
- Webometrics score
Results
Descriptive statistics
The quantitative indicators were found to be appropriate criteria for institutional variables of clustered academic organizations in Kazakhstan. These criteria are as follows:
- foundation year
- number of academic units
- number of students
- number of academic staff
- number of academic programs offered
In addition, the following two ranking scores were found to be appropriate for the measurement and performance classification of academic organizations.
- IQAA rating ranking
- Webometrics score
These two rankings include all academic organizations in Kazakhstan: IQAA (125) and Webometrics (125). The analysis is based on all institutions.
Although there are some criticisms of the methodologies used in ranking systems, ranking systems still shape the behavior of academic organizations (OECD, 2006). Sequencing results are increasingly used as performance indicators to evaluate and monitor processes. Given this impact, academic organizations develop organizational policies and strategies to optimize their position in ranking systems (Marginson and Van der Wende, 2007).
Clustering Analysis Results
A dendrogram was used for graphical representation of the cluster results. The dendrogram is scaled from 0 to 25 units from left to right and the distances between the units are also equal. The horizontal lines in the dendrogram indicate the distance and the vertical lines indicate the component clusters. On the other hand, the cluster junction points on the scale indicate the groups between which groups are formed and the distance between the clusters. In this study, 29 state academic organizations, which are used as units, are grouped at scaled distances between 0-25 units. In terms of service items constituting the number of transactions, the most similar state academic organizations form a group by uniting at a distance of 1 unit, while the least similar state academic organizations form a group at a distance of 25 units (Ozer et al., 2010).
The dendrogram obtained from the number of 4 clusters using the between-groups linkage methodology is shown in Figure 1 and the results of the cluster analysis of State academic organizations on the basis of quantitative variables are shown in Table 1.
The results of the cluster analysis of the State academic organizations on the basis of quantitative variables are presented in Table 1.
When the dendrogram is examined, 4 clusters are formed at a distance of 5 units, and 2 main clusters form after a distance of 15 units. This is an indication that the path to determine the number of clusters is correct and that the number of clusters can be selected between 2 and 4.
Figure 1. Dendrogram showing clustering of state academic organizations Using the Between-Groups Linkage Method
Abay Alrriati Eyalet Universitesi
El-Farabi Kazak Ulusal Universitesi
L1N. Gumilev Avrasya Lllusal Oniversite
Kazak Ulusal Ziraat Qniversitesi
Karaganda Eyalet Teknlk Universitesi
D Serikbaev Dogu Kazakistan Eyalet Teknik Liniversitesi
K I.Satpayev Kazak Ulusal Teknik Universitesi (KazNTU)
Kazak Kadm Pedagoji Eyaleti Universitesi
Buketov Karaganda Eyalet Universitesi
M.O.Auezov Guiney Kazakistan Eyalet Universitesi
S Toraygirov Pavlodar Eyalet LJniversitesi
Korkut Ata Kizilorda Eyalet Universitesi
M H DuIati Taraz Eyalet Universitesi
' §akerim Semey Eyalet Universitesi
MKozibaev Kuzey Kazaklstan Eyalet Universitesi
H, Dosmukhamedov Atirau Eyalet Universitesl
Yesenov Hazar Teknoloji ve Muhendislik Eyalet
Universitesi
Atirau Petrol ve Gaz Enstitusii
Janglr-Han Bati Kazaklstan Zlraat ve Teknik Universitesi
Kostanay Eyalet Pedagoji Enstltusii
ABaytursinov Kostanay Eyalet Unlversitesi
$,Ualihanov Kok^etau Eyalet Universitesl
GQney Kazakistan Devlet Pedagojl EnstitUsii
K Jubanov Aktobe Eyalet Universitesi
Karaganda Eyalet Endiistri Universitesi
Taraz Eyalet Pedagoji EnstitUsii
I. Zhansugurov Zhetysu Eyalet Universitesi
DENDROGRAM USING THE METHOD OF BETWEEN-GROtTS LINKAGE
|
Cluster |
Higher educational institutions |
|
Cluster A |
Abay Almaty State University, Al-Farabi Kazakh National University, Gumilev Eurasian National University, Kazakh National Agricultural University, Karaganda State Technical University, D-Serikbayev Eastern Kazakhstan State Technical University, Satpayev Kazakh National Technical University, Kazakh Women Pedagogical State University |
|
Cluster B |
Buketov Karaganda State University, Auezov South Kazakhstan State University, Toraygirov Pavlodar State University, Korkut Ata Kizilorda State University, Dulati Taraz State University, Sakarim Semey State University, Kozibayev North Kazakhstan State University |
|
Kluster C |
Dosmukhamedov Atyrau State University, Yessenov State University of Technology and Engineering, Atyrau Oil and Gas |
Cluster A (8 academic organizations), Cluster B (7 academic organizations), Cluster C (3 academic organizations), Cluster D (9academic organizations).
|
University |
||
|---|---|---|
|
Kluster D |
Jangir-khan Western Kazakhstan Agricultural and Technical University, Kostanay State Pedagogy University, Baytursunov Kostanay State University, Ualikhanov Kokshetau State University, South Kazakhstan State Pedagogical University, Jubanov Aktobe State University, Karaganda State Industrial University, Taraz State Pedagogical University, Zhansugurov Zhetysu State University |
Table 1 reveals that clusters include the following number of state academic organizations:
Academic organizations in cluster A do not focus on growth. Al-Farabi Kazakh National University, Gumilev Eurasian National University and Karaganda State Technical University are research-oriented academic organizations. The total number of students in the academic organizations in cluster A is 11,162 and the number of academic staff is 647. The cluster consists mainly of high quality education and high performing academic organizations.
There are a total of 8,963 students and 343 academic staff in cluster B academic organizations. Academic organizations in this cluster include the oldest medium-sized academic organizations.
Cluster C is composed of academic organizations located in the Western part of Kazakhstan, with an average of approximately 4,906 students and 314 academic staff.
On average, cluster D academic organizations have more than 5,600 students and approximately 167 academic staff. The cluster includes mediumsized and relatively old academic organizations.
Discussion and conclusion
In this study, academic organizations in Kazakhstan were classified using a data-based classification,hierarchical clustering analysis. As Shin (2009) points out, this approach differs from traditional task classifications using predetermined criteria. The results of the clustering analysis of the academic organizations were formed as expected.
This is an explorable study to classify academic organizations in Kazakhstan according to institutional size and performance. Although academic organizations are categorized according to empirical data, policy-makers and researchers should be careful in using the results and take into account the institutional contexts of academic organizations. Every academic organization has strong and weak programs, and results can represent some areas more than others. As McCormick and Zhao (2005) emphasize, the value of a classification is closely related to its intended use rather than an absolute standard for best classification.
This study has some implications for the higher education area. It is clear that there is a significant quantitative growth in the field of higher education in Kazakhstan. There are also ongoing strategy, structure and policy changes in Kazakhstan's higher education. With regard to the needs of higher education in Kazakhstan, we should conduct multiple clustering analyzes of academic organizations on a purposeful basis.
Possible topics for clustering include efficient resource allocation, resource creation, research and publication priority, training investments, informing the public and stakeholders, and self and external assessments of institutions. These topics represent areas of policy development for institutions as well as research topics for researchers. Another issue for cluster analysis may be to focus on the sub-dimensions of the higher education system (research, education, institutions, services, etc.). Before clustering analysis, researchers need reliable data sets and valid measurements. The valid measurement methodology is the responsibility of the researcher and reliable data sets require the support, cooperation and coordination of institutions in the higher education system and other relevant institutions.
Clustering or measuring the performance of academic organizations requires definitions of variables that are appropriate to measure the quality and effectiveness of academic organizations. A multivariate analysis should be applied to measure the various components of higher education institutions such as education, research, services and institutional variables. As Avkiran (2001) suggests, when evaluating academic organizations on the basis of measurable data, it should be taken into consideration that academic organizations have certain characteristics that distinguish them from other types of organizations.
Limitations of the study
Finding reliable data in the classification and measurement of the performance of academic organizations is a worldwide issue. As any other research on the clustering this study also has some limitations. One of the main constraints is the lack of data on organizational performance. It seems relatively easy to find data on the size of academic organizations, but it is very difficult to collect data on performance.
Another limitation of the study is the use of data on ranking scores, which do not include all academic organizations. More importantly, the intangible assets and products that make up corporate identities could not be included in an empirically based classification system.
References
- Avkiran, N. K. (2001). Investigating technical and scale efficiencies of Australian universities through data envelopment analysis. Socio-Economic Planning Sciences, 35(1), 57-80
- Bartelse, J., & Vught, F. (2007). Institutional profiles: Towards a typology of higher education institutions in Europe. IAU Horizons, 13(2-3), 9-11.
- Carey, К. (2006). College rankings reformed: The case for a new order in higher education.
- Retrieved from Education Sector website: http://educationpolicy. air.org/ sites/ default/files/publications/CollegeRankingsReformed.pdf
- Chu Ng, Y., & Li, S. K. (2000). Measuring the research performance of Chinese higher education institutions: an application of data envelopment analysis. Education Economics, 8(2), 139-156.
- Gunay, D., & Gunay, A. (2011). 1933'den gunumuz Turk yuksekdgretiminde niceliksel geli§meler. Yuksekogretim ve Bilim Dergisi, 1(1), 1- 22.
- Hair, J. F. Jr., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate data analysis (5th ed.). Upper Saddle River, NJ: Prentice Hall.
- Harvey, L. (2008). Rankings of higher education institutions: A critical review. Quality in Higher Education, 14(3), 187-207.
- Ibanez, A., Larrahaga, P., & Bielza, C. (2013). Cluster methods for assessing research performance: exploring Spanish computer science. Scientometrics, 97(3), 571-600.
- Ioannidis, J. P., Patsopoulos, N. A., Kawoura, F. K., Tatsioni, A., Evangelou, E., Kouri, I., & Liberopoulos, G. (2007). International ranking systems for universities and institutions: A critical appraisal. Bmc Medicine, 5(1), 30.
- Kugukcan, T., & Gur, B. S. (2009). Turkiye'de yuksekdgretim: Kargilagtirmah biranaliz. Ankara: SETAYayinIan.
- McCormick, A. C. (2008). The complex interplay between classification and ranking Ofcolleges and universities: should the Berlin Principles apply equally to classification?. Higher Education in Europe, 33(2-3), 209-218.
- McCormick, A. C., & Zhao, C. M. (2005). Rethinking and reframing the Carnegie classification. Change: The Magazine of Higher Learning, 37(5), 51-57.
- Marginson, S., & Van der Wende, M. (2007). To rank or to be ranked: The impact of global rankings in higher education. Journal of Studies in International Education, 11(3-4), 306-329.
- OECD. (2006). Education policy analysis: Focus on higher education 2005-2006. Retrieved from http://www.oecd.org/edu/skills-beyondschool/ education Policyanalysisfocusonhigheredu cation-2005-2006edition.htm
- Ozoglu, M., Glir, B. S., & Gumu§, S. (2016). Rapid expansion of higher education in Turkey: The challenges of recently established public universities (2006-2013). Higher Education Policy, 29, 21-39.
- Raponi, V., Martella, F., & Maruotti, A. (2016). A biclustering approach to university performances: An Italian case study. Journal of Applied Statistics, 43(1), 31-45.
- Shin, J. C. (2009). Classifying higher education institutions in Korea: A performance-based approach. Higher Education, 57(2), 247-266.
- Thakur, M. (2007). The impact of ranking systems on higher education and its stakeholders. Journal of Institutional Research, 13(1), 83-96.
- Tosun, H. (2015). Devlet Universiteleri: Performans Ciegerlendirme finansman modeli ve yeniden yapilanma. Ankara: Uzman Matbaacihk.
- Stella, A., & Woodhouse, D. (2007). Benchmarking in Australian higher education: A thematic analysis of AUQA audit reports. Melbourne, VIC: Australian Universities Quality Agency.
- IQAA (2015). Independent agency for quality assurance in education - ranking from https://iqaa-ranking.kz/rejting-vuzov/rejting-vuzov-kazakhstana- 2018/natsionalnyj - rejting-luchshikh-mnogoprofilnykh-vuzov-kazakhstana-2
- Webemetric. Ranking Web of Universities. Retrieved from http:// www.webometrics.info/en
- Usdiken, B., Topaler, B., & Kogak, O. (2013). Yasa, piyasa ve drgut tiplerinde ge$itlilik: 1981 sonrasmda Turkiye'de Oniversiteler. Ankara Universitesi SBF Dergisi, 68(03), 191-227.
- Van Dyke, N. (2005). Twenty years of university reports cards. Higher Education in Europe, 30(2), 103- 124.
- Ward, J. H. J. (1963). Hierarchical grouping to optimize an objective function. Journal ofthe American Statistical Association, 58, 236-44.
- YOK. (2015a). Higher Education System in Turkey. Retrieved November 15, 2015, from http://www.yok.gov.tr/en/web/uluslararasi-iliskiler/turkiye-de- yuksekogretim-sistemi