Modern Approaches to Evaluating the Effectiveness of Higher Education Programs

Abstract

Object: The application of modern approaches to assessing the effectiveness of academic programs in the university is a key task of the higher education management, ensuring the quality of the educational process and the formation of professional competencies of graduates.

Methods: The study included methods of systematizing the information of the analyzed literature sources, among them analysis, synthesis, structuring. Methods of expert evaluations were also applied, the peculiarity of which lies in the scientific understanding of the organization of all stages of the examination, as well as in the application of quantitative methods at each stage. The analytical part of this study was based on the use of statistical methods of data processing.

Results: The proposed methodology for assessing the effectiveness of educational programs has been tested in the bachelor's academic programs of D. Serikbayev East Kazakhstan Technical University. This methodology makes it possible to assess the demand for educational programs on qualitative and quantitative parameters and to develop management decisions to develop their potential.

Conclusions: The study of domestic and foreign experience allowed recognizing many problems of methodological nature in the field of modern approaches to assess the effectiveness of academic programs of higher education institutions and propose modern methods of their evaluation.

Introduction

The higher education in Kazakhstan is currently developing quite dynamically. The key innovation is the granting of academic autonomy to higher education institutions (HEIs). One of the purposes of academic and scientific freedom for universities is to train in-demand personnel for the national labor market. The transition to new economic conditions of universities and increased competition in the market of educational services also affect the appearance and content of the modern university. In this regard, one of the main objectives of the corporate management of universities is to ensure the effectiveness of academic activities, which is determined by the balance of the portfolio of academic programs (AP). The national Law “On Education” defines the concept of “academic program” as a single set of basic characteristics of education, including the objectives, results and content of training, the organization of the educational process, methods and techniques for their implementation, the criteria for evaluation of learning outcomes (The Law RK “Оn Education”).

The change in the management and legal form of the national and state universities of Kazakhstan has opened up new opportunities for academic and financial freedom, which in turn forms a new framework of responsibility and expands the scope and range of risks.

The authors identified negative trends in the policy of formation and development of the AP portfolio in domestic universities, which may ultimately lead to very negative consequences (reducing the quality of students, inefficient allocation of all types of resources, the lack of qualified personnel, an increase in unprofitable study groups, etc.):

1) Unreasonable increase in the number of educational programs. There are 9,288 records in the USHEI in the Register of academic programs, on average there are 77 academic programs per university.

2) Opening of new areas of training that do not correspond to the main profile of the educational activity of the university, especially in multidisciplinary universities.

Such distribution of resources leads to underfunding of educational programs in complex technical areas of training and, as a consequence, a decrease in the quality of educational services. In this regard, the use of methods to assess the effectiveness of AP of the university is relevant in terms of improving the competitiveness of domestic higher education.

In modern literature and scientific papers, there are many definitions of the concept of “economy efficiency”:

1) Characteristic of the quality of the system in terms of the ratio of costs and results of its operation.

2) Effectiveness of the process, operation, project, defined as the ratio of the effect, result to costs, which led to its receipt, etc. (Guseva, A.I., Vesna, E.V., 2013).

As part of this study we will stick to the following definition of the term efficiency: it is an indicator of the desire for the final result, but not the result itself, a version of the correctness, the accuracy of the direction to it.

Literature Review

Attempts to summarize the conceptual foundations of the formation and development of the modern system of higher education have recently been undertaken by many scientists. The scientific positions of the authors of the article were formed on the basis of the study and analysis of works dealing with this issue (Stukalova, I.B., Seljanskaja, G.N., Ponomarev, M.A., Bobkov, A.L., Mastalygina, E.E., 2014), (Kirkpatrick D., Kirkpatrick J., 2007). In this way, O.V. Saginova's works for the first time focus on the fact that it is the educational program that is the main product of any educational institution (Saginova, O.V., 2005).

Scientific publications available in the subject area can be roughly grouped thematically as follows: some focus more on the program itself and its content, the structure; in others, the priority criterion is the effectiveness of the program, others, assess the resource (Anderson, L., Krathwohl, D., 2013), (Freeberg, N.E., 2005).

Methodologies of AP analysis are quite often associated with the assessment of demand, significance for different types of stakeholders and assessment of their satisfaction. Quite a number of techniques affect the conditions of implementation, since the conditions, ranging from organizational and pedagogical to infrastructural, depend on the internal logic of the program and its effectiveness (Tatum, Ch. B., 2017). Academic programs differ in their inputs, which can be considered in terms of quantity, quality, and uniqueness (Korovkin, M., 2013), (Ruane, M., 2016). Nevertheless, all methods focus on internal aspects, on what is happening within the AP: process, products, resources, without taking into account other factors. The authors concluded the following points:

 the “effective” AP’s profile must include:

1. relevance (market trends, industry development, demand);
2. productivity (qualitative and quantitative achievements);
3. satisfaction (employment rate, NPS (Net Promoter Score);

 need for a comprehensive methodology that will evaluate the AP from different sides to determine its competitiveness.

Methodology for evaluating the effectiveness of AP establishes a procedure for monitoring the state and effectiveness of the implementation of APs on the key indicators that characterize the quality of their implementation and is designed to serve as the basis for operational management of the university's AP.

Methods

Research methodology is based on the laws of dialectical logic, systematic and integrated approaches. The main methods of cognition used are synthesis, structuring, expert-analytical, analytical hierarchy, and others.

The method of synthesis made it possible to combine, according to the general classification features, the existing disparate approaches to assessing the effectiveness of the AP.

The method of structuring allowed to organize all the obtained information about the approaches to assessing the effectiveness of educational programs in the university into a system that is easy to understand and interpret further.

Expert-analytical methods are used to assess the effectiveness of educational programs. In this case, on the basis of the system analysis by analytical way, a set of indicators is identified, with the help of which an evaluation of any property can be conducted, the system of evaluation scales and evaluation criteria themselves are developed.

The “expert” determines the threshold acceptable values for the indicators, the final target function for the evaluation is formed.

Ranking the effectiveness of the AP is a classic problem of ordering multi-criteria alternatives, which makes it advisable to apply the method of analytical hierarchy.

Results

To improve any assessment space there must be a common basis, according to which it is possible to combine the effective interaction of all participants in the process. According to the research outcomes of the concept of effectiveness and features of the implementation of the AP three groups of indicators were formed, taking into account both quantitative and qualitative individual indicators of AP development:

1. Image effectiveness — “Productivity”.
2. Positioning the university in the international and scientific space — “Relevance”.
3. Collaboration of APs with the main stakeholders — “Satisfaction”.

The scales and significance of the assessment indicators, obtained through expert evaluations and coordinated with each other, are shown in more detail in Table 1. The threshold acceptable values for each individual indicator, which are taken into account in the formation of the final score, are defined.

Table 1. Hierarchical system of KPIs of academic programs

Group indicators combine individual indicators for a homogeneous group of parameters:

A rule of thumb is used to make a decision about the effectiveness of the academic program, which was established by expert judgment (Table 2).

Table 2. Rule for evaluating the effectiveness of the academic program

The indicator “effectiveness of the educational programs”, according to the established rules, should not exceed some acceptable limits (Figure 1).

Evaluation of the academic program effectiveness by this methodology will identify actions needed to plan improvements, corrective or preventive actions to improve the quality of educational services and achieve the key objectives of the university.

Discussions

The following modern approaches are used in universities to develop a competitive AP.

1. The analytical approach examines the labor market and employers' requirements for future professionals. The reasoned proofs (demands of society, needs of the labor market) of necessity of realization of AP with a reference to a vector of development of a corresponding direction of science and technology are described.
2. The competency-based approach provides for the formation of basic and professional competencies in accordance with national and industry qualifications frameworks, as well as future professional activities. Information is provided on employment opportunities, the professional field of work and on the list of positions for which graduates can work on the basis of professional standards.
3. The practice-oriented approach provides that the developers of the AP build relationships with employers on the terms of partnership. Employers participate in the AP development, in the examination of the AP’s content. Responding to the requests and wishes of employers, the developers of AP change the content of the program, make the necessary changes, expand the range of disciplines in the areas of training.
4. The modular approach forms the AP on the basis of a modular representation of educational information and interdisciplinary correspondence. The modular structure of the AP allows to quickly and qualitatively expand and deepen the AP through the creation of educational modules, as well as change the depth and focus of training, respond quickly to the needs of the participants of the educational process, employers and the labor market, to develop competencies that represent a logical completed part of the learning process.
5. In the integration approach to improve the quality and competitiveness of educational services innovative APs, including double-diploma ones, are developed jointly with leading foreign partner universities. The integrative approach to the development of the AP contributes to the attractiveness for foreign stakeholders (students, faculty, researchers).

D. Serikbayev East Kazakhstan Technical University (EKTU) is a strategic educational center for training engineering and technical personnel, primarily for non-ferrous metallurgy, energy, and mechanical engineering. At present EKTU is one of largest HEIs in Kazakhstan, the leading center of science, education, and culture in East Kazakhstan region. There are 9 departments, 14 sub-departments in the university.

The EKTU is constantly improving the mechanism for managing the effectiveness of the academic programs, which includes the four specific stages:

Stage 1. Strengthening profiling/specialization (effective allocation of resources — market seg- ment/point positioning).

Stage 2. Optimization and balance of the program’s portfolio (Boston Consulting Group (BCG) Matrix — reduction/repositioning of unprofitable APs).

Stage 3. Ensuring the sustainable development of the academic program (the “Golden Triangle” (also called “relevance-result-satisfaction”), the AP development strategy).

Stage 4. Assessment and monitoring (an automated evaluation system on the platform e-monitoring) — adjustment of the development trajectory of the academic program (Figure 2). There are eleven indicators which exported for calculation in automatic mode daily, the rest are entered into the system manually as they are achieved.

Based on the results of the integral assessment of the effectiveness of educational programs in D. Serikbayev East Kazakhstan Technical University using BCG matrix (Boston Consulting Group), a strategic portfolio of EP development is formed (Figure 3):

 The “Question Marks” portfolio includes academic programs with a small market share but high sales rates. This portfolio also includes innovative or brand new APs of the university, training in which is at the intersection of several industries and is a major trend in the world of professions. It takes a lot of effort and expense to increase market share.

 The “Star” portfolio includes APs that are in demand on the educational services market and occupy a leading position now and in the near future, and require significant investment in them.

 The APs in the “Cash Cows” portfolio are characterized by a low growth rate of sales volume due to the regional economy and a high proportion of employment in these professions in the region. These programs do not require costly investments, generating stable and high income. Due to this income the university finances other academic programs.

 Academic programs in the portfolio of “Dog” are unprofitable and inefficient for the university, their possible strategy is to leave the market or to upgrade in view of new industry trends.

In this way, D. Serikbayev EKTU has an effective AP portfolio, as the APs in the “Cash Cows” portfolio are twice as large as those in the “Stars” and “Dogs” portfolios.

The calculations show that there is a direct dependence on the portfolio in the BCG matrix and the evaluation of the academic program effectiveness by the list of indicators. Consequently, to ensure a balanced portfolio of educational programs, it is necessary to abandon unprofitable areas of training.

Based on the methodologies outlined above, the transformation of the university's academic activities is currently underway at EKTU: diversification of educational programs depending on the needs of production, development of educational programs in cooperation with leading foreign universities and national enterprises, compliance with professional standards, integration of professional practices in theoretical training, introduction of modular training system; advanced training of teaching staff in innovative learning technologies, Posting MOOCs and blended courses on the OpenEdX and MOODLE platform, transferring 100% of courses to Blended Learning.

Thus, as an approach to a comprehensive assessment of the effectiveness of the AP these methods are preferred, as they reflect the current state of the AP in the market of educational services and help to develop a strategy for their further development, taking into account the conditions of new regulations.

Modern Approaches to Evaluating...

Modern Approaches to Evaluating...

Conclusions

Ensuring the quality of education throughout the developed world is seen as a key factor in the stability and development of society and an important vector of progress and sustainable growth of the country. Higher education institutions of the countries and society should be deeply interested in the promotion of educational programs of universities and in improving the competitiveness of educational services.

The change in the organizational-legal form of universities in Kazakhstan opens up new opportunities for academic and financial freedom, which in turn forms a new framework of responsibility and expands the scope and range of risks. For the sustainable development of a university one of the most important components of corporate governance is academic risk management. Today many regions are experiencing an exodus of top school graduates to major metropolitan areas and foreign universities. This circumstance is exacerbated by the decline in the productivity of the country's economy in the post COVID-19 period under conditions of risk and uncertainty, which requires the university to change its approaches to the development of educational programs using foresight analytics.

Application of modern approaches to assessing the effectiveness of educational programs allows universities to analyze the relevance of educational products depending on the growth of the market. The evaluation methodology, which includes a hierarchical system of indicators and a criterion rating scale for each indicator, is oriented to ensure the balance of the portfolio of educational programs.

References

1. Anderson, L. & Krathwohl, D. (2013). Bloom’s Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. New York, NY: Longman.
2. Freeberg, N.E. (2005). Assessment of disadvantaged adolescents: A different approach to research and evaluation measures. Journal of Educational Psychology, 61, 229–240.
3. Kirkpatrick D., Kirkpatrick J. (2007). Implementing the Four Levels: A Practical Guide for Effective Evaluation of Training Programs. San Francisco, CA: Berrett-Koehler.
4. Korovkin, M., Mogilnitsky, S. & Yanushevskaya, M. (2013). Quality management system at national research university. Journal of International Scientific Publication: Educational Alternatives, Vol. 11. — 3, 122–128.
5. Ruane, M. (2016). Action learning in postgraduate executive management education: an account of prac- tice/M. Ruane // Action Learning: Research and Practice, Vol. 13. — 3, 272–280.
6. Tatum, Ch.B. (2017). Developing and Evaluating Educational Programs. Cincinnati, OH: Southwestern College.
7. Guseva, A.I. & Vesna, E.B. (2013). Ocenka rezultativnosti i effectivnosti setevyh obrazovatelnyh program [Evaluation of the effectiveness and efficiency of network educational programs]. Sovremennye problemy nauki i obrazovaniya — Modern problems of science and education, 6, 353–358 [in Russian].
8. Saginova, O.V. (2005). Priroda i harakter obrazovatelnyh uslug [Nature and nature of educational services].
9. Marketing, 3, 53–63 [in Russian].
10. Stukalova, I.B., Seljanskaja, G.N., Ponomarev, M.A., Bobkov, A.L. & Mastalygina, E.E. (2014). Napravleniya povyşeniya konkurentosposobnosti obrazovatelnoy dejatelnosti vuza (na primere REU im. G.V. Plehanova) [Directions of increasing the competitiveness of educational activities of the university (on the example of Plekhanov Russian University of Economics)]. Vestnik Rossiiskogo ekonomicheskogo universiteta imeni G.V. Plehanova — Bulletin of the Plekhanov Russian University of Economics, 1(67), 15–24 [in Russian].
11. Zakon Respubliki Kazakhstan “Ob obrazovanii” [Law of the Republic of Kazakhstan “On Education”]. Retrieved from https://оnline.zakon.kz/document/? doc_id=30118747#pos=8 [in Russian].