The external review feedback results

In order to improve the quality of external (on-site) visits and activities of the Independent Kazakh agency for quality assurance in education systematically conducts a cross-survey of coordinators, experts and representatives of accredited educational institutions (higher education institutions and colleges) after each audit (feedback). The article reveals the findings of the on-site visits feedback of 2015, which are presented in detail thematic analysis on the website of the agency. The results of the feedback show that the work of the agency`s coordinators and experts was of high quality, however there were some places for improvement. Heads of the external review teams and experts, involved in expertise of self-assessment reports, need to be trained to develop leadership skills in first case and improve review skills in the second case. 

The Bologna process, which aims to create European Higher Education Area (EHEA), in last few decades, has been the object of attention and interest of scholars, politicians and global community in general. In higher education systems of 50 country-members of the Bologna process, the reforms are implemented in the frame of the following basic principles:

• a comparable diploma of higher education or postgraduate education;
• a unified mechanism of accounting scope of gained knowledge of students in terms of ECTSEuropean Credit Transfer System;
• the mobility of students and teaching staff;
• a three-tier system of education;
• a life-long learning;
• an integration of research and education;
• the openness of the European Union in research and education;
• a quality assurance of higher

The latter has always been a core value of the European universities in their 800-year history of formation and development. In particular, in the early 90's in a massification of higher education and a sharp reduction in the budget, the quality of education was challenged and became an issue.

To increase the level of student mobility through the Erasmus exchange program the European countries intended to ensure the comparability of the quality of foreign and domestic education. In this regard, the European Commission, originally organized a small group, consisting of the Ministers of Education of several countries and agencies. At that time, not all European countries had national agencies. For instance, the external quality assurance agencies were in place only in Denmark, France, the Netherlands and the UK,while Finland and Norway were preparing for the establishment of quality assurance agencies in the frame of the pilot projects. Sweden and Spain were only at the stage of consideration of this issue [1].

Thus, in 1991, to assess the state of quality assurance mechanisms of European countries, the European Commission launched a pilot project in which EU member countries were surveyed. The results of this project, published in 1994-1995, demonstrated the feasibility of dissemination of good practices and cooperation in the field of quality  assurance. Therefore, the quality  assurance of higher education has become     a cornerstone of the Bologna process, which involves the European cooperation and the use of unified  criteria and methodology [2]. In turn, cooperation in the field of quality assurance aimed at achieving comparability of qualifications and the mobility of students, academics and labor on the entire EHEA.

In 1998, in line with the recommendation, derived from above stated pilot project of the European Union, the Commission supported the idea of creating a network of quality assurance agencies, which will contribute to ensuring the quality of higher education in the EHEA and support accreditation bodies [3]. As a result, in 2000 Network for Quality Assurance in Higher Educationwas created. Later in 2004, it was transformed into European Association for Quality Assurance in Higher Education (ENQA) an umbrella organization of quality assurance agencies, devoted to promoting and support quality assurance in EHEA. Today, apart from the affiliated members,46 agencies from 27 countries and 2 international organizations have a membership of ENQA [4-6].

In 2005, to develop a  unified  methodology and quality  assurance criteria,  with  the  participation of  E4 Group, composed of ENQA, the European Students Union (ESU), the European University Association (EUA), European Association of universities (EURASHE), ENQA, have developed the first European standards and guidelines for quality assurance in the EHEA (ESG). Subsequently, ESG twice subjected to changes and additions in 2008 and 2015. ESG is a tool to ensure the quality of both higher education institutions (HEIs) and accreditation agencies [7]. Implementation of these standards is a basic requirement for the inclusion of quality assurance agencies in ENQA membership and European register of quality assurance in higher education (EQAR), which was established in 2008 in order to improve the transparency of higher education.

The EQAR registered 42 quality assurance agencies from 20 countries, most of which are represented by the agencies of Germany (8) Spain (7) [8]. In 2012, according to the results of the Bucharest  communiqué, it was decided that the agency, registered in the EQAR, will be able to carry out its activities in the EHEA, if it does not contradict to national legislation of the state, where the accreditation will be conducted [9]. The findings of the international activities of EQAR quality assurance agencies have revealed that more than half of agencies, registered in this register,work at cross-border level. For example, in the period from 2009 to 2013, 26 accreditation agencies, registered in the EQAR, evaluated educational  programs and educational institutions of 39 EHEA countries and 46 non-EHEA countries.However, in practice, the scope of international activity of the agencies is mostly restricted to EHEA due to national legislation barriers, while it is not always the case in non-EHEA countries. There are countries that have national accreditation agencies, and legislation, which allows counting the evaluation results of these agencies in program recognition or fund allocation. In EHEA, the largest amount of higher education programs were accredited by the agencies, registered in EQAR, in Kazakhstan (57), Russia (16) and Austria (8), which have their national agencies [10, 11].

In Kazakhstan, there are 10 agencies registered in the register of the accrediting bodies of the Ministry of education and science. Two of these agencies are national agencies, while others are foreign bodies. One  of the national agencies is Independent Kazakh agency for quality assurance in education (IQAA), which has accredited the vast majority of programs and HEIs in Kazakhstan. The accreditation procedures of the IQAA involve:

• preparing a self-evaluation report by HEIs;
• preliminary review (in case of institutional accreditation) of HEI;
• external review;
• Accreditation Council`s decision making;
• post-accreditation monitoring (follow-up procedure).

After the external review and expertise of the self-evaluation reports the experts and coordinators are evaluated, the core findings of which is elaborated further.

Methodology

In order to improve the quality of external (on-site) visits and IQAA activities in general,the agency systematically conducts a cross-survey of coordinators, experts and representatives of accredited educational institutions (higher education institutions –  HEIs/colleges)  after  each audit. For  these purposes  following  6 forms of survey were designed by the agency:

• Form 1 for evaluation of coordinators (for external visit experts);
• Form 2 for evaluation of coordinators (for accredited educational institutions);
• Form 3 for evaluation of external visit team chair (for coordinators and experts);
• Form 4 for evaluation of external visit experts (for coordinators);
• Form 5 for evaluation of external visit team (for accredited educational institutions);
• Form 6 for evaluation of experts, dealt with expertise for self-evaluation (for coordinators).

The list of respondents was drawn up in line with the audits schedule and lists of members of expert teams of the external audit.

The survey has been carried out since 2014, however, to gain comparability this study presents the results of 2015.

Initially, the forms were piloted, during which a number of difficulties were revealed. They were taken into account in the application of the forms.

The survey was conducted via an online questionnaire www.monkeysurvey.com

Evaluation of coordinators

The questionnaires (form 1)were sent to 487 experts, who took part in the external audits of specialized accreditation of 18 HEIs and 9 colleges, as well as in institutional accreditation of 10 HEIs and 7 colleges.

As a result, the responses of 245 experts were obtained (50%). The form №2 was sent to 27 representatives of reviewed educational institutions, including 18 HEIs and 9 colleges. The answers were obtained  from 16 educational institutions (60%), including 13 HEIs and 3 colleges. Overall,10 coordinators of the agency were evaluated by external audit experts and representatives of educational institutions in the following activities:

• timely ensure external visit experts with materials;
• conduct webinar;
• provide support;
• consultation;
• follow procedures of on-site visit and ethical 

Analysis and discussion

According to opinion of the vast majority of respondents, coordinators facilitate preparation of experts to audit by providing them with the necessary materials, including:

• the standards and criteria for institutional accreditation of educational institutions (HEIs/colleges) or standards and criteria for specialized (program) accreditation of educational programs;
• the handbook of student-experts, participating in the accreditation procedures;
• the handbook of external audit;
• the self-evaluation report of educational institution educational program;
• the code of good practice;
• the list of the experts in external visit

In addition, coordinators support experts through arranging webinars via communication tools, among which skype or Polycom is the most popular. Below is the list of resources stated in line with their frequency of application by coordinators of the agency:

1. skype/polycom.
2. e-mail.

During the webinar, the coordinators discuss a number of key issues, related to the activities of the external visit experts. They are:

• the responsibility of experts in the accreditation procedure;
• ethical issues in the audit;
• interview questions;
• requirements for external visit report, terms of preparation and delivery of the report;
• delegation of the

Significant progress has been taken place in the coverage of the main topics, discussed during the preparation of the audit team. Currently, in the opinion of almost all the respondents,the aforementioned aspects are explainedfully by the coordinators. For comparison, in the first half of 2015 the coordinators emphasized only these 3 core components of expert activities: the role and responsibility of experts, the requirements for the external audit report and the delegation of authority, while ethics and interview questions were considered least. However, in comparison with the 1 half of 2015 in the 2 half of 2015 the number of experts, saying that the interview questions were preliminarily discussed collectively, increased from 50% to 82%. Moreover, in last year the experts and the expert team chairs became more active in consideration of the interview questions during the webinar for more than 2,5 times from 24% to 61% and more than 10 times between 2% and 25% respectively. Nevertheless, these results show the need to increase the activity of expert team chairs in the webinar.

In addition, there is a steady improvement in thework of the coordinators in audits particularlyin ensuring the voice of the experts and collegiality in decision-making. Experts say that their opinion is taken into account in full (the 1 half of 2015 85%, the 2 half of 2015 97%). For example, as of the year-end, the recommendations of 91% were fully taken into account. However, some state that in the on-site visit in the frame of the institutional accreditation the view of the experts should be considered regardless the standard they in charge of. Furthermore, experts propose to extend the audit period from 2 to 3 days, which is not always possible. Since the additional audit days lead to more financial and moral burden on the educational institution being accredited, which the agency seeks to avoid. Another suggestion of the experts is to developcommunication skillsof the coordinators and provide experts with more support, particularly, by answering questions appropriately.

In general, experts and representatives of educational institutions appraised highly the agency`s coordinators. According to their opinion, the coordinators arrange the external audits, distribute the responsibilities of all members of expert groups properly (91%), provided the educational institutions with the support in all stages of preparation and conduction of the review (81%). In addition, they believe that coordinators comply with ethical norms (representatives of HEIs -81%, external review experts 93%).

In line with the last survey results, the coordinators were asked to provide less time to experts to work with the departments, where they were left alone with their representatives. In this regard, during the audit in the 2 half of 2015,the experts had less time to work with the departments. Additionally, some measures were taken on the following recommendations of representatives of educational institutions:

• to deliver instruction to the members of the expert team regarding the difference between accreditation and certification;
• provide colleges with more information on the importance of accreditation and give an understanding of the fact that the accreditation is not much of inspection but anassurance of

The first was carried out by coordinators during the webinars while the second was realized through workshops, conducted on the basis of educational institutions. Nevertheless, some experts and  representatives of educational institutions do not fully distinguish accreditation from certification, which requires more effort of the agency to work in this area.

Evaluation of the head of the expert team

The heads of the experts` teams were evaluated by the coordinators and experts. The survey responses were 208.

Analysis of the responses showed a generally high level of preparedness of the leaders of the expert teams to the external audit, and their awareness of the process of preparation of the on-site visit report and its main functions. The vast majority (86%) of the received feedback is positive.

However, some leaders of the expert groups were not prepared fully for the external audit (3%). 2% did not meet their duties properly and 4% did not participate in the development of the recommendation. Moreover, 2 of the experts were included in the “black list” of the experts` database and not recommended for further participation in any procedures of accreditation.

Nevertheless, there is a significant improvement in the activities of the team leaders within 2015.The team leaders became more aware of the purpose of the accreditation and their role in the external audit (1 half of 2015 -75%, 2 half of 2015-95%), in addition, they were more often involved in the organization of work with every member of the expert team (1 half of 2015 -67%, 2 half of 2015-79%).

In the 1 half of 2015, it was recommended to provide extra training for team leaders with negative feedback. For example, one of the experts, attracted by the agency as the team head, was suggested to develop  the leadership skills:

• in the organization of the work of the panel;
• in the distribution of responsibilities to other members of the panel;
• in peer discussion of comments and recommendations of the external

However, the vast majority of the coordinators stated that the group leaders were good enough to cope with their roles. 

Evaluation of external review experts

The main parameters, against which the on-site visit experts were evaluated,are:

• awareness of theexternal visitprocedures, self-evaluation report of the reviewed educational institution and other materials, provided by the agency; 
• participation in the interviews and relevance of the questions;
• timely arrival and departure;
• accuracy and timeliness in the preparation of the external audit reports;
• participation in the discussion of the on-site visit program;
• the amount of contributed effort in the activities of the team;
• awareness of the purpose of the accreditation and his/her role in the on-site visit;
• following the ethical standardsand demonstration of the respectful attitude to the reviewed educational institution;
• whether the coordinators recommend experts for further

During the survey 10 coordinators of the agency and 19 representatives of reviewed educational institutions evaluated 136 experts, 132 out of 136 answers were full and eligible for analysis.

Overall, the analysis shows a fairly high level of the experts` work, involved in the external audit. 76% of coordinators and representatives of educational institutions gave a positive feedback to their activities.

In considered year, there is a progress in all major areas of experts` activities. In particular, it should be noted that in comparison with the 1 half of 2015, experts increasingly began to provide audit reports timely  (1 half of 2015 90%, 2 half of 2015 99%). In addition, a significant improvement is can be seen in the experts`preparedness for external audit (1 half of 2015 96%, 2 half of 2015 99%). As it is known, the expert preparedness plays an important role in the accreditation of educational institutions and educational programs, as fixed deadlines and high responsibility of the agency, its coordinators and experts for the results of the accreditation require their maximum fitness to the procedures. Before stated positive results can be explained by to the fact that the experts gained more experience and became more aware of the purpose of the accreditation as well as their role in the external visit, in addition, the work of the coordinators in explaining procedures of the external visit and accreditation, in general, was intensified.However, a small number of surveyed representatives of educational institutions were not satisfied with the quality of the expert preparedness for external audit. Thus, 4% of the experts have not been recommended for further cooperation, the reason for this was the lack of understanding of the accreditation`s aim and their role in this process, and the inability to work in a group. 

Evaluation of the experts, involved in the preparation of an expertise to the self-evaluation report of educational institutions programs

In 2015, 55 experts, in charge of preparation of the expertise on self-evaluation reports, were assessed by the agency`s coordinators. The results of questionnaire data analysis showed that 100% of the agency coordinators positively evaluated the quality of these experts . However, according to some of the coordinators the experts, involved in the examination of the self-evaluation report of educational institutions and educational programs, particularly those, attracted for the first time, require additional training. 

Conclusions and recommendations

In general,the work of the coordinators was highly appreciated by the experts. Due to the activities of the coordinators,the work of the experts` team was arranged properly. The voice of the experts was assured, however, some suggested to extend it regardless the standard the expert responsible for. In addition, the  heads of the teams need to be more active in discussions especially in consideration of interview questions, which is one of the key aspects of data collection of the experts in the audit. The coordinators should pay more attention to preparation of the team leaders for their roles and stimulation their activity. Furthermore they are expected to:

• strengthen the role of the team leaders in the discussions, particularly in consideration of the interview questions;
• ensure the voice of all experts, regardless of their designated standards;
• improve the communication skills of the coordinators;
• provide more support to

In 2015, the majority of the team leaders fulfilled their functions at the highest level. Substantial progress was observed with regard to several areas of their activities. Team leaders became more aware of the purpose of the accreditation and their role in the external audit. However, some of the team heads are not sufficiently prepared for their role. Nevertheless, the agency coordinators believe that the team chairs met their obligations quite good. Thus, they are recommended:

• to pay attention to leadership skills of candidates to team heads in the recruitment process;
• to increase the activity of the team leaders in the

The work of the external review experts was positively assessed by the coordinators of the agency and the representatives of the reviewed educational institutions. During the year, significant improvements were noted in their activities, in particular, this applies to the timely preparation of the audit report. However, a small number of representatives of reviewed educational institutions are not fully satisfied with the awareness of the experts with external review materials. Moreover, some experts due to a number of reasons are not recommended for further work by coordinators of the agency. Taking in account these findings the coordinators were advised to:

• pay attention to ability to work in a group and to sense of responsibility in the selection of the external review experts;
• strengthen the preparation of the experts for an external

The experts, dealt with preparation of the expertise on self-evaluation reports, were positively assessed by the agency`s coordinators. However, some of the coordinators state that these experts, particularly those, attracted for the first time, require additional training. Thus, the coordinators should consider the possibility of delivering training courses for experts involved in the preparation of an expertise on self-evaluation reports of educational institutions and educational programs.

References

1. A decade of European co-operation in quality assurance in higher education. ¾ Helsinki, 2010. ¾ [ER]. Access mode: http://www.enqa.eu/wp-content/uploads/2013/06/ENQA-10th-Anniversary-publication.pdf.
2. European Commission: Directorate General XXII: Education, Training and Youth, Socrates: European Pilot Project for Evaluating Quality in Higher Education. The European Report, November
3. Council recommendation of 24 September 1998 on European cooperation in quality assurance in higher education (98/561/EC) // Official Journal of the European Communities. ¾ 7.10.98. ¾ P. 56-58. ¾ [ER]. Access mode: http://www. enqa.eu/wp-content/uploads/2013/06/Recommendation-of-the-Council-98.561.EC-of-24-September-1998.pdf.
4. History. ¾ [ER]. Access mode: http://www.enqa.eu/index.php/about-enqa/enqa-history/.
5. Affiliates. ¾ [ER]. Access mode: http://www.enqa.eu/index.php/enqa-agencies/affiliates/.
6. Members. ¾ [ER]. Access mode: http://www.enqa.eu/index.php/enqa-agencies/members/full-members/.
7. Standards and Guidelines for Quality Assurance in the European Higher Education Area. ¾ Brussels: ENQA, 2015.[ER]. Access mode: http://www.enqa.eu/index.php/home/esg/.
8. Quality Assurance Agencies Registered on EQAR. ¾ [ER]. Access mode: https://www.eqar.eu/register/search.html).
9. Bucharest  Communiqué.  ¾  Bucharest,  2012.  ¾  [ER].  Access  mode:  http://www.ehea.info/Uploads/%281%29/Bucharest%20Communique%202012%281%29.pdf.
10. Annual report. ¾ Brussels: EQAR, 2014. ¾ [ER]. Access mode: https://www.eqar.eu/documents/reports/annualreport-2014.html.
11. Recognising International Quality Assurance Activity in the European Higher Education Area (RIQAA). ¾ Brussels: EQAR, 2014. ¾ [ER]. Access mode: http://www.eqar.eu/fileadmin/documents/eqar/riqaa/WP5_RIQAA_Report_final.pdf.