Comparison of Russian and Kazakhstan proctoring systems

The article analyzes the problems faced by teachers in the context of the forced transition to distance learning in connection with the COVID-19 pandemic, when interaction with students was carried out only using information technologies. There are problems of organizing and conducting exams in remote access: the difficulties of identifying students, the use of proctoring technologies. The main problem of distance learning is objective assessment of knowledge. Online proctoring systems are one way to discourage student academic fraud. The article discusses the possibilities and problems of using the online proctoring system for online knowledge control. The possibility and feasibility of using proctoring technology in remote monitoring of the progress of university students, as a tool for personal identification are analyzed. Varieties of the proctoring procedure, classification of technologies based on the products of ProctorEdu, Examus, Oqylyq, Aero, Oes are considered, as well as the advantages and disadvantages of the proctoring system. The article notes that the Kazakhstani proctoring systems Oqylyq, Aero, Oes are not inferior to the Russian proctoring systems ProctorEdu, Examus. On the basis of the analysis of proctoring systems, the validity of its application was confirmed from the point of view of efficiency in terms of such indicators as the reliability of identity verification, reduction of time and material costs.

Introduction

Until recently, concepts such as distance learning, correspondence learning, open learning, etc., were practically not divided. However, at present, distance learning (DL) has proven its importance and relevance. The educational community recognizes that distance learning has good prospects associated with the implementation of learning throughout life [1].

In addition, it is important to note that DL is fundamentally different from traditional education because it creates a new educational information environment, where a student knows exactly what kind of knowledge, skills and abilities he needs. It can also be considered that a distinctive feature of DL is providing the opportunity to get the required knowledge themselves, using developed information resources (databases and knowledge, computer, including multimedia, training and monitoring systems, video and audio recordings, electronic libraries, as well as traditional textbooks and teaching aids) [2].

Distance learning is a new form of education, different from the usual forms of full-time or part-time education. It supposes other means, methods, organizational forms of teaching, different form of interaction between the teacher and students, students to each other. At the same time, like any training system, it has the same components: goals, content, also largely determined by the current programs for a specific type of educational institution, methods, organizational forms, teaching aids. Also, it can and should go beyond basic education, including additional education and everything related with it. Thus, on the one hand, distance learning should be considered in the general education system, and certainly in the system of lifelong education, thereby providing not just a certain system, but the continuity of its individual nodes. On the other hand, it is necessary to distinguish distance learning as a system and as a process. In other words, it is necessary to envisage and theoretically comprehend the stage of pedagogical design of this activity, its content and pedagogical (in terms of pedagogical technologies, methods, forms of education) components. The creation of electronic courses, electronic textbooks, complexes of teaching aids, the development of pedagogical technologies for organizing the learning process using networks — all these are the issues of the pedagogical design stage. That is why, a certain stage of preparatory work is required, which provides theoretical comprehension of a new form of education and training. The development of distance learning courses is a more laborious task, even than the creation of a new textbook or teaching aid, since in this case, a detailed study of the actions of the teacher and students in the new information-subject environment is necessary.

Distance learning forms have appeared for a long time, but the surge in their popularity was due to the regime of self-isolation. Although educational tasks were implemented more or less quickly, the remote control of knowledge still raises many questions — from the security of personal data to a huge number of opportunities to violate the knowledge testing procedure. Specialized proctoring systems help to solve this problem [3].

Experimental

Distance education allows to acquire knowledge at a convenient time with minimal costs, but to confirm them, to obtain diplomas and certificates, certification is needed. Proctoring provides the certification, allowing online exams to be conducted without the participation of an examiner (proctor), automatically calculating the degree of confidence in the results [4].

For the first time, the American company Proctor U began observing students in 2008: the proctor followed the students through webcams. Then, they began to check the integrity of the students after the exams, watching videos to find violations, but this method still involved a human factor, and the verification was still carried out by a person. Currently, the following foreign services provide proctoring services: Kryterion Inc., ProctorU, Tegrity, Respondus, B Virtual, Software Secure, ProctorCam and Loyalist Exam Services, there are also Russian services Examus, the Proctor Edu system, etc. The services of these proctor- ing providers are used by more than 1,000 educational institutions in 129 countries [5].

The term «proctoring» is derived from the English. «Proctor» is a person who monitors the course of the exam at the university. Proctoring is a procedure for monitoring and controlling remote testing: testing, task completion [6].

Results and Discussion

Nowadays, Russian educational organizations mostly use ProctorEdu and Examus proctoring systems. The Examus project was created and launched in February 2015, and in August, the program was used in test mode at the first entrance exams at the Ural Federal University. In October of the same year, the project received a grant from the Innovation Promotion Fund. These funds were used to improve the first version of the system. To conduct exams, universities such as the Higher School of Economics, RANEPA, St. Petersburg State University have chosen «Examus». Proctoring will help to objectively assess the knowledge of employees and exclude any attempts of fraud [7].

Initially, the ProctorEdu proctoring system was designed to provide online test control in recruiting or appraisal situations, solving the problem of personal identity, cheating, etc. The credibility of the results is achieved through facial recognition throughout the session and intelligent analysis of behavior during testing. The proctoring system supports two main proctoring options and their combination.

Auto-proctoring is automatic observation and confirmation of identity, the assessment of trust is set by the system automatically. «Live» proctoring needs the presence of an observer during the exam, interaction with the participant, the conclusion is presented by the proctor [8].

System features:

1. automatic assessment of test results and biometric identity verification;
2. support proctoring work on Android and IOS mobile devices;
3. seamless integration with the testing system, it works in a browser and does not require the installation of any extensions, plugins and third-party software;
4. communication with the proctor via video and audio communication, chat;
5. connection of an additional mobile camera for better viewing;
6. minimum network requirements — 128 Kbps (50 MB / h);
7. automatic recovery after communication breaks[9].;
8. integration with the testing system of the company or educational organization, for example, with the StartExam, Stepik and Moodle platforms.

Relevant proctoring assessments that can be trusted by users: trust assessment with details of violations, biometric verification, video protocol, PDF-report, uploading results via API and in the form of tables [10].

The license for this product is not permanent and therefore, the price may increase annually. The ProctorEdu proctoring system is too expensive.

Examus is an online proctoring service. Automatic algorithms identify the student's behavior and analyze his behavior in terms of threats to violate the norms of the exam, and report the results of the test to the proctor, that is the person who controls and adjusts the operation of the automation. User identification is based on face recognition algorithms. The image received from the student's webcam is compared by the system with one of the reference images or their combinations. The proctor then makes a visual comparison.

Benefits of the «Examus» service:

1. the complex use of several identification methods allows achieving 99.9 % accuracy;
2. receiving by the client in a convenient form the entire set of materials recorded during the exam: video, screenshots, chat log with the proctor. The built-in scoring system automatically calculates the probability of passing the exam fairly;
3. the presence of a flexible API allows to quickly integrate proctoring with any LMS;
4. in interaction proctors with the students will be exactly according to instructions for the particular exam;
5. precise identification system and constant monitoring of student behavior by proctors leave no chance for frauds;
6. convenient system of viewing the archive and reporting allows clients to conduct a comprehensive analysis of student’s behavior during exams [11].

Today, in Kazakhstan, 25 % of educational institutions use proctoring systems. There are a number of domestic platforms serving Kazakhstan universities: Oqylyq, Oes, Aero. Despite the existing modern developments and technologies in this area, the problem of preventing falsification of test results remains not fully resolved and the material resources of not all universities can afford to purchase or rent this product.

The Kazakhstan proctoring system «OES» allows to monitor the progress of the examinations. The system, using artificial intelligence, automatically verifies the student and monitors violations from the beginning to the end of the exam, then issues the information in the form of a detailed report. The system also records the webcam, microphone and user screen content. All videos are stored on the server and can be viewed at any time. 30 educational institutions of the Republic of Kazakhstan cooperate with the proctoring system (Kazakh-American University, Turan University, KRU named after A. Baitursynov, KazGASA, South Kazakhstan State Pedagogical University, Bolashak Academy, Eurasian Humanitarian Institute, Atyrau Engineering Humanitarian Institute, Aktau Humanitarian and Technical University and etc.).

The Kazakh proctoring system «Aero» enables to conduct online exams with monitoring, quick analytics, and detailed reports. In the proctoring system «Aero», 10,000 students have passed exams at the same time. More than 10 educational institutions of the Republic of Kazakhstan cooperate with the proctoring system (KazNPU, Zerde Holding, Nazarbayev University, Akhmed Yasawi University, KarU, Astana Medical University, etc.).

Kazakhstan proctoring system «Oqylyq» includes additional modules for automated proctoring and anti-plagiarism. This allows to be performed according to the principle of «one window» without switching to third-party systems, which gives convenience to all users of the system (Kazakh National University named after Al-Farabi, Republican School of Physics and Mathematics, Kazakh National Pedagogical University named after Abai, Kazakh National Pedagogical University, etc.).

• able 1 provides an analysis of Russian and Kazakhstani proctoring systems according to some criteria.

Table 1

Features of Russian and Kazakhstani proctoring systems

Continuation of table 1

In the auto-proctoring mode, the student goes through the verification stage. Then the system collects data from his webcam, microphone, and computer screen (cursor movements, keystrokes on the keyboard) and fixes violations. Adherence to predefined rules throughout the session will maximize the credibility of the exam results.

• able 2 depicts a comparative analysis of Russian and Kazakhstan proctoring systems by the type of blocking for violation.

Table 2

Features of blocking

As can be seen from these two tables (Table 1, Table 2), the Russian proctoring systems Examus and ProctorEdu outperform Kazakhstan proctoring systems (Oqylyq, Aero, Oes) in some indicators (locking the operating system, preventing copying or pasting, etc.).

Domestic proctoring platforms are developed without sufficient research of the scientific and methodological foundations for organizing control during exams in the context of distance learning; psychological and pedagogical recommendations for organizing control during exams in distance learning have not been developed. In addition, the following are not provided:

• substantiation of mathematical models, algorithms and hypotheses of artificial intelligence;

87

• support for proctoring work on Android and iOS mobile devices;
• online monitoring of several participants at the same time for each proctor using API technology;
• blocking of parallel input;
• definition of the second monitor;
• biometric control (on the iris of the eye).

Students' reactions to such programs are different. While some Russian students are looking for ways to circumvent the protection, Western students are protesting against the implementation of such systems. In their opinion, the use of remote surveillance systems is tactless and unethical, and the systems themselves are a dubious solution to the problem of online identification [12]. Nevertheless, many representatives of educational institutions believe that only with the help of such applications, teachers can guarantee an objective assessment of knowledge during online control. However, online surveillance systems do not provide a 100 % guarantee. Students can find a number of ways to get around them, for example:

• use headphones that the webcam cannot see;
• start with the person who will pass the exam first and remember the questions;
• use the blind spot of the webcam, where you can position your smartphone, to which the assistant will display answers.

The proctoring system is being developed based on artificial intelligence. Artificial intelligence is trained to track all passing participants at once and «notices» a large number of details: whether there are «extra» people in the frame, whether there are voices in the room, whether the student is trying to switch a tab in the browser or turn on extraneous programs, how often he looks away from the monitor. Having registered suspicious actions, the system notifies the human proctor. It connects to a specific webcam and personally monitors the student (Figure 1).

Artificial intelligence records during the exam:

• absence of the examinee's face in front of the camera;
• a stranger in front of the camera;
• an unknown person in front of the camera;
• conversation or noise;
• mute the microphone or low volume;
• new keyboard handwriting;
• the window with the exam page is not maximized to full screen;
• switch focus to another application or tab.

Despite all the advantages and convenience of proctoring, today, there are still those who are suspicious of proctoring. For example, someone is embarrassed that the examiner is looking at them during the exam, and this interferes with concentration. Furthermore, there is always the possibility that something will go wrong: the Internet connection or the program is lost, or the proctor misinterprets the looking away — a way to build thoughts, and not a desire to spy. In addition, the bandwidth of the Internet for streaming video plays an important role, especially in remote small settlements, where remote technologies are especially relevant. The complexity of organizing and conducting control in distance learning is the need for accurate identification of the student's personality and assessment with details of the violation. The advantages and disadvantages of online proctoring exams are shown in Table 3.

Table 3

Advantages and disadvantages of online proctoring exams

Conclusion

Due to the recent events related to the pandemic, the use of online proctoring systems in universities is a necessity, caused by the requirements for an objective assessment of students' knowledge and compliance to the conditions of quarantine. The need to use proctoring systems during the COVID-19 pandemic has become a problem for the entire community. Universities have faced problems when it is necessary to fulfill sanitary requirements and at the same time, ensure the quality of education through the control of the assimilation of educational programs. A significant limitation of the use of such systems is its cost. Also, based on the comparative analysis of the capabilities of Russian and domestic proctoring systems performed in this study, it follows that the existing proctoring systems do not fully meet the requirements for ensuring the objectivity of knowledge assessment in the online mode.

Distance exams are a real way to define knowledge of students. According to order of the Ministry of Education and Science of the Republic of Kazakhstan, it must be mastered by all universities in Kazakhstan. The search for a solution to how to establish the proctoring process, on the contrary, is becoming increasingly important today.

Thus, although proctoring systems have appeared recently, such systems quickly became a popular service since they provide the best protection today against cheating students during remote control of knowledge. The Internet offers a sufficient number of services for organizing proctoring, at affordable prices. Teachers have the opportunity to use remote monitoring systems when creating online courses, during testing, for conducting exams and in other situations.

Acknowledgements

This scientific article was prepared within the framework of the project No. AP09259657 «Research and development of an automated proctoring system for monitoring students' knowledge in the context of distance learning» under program 217 «Development of science», subprogram 102 «Grant funding for scientific research».

References

1. Андреев А.А. Дистанционное обучение и дистанционные образовательные технологии. [Электронный ресурс] / А.А. Андреев, В.И. Солдаткин // Облако науки. — 2013. — № 1. — С. 14–20. — Режим доступа: https://cyberleninka.ru/article /n/distantsionnoe-obuchenie-i-distantsionnye-obrazovatelnye-tehnologii.
2. Полат Е.С. Теория и практика дистанционного обучения: учеб. пос. для студ. высш. пед. учеб. завед. / Е.С. Полат, М.Ю. Бухаркина, М.В. Моисеева; под ред. Е.С. Полат. — М.: Изд. центр «Академия», 2004.
3. Скибицкий Э.Г. Теоретические основы дистанционного обучения: моногр. / Э.Г. Скибицкий, Л.И. Холина. — Новосибирск: Изд-во НГПУ, 2002. — 136 с.
4. Полат Е. С. Современные педагогические и информационные технологии в системе образования: учеб. пос. для студ. высш. учеб. завед. / Е.С. Полат, М.Ю. Бухаркина. М.: Изд. центр «Академия», 2010. — 368 с.
5. Сагиндыкова А.С. Актуальность дистанционного образования / А.С. Сагиндыкова, М.А. Тугамбекова // Молодой ученый. — 2015. — № 20 (100). — С. 495–498. [Электронный ресурс] — Режим доступа: https://moluch.ru/ar-chive/100/20703/.
6. Seaman J.E. Grade Increase: Tracking Distance Education in the United States / J.E. Seaman, I.E. Allen, J.Seaman. Oakland, CA: Banson Survey Research Group, 2018.
7. Moten J. Examining online college cyber cheating methods and prevention methods / J. Moten, A. Fitterer, E. Brazier et al. The Electronic Journal of eLearning, 2013. — Vol. 11. — Iss. 2. — 139–146.
8. Кусаинов А.К. Технология и методика дистанционного обучения в Республике Казахстан / А.К. Кусаинов, А.А. Шарипбай / Междунар. науч.-практ. конф. «Профессиональное образование и занятость молодежи: XXI век. Подготовка кадров для цифровой экономики». — М.: Профессиональное образование в России и за рубежом, 2019. — 2 (34). — С. 23–28.
9. Kentnor H. Distance Education and the Evolution of Online Learning in the United States. Curriculum and Teaching Dialogue / H. Kentnor. 2015. — Vol. 17. — 1–2. — 28–29.
10. Экзамус — система онлайн-прокторинга. [Электронный ресурс]. — 2013. — Режим доступа: http://ru.examus. net/.
11. King C.G. Online exams and cheating: An empirical analysis of business students’ views / C.G. King, R.W. Guyette, C. Piotrowski. The Journal of Educators Online, 2009. — Vol. 6. — No. 1. — P. 1–11.
12. Phillips P.J. The FERET evaluation methodology for face-recognition algorithms / P.J. Phillips, H. Moon, S.A. Rizvi et al. Pattern Analysis and Machine Intelligence, IEEE Transactions on. 2000. — Vol. 22. — 10. — 1090–1104.