Criteria of efficiency of state programs of innovative development of the economy

The article is devoted to the analysis of the effectiveness of state innovation programs. In this connection, the functional features of the evaluation of the effectiveness of programs for the progressive development of the economy, the principles of the methodology for this assessment are justified. To concretize the efficiency parameters, the advantages of program - target planning and budgeting are focused on results oriented and the principles of evaluating the effectiveness of program implementation arising from them. Based on this system-wide methodology, the specificity of evaluating the effectiveness of the actual state innovation programs is revealed. Therefore, the article focuses on program activities, methods of program implementation, in particular, the provisions of the PART methodology. In addition, the article examines the types of performance evaluation, the distinction between the concepts of «efficiency» and «effectiveness» in order to obtain the most adequate assessment of the results of state innovation programs. The significance of the complementarity criterion and its components to account for the quality of innovative development programs is substantiated.

Analysis of the state of industrial-innovative development in the Republic of Kazakhstan indicates positive trends in the modernization of the technological basis of the economy and the expansion of the area of functioning of the innovation infrastructure. But at the same time, the implementation of state programs for industrial-innovative development demonstrates the problem areas.

In particular, the innovative potential of the country's economy in the framework of state programs is of an additional character, as a means for changing the technological and structural basis of the economy. Therefore, there is no detailed study of real prospects and meaningful content of the innovation potential. Further, the scale of the programs, stimulating the main sectors of the economy to modernize, creates multitasking and therefore a certain imbalance in the applied aspect. This same multitasking and generates changes in the goal-setting programs. The ultimate goal of the program is becoming a broad system of target indicators. And this leads to a vagueness of their implementation. Therefore, success in one block of the program can veil problems and resource, organizational, methodological failures in others.

Before highlighting the parameters of the effectiveness of state innovation development programs, it is necessary to delineate the range of general criteria for the effectiveness of government programs in general.

The achievement of strategic objectives of social and economic development of the country requires not only the state impact on the operated system, but the use of the tools allowing to estimate consequences of such influence. Therefore, the evaluation of the effectiveness of state programs is an important element of public administration, performing a controlling, stimulating and corrective function. Assessing the effectiveness of government programs allows you to determine:

• – compliance of made decisions with strategic socioeconomic guidelines;
• – compliance of made decisions with public interests, i.e. acts as a means of public control;
• – rational use of economic and organizational resources;
• – standards and criteria applied in the development, implementation and evaluation of subsequent state decisions;
• – effective government solutions for the dissemination of positive experience;
• – deviations from targets and their causes;
• – ways to adjust when revealing the ineffectiveness of the state decision.

Therefore, the method of assessing the effectiveness of the state program is of particular importance. It takes into account the need for the following assessments:

1. the degree of achievement of the goals and objectives of the subprogrammes and the state program as a whole;
2. the degree of compliance with the planned level of costs and efficiency of use of budget funds;
3. the degree of implementation of departmental target programs and major activities (the achievement of the expected immediate results of their implementation). Evaluation of the effectiveness of the implementation of state programs includes criteria that should reflect the conceptual provisions of program-oriented planning and management – management by results.

A characteristic feature of results-based management is a clear linkage of budgetary funds to planned results, in contrast to standard (costly) budgeting. World practice allows to highlight the following advantages of program – target planning and budgeting based on results:

• – Funding of those public goods and services is provided, the quantity, quality, cost, time and place of provision of which most meet the needs of society and is characterized by the highest indicators of social efficiency at the set resource restrictions.
• – It is possible to critically comprehend the existing lines of spending and abandon many types of expenses that are made without a proper socio-economic justification for the need of these expenses.
• The choice of solutions is made taking into account not only short-term, but more remote consequences; the responsibility of state ministries for the end result is being strengthened, by which we mean not just the provision of a certain volume of services or the fulfillment of a certain scope of work, but also the achievement of certain quality indicators.
• Regular publication of reports on the achieved results of the ministries' activities allows the society to realistically evaluate the activities of the government.
• Increased justification for public spending decisions.
• The information base of government budget decisions is significantly increased. In particular, due to the availability of information on how different levels of program funding can affect the social and economic efficiency of public spending, it has become possible to reduce costs without reducing programs.
• It is possible to realistically assess the financial condition of the state, thanks to a more complete picture of the resources at its disposal. This is particularly important for managing future risks, above all, the risk that future generations will be burdened with an exorbitant burden of financial obligations resulting from implicit or unrecorded obligations.

The basic principles for evaluating the effectiveness of the implementation of state programs: independence, objectivity, ensuring maximum socio-economic efficiency, professional competence, openness and transparency.

Taking into account the existing principles for evaluating the effectiveness of the implementation of state programs, we define the main criteria for evaluating the effectiveness of state programs:

• Expediency (relevance) - compliance of the project goal with the tasks that were to be addressed within the project, as well as physical and strategic conditions in which the project was carried out, including analysis of the quality of preparation and project structure - the logic and completeness of the project planning process, as well as internal logic and consistency with the project structure.
• Efficiency - how well the project activities are implemented. The fact that the results were achieved at a reasonable price, i.e. how well the invested funds were transformed into achieved results in qualitative, quantitative and temporal terms, as well as into the quality of achieved results: targeted allocation of financial resources, personnel compliance, technical compliance, temporary compliance in phases and activities.

Effectiveness - analysis of the role of results in achieving the project goal, and how the predictions made have affected the project achievements: achievement of the program's final result, achievement of the program's immediate results.

• Controllability of the project - the presence of a clear functional, resource and temporary fixation, the presence of monitoring mechanisms and media coverage, the presence of a self-assessment system at various stages of the program.
• Project effect (social and economic effect) - the project's significance for a wider environment, and its contribution to the expansion of (sector) tasks, brought together in the overall objectives of the project, as well as in achieving the goals of all other state policy objectives.
• Sustainability - an assessment of the likelihood that the benefits and benefits of the project will continue to flow after the completion of external financing, with specific reference to factors related to the ownership right of the beneficiaries, to the policy support factor, to economic and financial factors, to sociocultural aspects character, gender equality, availability of adequate technology, environmental aspects, as well as institutional and managerial capacity factors.
• – Systems of indicators and indicators for assessing the effectiveness of programs, at the level of which the evaluation criteria are specified, are developed in the process of working on the program, which allows avoiding many mistakes.
• – Declarativeness of the program goal and, as a result, the impossibility of assessing the effectiveness and efficiency of the program.
• Lack of connection between the declared abstract goal and the same declarative tasks.
• – the indicators for monitoring the program, if any, are not linked into a logical system, which does not allow conclusions to be drawn about the progress of the program.
• – the process of intermediate and final evaluation of the program is extremely time-consuming, and, sometimes, impossible, which does not allow to use the experience gained in the implementation of the program for the development of subsequent program documents.

Thus, the use of common assessment standards is aimed at unifying approaches and ensuring the possibility of comparing the results of various programs. However, in relation to assessing the results of state regulation of the innovation sphere, this approach is limited and can be largely formal.

In international practice, extensive experience has been gained in assessing the effectiveness of government decisions and government targeted programs, expressed in a wide variety of methods, models, tools, algorithms, indicators. Evaluation of the effectiveness of state programs is evolving in parallel with the development of measures of state support for the object of management. This explains why in such a strategically important and dynamic area as state regulation of innovation activity, the existence of a universal methodology is impossible and a constant search is needed for the best ways to assess the effectiveness of government decisions. At the same time, the generalization of the accumulated experience, flexibility in the application of the most appropriate assessment tools makes it possible to evaluate the innovative development program in a multifaceted manner and to identify the limitations of a different nature.

As a general difficulty of evaluating government programs, we can single out: the laboriousness and resource intensity of accounting for all the results of program implementation; the difficulty of determining the result in the form of profit, which complicates the comparison; the difficulty of identifying and evaluating indirect results of the program.

At the same time, the evaluation of the effectiveness of innovation development programs has a number of features that are associated with specific features of innovation activity:

• – support for innovation is multidimensional, its object may be the development of venture financing, the creation of technology parks, the formation of innovation clusters, the development of exports of high- tech products, etc. For use as target indicators of programs in practice, several dozens of indicators have been developed. Therefore, the complexity is the choice of the right indicators and evaluation criteria for a specific event of the state program;
• – despite the active development of state programs in the field of innovation, there is no single definition of the term «innovation». This reduces the objectivity of the assessment;
• – innovative projects are unique, experimental, and sometimes random, which makes rigid targeting inappropriate. Under these conditions, it can be difficult to assess the relevance of the results to the goals;
• – a significant distribution of the effect over time. The effect of the project to create an innovation infrastructure facility may appear a decade after the creation of the facility. When evaluating government programs in the field of innovation development, there is a growing need to evaluate the long4erm effects of their implementation, a method of studying the history of projects (historicaltracing). For example, not only the indicator «Number of newly created small innovative enterprises» is important for evaluating the effectiveness of the state program in the field of supporting innovative entrepreneurship, but how many of these enterprises remain on the market and their growth prospects are the longterm stability of the program results;
• – in the course of innovation, there are several types of effects that must be taken into account when assessing the effectiveness of government programs: economic (including budget and commercial), social, scientific and technical, and environmental. The basis of innovation is the interaction between the subjects of innovation systems, in this regard, it is important to evaluate the effect for each participant in the program. Difficulties arise in assessing the synergistic effect of their interaction.

In assessing the effectiveness of government decisions in the field of innovation development, the importance of targeted application of tools that allow most fully quantitatively and qualitatively characterize program activities increases.

When selecting indicators to assess the effectiveness of projects, they usually use the allocation of four groups in accordance with the logical model of the program: inputs; direct results (outputs); outcomes; impact [1].

The most objective, available for measurement and comparison of quantitative indicators, which can be:

• absolute (volume of innovative products shipped);
• structural (share of enterprises engaged in technological innovation);
• relative (volume of innovative products per capita).

However, when evaluating the effectiveness of innovation development programs, given the uncertainty of the results of innovation, the role of quality indicators and the need to use methods such as benchmarking increase. The use of quality indicators makes it possible to identify causal relationships in the innovation system, determine the causes of the positive and negative effects of the program implementation, identify promising areas of government support (for example, how innovative enterprises assess access to financial resources).

The analysis of foreign experience shows that the evaluation of the project results is accompanied by an evaluation of the program implementation process. A striking example is the PART (Program Assessment Rating Tool) technique used in the United States until 2009 [2]. In evaluating the effectiveness of budget programs, the assessment was carried out in four aspects:

• Program Purpose and Design;
• Planning;
• Management;
• Results.

Such a distinction makes it possible to more accurately identify the factors that determine the effectiveness or inefficiency of government support measures. The effectiveness of the program can be reduced due to the inefficient implementation of the measures envisaged.

Conversely, omissions during the program development phase can lead to a lack of a positive effect during program implementation.

Since the participants in innovation activity are a wide range of participants, it is necessary to apply both internal and external assessment of the effectiveness of government programs in the field of innovative development. As an external assessment, an assessment of the state program by independent experts and the public can be used. Permanent external monitoring can be used in conjunction with a detailed external assessment of the effectiveness over certain periods of time, as is done in the Framework Programs of the European Union for the development of scientific research and technology.

According to the time of the assessment of the effectiveness of state programs, the following types of assessment can be distinguished [3]:

• a preliminary assessment (ex-anteevaluation) is carried out before the start of the program implementation, at the development stage, it is the basis for determining the target indicators of the program;
• intermediate assessment (intermediate evaluation) involves evaluation at the intermediate stages of the program;
• evaluation of the program in real time (real-timeevaluation) is close to the concept of monitoring, is a tracking of the results of the program during its implementation;
• final evaluation (ex-postevaluation) - evaluation of actual results after the completion of the program. To assess the effectiveness of programs in the field of innovation development, the ability to repeatedly perform this type of assessment is necessary.

The final step may be the determination of the quality of the assessment. Given the above difficulties associated with the evaluation of programs in the field of innovative development, this stage, often overlooked in practice, seems necessary. His task is to determine how well the assessment was carried out, whether its results were taken into account in the future. Lack of evaluation prevents the spread of successful experience. Without subsequent use of the results, the entire assessment process is a formality.

The results of the evaluation should be taken into account when developing follow-up programs, ensuring the flexibility of government regulation. Thus, the European Commission's Research and Innovation Directorate submits an annual report on the evaluation procedures carried out during the year.

Considerable methodological complexity is due to the fact that in the works of both domestic and foreign authors a clear conceptual apparatus has not yet developed with respect to criteria determining the effectiveness of government programs (for example, the concepts of efficiency and effectiveness are equated [4].In foreign practice, when evaluating the effectiveness of programs (efficiency) and performance (effectiveness) are differentiated and relate to the indicators reflecting the ratio of price and quality of the program (valueformoney).

Under the effectiveness refers to the ratio of results and costs. The criterion of efficiency is to maximize the result for a given amount of resources or to minimize the amount of resources for a given result while maintaining quality [5].

Efficiency is interpreted more multifaceted. Efficiency is understood as:

• Conformity of goals, volume of public spending and the results obtained with their help [6];
• obtaining the planned results in the implementation of activities [7];
• the degree of achievement of the planned results [8].

In general, when evaluating performance, we are talking about a comparison of planned and actual results. In this case, the result can be a negative value.

This group of criteria sometimes includes efficiency, understood as a reduction in the cost of resources used in the implementation of activities, while maintaining quality.

As other aspects of performance evaluation, it is possible to single out expediency (efficacy, appropriateness, strategic policy alignment) – to what extent the program is in line with strategic goals and contributes to their achievement. Evaluation of state programs in the field of innovation development should be performed not in isolation, but in the system of strategic goals of the country's socio-economic development. For example, the creation of an innovation infrastructure facility in a region should not lead to an imbalance in its territorial development.

However, the main purpose of the assessment should not be the calculation of indicators, but the identification of the reasons for the results of the program. At the same time, the formation of program results can be influenced both by internal (program management) and numerous external factors (interaction of participants in the innovation system, level of technology development, etc.).

It is important to determine if other measures could ensure a higher efficiency of state support for innovative development. Therefore, one of the key issues in evaluating the effectiveness of state programs in foreign studies is the assessment of the additionality indicator.

The additionality indicator reflects the difference between socio-economic development, taking into account the implementation of the state program and socio-economic development in the absence of government intervention. This indicator allows to evaluate the effect, the increment of socio-economic indicators obtained through the implementation of the state program. For example, the US General Accounting Office within the framework of the implementation of the Advanced Technologies Development Program (AdvancedTechnologyProgram) conducted an assessment - would receive a privately financed project supported by the program in the absence of budget funding?

The answer to this question cannot be unambiguous, since various internal and external factors can influence the increase in innovation activity. Therefore, it is difficult to assess how much of the program results were obtained through the implementation of program activities, and how much of the results were achieved due to the influence of independent factors. In this regard, the measurement of the additionality indicator remains an extremely difficult, but methodologically important problem.

In accordance with the OECD approach [9] there are three types of indicators of additionality.

• Additional resources (input additionally) characterizes the degree to which regulatory intervention ensures the attraction of additional resources or simply replaces resources that could be obtained from other sources (market, own resources of the enterprise, etc.)
• The additional results (output additionally) characterizes that part of the final results of the program, which could not be obtained without regulatory intervention.
• The effect of the implementation of the program can be not only obvious and accessible for quantitative measurement (obtaining additional funding), but also hidden (the development of cooperative ties). And it is these hidden, difficult to unambiguously measured effects that may be the most significant. They are characterized by behavioral additionality (behavioral additionally) —the change in the behavior of the control object after the regulatory intervention.

As noted, there is no universal methodology for assessing the effectiveness of government decisions in the field of innovative development. In this regard, we can talk only about the accumulation of positive evaluation experience, the correct determination of the focus, method, evaluation steps, taking into account the specifics of program activities. Features of innovation and a number of ambiguous methodological issuesrequire the most flexible approach. While providing flexibility, the development of reporting and external evaluation contribute to maintaining the quality of the assessment.

Improving the assessment of the effectiveness of state decisions in the field of innovative development will contribute to improving the efficiency of public administration and the achievement of the strategic goal of innovative development of the country.

References

1. Шаркова А.В. Совершенствование оценки эффективности государственного регулирования инновационного предпринимательства / А.В. Шаркова // Научное обозрение. — 2012. — № 3. — С. 297–305.
2. Шаркова А.В. Регулирование инновационного предпринимательства в регионе: теория и практика / А.В. Шаркова // Предпринимательство. — 2009. — № 6. — С. 140–145.
3. Smart Innovation: A Practical Guide to Evaluating Innovation Programmes. ECSC-EC-EAEC Brussels-Luxembourg, 2006 — 196 с.
4. Бреусова А.Г. Оценка эффективности государственных программ / А.Г. Бреусова // Вестн. Омского ун-та. — 2015. — № 2. — С. 128–136.
5. Jackson P. Value for money and international development: Deconstructing myths to promote a more constructive discussion. OECD Development Co-operation Directorate [Electronic resource] / P. Jackson. — Access mode: http://www.oecd.org/dac/effectiveness/49652541.pdf.
6. Афанасьев М.П. Инструментарий оценки эффективности бюджетных программ / М.П. Афанасьев, Н.Н. Шаш // Вопросы государственного и муниципального управления. — № 3. — 2013. — С. 48–69.
7. Газизуллин Ф.Г. Инновационные и институциональные прорывы — доминанты развития экономики России в условиях новой индустриализации / Ф.Г. Газизуллин, Н.Ф. Газизуллин // Проблемы современной экономики. — 2013. — № 2 (46). — С. 28–29.
8. Evaluation of Innovation Activities. Guidance on methods and practices. — European Union, 2012 [Electronic resource]. — Access mode: http://ec.europa.eu/regional_policy/sources/docgener/evaluation/pdf/eval2007/innovation_activities/inno_ activi- ties_guidance_en.pdf.
9. Evaluation. OECD Innovation Policy Platform, 2010 [Электронный ресурс]. — Режим доступа: