Output and result indicators constitute an essential part of a more general RIS3 step, focusing on monitoring and evaluation. Output and result indicators are the main instruments of monitoring and evaluation processes, and try to capture the effectiveness of the implemented programs’ performance. The definitions for each one of these two kinds of indicators are given below. According to the European Commission (2015, p.7):
- Output indicators relate to operations supported. An output is considered what is directly produced/supplied through the implementation of an ESF operation, measured in physical or monetary units. Outputs are measured at the level of supported people, supported entities, provided goods or services delivered. They are set at the level of investment priorities or specific objective.
- Result indicators capture the expected effects on participants or entities brought about by an operation. Result indicators shall correspond to the specific objectives set out for each investment priority selected. Result indicators go beyond output indicators in so far as they capture a change in the situation, in most cases related to supported entities or participants.
Given the fact that output and result indicators are the main tool for performing monitoring and evaluation activities, it is important to better understand the main concept of these two processes, for effectively choosing the appropriate indices in each case. At the same time, the fact that RIS3 strategic planning has turned out to be a fundamental ingredient of policy making process within EU, indicates the need to adjust M&E mechanisms and thus, the specification of output and result indicators under this framework.
A list of the regions that have applied this method in RIS3 strategies includes: Northern Ireland, Central Portugal, Mazowieckie (PL), Zuid-Holland and Noord-Holland (NL) and many others. The mapping exercised indicated that the definition of output and outcome indicators has been applied in 90% of regions, while more structural change and context indicators have been set by some 70% of regions.
According to the European Commission (2014) it is important to have a common set of guidelines that will enable the project managers to effectively implement the chosen programs under the RIS3 framework. Based on this fact, the identification process of output and result indicators should follow a specific set of criteria, that will maintain the quality of the monitoring and evaluation processes.
More specifically, these indicators should be: i) responsive to policy; ii) normative; iii) robust; and iv) they should be available to be collected or calculated over time. For each of them a baseline value should also be required, in order to make comparisons and evaluations of the implemented policies. It is essential to note that before identifying the output and result indicators, a set of targets and priorities shall be established, under the RIS3 framework. The selection of the appropriate indicators will then be based upon these more general targets.
As it is stated, the result indicators are the expression of the objective of an investment priority (European Commission, 2014). At the same time, output indicators should be program specific and should be derived from the intervention logic of the program. This is a crucial difference between these two kinds of indicators, as it implies that for the case of output indicators, there is no need for setting baseline values. This means that the baseline in these cases is zero (Kokkinoplitis et al., 2015).
Finally, in order to further promote the use of high quality indicators, that could additionally provide the possibility of performing regional comparisons, a set of common indicators has been proposed by the European Commission. These indicators have standard definitions and units of measurement across different countries, when being used for measurement throughout Operational Programs. Thus, it is also possible to aggregate the results to higher spatial levels of analysis, such as national and EU.
EU regions are using different types of indicators to capture the impact of RIS3. The use of a common system of indicators will provide meaning to the data and enable benchmarking of the performance and efficiency of policy measures.
The choice of output and result indicators is implemented based on a set of already defined objectives and priorities, interconnected with the general concept of well-being and progress of people. After having identified the desirable results and targets for each program, the specification of proper variables that would enable a valid measurement of the program’s effectiveness is an essential procedure.
In general, it is important to define highly appropriate output and result indicators, in order to be able to specify and analyse any possible problems that might arise during the implementation process of the programs. All the available information deriving from these indicators, then will be used for understanding the problems and finding solutions to adjust the implemented policies, in order to meet the desirable target values (European Commission, 2014). It is important to note that output indicators are derived as direct effects of the programs, contributing to the further composition of the result indicators, that try to measure the effectiveness of the implemented policies, in terms of wider objectives.
Under this concept and based on previous research (Ekins & Medhurst, 2006; European Commission, 1999), Figure 17 illustrates a diagram produced by the European Commission (2014, p.5), which tries to depict more specifically the use of output and result indicators. As it is shown in this diagram, the achieved outputs are used as an input for measuring each program’s contribution/impact to the overall actual result of the implemented policy. The achieved outputs are measured via output indicators, whilst the actual result is defined by the synthesis of the result indicators. One of the most crucial parts of this process is the definition of the baseline and target values for each one of these variables.
Figure 17 Diagram illustrating the main idea for an effective use of output and result indicators.
Source: European Commission (2014)
Also important, is the fact that other additional factors exist and act during the implementation of the programs, affecting their final actual results. Thus, it is essential that throughout the evaluation process, a distinction between these additional factors and the contribution of policy interventions be made. This can be represented schematically as follows (European Commission, 2014):
|Change in result indicator||=||contribution of intervention
contribution of other factors
According to Nauwelaers and Wintjes (2008) the evaluation process should also play a key role to understand the ways in which complex innovation policy systems operate. Although, evaluation systems were traditionally considered to have an ex post character, there is evidence that they can also be used for improving the design of innovation policies in real time (Magro & Wilson, 2013). Thus, the development and use of proper output and result indicators could work as a propulsive force towards the further development of these policies.
Data for calculating these indicators are available from EUROSTAT, at a national or regional level, for specific time periods. Another indicative example of defining the appropriate indicators for monitoring and evaluating the results of the implemented policies is also presented in the case of the Horizon 2020 strategy by the European Commission (2015).
A valuable source of information regarding the potential variables that can be used as output and result indicators is the List of Common indicators published by the European Commission in 2014. This list provides a comprehensive set of indicators that can be used for monitoring and evaluation, as long as their definitions.
According to the European Commission (2014), the 46 proposed common indicators are grouped based on 10 wide areas of interest: (1) productive investment; (2) ICT; (3) Transport; (4) Environment; (5) Research Innovation; (6) Energy and Climate change; (7) Social infrastructure; (8) Urban Development specific indicators; (9) ETC specific indicators; and (10) Labour market and training.
An analytical roadmap regarding the steps that should be followed before and after the selection of output and result indicators is given below (Figure 18). As it can be seen, the selection of challenges and the specific actions that will be used for their implementation constitute essential prior steps to the definition of indicators.
Figure 18 Roadmap for the selection of output and result indicators for monitoring and evaluation.
- Ekins, P., & Medhurst, J. (2006). The European Structural Funds and Sustainable Development A Methodology and Indicator Framework for Evaluation. Evaluation, 12(4), 474-495.
- European Commission. (2015) Guidance document on monitoring and evaluation of European cohesion policy. The programming period 2014-2020. Brussels, June 2015. Available at http://www.esf.ie/en/ImageLibrary/Repository/Files/Monitoring-Evaluation-Guidance-June-2015.pdf.
- European Commission. (2015) Horizon 2020 indicators: Assessing the results and impact of Horizon. Brussels, 2015. Available at https://ec.europa.eu/programmes/horizon2020/en/news/horizon-2020-indicators-assessing-results-and-impact-horizon
- European Commission. (2014) Guidance document on monitoring and evaluation. The programming period 2014-2020. Brussels, March 2014. Available at http://ec.europa.eu/regional_policy/sources/docoffic/2014/working/wd_2014_en.pdf.
- European Commission. (1999) Indicators for Monitoring and Evaluation: An Indicative Methodology. Working Paper 3 in The New Programming Period: 2000–2006: Methodological Working Papers. Brussels: EC. Available at http://europa.eu.int/comm/regional_policy/sources/docoffi c/working/doc/indic_en.pdf.
- Kokkinoplitis, K., Komninos, N., & Tsipouri, L. (2015) RIS3 monitoring indicators and assessment system. Report (non-paper) for the European Union, December 2015.
- Magro, E., & Wilson, J. R. (2013). Complex innovation policy systems: Towards an evaluation mix. Research Policy, 42(9), 1647-1656.
- Nauwelaers, C., & Wintjes, R. (2008). Innovation policy in Europe. Edward Elgar Publishing.