National
Development Agency

The National Development Agency commissioned the Scientific Research Institute of Western Hungarian of the Centre of Regional Research of the Hungarian Academy of Science (HAS) to carry out the evaluation targeting the development of the monitoring system and the indicators of NDP ROP, which was completed at the end of 2007. The objective of the assessment was to screen the monitoring system of NDP I ROP in the 2004-2006 programming period by conducting structured face-to-face interviews and carrying out document analysis.

Evaluators identified the following:

• As regards programme-level indicators, indicators defined for monitoring ROP are in part suitable for appropriately presenting and measuring the implementation of ROP objectives.  
• Output indicators outweigh other indicators in almost every case and result indicators need to be increasingly emphasised at a measure level.  
• ROP programme level indicators explicitly only enable the evaluation of efficiency (comparison of real and anticipated outputs, results and impacts); however, the criterion of effectiveness is not assessed at all.   
• Evaluation and measurement technique type problems and contradictions persist in the case of every measure.
• The majority of individuals interviewed are satisfied with the indices evolving at the end of the consultation procedures.

Evaluation of measure NDP I ROP 2.1 "Improving access to disadvantaged regions and micro-regions"

The National Development Agency commissioned AAM Management IT Consultancy Ltd. to carry out the evaluation of the NDP I ROP 2.1 measure „Improving access to disadvantaged regions and micro-regions, which was completed at the end of 2007. The objective of the assessment was to increase the usefulness of similar types of measures. The analysis of statistical data and sector trends was completed and interviews with beneficiaries and implementing organisations were conducted within the framework of the assessment.   

Evaluators identified the following:

• The development scheme was justified; the measure complimented missing components and implemented relevant developments.
• Component ROP 2.1.1 (minor by- and access roads): the technical state and accident statistics of these roads improved, which demonstrably contributed to developing employment; however, the measure did not manage to decrease the ratio of dead-end settlements. 
• Component ROP 2.1.2 (roads leading to industrial and tourism zones): these measures effectively contributed to improving the economic and employment status of the given settlements; however, it was not possible to resolve the measurement of other impacts.
• Component ROP 2.1.3 (development of local transport): useful developments justified on their own accord; however, explicitly due to this – and the lack of a system-level approach – these measures were incapable of changing unfavourable general trends.   
• The development scheme reached the targets set; its effectiveness complied with benchmark requirements; maintainability was ensured; however, problems are expected to arise following the 5-year maintenance period specified. 

Evaluators made the following recommendations:

• Stronger enforcement of regional development and regional criterions in the case of road development calls, since in such a network, it is not possible to pinpoint the optimal level of potential developments by assessing individual projects and without examining correlations.     
• From among mass transport developments, it is necessary to prioritise complex integrated developments and it is worthwhile considering integrating these types of developments with the ROP measure targeting settlement rehabilitation. 
• Modifying the scope of eligible expenditures is recommended and it would be worthwhile authorising the allocation of reserves in the budget.
• Carrying out a procedure audit is recommended, in addition to assessing the staff size and level of preparation of Regional Development Agencies.

Evaluation of the measures NDP I ROP 2.2 aimed at the rehabilitation of urban areas 

The National Development Agency commissioned the Centre of Regional Research of the Hungarian Academy of Science (HAS) to carry out the evaluation targeting improving the effectiveness of the NDP I ROP  2.2 measure "Rehabilitation of urban areas"; the evaluation was completed at the end of 2007. The evaluation covered the specification of definitions and literature underpinning the rehabilitation of urban and brown field areas and gathering EU experiences, more specifically, primarily experiences in connection with Hungarian developments implemented so far. This underpinned the objective of making new calls setting out similar tasks more efficient in relation to current practices, for these calls to reach a wider scope of settlements and be more consistent with needs stipulated by local governments. 

Main findings of the evaluation:

• The most relevant and largely consistent finding of the field surveys and interviews conducted in 35 towns/cities in connection with the objectives and efficiency of the tendering system focusing on urban rehabilitation was that the call met the needs of local governments and that the projects, with a few exceptions, were successfully completed.   
• In almost every case, the investments that were implemented considerably improved and shaped the profiles of smaller towns.
• Developments implemented in connection with public spaces and building reconstructions contributed to strengthening the sense of settlement identity, raising the standard of public services and stimulating the economy of the town  or the settlement (boosting tourism in the case of certain projects), i.e. to creating settlements with better access. 
• The call – in line with the basic principle of urban rehabilitation – targeted the renewal of run-down derelict residential zones; however the majority of investments (over half in terms of the amount of funding approved and the number of applications) were mostly implemented in downtown precincts.    
• The way in which all applicants approved funding managed to comply with horizontal objectives, however equal opportunity and environmental sustainability objectives were not key components of any of the investments, but were instead considered as compulsory tasks to „get over with”  is a similarly unique feature of the call.

Enforcement of the principle of equal opportunity, as horizontal principle – evaluation study aimed at assessing the efficiency of NDP I ROP

The National Development Agency commissioned the Békéscsaba Unit of the Great Plain Scientific Research Institute of Centre of Regional Research of the Hungarian Academy of Science (HAS) to carry out the evaluation of the enforcement of the principle of equal opportunity, as horizontal principle, in the Regional Development Operational Programme; the evaluation was completed at the end of 2007. The evaluation had a dual focus. It on the one hand evaluated the enforcement of equal opportunity, as horizontal principle within the framework of the Regional Development Operational Programme, whilst on the other hand exposed the impact of its mechanisms on equal opportunity in the case of projects implemented within the framework of ROP.    

Main findings of the programme-level evaluation:

As regards the equal opportunity objectives defined by Community policy, ROP objectives primarily contribute to improving equal opportunity in the following areas:   

• preventing violation of human rights;
• eliminating exclusion from decision-making;
• eliminating economic dependency;
• ensuring access to knowledge and information;
• ensuring right to health;
• ensuring access to goods and services, disposal of resources, and
• reducing regional disparities.
• Summary findings covering specific age groups identified that the anticipated equal opportunity objectives were not specified in the documents and a separate impact analysis was not conducted in connection with these.
• Ensuring equal opportunity was attempted within the scope of programming during the course of setting up the authority and the management. From among target groups, it is possible to follow the inclusion of Roma representatives and individuals representing persons with disabilities; however, the monitoring committee does include a women’s organisation (women were not represented through organisations, but instead women are directly represented through committee members, 30% of whom are women).    
• In the area of equal opportunity, ROP documents fulfilled compulsory tasks in part stipulated by the equal opportunity policies of the EU and in part by Hungarian policies; however, raising the awareness of applicants to identify equal opportunity related problems and promote finding potential solutions for these   was only successful to a limited degree.

Main findings of the programme-level evaluation:

• The projects selected for analysis justify that the principle of equal opportunity can work in the case of projects implemented in connection with any given ROP priority.   
• On the basis of available information, the assessment of equal opportunity impacts can either not be performed, or can only be performed on the basis of considerable estimates, since impact assessments were not compiled during the course of planning, there is no „reference basis” and the vast majority of project managers were not capable of providing suitable information for objectively measuring impacts.   

Evaluation of the measure NDP I ROP 2.3 "Developing the infrastructure of kindergartens and primary education institutions"

The National Development Agency commissioned AAM Management IT Consultancy Ltd. to compile an (ex-post) evaluation study on the National Development Plan I Regional Operational Programme (ROP) Measure 2.3 „Developing the infrastructure of kindergartens and primary education educational institutions”; the study was compiled  the end of 2007. The most relevant objective of the study was to present the extent to which applicants approved funding contributed to the fulfilment of objectives, what results the projects reached, as well as what new impacts can be expected in the future. In the study, evaluations were searching for an answer to the question relating to what changes are needed in measures with similar types of development objectives to be able to increase usefulness. 

Evaluators identified the following general findings on the basis of the assessment:

• The development scheme was justified, since the basic problem truly exists and can be felt in the given area. The measure implemented relevant developments: it improved the infrastructure of institutions in a statistically measurable way, which contributed to reducing the extent to which these institutions are lagging behind. Projects of ROP Measure 2.3:
• significantly improved the infrastructure of the concerned institutions,
• effectively reduced over-crowdedness,
• increased the opportunities of children enrolled in these institutions for continuing their educational career through the development of quality infrastructural components and methodological development.
• The development scheme fully reached the targets set.
• It would have been possible to improve the efficiency of the tendering system. The open tendering framework, use of two intermediate bodies and the relatively complicated administrative requirements in relation to other schemes made it necessary for the implementation system and beneficiaries to invest a higher rate of resources.
• The time lapsing up to the decision made in connection with the approval of funding also exceeded the expected timeframe.
• Moreover, the high volume of applications accepted and assessed following the depletion of the framework, as a result of which considerable resources were (unnecessarily) invested on both sides, caused problems.

Enforcement of sustainable development, as horizontal principle in NDP I ROP

The National Development Agency commissioned the Békéscsaba Unit of the Centre of Regional Research of the Hungarian Academy of Science (HAS) to carry out the evaluation of the enforcement of the principle of sustainability (sustainable development), as horizontal principle, in the Regional Development Operational Programme for the 2004-2006-period; the evaluation was completed at the end of 2007. The evaluation is largely based on document analysis (Regional Operational Programme and the ROP Programme Compliment document, Environmental Sustainability Guide, Strategic Environmental Assessment and application guidelines) and case studies (projects selected by VÁTI Hungarian Public Non-Profit Company for Regional Development and Town Planning). After clarifying the notion of sustainability, planning documents are compared with sustainability criteria specified by expert according to measure.     

In overall terms, the evaluation positively assessed the Operational Programme, although serious shortcomings were identified in several places. In addition, the evaluation formulates a series of recommendations for the 2007-2013-period.  

Evaluators identified the following main findings:

• Measures of the Regional Operational Programme tying in with tourism and urban rehabilitation fundamentally satisfy environmental requirements underpinning the principle of sutainability; however, they need to satisfy deeper criteria truly enforcing the environmental requirement underlying the principle of sustainability in a more diverse and broader context than today.   

Main findings made by evaluators:

• Environmental policy objectives should be applied in an extremely wide scope within the framework of the evaluation system; preferably those that set more rigorous limits. In the case of environmental policy objectives, it would be practical to define objectives that must, as a compulsory requirement, only be reached at a later date on the basis of legislation. 
• In the future, principles tying in with environmental conduct under-represented in the programming procedure (preserving the opportunity to choose lifestyles, local resources benefiting locals and making the principle of sustainability an ethical norm) need to be enforced more rigorously.
• Sustainability messages should be tailored to themes and adjusted to the theme of the given call. A special sustainability evaluation criterion and guide should be compiled in connection with, or to compliment specific operational programmes, or even specific priorities, instead of using central sustainability evaluation criteria and sustainability manuals.  
• From among the criterions stipulated in connection with the project management, preferential treatment should be ensured if the manager holds qualifications that presuppose knowledge of sustainability (environmental engineer, geographer), of if the management organisation gained experiences in this area earlier. 
• Sustainability criteria needs to be increasingly taken into account within the context of ongoing planning, i.e. during the course of compiling and updating (performed every two years) action plans tying in with operational programmes 

Evaluation of NDP I ROP tourism development measures

The Transdanubian Scientific Research Institute of Centre of Regional Research of the Hungarian Academy of Science (HAS) carried out the evaluation by the end of 2007. The objective of the evaluation involved evaluating the implementation of the following measures of the Regional Development Operational Programme (ROP): 1.1 Development of tourist attractions’ and 1.2 Improving tourist accommodation capacities. The following main methodology was used: document analysis, interviews conducted with project managers, questionnaires conducted with the staff employed at RDAs, as well as the analysis of SMIS (indicator) data.  

The evaluation starts by briefly analysing the status of the Hungarian tourism sector, following which it assesses the effectiveness of tourism interventions in a complex way and broken down according to region and measure and formulates specific recommendations in connection with the implementation system and calls for applications alike. 

Main findings identified by evaluators:

• Lack of creativity during the course of the elaboration of the measures;
• Projects were not appropriately and sufficiently adjusted to the tourism development strategy of regions; projects mainly focused on accommodation developments.
• It was difficult to monitor the projects via SMIS owing to differences in interpretation and low volume of uploaded data.
• More developed regions performed better in the case of both measures as regards approval of funding, even if we take account of the different tendering criteria specified in connection with the two groups of regions. 

Main recommendation made by evaluators:

• Standardisation of SMIS data uploading procedures and methods and setting data uploading deadlines would be important in order for SMIS to function as a truly updated and reliable information source and for the system to effectively facilitate the assessment of social and economic impacts generated by programmes.    
• The complexity of administration should be considerably reduced.

Calls for applications should be more transparent and should use clearly defined notions.