Within the framework of our government assignment, we have been commissioned to carry out impact evaluations of the nine regional fund programmes for growth and jobs during the 2014-2020 programming period.

The government assignment has consisted of several different evaluation studies focusing on the evaluation of specific structural fund interventions in support of companies. The government assignment is presented in the form of three interim reports and a final report. The following final report is a summary analysis and description of the results of the evaluation as presented in the three previous interim reports. For more detailed information regarding specific results, consult previously completed interim reports as necessary.

Questions

In the final report, we have sought to answer the following questions:

• To what extent do companies with growth potential participate in ERDF structural fund interventions?
• What differences exist in regional impacts between project interventions implemented in programme areas located in metropolitan regions compared to other regions?
• Have the project interventions had any short- or medium-term effects?
• Have the project interventions had any long-term effects? - i.e., effects on the growth and competitiveness of participating companies.
  

Results

Regarding whether companies with growth potential participated in ERDF structural fund interventions during the 2014-2020 programming period, our computations indicate that structural fund interventions have reached important groups of companies with development potential outside metropolitan regions. The programme's objective of influencing key groups of companies in regions with less favourable regional conditions appears, based on our data, to have thus been achieved. We can also see from our empirical calculations that structural fund interventions have targeted a higher proportion of younger, higher educated entrepreneurs and employees than is common across sectors. Finally, significant differences have been found showing that companies participating in structural fund interventions have on average higher work productivity than the control group.

Regarding whether project interventions led to any short-term effects on participating companies during the 2014-2020 programming period, we can conclude that about half of participating companies consider that knowledge enhancement and concrete changes occurred in the company to a great or very great extent. We also see from the survey responses that more than half of participating companies spent a relatively large amount of time participating in the interventions. Given these data, it is difficult to claim that the project interventions have not had an impact on the participating companies, but rather seem an important component in triggering changes in the company and thus contributing to long-term effects. The actual impact on these companies is more difficult to assess using these methods, but it is a good way to gain insight into the conditions for the intervention's long-term impact. One weakness of these methods is the difficulty of estimating the impact on the total population of participating companies due to non-response and uncertainties regarding variation in the assessments of the interventions. The results show that the interventions provide added value, but for various reasons it may still be difficult to involve entrepreneurs sufficiently in these project interventions. In addition to the quality of the interventions, the scale and level of participation among companies in the different interventions is central to the impact. We can see from our data that participation variation is great and also influences assessment of the project intervention's impact.

Regarding whether the project interventions have led to any long-term effects, our main results show that the ERDF has positive effects on variables related to the overall ERDF objectives. It has a long-term positive impact on growth (net turnover), competitiveness (value added) and employment. However, the impact differs depending on which project intervention the companies participated in. However, for the group of companies as a whole, the long-term effects are positive, albeit small. The overall positive effect is significant and the participating companies increased net turnover by SEK 121,900, value added by SEK 57,000 and the number of employees by 0.79 compared to the control group. There is a greater uncertainty regarding the outcome variable number of employees due to its failing to meet the assumption of parallel trends. However, regardless of whether the overall positive impact on the total company population is smaller, we find at project level that strategic projects can generate large, positive and significant impacts on participating companies, even those located in peripheral areas. This is significant because it shows great potential to improve the efficiency of future project interventions.

Regarding regional differences in impacts between project interventions implemented in metropolitan compared to non-metropolitan programme areas, we find no significant differences at the 5% or 10% levels between participating companies in metropolitan and non-metropolitan programme areas. The results thus suggest no clear difference in outcomes related to the programme area in which the project interventions are implemented. Nor could we clearly demonstrate that the level of knowledge among regional representatives has increased during the programming period or that their implementation capacity has improved, although we did see examples of different attempts to work with monitoring and creating synergies. We therefore conclude that entrepreneurship interventions have thus had an impact at the level of entrepreneurs and companies, but have not led to a clear synergy or spill-over effect at regional level, and it remains to be seen whether the current programming period can better strengthen the regional link between ERDF programmes and the idea of smart specialisation.

Method

In our evaluation studies, we have combined a number of qualitative and quantitative methods to get a better idea of the different types of impacts that can arise from a structural fund intervention. In addition to identifying short-term effects through interviews and questionnaires addressed to participating companies, we have used quasi-experimental methods to identify long-term effects on company growth and competitiveness. A problem highlighted in recent evaluations spanning long periods, with implementation and outcomes occurring in different years, is time heterogeneity. Time heterogeneity entails that the same project can have different effects if it is carried out in different time periods. To measure the effects on companies that have participated in ERDF projects, we have used a method rare in a Swedish regional policy context, namely a variant of the Difference-in-Difference (DiD) method. Unlike many other methods, DiD is able to cope with so-called time heterogeneity. As ERDF projects, like many other EU-funded interventions, have this time heterogeneity, the methodology is particularly well suited to this type of evaluation.

Our methodological approach has been to try for decent internal validity in the first place, but at the same time this has meant a reduction in external validity. This means that we must exclude from the investigation participating companies that cannot be followed during the whole observation period. In our case, we have required a two-year pre-treatment period but use matching of outcomes four years earlier. The companies therefore must exist throughout the period 2011-2020. However, we believe that the approach used in this study also has several advantages. In particular, we use matching pre-treatment that makes the assumption of parallel trends easier to achieve. However, this effort has a cost in terms of reduced external validity in that many of the participating companies are not available for monitoring. The representation of the total number of participating companies is thus smaller than if we chose a shorter investigation period. It is important to see and interpret the uncertainties of the results from such a perspective.

In the regional impact analysis, we have chosen to examine the relative impacts. In the relative impact measurement, we have made comparisons using panel data analysis and thus studied possible differences in turnover, value added and number of employees between metropolitan and non-metropolitan companies receiving support. Examining the relative impacts is not a standard impact measurement but the focus is only on comparing and measuring differences in impacts between such programme areas.

Recommendations for monitoring and evaluation of the next programming period

To get a better idea of the potential impact of project interventions on participating companies at an early stage, Growth Analysis recommends that the time spent by participating companies on project activities should be better monitored during the next programming period. Such information can provide an early indication of whether the projects will be able to deliver impacts to the participating companies.

Growth Analysis would also like to emphasise that our data basis for evaluating the impact on participating companies during the 2014-2020 programming period was greatly improved. However, there is still room for improvement in terms of the evaluability and quality of ongoing project evaluations carried out in the next programming period. Regarding project documentation, great variations persist in the quality and level of documentation of projects. It has been very time-consuming and in many cases impossible for evaluators to obtain and use the project documentation that exists from an impact perspective.

Finally, we would also like to emphasise that, whatever the method, impact evaluation of structural fund interventions is not easy. Many methodological challenges are difficult to solve whether we use quasi-experimental methods or other forms of methods. In particular, we experience high non-response rates regardless of the method, which affects the ability to evaluate all projects and participating companies in different programme areas. It is therefore essential to also work on strengthening the project evaluations carried out in the next programming period so that they can be better used in connection with the impact evaluation of projects and programmes at the end of the programming period. In our experience, we need and must use a variety of methods and sources to get a more reliable idea of whether and why we have an impact. We can now say from our impact evaluations that structural fund interventions (business-support interventions) have an impact and, in some cases, a large impact on participating companies, but it is more difficult to speak about the whole population of participating companies. Much uncertainty persists regarding the results and the magnitude of the overall effects, for which we need to find new methodological solutions in the future.

Please note that:

• The Swedish Agency for Economic and Regional Growth uses comprehensible and consistent Theory of Change terminology, including for the different elements of the theory of change for programming period 2021-2027.
• The Swedish Agency for Economic and Regional Growth works for a consistent and high level of project progress and final reporting between different programme areas.
• The Swedish Agency for Economic and Regional Growth develops templates and instructions for progress and final reporting pursuant to the Theory of Change.
• The Swedish Agency for Economic and Regional Growth produces this document using the guidelines for collecting and measuring activity indicators and ensures that the activity indicators are used in a consistent way in all projects and all programme areas. If possible, the activity indicators decided by the EU can be supplemented with additional indicators that are relevant in a Swedish regional policy context.
• The Swedish Agency for Economic and Regional Growth reviews its internal processes to provide evaluators, researchers and other stakeholders with programme and project documentation for use evaluating or developing new knowledge regarding ERDF.