Fraunhofer Institute for Systems and Innovation Research ISIBührer, S.; Edler, J.; Kuhlmann, S.; Seus, S. (2024): Evaluating Public Research and Innovation Policies: A Short History of Co-evolution. In: Edler, J.; Walz, R. (Ed): Systems and Innovation Research in Transition. Research Questions and Trends in Historical Perspective. Cham: Springer.
Evaluation of STI policies over five decades
German and European evaluation practice through the decades
Since the early days of science, technology and innovation (STI) policies, the objectives, concepts and methods of evaluation in this field have undergone a significant evolution. The developments within evaluation research and practice should always be seen in the context of research and innovation policy as a whole, since evaluation concepts and instruments tend to mirror the respective "fashions" and waves in STI policies.
Highlights here include the establishment of competence centers at the end of the 1990s, the new mission orientation, and the introduction of New Public Management approaches in public research organizations.
Professionalization from the 1960s onward
The article in the anthology focuses on evaluations of funding measures, but also addresses questions about the evaluation of research institutions. It focuses mainly on German and European evaluation practice, but also includes the Anglo-American literature on evaluation theory.
In the 1960s, 1970s, and 1980s, the specific evaluation of public research and innovation activities and policies took shape, encompassing a growing range of topics and gradually becoming more professionalized.
In the 1990s, the key approaches and work of the previous decades were developed further, but there were also important new advancements such as the evaluation of multi-actor, multi-measure programs, and the strengthening of a national and international evaluation community.
More evaluations and increased self-reflection
There was an increasing number of evaluations in the decade from 2000-2010 that sought to capture system-level dynamics and provide an understanding of the role of policy and policy portfolios in system development, combined with increased efforts to support policy making with formative approaches.
Conversely, the policy-making system continued to demand a summative evaluation; a quantitative figure to legitimize RTI policy spending. The divergent demands of STI evaluation led to increased self-reflection in the evaluation community with major peer group events, reviews, and analysis.
The years from 2010 to the present have been dominated by discussion on how to measure the impact of research, with a focus on the non-academic aspects of it. The evaluation community is now increasingly exploring the possibilities offered by Big Data analytics.
Evaluation of STI policies at Fraunhofer ISI: Five projects from five decades
Swipe through the years
1982: Evaluation of R&D support measures for SMEs
This study was one of the first evaluations of R&D measures, and especially SME grant initiatives, in Germany. It was the prelude to other evaluations of SME grant initiatives that ran until the early 1990s. Fraunhofer ISI was involved in various configurations.
1995-2003: Concurrent evaluation of the interdisciplinary centers for clinical research (IZKF)
Since 1995, the German Federal Ministry of Education and Research (BMBF) has funded eight interdisciplinary centers for clinical research (IZKF), selected through competition, at university hospitals. This evaluation included an analysis of how IZKF had developed over the grant period; a comparative study of the IZKF; an analysis of the importance of IZKF for modernizing clinical research; and an illustration of the outlook for how university medicine could develop against the backdrop of changing conditions.
2002-2003: Assessment "Future of the competence center programs (Kplus and Kind/net) and future of the competence centers"
This project was an ex-post evaluation aiming to advise the responsible Austrian ministries on whether and, if applicable, how they should continue with these programs. The new thing about the programs and the challenge for the evaluation was that large, medium-term cooperative structures needed to be established between highly heterogeneous partners from industry and the scientific community, whose impacts would be felt at very different levels and over very different time periods. The evaluation not only considered scientific and technological outputs, it also looked at the formation of a cooperative system of corporatist strategic governance within the new structures. The challenge of this project was further exacerbated by the fact that the political landscape, with two ministries following two very similar programs, was incredibly antagonistic.
2010: INNO-Appraisal. Perspectives on Evaluation and Monitoring
The aim of the INNO-Appraisal project was to contribute to a better understanding of how evaluation is currently used in innovation policy in Europe, and how evaluation contributes to policy-making. INNO-Appraisal was the first systematic attempt to provide an overview of evaluation practice in Europe. It helped make evaluation practice accessible to the policy and evaluation community. Furthermore, it contributed to a better-informed evaluation discourse and networking in the European STI evaluation community.
2020: Evaluation of the BMBF Framework Programs Research for Sustainability (2005-2009) and Research for Sustainable Development (2010-2014)
The focal points of this ex-post evaluation were to measure the extent to which objectives had been achieved and record the impact of research funding, particularly the sustainability research funded by BMBF from 2005 to 2018. Alongside the traditional measured variables of excellent research, the evaluation also examined the influence of features specific to FONA, particularly transdisciplinarity and interdisciplinarity, and research. It also looked at the program’s focus on transferring research results. To this end, the evaluation team developed prototype impact pathways and impact mechanisms for research funding that go beyond the effects on the scientific community.
Layout: Renata Sas; Icons: Anatolii Babii/creativemarket, Renata Sas