Results – general findings
Based on the evaluation of the selected NRPs, the national research programmes prove to be a powerful set of instruments with a wide and profound spectrum of effects. These effects may appear in the following dominant general forms:
-
Generating new knowledge,
-
providing stimuli and preparation,
-
networking and polyvalent combination of effect contexts, theme and discipline areas, players and institutions (inside and outside the field of science).
NRPs may contribute research results at the cutting edge of science (publications in respected international journals such as Science), however, this is not essential in order to achieve effects.
The direct effects of the programmes, in particular also their longer-term effects and contributory causes are clearly underestimated. The reasons for this lie on the one hand in the documentation being insufficiently directed to evidence of effect and in an insufficiently targeted and structured reporting; on the other hand, NRPs – in accordance with their character as scientific programmes – are positioned mainly in initiating and preparatory phases of cause-effect chains, so that the specific effects will only become fully effective and apparent after the end of the programme (sometimes even considerably later).
Results – conclusions in detail
1. Effects – repercussions and causalities
NRPs exhibit many, diverse and sometimes far-reaching effects. As a general rule, however, no “linear” unambiguous proof of causality can be produced. One of the reasons is that NRPs tend to have a preparatory effect and that the interaction of various players and constellations is of great importance for perceived effects (usually only after a programme). In the effect phase after the conclusion of the programme, activities of earlier NRP players may be decisive. Characterising the NRPs as stimulating and contributing to the effect therefore seems appropriate.
2. Types of effects for which NRPs are suitable
As main types of effects our selection of NRPs included
-
in the context of “science/research”: knowledge generation, promoting young talent, international positioning, establishment of science-based areas of activity at the international level;
-
in the context of “practical application”: preparing the way for new industrial fields of development, starting points for longer-term cooperations between universities and industry, creating aware-ness of problem situations; development of solution strategies, of action procedures and of concepts, proposals for and reviews of measures.
3. Types of effects which for which NRPs are unsuitable
NRPs are unsuitable:
-
for delivering user-ready problem solutions for administrative and political organisations and
-
for providing ready-to-use processes or market-ready products for industry;
-
for satisfying or legitimising current (political) interests or uncertainties through a rapid scientific product;
-
for responding to short-term “cycles” of a economic, social or political nature.
4. Instrument-specific effects
It would appear plausible to us that instrument-specific effects exist that could not be achieved in this way with any other instrument. We recognise them for example in the networking processes and in connection with the occurrence of case-specific inherent dynamism, which also comes to bear outside an NRP, especially after its conclusion. Corresponding effects are a result of the instrument-specific possibility for the combination of subjects, tasks, disciplines (discipline areas), players and effect contexts, also in an unconventional constellation, which inspires to effect-oriented work. In particular, compared with stand-alone research projects, NRPs possess the necessary critical mass to achieve the appropriate effects.
5. Profile of NRPs as instruments
NRPs are instruments conforming to the above-described profile. With regard to the effects, it may present itself in one of two forms (type I and type II, cf. section 6.1.4):
1. NRPs are able to produce innovative cutting-edge achievements in (scientific and technical) research, combined with the promotion of young talent and with application potential for industry (including the medical sector);
2. NRPs are also able to create the preconditions for new directions with application potential for the state and society, based on scientific principles.
When evaluating the effects, distinct hybrids became apparent. Despite all their advantages, they may run the risk that conflicts between different scientific cultures arise and that these may possibly detract from successes of the effects.
The hybrids show that it would be wrong to assume two totally different part instruments within the overall NRP instrument. Rather, the forms mentioned are the opposing ends of a spectrum in the overall potential of the instrument. Nevertheless, the differentiation from the effects perspective has proven sensible since the study of “successes” from the perspective of the effects makes clear that different types of yardsticks apply for the evaluation, depending on the effect context. From the point of view of their fundamental performance capacity however, the NRPs – irrespective of a type distinction – prove to be science-based initiators, forerunners, knowledge generators and network creators, each with its own spectrum of effects.
6. No NRPs in the selection under investigation without results
Our study of the effectiveness showed no NFP that failed to achieve substantial individual results. In our opinion, just a few programmes in the selection under investigation remained at a modest level, either at the level of the overall result or the individual projects.
7. Problem areas
Problem areas arise concerning the questions
-
of the interaction and allocation of tasks between NRP and the government departmental research and
-
of the feasibility of dealing with problems in respect of the availability of the relevant research potential and of consultancy firms “leaping in to fill the gap”.
8. Reporting and evaluation
The NRPs lack systematic documentation orientated towards evidence of effects and an adequately goal-oriented structured reporting to round off the programmes. This was also reported in earlier NRP assessments. The same applies to routine, accompanying and/or ex-post evaluations or assessments of the effects. Evaluations of the effects were also called for by various programme managers during the discussions.
In our experience, an important factor with regards to the evaluations of the effects is the taking into account of the time scale, i.e. especially the incorporation of an appropriate period of time between the end of a programme and the review.
