Introduction: the CADIC evaluation concept
Within the CADIC project plan, altogether two evaluation cycles were planned, each of them directly following-up a phase of implementation of prototypical CADIC tools and services. The aim of this close connection between implementation and evaluation was a direct collection of feedback from pilot users (catalyst SMEs, pilot clusters, Cluster Facilitators and SME-AGs) on the CADIC developments for SME cluster support. Hence, the evaluation approach chosen for the project mainly served as an instrument to collect input and recommendations for tools & services development: how were tools and services implemented and applied? What should be improved during the next RTD work stream in order to better fulfil the needs and requirements of catalyst SMEs and their clusters?
Beyond that, a project that deals with cluster development should consider another aspect of evaluation: which effects did the application of developed tools and support services have on cluster development as a whole? Which were short term effects, which ones became only visible with a certain time-lag? Which aspects of cluster work were supported most by which CADIC service/tool and which aspects still need more or different support? Hence, beyond methodological and technical usability and application, also the effectiveness and benefits/limitations of developed CADIC solutions should be a topic for evaluation in this project.
Even though it is highly desirable – from a scientific as well as from project management point of view – to combine both aspects for the CADIC evaluation approach, empirical studies imply that cluster development processes and the underlying supporting/obstructive mechanisms can only be identified ex-post with a time-lag: according to recommendations by OECD, cluster support programs should be designed at least for a runtime of three to four years. Shorter programs are unlikely to show any impact since cluster and network development are mid- to long-term processes.1 Hence, at the earliest after three years of cluster work, effects on cluster level are visible and might be detected and described by evaluation mechanisms. CADIC with a total runtime of two years and nine months barely hits this limit. The second phase evaluation was timed at the end of the second project phase in project month 28, 22 months after the beginning of the first implementation phase (first six project months were dedicated to basic research, requirements analysis and development of first prototypes). It marked the final point of the second implementation phase and is the last evaluation cycle scheduled in the CADIC work plan.
Consequently, designing an evaluation mechanism including both, analysis of application and effectiveness of CADIC prototypical tools and services, was not considered as a 100% proof concept, since 22 months after the first cluster actions effects and changes might not be identifiable clearly. Nevertheless, an evaluation of the cluster development since the status-quo at the beginning of the project is a desired intent. The evaluation methodology developed therefore contains two parts: on the one hand, a questionnaire was developed which captures the feedback and experiences from pilot users (catalyst SMEs and pilot clusters) when applying CADIC tools and services prototypes. On the other hand, a workshop procedure was designed which aims at capturing experiences as well as first effects/results of CADIC cluster development work during the last months. Furthermore, the workshop aimed at supporting the pilot clusters in defining next steps and learning from other clusters.
After phase I implementation, the evaluation procedure (phase I) was designed to capture whether the clusters are on the right track during the first six months of cluster work. The CADIC Framework for Effective Cluster Practice aimed at evaluating each CADIC cluster’s achievements in realising “pathways to value” at a particular point in time. As the cluster evolvement was rather young and CADIC tools and services were not yet fully implemented and applied at that time, the first phase evaluation procedure was designed to give a quick snap shot whether the clusters had found a good entry point into cluster work.
This short information sheet concentrates on the description of the methdolologies applied during the evaluation workshop held in Berlin during 22nd October 2012.
CADIC Evaluation Workshop – Concept and Set-up
In summary, the workshop had the following objectives:
The preparation to the workshop involved the following key steps:
Workshop Design and Methodology
To reach the objectives of the evaluation, the workshop has been designed divided into two key working sessions:
The country sessions served as a preparatory session for the Cluster World Café in the afternoon and applied an instrument supporting a thorough review of past experiences. Based on this review, the most important fields of action and next steps for the cluster work were identified for each cluster. The methodological steps are described into detail in the following section:
Step 1: Cluster’s goals and objectives: The country team members are asked to formulate their strategic goals and key operative objectives/challenges for the cluster. The moderator summarizes the goals and objectives on a flip chart/pin board.
Step 2: Importance of success factors: This step aims at identifying the main success factors for the cluster (input for Cluster Café Session) and includes the following actions:
- The participants are presented a list of success factors
- The participants are asked to rate the success factors based on their importance with regard to the identified strategic goals and objectives on a scale from 1 (“not important”) to 10 (“ very important”)
- Selection of the most important success factors by “filtering” the list: factors rated above 7 or 8 require most attention in the future (=> for the further detailed discussion).
Step 3: Evaluation of the selected success factors: Simplified evaluation of quality, quantity and status of most relevant success factors identified during Step 2. For each selected factor, the participants discuss:
Question 1: What has been done in the past with regard to the identified success factors (“Have we done enough? Did we choose the right approach?”)
Question 2: How successful where the previous measures? What has been achieved? (satisfactory/good/unsatisfactory results)
Question 3: How pressing is the need for future actions? The participants are asked to rate the future need for action concerning this success factor and actions using a simple semaphore (“Ampel”) ranking, where red colour signifying high need for immediate action and green colour indicates no immediate action for the specific success factor.
Question 4: With regard to the factors that require immediate actions (were marked red at the previous stage), the participants are asked to start brainstorming on necessary next steps or describe management measures and options that have already been planned.
Step 4: Preparation for the World Café Session: The participants are asked to summarize in key words three challenges or themes which they would like to discuss in the World Café Session with other country teams and present the results in form of a simple an initial mind map, where the central node contains the name of the cluster and three additional nodes specify the challenges/themes to be addressed in the World Cluster Café Sessions.
World Café Session
The World Café method is an established methodology for supporting knowledge transfer based in large group interaction. The main advantage of the World Café is that it allows for a quick and focused interaction among the participants and offers the advantages of the fresh perspective: the individuals outside of a cluster are more likely to see the new opportunities and focus on unexpected solutions from a new perspective. At the workshop, a simplified World Café methodology has been adapted to the needs of smaller group.
The World Café method in this workshop has been designed and adapted to the needs of this workshop as follows:
- Setting: each cluster is provided with a flipchart and numerous materials for recording their ideas, including pin cards, colourful markers and pins.
- Content: the World Café Session focuses on finding the solutions for the main challenges or themes that have been identified by the each cluster in the country sessions before
- Format: Country teams present their three key challenges in a form of an initial simple mind map. The mind map was then further elaborated with the help of other participants.
- Moderation: the sessions are chaired by the moderator who provides a welcome and an introduction to the session. He/she is responsible for keeping the time limit on discussion
- Timing: In this simplified version of the technique, the discussions are to be limited to ten minutes per cluster
- Small group rounds: there are ten-minute rounds of conversation at the flipchart of each cluster
- Closure: after the discussion rounds, the mind maps should represent the ideas discussed and generated during the cross-cluster/organisational discourse.