Evaluation and Monitoring Workshop

 

Introduction: the CADIC evaluation concept

 

Within the CADIC project plan, altogether two evaluation cycles were planned, each of them directly following-up a phase of implementation of prototypical CADIC tools and services. The aim of this close connection between implementation and evaluation was a direct collection of feedback from pilot users (catalyst SMEs, pilot clusters, Cluster Facilitators and SME-AGs) on the CADIC developments for SME cluster support. Hence, the evaluation approach chosen for the project mainly served as an instrument to collect input and recommendations for tools & services development: how were tools and services implemented and applied? What should be improved during the next RTD work stream in order to better fulfil the needs and requirements of catalyst SMEs and their clusters?

Beyond that, a project that deals with cluster development should consider another aspect of evaluation: which effects did the application of developed tools and support services have on cluster development as a whole? Which were short term effects, which ones became only visible with a certain time-lag? Which aspects of cluster work were supported most by which CADIC service/tool and which aspects still need more or different support? Hence, beyond methodological and technical usability and application, also the effectiveness and benefits/limitations of developed CADIC solutions should be a topic for evaluation in this project.

Even though it is highly desirable – from a scientific as well as from project management point of view – to combine both aspects for the CADIC evaluation approach, empirical studies imply that cluster development processes and the underlying supporting/obstructive mechanisms can only be identified ex-post with a time-lag: according to recommendations by OECD, cluster support programs should be designed at least for a runtime of three to four years. Shorter programs are unlikely to show any impact since cluster and network development are mid- to long-term processes.1 Hence, at the earliest after three years of cluster work, effects on cluster level are visible and might be detected and described by evaluation mechanisms. CADIC with a total runtime of two years and nine months barely hits this limit. The second phase evaluation was timed at the end of the second project phase in project month 28, 22 months after the beginning of the first implementation phase (first six project months were dedicated to basic research, requirements analysis and development of first prototypes). It marked the final point of the second implementation phase and is the last evaluation cycle scheduled in the CADIC work plan.

Consequently, designing an evaluation mechanism including both, analysis of application and effectiveness of CADIC prototypical tools and services, was not considered as a 100% proof concept, since 22 months after the first cluster actions effects and changes might not be identifiable clearly. Nevertheless, an evaluation of the cluster development since the status-quo at the beginning of the project is a desired intent. The evaluation methodology developed therefore contains two parts: on the one hand, a questionnaire was developed which captures the feedback and experiences from pilot users (catalyst SMEs and pilot clusters) when applying CADIC tools and services prototypes. On the other hand, a workshop procedure was designed which aims at capturing experiences as well as first effects/results of CADIC cluster development work during the last months. Furthermore, the workshop aimed at supporting the pilot clusters in defining next steps and learning from other clusters.

After phase I implementation, the evaluation procedure (phase I) was designed to capture whether the clusters are on the right track during the first six months of cluster work. The CADIC Framework for Effective Cluster Practice aimed at evaluating each CADIC cluster’s achievements in realising “pathways to value” at a particular point in time. As the cluster evolvement was rather young and CADIC tools and services were not yet fully implemented and applied at that time, the first phase evaluation procedure was designed to give a quick snap shot whether the clusters had found a good entry point into cluster work.

This short information sheet concentrates on the description of the methdolologies applied during the evaluation workshop held in Berlin during 22nd October 2012.

 

CADIC Evaluation Workshop – Concept and Set-up

 

In summary, the workshop had the following objectives:

 

  1. capturing experiences as well as first effects/results of CADIC cluster development work during the last months,
  2. identifying the most critical factors that have supported or hindered successful cluster cooperation and development,
  3. supporting the pilot clusters in defining next steps.
  4. provide the SME pilots with the opportunity to learn and get new ideas from other clusters as regards actions and next steps.

 

Preparatory Steps

 

The preparation to the workshop involved the following key steps:

 

  1. Literature review to identify the key empirical and theoretical approaches to cluster evaluation.
  2. Review of the empirical studies to identify the success factors pertaining to intellectual capital management in cluster 
  3. Development of a questionnaire on the CADIC tools and services aiming at identifying the key characteristics of the SME clusters and their status quo.
  4. Development of a questionnaire aimed at distinguishing networks based on their governance, network dynamics and key strategic goals. It also aimed at capturing the success factors at the network ecology level (factors pertaining to the environment in which networks operate, incl. geo-economic factors, access to funding, regulations, economic development in the region, and linkages to other networks or clusters)
  5. Preparatory telephone interviews with cluster facilitators to identify the status quo in the selected clusters conducted on the basis of the questionnaire developed at step 4

 

Workshop Design and Methodology

 

To reach the objectives of the evaluation, the workshop has been designed divided into two key working sessions:

 

  1. Country Sessions: In these working sessions, the participants of each cluster met in country teams involving the responsible cluster facilitator, an RTD partner and the respective pilot SME representative to conduct a brief participative evaluation of CADIC progress in their clusters.
  2. World Cafe Session: In this joint session, the participants of all clusters were given an opportunity to interact with each other to exchange ideas and the experiences. Based on selected next steps and actions which were identified at the country sessions, the exchange between the different country clusters was to be focused during the cluster world café session.

 

Below is a short description of the design of each working session.

 

Country Sessions

 

The country sessions served as a preparatory session for the Cluster World Café in the afternoon and applied an instrument supporting a thorough review of past experiences. Based on this review, the most important fields of action and next steps for the cluster work were identified for each cluster.  The methodological steps are described into detail in the following section:


Step 1: Cluster’s goals and objectives: The country team members are asked to formulate their strategic goals and key operative objectives/challenges for the cluster. The moderator summarizes the goals and objectives on a flip chart/pin board.

Step 2: Importance of success factors: This step aims at identifying the main success factors for the cluster (input for Cluster Café Session) and includes the following actions:

 

  • The participants are presented a list of success factors
  • The participants are asked to rate the success factors based on their importance with regard to the identified strategic goals and objectives on a scale from 1 (“not important”) to 10 (“ very important”)
  • Selection of the most important success factors by “filtering” the list: factors rated above 7 or 8 require most attention in the future (=> for the further detailed discussion).

 

Step 3: Evaluation of the selected success factors: Simplified evaluation of quality, quantity and status of most relevant success factors identified during Step 2. For each selected factor, the participants discuss:

Question 1: What has been done in the past with regard to the identified success factors (“Have we done enough? Did we choose the right approach?”)

Question 2: How successful where the previous measures? What has been achieved? (satisfactory/good/unsatisfactory results)

Question 3: How pressing is the need for future actions? The participants are asked to rate the future need for action concerning this success factor and actions using a simple semaphore (“Ampel”) ranking, where red colour signifying high need for immediate action and green colour indicates no immediate action for the specific success factor.

Question 4: With regard to the factors that require immediate actions (were marked red at the previous stage), the participants are asked to start brainstorming on necessary next steps or describe management measures and options that have already been planned.

 

 Figure 1: Screenshot from the Moderators Tool supporting the Evaluation Workshop

Figure 1: Screenshot from the Moderators Tool supporting the Evaluation Workshop


Step 4: Preparation for the World Café Session: The participants are asked to summarize in key words three challenges or themes which they would like to discuss in the World Café Session with other country teams and present the results in form of a simple an initial mind map, where the central node contains the name of the cluster and three additional nodes specify the challenges/themes to be addressed in the World Cluster Café Sessions.

 

World Café Session

 

The World Café method is an established methodology for supporting knowledge transfer based in large group interaction. The main advantage of the World Café is that it allows for a quick and focused interaction among the participants and offers the advantages of the fresh perspective: the individuals outside of a cluster are more likely to see the new opportunities and focus on unexpected solutions from a new perspective. At the workshop, a simplified World Café methodology has been adapted to the needs of smaller group.

 

 

Figure 2: Cluster World Café Session during the Evaluation Workshop – Basic setup

 

Figure 2: Cluster World Café Session during the Evaluation Workshop – Basic setup

 

The World Café method in this workshop has been designed and adapted to the needs of this workshop as follows:

 

  • Setting: each cluster is provided with a flipchart and numerous materials for recording their ideas, including pin cards, colourful markers and pins.
  • Content: the World Café Session focuses on finding the solutions for the main challenges or themes that have been identified by the each cluster in the country sessions before
  • Format: Country teams present their three key challenges in a form of an initial simple mind map. The mind map was then further elaborated with the help of other participants.
  • Moderation: the sessions are chaired by the moderator who provides a welcome and an introduction to the session. He/she is responsible for keeping the time limit on discussion
  • Timing: In this simplified version of the technique, the discussions are to be limited to ten minutes per cluster
  • Small group rounds: there are ten-minute rounds of conversation at the flipchart of each cluster
  • Closure: after the discussion rounds, the mind maps should represent the ideas discussed and generated during the cross-cluster/organisational discourse.