|dc.description.abstract||At the onset of the ALICE RAP project, the following objectives, description of work, and
main tasks were agreed for Work Package 21:
1. To evaluate the overall functioning of the collaborative research project, using a
state-of-art systems model of partnership functioning.
2. To document the interactions and linkages between project inputs, throughputs and outputs as
the project unfolds over five years.
3. To facilitate structured discussions involving all partners, to be held at the end of every
project meeting, to identify ways to optimise partnership functioning.
4. To produce a report about what is learned about optimising partnership functioning, to benefit
future European research collaborations.
The frameworks for the evaluation are the Quality Criteria of Transdisciplinary Research (QCTR) and
the Bergen Model of Collaborative Functioning (BMCF), an established systems model with five
(1) Inputs including projects aims, partner financial, intellectual and technical resources; (2)
Production processes (work directly related to meeting objectives) including positive
and negative interaction amongst inputs, and positive and negative aspects of leadership,
communication, and roles/structure;
(3) Maintenance processes (work directly related to keeping the partnership functioning
(4) Outputs (e.g., delivery on objectives) including additive, synergistic and antagonistic
(negative) outputs, and
(5) Feedback processes whereby outputs generated underway affect continuing input and throughput
These are the main Work package 21 tasks as agreed at the onset of the ALICE RAP Project;
subsequent modifications are noted [in brackets].
Task 1: During the first three months of the project, at the half way point and again near the end
of the project, all willing WP leaders and key administrative personnel will participate in 15-25
minute recorded, structured telephone interviews. The interview guide will include brief questions
related to all five evaluation elements, with greater detail given to the element(s) having the
most relevance at the various phases of the project. In others words, element 1 will be emphasised
at the first round of interviews and elements 4 and 5 will be emphasised during the last round of interviews. [The planned first round of telephone interviews was completed, while the 2nd and 3rd
rounds were substituted by self-report questionnaires completed either at partner meetings or
online. This change was made based on preferences expressed by the Work Package Leaders, and agreed
to by ALICE RAP leadership and the European Commission’s Project Officer.]
Task 2: Following each round of interviews, selected recorded material will be transcribed and
analysed and report will be generated and circulated to all partners prior to the next project
meeting. The reports will be structured to fully protect the confidentiality of interviewees. The
three reports will be synthesised into a final report following the third round of interviews. The
reports will be used by the WP leaders to illuminate ways to improve the functioning of the
[This task was completed with data generated by questionnaires, in addition to telephone interview
Task 3: At the conclusion of each project meeting, WP 21 will facilitate a structured discussion
(recorded, guided by a structured interview guide) that will briefly address each of the five
elements listed above. However, at each meeting, one of the elements will be intensively in focus,
corresponding to the progression of the project. In other words, element 1 will be the focus of the
discussion at the first meeting, elements 2 and 3 will be the main focus of the second meeting, elements 4 and five will be the main focus of the third meeting, and so on.
[All the planned project meeting discussions took place, but not all were recorded due to practical
limitations. When recording was not feasible, self-report questionnaires were used to gather needed
data. The discussions were augmented by online surveys.]
Task 4: After each project meeting, WP 21 will transcribe recordings and produce an analysis and
report to be circulated to all partners. The reports will be used by the WP leaders to illuminate
ways to improve the functioning of the project.
[Following each project meeting, data analysis and reporting were done as planned, but the data
came not only from recordings; qualitative and quantitative data were also used, collected by
self-report questionnaire (both paper-and-pencil and online).]
Task 5: As an ongoing task, WP 21 will produce a running analysis using all data as they become
available, using the BMCF as the evaluation framework. This will result in a final evaluation
report, publication of results in the international peer review literature and presentations at
To meet the goals and accomplish the evaluation tasks, a range of qualitative and quantitative
methods was used in a participatory/exploratory sequential mixed methods study design. The data
sources were evaluations following ALICE RAP Partner meetings, data from several computer-mediated
communication sessions at ALICE RAP partner meetings, self-report questionnaire data gathered at
partner meetings and via the internet, individual telephone interviews with Work Package and Area leaders, and members of the Media and Communications Group and the Global Science Group, and project documents. This is a ‘dynamic’ design including a series of sub-studies, conducted over the entire project period, with each new sub-study’s aim and design based on the accumulating knowledge. The entire Area 7 team participated in decisions about the aims and designs of the sub-studies, and all Work Package
Leaders had the opportunity to provide input.
The ALICE RAP official list of project participants is the population from which evaluation
participants were drawn. The list was modified over the course of the study as some participants
left the project and other joined. The current list was used at each stage of the evaluation. In
some instances the entire list was used and the sample was the population. In other instances the
subset of participants attending a partner meeting and present at the time of evaluation composed
the study sample. At yet other times a purposive sample was drawn, as was the case for the
collection of telephone interview data. The details of participation are given in the descriptions
of the Sub-studies, later in the Methods section.
Post-meeting evaluations At the conclusion of each of the annual partners’
meetings, a meeting evaluation questionnaire was distributed to all participants in attendance.
These data were analysed and fed back to ALICE RAP leadership to assist them in planning the next
meeting. As the meeting evaluations had mostly to do with the practical aspects of conducting
meetings, the results of these evaluations are not included in this report.
Computer-mediated Communication (CMC) participatory evaluation When actors from disparate
backgrounds and viewpoints gather to formulate and implement solutions, as in ALICE RAP, achieving
effective communication is a special challenge. Computer-mediated communication (CMC) is often
helpful in this regard, and it was used at some partner meetings to foster general discussions. A
CMC innovation addressed by this evaluation process was to use scientific methods to analyse,
interpret and feedback CMC data and results to ALICE RAP management.
Interviews Following the ALICE RAP meeting in Barcelona in May, 2011, telephone
interviews were undertaken with Work Package leaders and the Chairs of the Media and Communications
Group and the Global Science Group. The interview guide is in Appendix A.
Self-report questionnaires Self-report questionnaires were used in several
Sub-studies, distributed either during partner meetings, or via the internet.
Documents All official ALICE RAP documents were available for analysis, including
the original application and the evaluation that led to the funding of the
project. These were used, but were data sources of relatively minor importance.
Evaluation Sub-studies aims and main findings
Mission and Values Sub-study
This sub-study, undertaken early in the project’s development, aimed to initiate a wide- ranging
partners’ discussion on how the ALICE RAP partnership should function, ways to make it work
smoothly and efficiently, the project’s mission, and the degree to which there is a diversity of
core values among the ALICE RAP scientists. The issue of partnership functioning was a focus in all
the Sub-studies, while the issues of mission and values were the particular foci of this Sub-study.
The main result with regard to management practices, rather expectedly, was that the participants
wish for a leadership style that balances firmness and clear direction on the one hand, with
openness and a consultative style on the other hand. Perhaps less expected was the degree to which
participants focussed on challenges related to diversity in the ALICE RAP research team. A warning
was sounded about possible diversity-related dominance problems of two types: dominance by one or
several disciplines over the others, and dominance of native English language speakers over the
others. A finding of particular significance that emerged from the CMC has to do with values. It
might have been the case that as scientists, the participants would have focussed
on the eternal debate about value-laden versus value-free science. However, the widely held
position was that science is value-laden, that values should therefore be illuminated, and
that ALICE RAP should make an effort to develop shared values. Considerable attention was also paid
to societal values regarding the nature of addiction, and the need to reframe the problem of
addiction so as to reduce stigmatisation and marginalisation of people with addictions. However, it
is open to question which values actually predominate in ALICE RAP, since the CMC is not the right
methodology to study values, attitudes and predispositions. Further study of values, attitudes and
predispositions held by ALICE RAP scientists could be useful in understanding the unfolding of the
A telephone interview survey was used to gain insight about ALICE RAP’s leadership strategy, and
its coordination and communication methods. The results of this Sub-study were discussed by the
ALICE RAP leadership team, and it was agreed that ALICE RAP should develop Vision and Mission
statements. There ensued a round of consultations, leading to the adoption of these statements,
which are featured on the home page of the ALICE RAP web site:
Vision Statement -- Promote well-being through a synthesis of knowledge to redesign
European policy and practice to better address the challenges posed by substance use and addictive
Mission Statement --
Advance synergy among sciences that address substance use and addictive behaviours, through a
five-year programme of European trans-disciplinary research.
Science and Policy Dialogue Sub-study
The aim of this Sub-study was to address a key goal stated in the ALICE RAP Description of Work: to
include “clear and targeted dissemination and valorisation strategies, addressing and involving not
only scientific audiences, but also policy-makers, other key actors and stakeholders…” The starting
point for the CMC session on policy-science dialogue was the
assumption that ALICE RAP has to engage stakeholders, as that is an essential aspect of TDR.
The issue, then, is how to involve stakeholders. Scenarios were provided to illustrate possible
levels of involvement, ranging from simply engaging stakeholders in dialogue, to acting together
and taking responsibility together. The findings suggested a differentiated strategy for
stakeholder involvement, tailored to particular aspects of addiction, to country or community
levels of interest, and to the different stages of the project as it progresses. This could be
developed as an element in ALICE RAP’s Communication Strategy, in collaboration with Areas and Work
Packages. The essence of a Stakeholder Communication Strategy with this tone would be its emphasis
on ‘communication with’ stakeholders, rather than
‘communication to’ stakeholders.
Research Orientation Sub-study
This sub-study was undertaken to provide ALICE RAP with insight about its scientists’ orientations
to research collaboration after two years of the project, and again at the conclusion of the
project. Those in ALICE RAP with the greatest expressed proclivity for inter/transdisciplinary
research are older men in the biomedical sciences and younger women in the social sciences. Yet, as
Tables 6-8 show, respondents in this study generally expressed strong proclivity for
inter/transdisciplinary research, regardless of scientific background, sex and age. This quite
likely is due more to a selection effect than to an adaptation effect. The ALICE RAP leadership
intended to recruit scientists with the needed scientific expertise and with proclivity for
transdisciplinarity. While the experience in ALICE RAP between 2013 and 2015 may have affected
scientists’ attitudes towards transdisciplinary working both favourably and unfavourably, we
conclude that at near the mid-point of ALICE RAP (2013), the partners’ orientations to
collaboration was already consistent with the transdisciplinary Mission of the Project.
The 2013 and the 2015 rounds of data collection yielded data that should be compared with
substantial caution, since the sample frames (partners in attendance at the 2013 Barcelona and the
2015 Lisbon meetings) were different, even if drawn from approximately the same population
(participants listed on ALICE RAP lists). Notwithstanding this caution, we conclude that amongst
partners attending the two meetings and participating in the Research Orientation Scale data
collection, the mean Research Orientation Scale score was similar, at about 28 in 2013 and in 2015.
Experience in ALICE RAP seems to have neither decreased nor increased partners’ proclivity for
transdisciplinary research collaboration.
The aim of this Sub-study was to map collaborative contacts within and between ALICE RAP Areas and
Work Packages. ALICE RAP is conceived as a transdisciplinary research (TDR) project. Great effort
has been made to organise the project in ways that encourage the partners to cross administrative
boundaries (Areas, Work Packages). Substantial parts of partner meetings have been devoted to
discussions and consultations having the aim to stimulate the TDR atmosphere. The data suggest that
a great deal of inter-Area and inter- Work Package collaboration happened in the period before data
collection. Yet, the pattern was of substantial heterogeneity, with Area 3 seeming to be a beehive
of inter-Area collaboration, and Areas 2 and 5 beehives of intra-Area collaboration. In their
interpretations of the instructions for completing the survey, there is reason to believe that
respondents pondered personal instances of collaboration.
The Futures Sub-study
The Futures Sub-study’s aim was to use the Bergen Model of Collaborative Functioning, to document
partners’ perceptions of ALICE RAP collaborative processes and the degree to which they experienced
synergy in the project. Data were collected using the online ALICE RAP Futures Questionnaire in
2014, with repeated data collection using the same questionnaire in 2015. In the measurement of
ALICE RAP synergy, the data show consistently that a large majority of respondents perceived the
project to have synergistic effects. There were interesting sex differences. Males were more than
twice as likely as females to have experienced synergy frequently, but males were far more likely
to be critical of the scientific quality of ALICE RAP. There was more moderate evidence for synergy
in the other synergy construct – continuity -- as expressed by respondents’ intentions to
participate in ALICE RAP- related activities after the project is completed. In Futures I, the
continuity scale was close to normally distributed. In Futures II, the majority of respondents
indicted intentions to participate by endorsing and supporting activities, or playing a modest
role. For the more active types of future participation, women were more likely than were men to
indicate the possibility of extensive futures participation. This was the case for connecting to
networks with like interests, writing research applications, and writing papers. Men, on the other
hand, were more likely to intend active participation in more general activities like endorsement
Maintaining momentum Sub-study
This Sub-study focused on the outputs element of the BMCF (Figure 1), aiming to ascertain the
degree of synergy regenerated by ALICE RAP (indicated by partners’ intentions to keep aspects of
the project underway after the EU funding period). Two main themes arose, research and networks.
Five specific ideas were identified related to possible future social science research, seemingly
with an emphasis on descriptive aspects (not intervention research). The five topics shown are
listed in arbitrary order. The second research stream that received attention was related to
intervention for policy enhancement, for which three highly interrelated topics were identified –
again, these are listed in Figure 1 in arbitrary order. Together, eight social science research
topics received attention, while no bio-medical topics were suggested. The other emergent theme was
networks, loosely divided into ideas related to less and more formal networks. Five network ideas
received attention, with various purposes: the gathering of vested interests; advocacy; building transdisciplinary science capacity across addiction arenas and micro to macro levels of systems; research.
The findings suggest at least some degree of interest in continued ALICE RAP work. Interestingly,
no participant suggested just letting ALICE RAP fade away.||eng