inGOV evaluation strategy is organized in three iterations of piloting aiming at evaluating the main outcomes of the project. The piloting activities are performed in four countries

• Malta. Modernisation and integration of the digital common family household public service, affecting 200,000 householders.
• Croatia. Creating an AI-driven mobile virtual assistant and a common public services platform for the citizens of the City of Bjelovar. The virtual assistant will provide universal access to all local services, i.e. 32,000 adult inhabitants.
• Greece. Reengineering and digitalization of the issuing and renewal procedure of the disabled citizens discount cards for public transportation in the Region of Thessaly, affecting more than 11,500 disabled low-income citizens.
• Austria. Digitalisation and simplification of the tourism overnight stay tax collection in Lower Austria, affecting more than 3,200 accommodation providers.

Each piloting iteration follows a cycle of Plan – Implement – Analyse actions:

• Plan. Prepares the ground for the pilot implementation, e.g., describing aspects of the pilots, planning the co-creation piloting activities, planning for the pilot’s evaluation, securing ethical compliance, etc.
• Implement: Refers to the actual conduct and evaluation of piloting activities, i.e., preparing the pilots’ implementation, rolling out the piloting activities, gathering feedback for evaluation, etc.
• Analyse. Refers to documenting and exploring the results of the pilot implementation, e.g., reporting on the co-creation piloting activities, analysing the gathered feedback, assessing the success of the piloting activities, extracting lessons learnt, etc.

What is evaluated?

The evaluation dimensions that we will pursue in the inGOV project are:

• Evaluation of the enhanced CPSV-AP that models complex public services and life-events enhancing their interoperability.
• Evaluation of the IPS-Co (Integrated Public Service Co-creation) models, that provide a conceptual basis for the implementation of Government as a Platform for IPS.
• Evaluation of IPS holistic framework that includes guidelines and recommendations on IPS Governance and implementation.
• Evaluation via ISA2 methods and tools.
• Evaluation of the co-creation process followed at the pilots.
• Evaluation of inGOV platform & tools that serve and provide the IPS at the pilots.
• Evaluation of the pilots’ impact.

Who is involved at the evaluation?

The evaluation involved three different groups of evaluators:

• Experts, i.e., domain experts, academics, practitioners, etc., external to the inGOV consortium;
• Pilot partners, i.e., the four public administrations participating in the inGOV consortium; and
• Pilot stakeholders, i.e., public employees, citizens, businesses, etc., who will be approached by the pilot partners.

How is the evaluation performed?

inGOV adopts a different methodology for each of the evaluation dimension:

• Evaluation of the enhanced CPSV-AP model and the IPS-Co models is performed fully via experts’ evaluations via publications to journals and proceedings, presentations to conferences, webinars, workshops, focus groups or other similar activities that may provide feedback to the presented results.
• The evaluation of the IPS holistic framework is performed by the pilot partners as the owners and implementers of each pilot. A questionnaire was used to evaluate the recommendations proposed by the framework based on four criteria, namely relevance, effectiveness, coherence and utility.7
• Evaluation via ISA2 methods and tools is performed by the pilot partners as the owners and implementers of each pilot. The tools that were used are: i) the Interoperability Quick Assessment Toolkit (IQAT) that allows solution owners to quickly assess the potential interoperability of their software solutions supporting public services and ii) the Interoperability Maturity Assessment of a Public Service (IMAPS) that is a tool for evaluating the interoperability maturity of digital public services
• The evaluation of co-creation activities and the platform & tools is performed via the pilot implementation activities that are performed during each pilot iteration and the evaluators are the stakeholders of each pilot, i.e. citizens, businesses, public employees, etc. For this reason, dedicated physical or online workshops are organized.
• Pilots’ impact will be mostly evaluated during the third pilot implementation, although some impact-related aspects, e.g. baseline values of KPIs, will be examined beforehand.

At the first iteration of the piloting activities the focus was on the co-creation activities. The second iteration will focus on the platform and tools while the third iteration will focus on the impact.

Evaluation activities and results at the 1st piloting iteration

Summary of evaluation activities:

• At the first piloting iteration a total of 15 dedicated workshops tool place at the four countries (Malta, Croatia, Greece and Austria) involving multiple pilot stakeholders.
• Expert evaluations took place at three international scientific conferences and one journal.
• Two ISA2 tools were used namely IQAT and IMAPS.

Summary of results:

• The pilots found the recommendations of the IPS holistic framework useful and coherent.
• The majority of the pilot stakeholders that participated at the workshops felt that their opinion/feedback was taken into consideration and they are overly satisfied with the overall co-creation process.
• The scores received by the ISA2 tools are moderate. These are expected results since the pilots are at their early stages and the scores are expected to be improved.