inGOV evaluation strategy is organized into three iterations of piloting, aiming at evaluating the main outcomes of the project. The piloting activities are performed in four countries:

• Malta. Modernization and integration of the digital common family household public service, affecting 200,000 householders.
• Croatia. Creating an AI-driven mobile virtual assistant and a common public services platform for the citizens of the City of Bjelovar. The virtual assistant will provide universal access to all local services, i.e. 32,000 adult inhabitants.
• Greece. Reengineering and digitalization of the issuing and renewal procedure of the disabled citizens discount cards for public transportation in the Region of Thessaly, affecting more than 11,500 disabled low-income citizens.
• Austria. Digitalization and simplification of the tourism overnight stay tax collection in Lower Austria, affecting more than 3,200 accommodation providers.

The plan for the 2nd (interim) iteration of the pilots was based on feedback received from the 1st iteration as also from the comments of the interim project review. The plan was updated by:

• Including measurable KPIs for the evaluation of the co-creation process and tools.
• Considering the measurement of baseline KPIs related to impact and capacity building.
• Considering the evaluation of trust and satisfaction (using KPIs).
• “Strengthening” co-creation by using more co-creation activities at the pilots and also by making creative use of co-creation at the pilots’ workshops.

Evaluation activities at the 2nd piloting iteration

• At the second piloting iteration a total of 13 dedicated workshops took place in the four countries (Malta, Croatia, Greece and Austria) involving around 100 people in total from multiple pilot stakeholders.
• The participants of the workshops evaluated the co-creation process, the produced tools and the trust aspect.
• Two ISA2 tools (IQAT and IMAPS) were used to e valuate the status of the pilots.
• The pilots evaluated the IPS holistic framework – interim.
• Expert evaluations took place at two inGOV international workshops and one expert group.

Evaluation results

Workshop evaluations

The participants of the workshops filled out a Likert scale (1 … 5) questionnaire that led to the measurement of 24 KPIs. These KPIs were grouped into 4 categories: co-creation process, ICT tools, trust and information capacity. The average score of these categories, for each pilot, is presented in the following diagram. Based on the diagram areas were identified where each pilot is lacking (e.g. score is below 4) and should be ameliorated in the 3rd pilot round.

As a result of the co-creation activities, several improvements or new functionalities were identified, highlighting the effectiveness of the co-creation process. The following list includes some highlights per pilot:

• Malta: Identification of new uses of the produced ICT tool that were not previously considered, as well as ways to solve potential issues that may occur.
• Croatia: identification of potential improvements of the chatbot in terms of UI or addition of new content.
• Greece: identification of potential improvements of the application, especially for its accessibility to disabled people (e.g. deaf, visually impaired).
• Austria: simplification of the current process of the accommodation tax. Furthermore, it was found that by publishing the tax rates and classification of the municipality via OGD (Open Government Data), added value can be created for many stakeholders.

IPS Holistic Framework evaluation

The pilots evaluated the relevance, applicability and sufficiency of the recommendations of the IPS Holistic Framework. The evaluation results are summarized in the following image:

Evaluations via ISA2 toolkits (IMAPS and IQAT)

Comparing the scores received through the use of IQAT and IMAPS from the 1st and 2nd pilot round we observe that the scores have been improved. However, the scores could be further improved by applying and adopting some of the recommendations provided by the tools. Additionally, the scores are expected to improve since the produced pilot applications are becoming more mature.

Challenges and good practices

As a result of the pilot activities and evaluations a set of challenges and good practices related to co-creation were identified.

Challenges:

• Difficulty to motivate and engage the stakeholders that will participate in the co-creation activities.
• Difficulty to identify and assign roles to the co-creation activity participants.
• Technical issues for the organization of the workshops (e.g. no adequate number of PCs available).
• Difficulty of the participants to comprehend the scope of the co-creation activity (as a result they provided content irrelevant to the problem definition).
• Difficulty to motivate the participants to complete the questionnaire, resulting in a low number of respondents in some cases.
• Difficulty to find dates for the workshops due to the limited availability of the prospective participants.

Good practices:

• It is beneficial to use “Win-Win cases/scenarios“ for the co-creation activity organizers and the participants. This will facilitate the motivation and engagement of the participants.
• Working on existing cases/scenarios motivates the participants more, rather than identifying new cases/scenarios.
• An information session before the co-creation activity can help the participants to better understand the scope and the problem definition.
• It is important to consider/implement the feedback received by the participants.
• The use of open questions enables the collection of various useful feedback from the participants regarding the improvement of the tools, processes etc.
• The focus of the co-creation activity should be concrete (not too generic).
• The organizers of the co-creation activity should be open to criticism regarding the co-creation process as well as the objective of the activity (e.g. ICT tool).
• A meet-in-the-middle approach to the objective of the activity is beneficial. Both the activity organizers and participants provide and discuss their views leading to the outcome.
• It is beneficial to build channels (e.g. social media, mailing lists) with the participants to constantly collect feedback and communicate results.
• It is essential to keep contacting the stakeholders (by phone, by mail) to ensure their participation in the activities.

Regarding the way of organizing the co-creation activities, some of the activities were online while others were on-site. The online activities better fitted a small group of participants and also enabled the easy participation of people (e.g. people from other cities, no need to commute). The on-site activities better fitted larger groups of participants, the participants were better involved and more active, but in some cases the need to commute was a limitation.