Notes on Civic Technology and Open Development Policy
26/04/12 Digital Government , Research #

Why we need another composite index (on public e-Services)

The debate on composite indicators or synthetic indices in the e-government field has been ongoing since the publication of the first benchmarking exercises at the EU level back in 2002. Many analysts and researchers consider composite indicators as “black boxes” (see for example this paper by Frank Bannister, 2007). We put in still intelligible indicators and what comes out is a mysterious number, and, inevitably, a mysterious rank. The feeling is that it’s a weird combination of voodoo (or too complicated math), subjectivity, weak frameworks, unbelievable results (can you really believe that Italy has put 100% of public services on line with the highest possible level of interactivity?).


A 3-days seminar at the JRC-IPSC of the European Commission opened my mind. There I found a motivated and high-skilled team coordinated by Andrea Saltelli, which, by the way, was responsible for drafting the OECD-EC Handbook on Constructing Composite Indicators.

While it was clear to me that things like data quality, framework reliability and transparency – when it comes to show how the results have been computed – are always crucial, I learned that composite indicators quality and robustness can and must be checked, and that more advanced and reliable techniques can be applied. I suspect that if we applied tools such as the Sensitivity Analysis or the Uncertainty Analysis to the existing “black box” indicators we would get an idea of how ranks can vary and of therefore at what extent resulting policy indications can be week.

I’ve been working for quite some time on a composite indicator on eServices (eGovernment, eEducation, eTransportation, to be extended to eHealth and Smart Cities) for research project TAIPS funded by the European Investment Bank, together with my friends and colleagues Marco Biagetti, Davide Arduini and Professor Antonello Zanfei. I presented some preliminary results at the 1st EIBURS-TAIPS Conference at Urbino University (here you can find all papers and slides from the conference), in front of a bunch of innovation policy gurus including Paul David, Ian Miles, Edward Steinmueller and Keith Smith.
Here is the abstract and my slides.

Abstract The study aims at providing evidence on regional differences in the diffusion of ICT in the public sector in Italy, with a focus on different types of public e-services (eGovernment, eEducation and Intelligent Transport Systems). Data are obtained by merging four different surveys carried out by Between Co. (2010-11) and Istat - Italy’s National Bureau of Statistics (2009). We pursue a three-fold objective. First, we attempt to overcome the prevailing attitude to consider the various domains of public e-service provision as separate from one another. In other words, measuring the progress of digital government requires a holistic view to capture the wide spectrum of public e-services in different domains (e.g. local and national administrative procedures, transportation, education, etc.) and the different aspects of service provision (not just e-readiness or web interactivity, but also multi-channel availability and take-up). Second, we shall tackle a major drawback of existing statistics and benchmarking studies of public e-services, which are largely based on the count of services provided online, by including more sophisticated indicators both on quality of services offered and back office changes. Third, we develop a sound, open and transparent methodology for constructing a public eServices composite indicator based on OECD/EC-JRC Handbook. This methodology, which incorporates experts opinion into a Data Envelopment Analysis, will allow us to combine data on different e-service categories and on different aspects of their development, and will enable us to define a ranking of Italian regions in terms of ICT adoption and public e-service development.
0 likes no responses
31/10/11 Open Policy

The map of EU Structural Funds Transparency at regional level – October 2011

According to the current 2007-13 regulation, all regional and national agencies responsible for managing one of the 434 Operational Programmes funded by the 2007-13 Structural Funds must publish on the web a list of businesses or public authorities that have received public funding and the amount of funding received. But the way they do this varies greatly across Europe.

The second output of the evaluation activity of the availability and quality of open data on European Structural Funds is now being published. It’s a benchmarking report (in Italian only, at least for now) that I prepared with the help of my colleague Chiara Ricci for the DG Regional Policy of the Italian Ministry of Economic Development.
I had the chance to present it at the Annual Meeting between the European Commission and the Italian Managing Authorities of the European Regional Development Fund (ERDF), held on October 27-28, 2011 in Rome. It was an extraordinary opportunity to talk about the benefits of open government data in front of a number of high-level representatives of all regional and central institutions involved in the implementation of Regional Policy in Italy (here is my presentation).

The report features brand new data on detail, accessibility, formats, and other characteristics of the datasets on the recipients and the projects funded by European Regional Policy (“lists of beneficiaries”). It’s a new wave of data collected in October 2011, exactly one year after the first web-based survey. A total of 32 characteristics are taken into account in the evaluation process, including the presence of search masks and visualization systems.

The map of Structural Funds transparency reported below shows a core component of this research, that is the format of the data published by each region. The map shows the average score of all the regional and multi-regional programmes that have an impact of that specific territory. A very low score is attributed to PDFs and to HTML reports that split the data into multiple tables or pages (regions in red or orange) . Higher scores are assigned to the XLS format, which is machine-processable (in yellow). The highest scores are attributed to the few regions in Europe that publish data in an open format such as CSV (in green), since no data is currently published in XML or JSON or RDF (see the report for the details about the construction of the index). You can also find the link to the datasets by clicking on each region. The link to the Regional Programme is displayed where there is more than one dataset available.

One year after the first survey, the level of openness has not improved. About two-thirds of EU Operational Programmes still publish their data in PDF, while only 2% use open formats. A radical change is necessary to meet the requirements of new 2014-2020 regulation, as proposed a few weeks ago by the European Commission, which include the use of CSV or XML format.

0 likes no responses
10/10/11 Innovation Policy

Ex-ante conditionalities for Regional Innovation Policies

As I reported last week, the European Commission has presented the proposals for the new 2014-2020 EU Regional Policy regulations, and the EU regions are currently discussing the future of the policy at the Open Days 2011. Although these drafts need to be definitively adopted by the end of 2012 by the Council and the European Parliament, this step is going to represent a milestone in a long process. On the one hand, it is the final product of a two-years-long discussion started in the high-level group reflecting on future Cohesion Policy, a support working group composed by representatives of the Commission and the Member States. On the other hand, it represents the beginning of the 2014-2020 programming phase.

A new principle is introduced. Regional Policy will not finance a Member State Programme until a set of ex-ante conditionalities, that is specific conditions and pre-requisites at the national level, are fulfilled. Ex-ante conditionalities were first envisioned by the European Commission in the framework of the debate on the reinforcement of economic governance [pdf]. Then they were reaffirmed in the Fifth Cohesion Report and further developed in a high level group document [pdf] proposed by the Commission in December 2010.

A list of the conditionalities proposed can be found in the general regulation. As for research and innovation (R&I), the conditionality is as follows: “the existence of a national and/or regional innovation strategy for smart specialisation in line with the National Reform Program, to leverage private R&I expenditure, which complies with the features of well-performing national or regional research and innovation systems”.

The focus of the conditionality is in fact only on the strategic level, that is on the existence of policy documents that, based on tools such as the technological forecasting, select a limited number of policy priorities and industry sectors to support. In particular, the proposal of the EU Commission states that Member States must:

  • define an innovation strategy for smart specialisation that is based on a SWOT analysis to concentrate resources on a limited set of R&I priorities, outlines measures to stimulate private RTD investment and contains a monitoring and review system.
  • adopt a framework outlining available budgetary resources for R&D;
  • adopt a multi-annual plan for budgeting and prioritization of investments linked to EU priorities (ESFRI).

The need for a strategic view of regional policies in general and of innovation policies in particular (see for example the much-needed smart specialization approach) is evident. But is such a strategy-focused approach enough to ensure a better policy and a better use of public money?

First, given the essential nature of R&I policy, the answer should be “no”. As stated in one the underlying papers of the Barca Report on the future of Cohesion Policy, “Putting money into a strategic R&D plan may not always lead to a new technology, or even less to the adoption of that technology”. In other words, identifying regional strengths and comparative advantages is correct, but will may not lead to the desired results.

Secondly, a strategic document that is compliant with the Commission’s requests is easy to be obtained for a regional government. For example, the regional strategy could be written by a consultancy and adopted without an active involvement of regional and local agencies, and, even worse, of stakeholders. I don’t mean that this is the usual way to define a regional strategy, but it happens.
Thirdly, a strategy defines policy objectives, priorities, targets, indicators, but does not describe exhaustively how these objectives are going to be reached. And we know that devil is in the details.
So this is why other kinds of ex-ante conditionalities should be taken into account, specifically those related to policy implementation. For example, one conditionality could focus on the specific content and characteristics of the criteria for projects selection included in the Operational Programmes. Since Operational Programmes are generally set up after the definition of the “contract” between the Member State and the European Union, this solution may be more difficult to implement – because Member States should first approve the Operational Programmes and the selection criteria and than wait for the EU money – but will ensure a tremendous improvement in the quality of selected interventions.

For example, a conditionality could be applied to the selection criteria by making sure they incorporate the results of ex-post evaluation of past interventions and learning from policy failures. The conditionality could also be leveraged to systematically introduce a multi-stage approach in funding projects. In this case, a project-level conditionality is applied both on final and intermediate results through in itinere and ex-post evaluations.

0 likes no responses
1 3 4 5 6 7 8 9 13
Recent Comments
- Patrick to From closed data to open data ecosystems – stages of an evolution
Thanks for this interesting article. I agree with this evolution as you've laid it out – especiall...
- Putting research into practice: Training academics to use Open Data as OER: An experience from Uruguay | Thoughts on Open Education to OpenCoesione School – A scalable learning format using OpenData as Educational Resources
[…] developed by A Scuola di Open Coesione, and in the work t Chiara Ciociola and Luigi Reggi...
- Così fallisce l’Open Government: quando lo Stato fa auto-gol | to A (long) list of the risks of Open Government
[…] un post di un paio di settimane fa mi sono cimentato in una prima lista dei possibili ris...