Friday, May 20, 2016

The increasing demand for measuring

@SilvanavF

The increased use of the concepts of transparency and openness has fuelled the demand for measurements, rankings and assessments on FOI (Freedom of Information) legislation and OGD (Open Government Data) policies, especially as global comparative exercises. The past two decades have witnessed these FOI legislation and OGD policies become key developments in the transparency and openness areas. The number of FOI laws worldwide has exceeded 100. In less than a decade, OGD initiatives have also increased with the number of countries with OGD expanding significantly after the launch of the Open Government Partnership (OGP)(1).

Even though these developments (FOI and OGD) have taken place in a relatively short period of time, actors need to start analyzing what are the gaps between expectations and reality. It would be a sign of maturity to limit the grandiose statements and focus on what governments and society are achieving by publishing and reusing government data and information.

It is getting much clearer that measuring impact (despite it has become a buzzword these days) in the strict sense of the word, is extremely difficult, in particular when trying to link certain changes to a particular isolated intervention. 

Actors on both the demand and supply sides of government-held information and data are behind the requests for such measurements. On the demand side, advocacy groups and practitioners need to present impact-related figures to donors and governments, but it is difficult to do so when its impact is difficult to be empirically proved. On the supply side, the champions of increasing access to information within public administrations need to “sell” the benefits of these types of policies by showing the decision-making heads of agencies and/or governments, and the staff required to implement the policies, that their ‘pros’ surpass their ‘cons’.

International rankings, ratings, indices, barometers all refer to the idea of placing the elements of a given domain in a particular order. However, this order is not arbitrary. All the measurement mechanisms refer to the placement of a certain list of objects according to a set of criteria or variables, which in turn depend on the values of those creating the measurement tool. Even though measurements and assessments in both fields relate to the availability of different types of information/data, the main features of each field differentiates them. Thus, the criteria and variables selected in the FOI field differ from the ones applied in the OGD context.

Unsurprisingly, despite the many differences between the FOI and OGD fields, in both cases the measurements and assessments relate to the availability of different types of information and data. However, there are also other elements in common that could help organize (and possibly correlate) some of the most well-known assessment exercises, as well as explain the main differences between the two fields. Some of the main elements to differentiate both fields are the object (information, data, proactively and reactively disclosed, for example to measure transparency and/or broader goals), the geographic scope of the exercise (one-country, regional and global assessments), and the type of assessment (legal, performance/implementation, among others). 

Different assessments cover different jurisdictions as well as they examine different aspects of the disclosure of government information. In doing so, they use a variety of criteria and methodologies. Regardless of the particularities, comparative measurements and assessments are based on the idea that those holding the first position in the ranking are more valuable/useful/have greater impact than the lower ranked objects. However, what it actually means is that those policies or laws that are ranked highly are just closer to the preferred — or ideal — model set by the author of the scale. As the criteria and variables selected for measurement in the FOI and OGD domains differ, so the ideal conditions for each domain exhibit differences as well.

As rankings/ratings, and other exercises, assess particular aspects and characteristics in each initiative (FOI and/or OGD) they do not necessarily place a country/district in the same position, as they tend to examine different aspects of the disclosure of government information. Thus, comparing the results of such assessments should therefore be done with an eye on these differences, in order not to draw any quick conclusions. 

Examples of that partial and misleading picture could be found if we were to draw conclusions by simply extrapolating from the RTI rating results for India and Liberia. On paper, both of these countries have enacted strong FOI legislation, which are placed among the best laws in the field. In the case of India, despite its high score of 128/150 in the RTI rating, the country is placed 85/175 in the Transparency International Corruption Perceptions Index and the Open Data Barometer gives the country a 39/100. Liberia presents a similar scenario. The RTI rating gives the country’s FOI law a high score (124/150) but the World Justice Open Government Index 2015 (2) scores the country with a low figure (0.35/1), as does as the Transparency International Corruption Perceptions Index (94/175). 

Thus, to obtain a more accurate picture of a given jurisdiction it is necessary to gather information from a myriad of assessment’s exercises on different aspects of the disclosure of government information and data. This combination of information (from different assessments made by different actors on a variety of aspects regarding government’s disclosure of information and data) also could help to identify the cases of resistance to effective transparency by understanding administrative bottlenecks and the type of information/data that governments are withholding, among other elements. In a similar vain, because the letter of the legislation sometimes says little about the actual implementation of it in practice and thus the reality of access to government-held information and data in a given country, it is necessary to assess the production of statistics on compliance with FOI and OGD requirements. This information is mostly provided by governmental assessments (agencies in charge of implementation and/or oversight agencies) of their own policies and legislation. These systematic exercises allow advocates and academics to learn about the implementation status of the legislation and policies.

To sum up, there is a need for comprehensive exercises in terms of government-held and produced information and data. So far, we have partial snapshots and not the whole picture. Adding to this lack of comprehensive exercises, there is also a need for further research to develop a framework to understand not only the availability but also the use of information and data in a global comparative exercise. In order to take the first steps to achieve these improvements in terms of measurements and evaluation, a closer collaboration between the actors behind the demand and supply of data and information is needed. 

Just to close this posts that it is getting too extensive, it is necessary to highlight that there is a need for comprehensive exercises of the availability and use of government-held and produced information and data but, in order to achieve that, it is also necessary that the actors working in the field, in particular civil society actors making use of that information and data, start evaluating and planning their actions more carefully. If the actors can provide a clear idea of the results (I’m consciously avoiding the use of the word impact) it would help in the design and implementation of a global aggregated measurement exercise. To have a clear picture of the whole ecosystem would benefit all actors involved.

Notes: 

(1) Many of the ideas included in this post were already explored in the article “Understanding two mechanisms for accessing government information and data around the world”: http://webfoundation.org/wp-content/uploads/2015/08/UnderstandingTwoMechanismsforAccessingGovernmentInformationandData 

I’m grateful for the useful comments provided by Marcos Mendiburu. However, I’m solely responsible for the content of this post. 

(2) It is worth noting that several FOI specialists have questioned the World Justice Open Government Index. For example: http://www.freedominfo.org/2015/04/design-of-wjp-open-gov-index-skews-results/ and http://www.freedominfo.org/2015/04/wjp-open-government-index-raises-interesting-questions/

References:

No comments:

Post a Comment