crisscrossed Blog

Data dive: Measuring poverty through real-time data

2013-03-21T16:20:23+01:00

Crossposted from the Voices from Eurasia Blog by UNDP

Looking at the many dimensions of poverty, one can realise what a highly complex issue it is. The monetary perspective is just a start-off, there are also many other relevant areas such as health, acccess, location, environment, mobility, etc. Amartya Sen has done a groundbreaking work and much progress in recent decades to measure poverty. But one big problem persists: Poverty analysis is backwardly focused. To put it in blunt words, we can measure how it happened, but it is very difficult to measure it as it happens. Statistics are often months or even years old. When people apply for social welfare the personal crisis has already occurred. But how can the symptoms be measured before they even happen?

Such a new approach is to use real-time data (or big data) to measure poverty as it is happening. As we enter an era of massive data collection, thanks to the Internet and mobile devices, how could such data be also used to understand many dimensions of poverty? One word of caution : Better analysis can lead to better action to fight poverty, but it does not take people out of poverty by itself.

Data dive event Vienna

This new promise is a starting point for a series of data dive events organized by UNDP and the World Bank in the next months. One event already took place on February 22nd organized together with the Open Knowledge Foundation Austria, where I had the pleasure to facilitate the event. We were happy to have 26 participants from various backgrounds such as data analysis, spatial analysis, visualization, crowdsourcing, business intelligence, data protection and programming. For the event various [data sources] (http://wiki.opendataday.org/Vienna2013_Datasets_for_DataDive) were provided thanks to the generous contribution of TextToChange and the Qatar Foundation.

The overall good turn-out of participants, for such a niche topic, shows the high interest in big data analysis and proves the potential to reach out to external resource persons. It is promising to see data specialists investing their free time to participate on such events. Since big data analysis is in its infancy within development organizations, this is a chance to combine inside expertise with outside data specialists. Personally, I think it is particularly helpful to invest in a community for that topic, instead of relying on business intelligence solutions.

My first observation during the discussion was a bit of a clash of mindset between classical data collection for statistics and new opportunities with big data and data mining approaches. How can these two approaches can be used together? How can one support the other for better poverty analysis? But that did not stop participants from brainstorming interesting ideas to measure vulnerabilities.

Various ideas

One idea was to look at data from retail chains to track consumption patterns to analyze the risk of poverty on a regular basis. For example, having exact figures from recent weeks how certain commodities have been purchased. How staple food is maybe replaced by more luxury and vice versa. Bonus-points-collecting services have fine granular data for millions of consumers, where larger and smaller shifts in consumption patterns can be analyzed and localized in real-time. Maybe this data could be used once for a good cause, instead of tweaking the last bit from it to identify consumer interests for marketing purposes. But to what extent people affected by poverty, particularly in developing countries, are tracked with their consumption behaviors?

Another approach is data from insurance companies, which can be used as an early indicator for vulnerabilities. Particularly insurance companies have long tradition for risk assessment, and through micro-insurances their data reaches far into rural areas also in developing countries. A participant proposed measuring the distance for daily commuting and how this might change due to economical constraints. Or energy providers could provide data for payments as an early indicator when electricity or heating bills cannot be paid. If one gets access to companies' data, the flow of remittances could be analyzed in real-time to see larger and smaller trends of money transfer and what it might indicate about poverty.

Closed data

All these ideas lead to the challenge of closed or private data. Open data is one driver to hope for more access, but maybe more could be done under the umbrella of corporate social responsibility or through the concept of data philanthropy. But is there any non-social media company offering a real-time API yet? There is need for precise data requirements, solutions for data portability and data protection. Especially mobile phone data with movement logs of its users contains highly sensitive data and cannot, as different studies show, be fully anonymized. This is the great side-effect about such personal data. How can it really be protected?

Many participants shared the hope that corporate data philanthropy would become the norm soon and perhaps a requirement to bid successfully on contracts. There was also a discussion on the fact that development organizations needed to make a better case for why corporations should share data with them. The open data movement has focused so far on governments, so maybe a shift to companies could help here.

Crowdsource data

Another stream of discussion went around data collection or producing data in need, since most data is not at hand and is more or less closed data. Could crowdsourcing approaches help here to get accurate data about poverty? There was an interesting discussion on personal data philanthropy and the growing willingness of individuals to 'donate' personal data for the public good) (e.g. quantifiedself.com). Efforts for crowdsourcing in recent years, show that user reported data can be an alternative to traditional surveys. Can these approaches be scaled to the level needed to get accurate results? The development of low-cost sensors is widening the landscape of data tracking. Looking at the growth of smart phone usage in developing countries, this is a potential venue for data collection in the near future. There have already been experiments to use sensors to control water access or to analyze the water quality across a country.

This out-of-the-box thinking shows new potentials for using data to help understand complex problems better. But it also became clear how little data is available in the public space; and to fight poverty, such data analysis can be only a tool to then act on. And the second step is far bigger and even more complex.

Transparency Portal KfW Development Bank

2013-03-07T16:20:23+01:00

Organisations working in development cooperation generally have nice websites with descriptions of projects and plenty of texts on big issues. But when it comes to talking about money, most websites put the topic on the back burner. KfW Entwicklungsbank (KfW Development Bank) wanted to change this, and through my engagement at the Open Knowledge Foundation Germany (OKF), I helped to implemente their new transparency portal (Disclaimer: I am in the board of OKF GER).

We had already been working with KfW on their funding report for Germany, and as a result we were approached by KfW-Entwicklungsbank, who wanted to know if we could present their aid data in a new way. At that time we were already working on offene-Entwicklungshilfe.de and were pleased to be able to extend that approach to more data sets.

Transpareny portal

Recently the transparency portal (transparenz.kfw-entwicklungsbank.de) went live online: There we present data from KfW-Entwicklungsbank from a number of angles. Alongside the typical financial data, the results of project evaluations are also shown in a school grade format. This year we should also be adding an exhaustive data bank, and we hope that this will be just the beginning of a fuller disclosure of data.

The data which have now been made available can also be downloaded in machine-readable form. Fortunately we were also able to persuade KfW to use OpenStreetMap geodata and maps. We have also integrated World Bank indicators for each country. It is precisely this combination of data sets in real time that will really set the ball rolling.

A start has been made

But it’s quite clear that the data sets are only a start. A further step is publication of data on open contracting: a conference took place on this recently between the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) and the World Bank. It is precisely in contracting that greater transparency is called for, as there are frequent cases of corruption in this area. Data on this subject have recently been published by the World Bank, and they paved the way for some interesting analyses.

Interview

Interview extract

KfW: The criticism has been raised that transparency only benefits non-governmental organisations and IT specialists. So how do poor countries gain from more transparency?

Kreutz: It is precisely the administrations in developing countries that derive enormous benefit from it. Officials on the ground are faced with such a flood of project proposals that they quickly lose any overview. If we in donor countries can save them work by presenting them with data sets prepared to international standards, it becomes much easier for them to build these into their analysis tools and to control and distribute cash flows. It will pay for itself in hard cash!

KfW: Have developing countries even got the necessary infrastructure?

Kreutz: The greatest innovations in the internet and telephone fields are taking place right there in emerging nations! East Africa was the first region in the world where it was possible to transfer money by mobile phone. Of course we need to enhance analysis capacities and make people over here and in the emerging nations aware of the value added. We’re only just beginning. Data are still highly aggregated. It gets really exciting once you can see how much the staff expenditure is, what the material costs are. Of course that’s also very political…

KfW: What do donor countries get from this new transparency?

Kreutz: The position at the moment is that there is no overview. Right now there is nowhere I can find out how many health projects there are in Tanzania. Transparency is knowing who is doing what, where and when. A uniform, worldwide organisational standard will enable donor institutions to refer far more quickly and easily to crucial data.

KfW: …you are talking about the International Aid Transparency Initiative (IATI) standard, which the German government has also signed up to….

Kreutz: Exactly. If it just takes one click of the mouse to compare data from the World Bank, USAID and other sources, whole new horizons open up. It also improves communication between the organisations themselves. And if we combine the project data with tenders and evaluations, things really start to hot up! And one day we might even be able to get an insight into organisations’ planning, without possibly having to acquire such knowledge through formal or informal channels. This would save expense and avoid corruption and insider dealing. We’re only just beginning and we can see huge potential.

Open aid: Mapping over 100.000 German aid projects from the last 10 years

2013-02-12T16:20:23+01:00

There are practically only two options of German developing aid figures one can obtain; either highly aggregated data from the German Ministry of Economic Development and Cooperation, or complicated and academic designed pages such as aidflows from the OECD.

Offene-Entwicklungshilfe.de (Open Germain Aid) is in this regard different. Here, instead of first learning how to use the app, you can wander easily through figures about beneficiary countries from German aid and other individual projects.

Developing aid is, from a financial point of view, a large sector. The OECD countries alone spent 129 billion dollars in 2011 on ODA contributions. Large sums are distributed to thousands of organizations and for many projects. This tax money transactions are not always transparent. There is no overview of who is doing what, when and where in any single developing country. That this leads to inefficient spending is not a secret.

The ministry does not publish its own data

I used the open accessible data set from the Creditor Reporting System (OECD) for the implementation of the website. By the way, the complete raw data set is a bit hidden; click on the above link in the menu on export and then on "related files". Ironically, the ministry only delivers the data about a year later, but does not publish it on its own website. But that should change now; this spring, after a long delay, the ministry announced the publication of first data by the IATI standard. Better information in shorter cycles should be delivered this year. While analyzing data for this project, I realized that the historic perspective of data is particularly interesting.

Over 100.000 activities in the last 10 years

That is why I have included the Federal German aid activities for the last ten years in the website. So, there is an own description for every beneficiary country of German development aid, which describes and shows in detail every activity per development sectors like in Burkina Faso. The data from the projects also appears to be of insufficient quality; the titles of the projects are too general, in another language or partially incomplete. In addition, there is almost no information on the recipient organizations. The donor institutions are displayed with acronyms such as LG or Found, and are like in the case of the GIZ money, not completely traceable.

Analyse Trends

It is important for the website to obtain transparent overviews from different perspectives, so one can easily analyze, for example, in which sectors was money invested in the last years and where lie the sectors for the German development cooperation. Are the aid funds allocated for the adequate demands of the developing countries, or are they conducted from foreign or economic political interests? Are the expenditures sustainable, or are they done as a sudden single investment? The data not always provide a direct answer, but should at least help to identify the trends and introduce questions.

Combination of records

It will be exciting when other records are added. Thanks to the Open Data Initiative of the World Bank, who for example, include development indicators. These indicators show at a glance where do most important problems lie in particular countries. One can compare, if German expenditures are matching these needs. There is no question that the comparison is a bit weak and more records are necessary to be able to better retrace the expenditures for the sectors. And exactly in this direction is where I think it is import to go.

Outlook

Therefore, a commitment to open data must come from German organizations for development cooperation. Unfortunately, at the moment not a single non-governmental organisation is a model to follow. With some exceptions, there is almost no open finance data. At least a freedom information request has revealed an overview of all ministry funded organizations in 2011. I hope that offene-entwicklungshilfe.de becomes an inspiration for such demands.

We would have great apps, if most Open data were not so boring

2013-02-12T16:20:23+01:00

If I only had some good data, I would do some great apps...

It’s like that basic rule in nutrition: Food that is not eaten has no nutritional value. Data which is not understood has no value. Hans Rosling

When it comes to open data nowadays, one first tends to look at a range of data sets and then think about what could be done with them, though it should be the other way around. There is a real demand for data services, and it is 'only' about combining the right data sources together. Sadly, this demanded, good and valuable information is rarely available as open data. It could be that my criticism is due to the lack of my creativity to squeeze the best out of even lame data.

Open data portals

There has been an incredible effort done to make information available through open data portals for example, by cities or national governments and I applaud the effort. But here again I often stumble over the same problem. The data sets offered are so boring and look often like Alibi cases. I can so imagine a meeting of beaurocrats; "colleagues, as the last point of the agenda we need to talk about that open data portal." Sighes in the room. "Could you come up with some data sets?" Pause. "Well we give the latest count of animals in our zoo. We can offer the opening hours of our 10 museums. And we will publish the voting results from 1990 to 2006."

What is clear is that such data will not motivate a single developer. It is 100% boring and far away from being valuable and interesting. Check out the new European data portal from the European Commission, a great attempt no doubt and only a few months on the go. These are some example names of data sets:

The above image shows one such data set, which is probably only readable to an insider. In my research these are often the case on other portals as well. One can rarely find appealing and valuable data.

The case of geo data

Or take the case of geo data. It is the basis of many apps, but finding for example, administrative borders for cities, districts or post codes is almost impossible for many countries. But this is exactly what needs to become a very detailed data service to best serve a user. A lot has been done to offer more geo data openly, but unfortunately governments rarely offer such essential, mostly tax-payed information, for re-use.

In Germany, for example, the statistic office offers a ridicolous small amount of "geocoded" data. In 2011 they offered data sets of marriages, child care, unemployment rate, car accidents, election results. The problem not only is that it is very little, but also lacks "ground data" such as a list of kindergardens or playgrounds across Germany.

I imagine we will enter a complete different phase when interesting and valuable open data is offered. That is also the incentive we need to get many more developers involved. Crossed fingers governments will have no choice but to publish through external pressure. Otherwise it is just not going to happen.

Budget transparency: A few clicks to accountability?

2013-02-09T16:20:23+01:00

Frankfurt budget vizualization

"[..] if you don’t adapt your way of presenting to the way that people understand it, then you won’t get it through. You must prepare the food in a way that makes people want to eat it. The dream that you will train the entire population to about one semester of statistics in university: that’s wrong." Hans Rosling about the importance of good data presentation.

Budget transparency websites such as openspending.org are lately criticized that opening and vizualizing data is not enough. Journalists are required to find the stories behind such data as David Sasaki argues in his interesting post.

First, raw data is not enough — we need visualizations to understand it. Second, visualizations are not enough — we need journalists to investigate and tell stories about the data.

I agree it is essential that secrets or wrong spendings need to be disclosed by journalists, but accountability can work also in different ways. One of the biggest challenge at the moment is the lack of simple information services. For example take the OECD website aidflows.org. The website has a loading time of almost a minute and then offers data for scientist but not for a person, who wants quick access to data.

Good transparency means to me that I need maximum 2-3 clicks or 30 seconds to find a specific budget item. It means a journalist can get the exact numbers during a press conference. It sounds trivial, but the majority of data website lack these features and are often too complicated.

A while ago I received the budget for the municipality of Frankfurt, Germany. Nice raw, clean data with all revenues and expenditures for the last four years. Inspired by Openspendings.org I forked the software to vizualize the budget of Frankfurt. You can now easily browse from large to small budget items and analyze financial flows. Unfortunately some categorizations for budgets are still confusing, but instead of looking through 2874 PDF pages, you can now access the budget within a few clicks. That does not disclose any wrong spendings of the administration, but it makes financial data easily understandable and accessible.

„The problem with fiscal transparency portals is that there is no mechanism to oblige government agencies to defend the purchases they make, much less sanction them when they mis-spend public funds.“ David Sasaki

David Sasaki gives a great example from Mexico, how such disclosed data can also have no implications. However, in my opinion such portals are already a great step towards transparency, because they reduce search costs. We are just at the beginning of fiscal transparency. Soon we will hopefully have easy mobile access and location-based spendings.

At the moment we only scratch at the surface, present mostly aggregated data and experiment still a lot to find best ways to present data. Hopefully soon we have more fine granular data and better ways to make data even better accessible for quick analysis in different contexts.

A world of big (closed) data and its unused potential

2012-05-29T02:41:53+02:00

“Every two days now we create as much information as we did from the dawn of civilization up until 2003” said Eric Schmidt from Google in 2010. Behind most of this information there is Big Data. And although we Internet users contribute mostly to that vast data, we get little or nothing back. But what is big data? Big data is so far a closed shop with a wealth of information that urgently needs to change.

In older days most data, such as statistics, could be easily analyzed through a simple spreadsheet (e.g. Excel) or a database. Such data had hundreds or thousands of rows of information. It often needed to be collected in a survey form by a person walking door to door. But nowadays, data is collected 24/7 as we all browse and interact through the Internet. Let’s take for example log files: When you browse a website, all your clicks are saved in a log file with your ip address, your browser and a lot more. Now, imagine a website with millions of visitors, who are all protocolled. With mobile phones we now easily leave digital footprints wherever we are; just look at Twitter, which has about 6 Terabytes of tweets every day. These millions of rows are Big Data.

But here lies a problem. Big data is a little bit like open data. Many talk about it and only a tiny fraction is offered publicly even worse, half of government data is not usable. Governments sit on huge chunks of data, but like the corporate sector, share almost no data. If such data were open it would be possible to develop greater information services or held governments accountable easier. But shall all big data be opened?

Some argue that collection of data is a privacy nightmare. For example, in the case of the smart meter, your company knows by your electric consumption when you are on holidays and when you normally go to bed at night. Initially, this was meant to achieve a more responsible energy consumption. With a smart meter, electric consumption data is sent every few minutes, whereas in the old days it was read only once a year. No doubt big data can contain a lot of sensitive information and for a good reason it is not offered online.

A lot of data from consumers and citizens is collected, but one cannot have access to it. Imagine, if consumer rights organizations had access to anonymous transactions done through credit cards, you would get insight of the consumption habits of millions of people and would get to see on what the money is really spent. Many companies have such data; either they buy it or collect it themselves. So is the case of mobile providers, who have data about the movements of their clients. But hardly any company is sharing such data although science, media and civil society could benefit from it.

A study shows that tweets from Twitter can be used to analyze where an outbreak of Cholera happens first long before the media or government realizes it. Michael J. Paul and Mark Dredze used Twitter to analyze the situation of public health in the USA. They could locate the degree of various illnesses and deceases across the country. Or the Indiana University Center for Complex Networks and Systems Research has some interesting network analyzes on how worldwide protests are reflected on Twitter. It says a lot that most such data analysis is done with data from one company.

Another interesting project is UN Global Pulse, which is “Exploring innovative methods and frameworks for combining new types of real-time data with traditional development indicators to detect early impacts of global shocks”. Imagine you had access to mobile banking account data by location in Africa and could monitor sudden widespread changing spending behaviors. Are these early signs of an economical crisis or a drought? The hope is that with such data you can react and intervene much faster to crises. Global Pulse has partnered with organizations and companies to gain helpful insights from big data, but unfortunately next to nice reports, they do not offer a single data set on their website.

Robert Kirkpatrick talked about the importance of data data philanthropy back in September 2011, but as all the other players in this field, offers no data. I sent a tweet to the UN Global Pulse team and here is their reply: “This year, Pulse Labs projects will make every effort to share datasets for cross-verification and open access when possible” and “we’re talking to a mobile company about hosting a data mining competition with anonymized & aggregated call detail records.” Both points sound nice to me, but what I miss is a real attempt to share data, concepts and approaches to analyzing it with a community around the world. A good example is the Guardian data blog, which publishes each data set; readers started to play with the data and came up with more results.

The problem is that critical data for transparency is rarely available. Whereas UN global Pulse can partner with companies and institutions to get data, other initiatives have to work hard to get any data. For example, the International Land Coalition has released, with the support of the Tactical Technology Collective, a great overview of land acquisitions world wide. These one thousand rows of data about land acquisition were difficult to collect, but are such an important step for transparency in the massive field of land acclamation world wide. That’s why innovative solutions, such as the Forestwatcher project to crowdsource deforestation monitoring worldwide, are needed.

If one looks at publicly available big data sets the view is terribly disappointing. Ridiculously little is available and more than often data sets are very old. Amazon hosts 54 large public data sets. Compare that to the estimate made by Eric Schmidt at the top of this post. Or check the list on Quora, which is quite comprehensive, but tiny compared to data hidden behind corporate and government walls. The open data exceptions are projects such as Openstreetmaps, Wikipedia, Dpedia, Datahub, Opencorporates or the World Bank. Please hint me to more resources, which I have not mentioned.

The release of data can only be a first step, open collaboration around data and solutions has to further develop helpful services. There is a great promise in big data, but it has to be treated as carefully as any other data set or statistic and has also many pitfalls as Danah Boyd and Kate Crawford summarized nicely in their six provocations for big data.

"Big Data offers the humanistic disciplines a new way to claim the status of quantitative science and objective method. It makes many more social spaces quantifiable. In reality, working with Big Data is still subjective, and what it quantifies does not necessarily have a closer claim on objective truth – particularly when considering messages from social media sites.“

Knowledge management in decline and the prove of data

2012-02-23T15:30:59+01:00

Knowledge management is in decline or even once again dead according some writers. Knowledge management has probably been a misleading concept ever since the times when we used to believe we could store our wisdom in a little neat database. And that's also probably why document management is also plunging.

Have a look at these Google search trends; they show how all things management are downturning. Is this finally an attribute to complexity?

But wait, isn't it nowadays all about data, open data, big data? So are we walking back in time? I hope not and the above example makes a good case. There are discussions about the decline of knowledge management, but rarely any evidence. The above search queries from millions of people give us an insight. That is why the combination of data with knowledge is helpful. I will write soon more on that.

And what comes after knowledge management? Social business is the new kid on the block. I thought social business was an entrepreneur with an altruistic perspective. But it seems like it is also a new cluster to describe "what" social media can change in an organization. That is indeed funny – as if organizations are not inherently social. How could they even function without social relationships? Internet based exchange can foster social relationships, but it is only one type of exchange. Or maybe it is time to stop the people vs. technology debate? So what can we learn from this? Concepts are helpful clusters, associated in all directions and have their life span.

Results of the Open Aid Data Hackday

2011-10-17T14:01:40+02:00

Around 150 participants joined the Open Aid Data Conference in Berlin. The event was full with discussions and exchange on how open data can be used to achieve more transparency in the developing aid sector. The first day was split into two workshops – an 'Aidinfo Data Training' and 'Hackday,' to explore potential data sets and applications to make developing aid more transparent.

In this post I would like to share my experiences from the Hackday as a co-organizer. There was an interesting mix of participants, from different backgrounds, who worked in teams on different issues greatly facilitated by Marek Tuszynski from Tactical Technology Collective. Here is a link summarizing the discussions, some available data sets, and these are the main questions we dealt with:

What do we need to know about Open Data for aid transparency?
What data is out there?
Who benefits from developing aid? Organizations?
How does an organization implement Open Data?

The last question was answered by one of the groups and put together in a document. Other teams took a look at the kind of data offered in the German development aid sector. To begin with, the status of information is disappointing and proves the main purpose of the conference: Information is fragmented, almost no data is offered as open, and detailed data for financial spending is not available at all.

It is also impossible to find a list of organizations (NGOs, Government Institutions, etc.) funded by the Federal Ministry for Economic Cooperation and Development (BMZ). Not to mention a list if disbursed funds for each project. That led to one outcome at the Hackday: To initiate a freedom of information act request to the BMZ.

Another team found out that not a single German developing organization offers project information in an open data format, so that it could be easier analyzed. Furthermore, it is very difficult to find out in which countries all these organizations have projects. It can only be found out by clicking through all the websites of hundreds of organizations funded by the BMZ. Imagine how long that takes and how non-transparent that is!

However one interesting database is offered by the OECD, which is called Creditor Reporting System and offers project information for over 50 years. Member countries such as Germany contribute their data about their funded activities under the umbrella of the Official Development Assistance (ODA). We took a look at the data and potential visualizations.

As an outcome of the Hackday, it became clear that we need to invest more time into data analysis to bring more transparency in the sector. Therefore, I will start developing a data catalogue to monitor German aid money better. More to come soon.

Open Aid Data conference and Hackday Berlin

2011-08-23T17:33:57+02:00

The past year I have written on many occasions about the potential of open data and why it is much needed particular for the development aid sector. So I am happy to announce a Hackday I am organizing as part of the Open Knowledge Foundation Germany. The Hackday is linked with the Open Aid Data conference held in Berlin, which is organized by Openaid.de, Boell-Foundation among others. Click here for full further information.

Join the Open Aid Data Hackday!

Help us find innovative solution for aid transparency and make development aid more effective.

Germany is one of the largest donors in development aid worldwide. Every year over 6 billion euros are spend alone by the Federal Ministry for Economic Cooperation and Development to provide humanitarian relief and tackle poverty around the world. The Open Aid data conference will bring together practitioners from various organizations to discuss and exchange about new solutions and how technology, the Internet and particular open data can make aid more transparent – because not all of the money is spent effectively.

Prior to the conference, we will organize on the 28th of September a Hackday at the Böll-Foundation in Berlin to bring developers together to experiment on technical and data solutions to improve development aid. We are looking for programmers, designers, coders and others who want to learn more about the field of development aid and would like to share their wisdom. In the morning, we will introduce you to the theme and then brainstorm on possible approaches to make aid more transparent. During the rest of the day we want to work through a code sprint on a real solution. Be part of the event!

There are a range of activities around open aid data worldwide, such as the recent conference in Amsterdam or a Barcamp in Kathmandu for aid transparency. By the way, an interesting fact: The Kenyan government has offered an open data portal, while the German government is still debating on such a platform.

Open data and new bottom-up solutions for development aid are a rather new field but with some promising developments. Around data there is an initiative called IATI (International Aid Transparency Initiative), which propagates a common standard for data sets for financial and other project related data. So far in development cooperation, only a tiny fraction of financial data is openly available, which is, at the end of the day, the tax payers' money. Watch this excellent movie from Publish What You Fund on why financial aid transparency is needed.

One driver of IATI is Aidinfo.org, a co-organizer of the conference and member of the IATI secretariat, who has done some pioneering work in the area. Check out the AidInfo Labs to see what is possible through such data sets. We are curious to hear your ideas and projects.

Another driver of open aid data is the World Bank, who will also present their work on the conference. The World Bank has not only opened up its data, but also made an app competition, where many great solutions have been developed, to use the data, for example games about development indicators, amazing visualizations and crowdsourcing approaches. The aim is to make development aid more effective. Initiatives, such as Ushahidi in Africa, demonstrate the potential of new forms of technology. Come join us at our Hackday to network with great people from the community.

You can apply here or contact Christian.Kreutz {at} okfn.org for further questions. The Hackday is organized by the Open Knowledge Foundation Germany.

Learning cycling and the persistent illusion that all knowledge can be accessed online

2011-08-19T15:59:39+02:00

Remember when you learnt cycling? The first time you stepped into a pedal and tried to balance the bicycle; the voice behind your neck telling you to keep pedaling and you will be fine. Learning cycling is a wonderful example of how difficult it is to "transfer" knowledge and that most of our wisdom is not just published in the Internet.

Have you ever heard of or read a book about how to learn cycling. I am sure there are books for that out there, but would you learn cycling from one? Without practice, patience and using all your senses, you will not succeed nor overcome the fear of falling down. Learning cycling shows how difficult it is to learn other than by just doing and experimenting. About 80% of knowledge in our brain is tacit and cannot be written down.

I have recently stumbled over a nice article by Noam Cohen, in which he argues that the citation rules by Wikipedia, introduced as a quality mechanism, can also hinder knowledge sharing.

"In the case of dabba kali, a children’s game played in the Kerala state of India, there was a Wikipedia article in the local language, Malayalam, that included photos, a drawing and a detailed description of the rules, but no sources to back up what was written. Other than, of course, the 40 million people who played it as children."

The same article also linked me to a great video called "People are Knowledge - Exploring alternative methods of citation in Wikipedia" by Achal Prabhala.

People are Knowledge (subtitled) from Achal R. Prabhala`

At this point one could ask, what about social media? Knowledge sharing happens through conversations. But, let's be honest, how many tweets would you need to explain sufficiently how to cycle? There is an inherent limitation in written exchange compared to face-to-face exchange. Ana what about video? Visual exchange can make learning easier; for example, I have read a dozen articles about how to repair my old espresso machine, but only the video explanation made me fully understand how to do it. On the other side, Mike Davies argues in recent post:

"The lack of very local digital content is acute in Africa and is one reason even Google’s strategy is challenged here. Google Trader (among others) have offered cool new technologies, but without any real content they're just not being used." (http://blog.esoko.com/2011/08/mark-davies-part-ii-content-is-king.html)

But he also conludes that "we should also recognize that content can come from a multiple number of sources. Isn’t that the lesson we’ve learned over the last ten years? That content provided by your neighbour may be equally or even more relevant than that provided by your government, or by CNN."

Through my experience with the hyperlocal open data platform in Frankfurt, I have realized how little information is available on the local level in Germany – information such as finding out why a red traffic light lasts longer or shorter (by the way, there is a regulated framework called the "Richtlinien für Lichtsignalanlagen" – Guideline for Signal and Street lights). It is maybe a small detail, but that is what citizens care a lot about in Frankfurt.

So, it seems we need to work on two issues, making more knowledge explicit, particularly local content. And we also need to be aware that most knowledge won't be on the Internet, and especially piles of opened data will not change that either.

Who to feed? The open vs. the commercial race for data

2011-05-12T10:12:21+02:00

Google Maps has been an incredible service in the past years. Not only it was Google's engineers, who invented the slippy map, which revolutionized digital maps, but its approach to offer such a service for free and shock competitors with a free routing service. Google has a tremendous overview on all activities on the Internet; billions of search queries everyday say a lot about people's personalities. With analytics in websites, Google tracks people's paths from one page to the next.

We are just at the beginning of this massive data collection endavour. TomTom now throws out their gadgets for free just to get real-data from their users. Ironically, it came out recently that they sold the real-time traffic data to the Dutch police. That kind of data collection is not appreciated, all other data collection is agreed on with a small click by accepting the terms of service.

The author Daniel Suarez is worried about the future with his new book: "Understanding the Daemon." It is still fiction, but that can change soon: "Computers have learnt from us with every Google search, with every „I like“-click. Now they are beginning to change us". Every time we click, we feed the system called Internet and the outcome is not yet known. Ironically, the features of web 2.0 have incredibly helped to feed the system. Each recommendation, rating, each link, makes the data analysis better. But I am really worried that this is not always for the better. The open available data sets are peanuts compared to data sets of Google or Facebook likes. The question is also what data shall or can be public?

Data becomes more important than hardware. Apple is so eager to collect data that they spy on iPhone users activities and obliged users to do so over their terms of services. I have read a nice comparison: It is like buying a car and you are obliged to not use the seat belt. But things are not for free, although many services draw that illusion. Companies will at one point need to earn money with these adventures.

That's why Google Maps has changed their terms recently. Websites, which use Google Maps are from now on obliged to "forward display any advertising delivered in the maps imagery". An obvious and, from a company perspective, understandable move. Perhaps Google will invade millions of pages with advertisement soon. I am sure it is only a matter of time when these companies start to make money with personal data.

That's why projects like Openstreetmaps are so important because there is an attempt to offer valuable geo data without restrictions. There is an uneven race for getting data open for more transparency and, for example, for better citizen services. Recently, companies, such as Nike, have started to provide open data. However, I get the sense it is just a public relation move; or will they soon provide data up and down the supply chain, so one can follow up how sneakers are being produced? The crowdsourcing potential for such open and free data will keep growing if more and more people join the process. However, these type of data collection is tiny compared to the huge commercial data sets. Ironically, commercial companies now "exploit" Openstreetmap data because it is so good.

That is one big reasons why I co-founded the Open Knowledge Foundation Germany two months ago – to work more on transparency projects.

What can we learn from Africa on the use of mobiles for social/digital inclusion?

2011-02-01T14:04:08+01:00

David Wilcox asked me on Quora this question, which I also want to publish as a response here to discuss the topic further. Would be great to get some more thoughts on that topic from you. I imagine we can learn a lot from digital inclusion in Africa. Here are some points:

The art of improvisation

When it comes to access, the innovation under constraints is amazing in Africa. Look, for example, how the challenge of energy supply has been greatly mastered. I think that in Europe we address inclusion only from few angles and should be more creative. We could focus a lot more on mobile phones and offer real needed services even through SMS. Suddenly, we can potentially reach over 90% of people, but most important, we need to play a lot more with technology and hack it where we can. A lot is happening in this regard in the UK, on the contrary Germany, where technological skepticism is still hampering innovations, or where one faces legal implications when offering open wifi.

I like these to posts very much:

Innovation from Constraint by Ethan Zuckerman
If It Works in Africa, It Will Work Anywhere by Erik Hersman

However, to me one of the biggest challenges is media competency, and not only in Germany but in the rest of the world.

Service models

If we have not reached enough people through the Internet, it might be that most services do not address a real need and do not offer sufficient help. Isn't the Internet in Europe largely focused towards the middle class? Where are web solutions or services focused on marginal groups? Here information literacy is the key: "… empower people in all walks of life to seek, evaluate, use and create information effectively to achieve their personal, social, occupational and educational goals" UNESCO

I wish there would be more solutions such as "Rate my Prison" from the first social innovation camp or or "My Police". Unfortunately "Rate my Prison" seem not to have been developed further.

Look at the public sector for example and see how little is offered here in Germany. There is a city website, but hardly any online services. On the other hand, Fixmystreet is still a rare exception. The whole world of apps unleashes here a new creativity, but if you really want to get inspired for future mobile services, you need to look at Africa. In Africa solutions such as Farmer's friend (SMS price information) attempt to reach also poor people in remote areas. Where are such business models in Europe?

Revenue? Examples of nonprofit or business model for open data

2011-01-20T13:41:37+01:00

As open data becomes more popular, I wonder where are the nonprofit and business models for open data? It is clear that somehow open data needs to generate revenues, because it will not only work with voluntary efforts. I did a little research to find interesting approaches to do more with open data.

A good starting point are existing open data initiatives, such as London or San Fransisco. One area of applications are all types of visualizations, which can help to highlight hidden information behind the data. A nice example is Betterflux, which offers a nice visualization tool for the open data World Bank API. Carolyn Mellor desribes in her post “Mining World Bank Data” how to offer paid analysis services using the World Bank API.

Fireworkers

Lorenz Matzat, a fellow blogger from the open data blog of the ZEIT magazine in Germany, wrote about an intriguing case to use open data at the Amsterdam fire brigade. Once a fire alarm starts, all sorts of data is collected about the location and the route to the emergency: Constructions on the way, latest updates from Openstreetmap, the type of house and if possible more data such as construction dates, materials, people living there, etc. A great case of how open public institutions themselves can benefit from open data. However, it is an example of how open data can easily collide with privacy. How many data should be freed for the sake of emergency.

Public transport

Everybody who has a smart phone might have already benefitted from a location-based public transport application, which gives you for example information on bus or train lines close to you. These applications would not have been possible without access to public transport information. In Germany, from my experience, in almost all cases the private applications are superior to the ones from public transport companies. An interesting example of what can be done with such data is the London Live Tube.

Also, ICommute takes the available data from the San Fransisco open data store and offers a mobility check tool. “ICommute SF helps you locate, organize and access route information and real-time arrival predictions for San Francisco's Muni system. Get the most of public transit and improve your daily commute.” The app costs $2,99 dollars. I would be curious to know how many sales it takes to get at least the development costs back or even make a profit.

Kids life

Again, in San Fransisco an idea came up to provide better information for kids' lives. “What choices are there as kids travelling to & from school”. After School provides a map for specific locations: Schools, libraries and playgrounds. It also offers places to eat – questionable places such as McDonalds. A commercial approach, again through an Iphone app, is done by MomMaps – It seems they do not offer a “Dadmaps.” Mommaps offers places such as parks, playgrounds, restaurants, museums in over a dozen cities in the USA. The app is for free, but I could not identify the business model.

Food

Nutrition is another interesting sector to use open data, which I discovered lately. Everyblock has for years food inspection data on their website and in the UK there is an Iphone app by the Lichfield district council: Ratemyplace. “Every time a council in the Ratemyplace scheme carries out an inspection of a food business's kitchen, it's listed on the Ratemyplace app.”

Another really interesting approach is Food Sprout. It combines different data sets to make transparent how the food is produced, up and down the supply chain. And they also come up with various revenue models. Check out the interview at the great Food and Tech blog. Interestingly companies seem to have growing interest to make their supply chain transparent in their corporate social responsibility efforts. These are the data sources of Food Sprout:

Data our internal team at Food Sprout gathers
Data a user inputs into the system that we then have to verify
Third parties like non-profits supporting farmers that have data
Government agencies and databases of food
Investigative reporting where our team seeks out hard to find data.

A last example for food is the whole potential behind barcode scanning – you take your mobile phone to the supermarket and scan products to get the information behind the fair trade certificate or behind the company. In the recent dioxin scandal in Germany, the company Barcoo took information from the ministry of agriculture in Germany, of which farms have intoxicated eggs and offer the info in their app. So, you can check in the supermarket the eggs that are fine and not with your mobile phone.

Conclusion

There are still very few business models for open data. Maybe because there is still little open data available and that might be hampering the development. Although if you look at Openstreetmap or CKAN, there are large data sets offered. Besides Iphone apps, there is also no revenue model and any other is more of an experiment still. It seems way easier to start with open data as a nonprofit project.

A working-day of a knowledge worker in 2030

2011-01-12T15:00:31+01:00

Let’s say one day you arrive at work – a mainly knowledge-driven organization, such as a consultancy, where you don't have an office, not even a position, nor a particular function. So to start your day, you first get a selection of all projects, ideas and problems that your organization is dealing with at the moment.

Half of your working day is already subscribed to ongoing projects, and the other half you could jump into something new. You look at various open tasks, questions, ideas or requests for solutions – all these items have a chronology of contributions and interactions. You can see what has been already done and what is needed. You find an interesting challenge, estimate the working time and send an invitation to a colleague, who has the skills and might be interested on working together on it.

Now, you have 20% of the day left. You take a look in your competence section and see several questions and help requests for topics. You pick a few tasks, which you can solve quickly and teach others how to do it themselves next time. The daily work plan is done and you go on to a workspace, where colleagues are gathered to work on your main project.

By the way, instead of having a boss, you have different scores you give yourself on your work performance. You might prefer the creativity score, which gives you a lot of time to find solutions and to push for innovations. Or you focus your work on your teaching score, which is evaluated by your colleagues. Or you pick another score, which fits best your working style. And to top it all, strategies do not exist either. This is done by a prediction market.

Is that absurd?

Maybe to an organization it is, but the social web pretty much works with this concept. Many people engage that way. After they have left the office, they privately engage in the social web. Take a look at the newest hyped tool Quora, which is basically a questions and answers tool.

You can ask any questions.
It is horizontal. Everybody can answer or edit questions (collaborate).
You gain reputation (score) in many different ways: As your questions are followed up, the answers move up to a higher ranking, or when people vote your question moves up.
You can also address questions to certain people or invite others to answer and so on.

Quora also has an interesting solution to find information from within the ocean of questions. Words of questions become key words (tags), which are then associated with similar questions and clustered under one topic. Imagine such a thing in an organization. You would create organically an organizational wisdom. Why cannot whole projects be organized in such a fashion?

Of course that nice set of features does not automatically lead us to the utopian first part, but maybe it can contribute to it. If we look at the incredible inefficient and non-creative problem solutions capacities of organizations and companies and can overcome the cultural resistance, such open collaboration form would bring us closer to the utopian first part, where you work what you really want.

Automated vs. manual mapping - consequences for crowdsourcing

2010-12-16T11:48:18+01:00

Digital cartography has made map making a lot easier. But If a map contains a lot of data or specific data, it can become a complex or costly adventure. Despite the efforts around open data, still the majority of data is not publicly available, and if so only for high costs. Crowdsourcing is one alternative to collect data for maptivism, but maybe some of these approaches will not soon be needed if automated mapping is further progressing.

Automated mapping

It is quite impressive and a bit scary to see the pace of innovation around digital recognition. Its aim is to make more information available from the offline world. Google is on the frontrun of digital recognition with another example: 3D trees in Google Earth. Google has chosen parks in 50 cities around the world to identify in an automated process trees out of satellite images.

“With 3D trees in Google Earth, we’ve brought characteristic trees to life, from the palm trees that dot San Francisco's bayfront Embarcadero Street, to the olive trees that cling to the Acropolis in Athens, to the flowering dogwoods found in Tokyo’s parks. All told, there are around 50 different tree species to explore in Google Earth and counting!”

Consequences for mapping

A while ago I blogged about the crowdsourcing Urban Forest Map in San Francisco. Its goal is to map all trees in the city. Now at least the work for the park is not needed anymore if Google is giving out the data. Thousands of people from the Openstreetmap community use satellite imagery from Yahoo to draw shapes of buildings into maps. Is that becoming obsolete soon? What needs to be manually mapped? Of course a lot, because most of such data will not necessarily be publicly available. One example is real-time data. Check this post on maptivism: live tactical mapping for protest swarming.

Here, we are also getting in a dilemma. Such a virtualization of trees can contribute to the protection of forests. Imagine the mapping happens within days and deforestation in the rain forest can be act on quickly. However, what else can be mapped? If trees can be classified, all kind of objects can be classified if digital recognition software becomes increasingly powerful. Check for example the bird’s eye view from Bing, where you can see detailed aerial imagery (see image), not to mention Google street view.

Bandwidth divide: What's fast to you, isn’t fast to others

2010-12-10T17:25:54+01:00

On my last post, I talked about how slow websites can trigger higher search costs. In this post I want to further elaborate on the bandwidth divide that exists within and between countries, and which is largely ignored by website developers. There is also the notion that we have unlimited capacities for websites – the more the features, the better the website. On the contrary, especially now with the increasing usage of the web, low-bandwidth websites are more important than ever.

Connection speed comparison

Check, for example, the latest “real” connection speed overview from Pingdom based on Akamai data. South Korea has been leading in terms of bandwidth for years, while other Western countries have considerable less capacity available. I did a little heat map to show the geographical variation across the world. Would you have guessed that Romania has the fastest connection speed in Europe?

On the bottom of the list are countries such as Nigeria, Indonesia and Iran, which have 1/3 of users with less than 256kbs per second connection. If that speed is really available, it still needs 6 seconds to load the Wikipedia page example from my last post. Take a book and select a page and then wait 6 seconds before you open it. Do that for a few pages and you will see how frustrating can slow speed can be when doing research.

Website speed and search engine ranking

Although the mobile web is growing exponential, only a minority offers customized websites because most such changes take time, skill and resources. Even Google is taking website speed into consideration. Its primary goal probably is to save resources for their crawlers. In the tool Google webmaster you can see your website's speed performance. This example is with 5 second slower loading time than the majority of other websites.

Some hints to dive into website speed

You can also use tools such as webpagtest.org and see that, for example, the new World Bank page still has 900 kb to load. This means that with a fast connection, it still takes 8 seconds to load. Check here for details. With a low-bandwidth connection, which is the situation in many developing countries, it takes over one minute to load the front page.

Here are a few first steps to get faster loading websites:

Think about whether you really need a special feature – certain widgets and slideshows (e.g. World Bank's websie) need a lot of kilobytes to load. Less is more, and your reader will thank you.
Check your website speed with tools such as Yslow or Page speed, and analyze how many files your website has and if there is potential to minimize or at least to combine them.
Check whether it is possible to cache your page. This means that it can also be available in static html and updated regularly. For Drupal, there is, for example, the Boost module, and for Wordpress the Super Cache or W3 Total Cache plugin.
Use HTTP compression (http://en.wikipedia.org/wiki/HTTP_compression).
Throw out all external widgets, which are not very necessary, since they often load a lot extra coding to your website.
Consider to work with a Content Distribution Network.

This is just a small list. There is a lot more to tweak for better performance.

Content Delivery Network

One other great approach is content distribution network (CDN). They distribute your files around the world and put them closer to the user's end. So, if a user visits your site from Asia, then he gets the files from a server in Asia instead of Europe. Basically, you distribute the same files across servers. Wonder why is Google so fast? Because they have servers around the world. If your audience is let's say in Uganda, you better not only host the website in the US, but also in Kampala or Kenya. However, it all depends on your audience.

Slow website speed and consequences for search costs

2010-12-06T11:40:05+01:00

There are a number of reason why websites are slow; and the results of this lead to raising searching costs and leaving users. A faster website requires skills and resources often not available to great information-rich pages. With a low-bandwidth connection you have two amin major challenges: A slow connection needs to deal with increasing loaded websites, and low-bandwidth is often what you get with a mobile data connection.

Search costs

Let’s take, for example, Wikipedia’s entry about itself. The page has a file size of 1.54 MB! This means you need to wait 212.52 seconds with a 56kbs modem connection, 82.84 seconds with a 128kb (ISDN) connection and, even with a 1.44Mbps, still 30.57 seconds to see the full page. Imagine the time to browse Wikipedia. Believe it or not a few milliseconds can make a difference, whether we stay on the website or change to a different one. Google found out in a study that the slower their search results appear, the less people search.

“Our experiments demonstrate that slowing down the search results page by 100 to 400 milliseconds has a measurable impact on the number of searches per user of -0.2% to −0.6%.”

Akamai did a similar study on retail websites about speed and expectation of Internet users back in 2009:

“The most compelling results reveal that two seconds is the new threshold in terms of an average online shopper’s expectation for a web page to load and 40 percent of shoppers will wait no more than three seconds before abandoning a retail or travel site.”

So, are we only impatient or do we still have to deal with too many slow websites? Imagine you have a book and you are looking for something, but changing the pages is somehow delayed.

Bandwidth is not keeping pace with page size

The challenge is that bandwidth is not growing in average as fast as the page size. Thanks to web 2.0 in particular, pages are loaded with widgets and many nice features, each tremendously raising the website speed. Alone the facebook "like" button or "fan" box needs more than a 100kb of java script to load. But 100 kb should be the limit for a real lightweight fast website. Aptivate has a good blog about it, with a graph showing that the average page size of websites is growing much higher than the bandwidth.

In the next post I will describe the unequal distribution of bandwidth worldwide and why lightweight pages are important, especially for mobile phone access.