In recent years, there has been an explosion in the quantity and diversity of high frequency digital data. A massive amount of data is regularly being generated and flowing from various sources, through different channels, every minute in today’s digital age.
The amount of available digital data at the global level grew from 150 exabytes in 2005 to 1200 exabytes in 2010. It is projected to increase by 40% annually in the next few years, which is about 40 times the much-debated growth of the world’s population. This rate of growth means that the stock of digital data is expected to increase 44 times
between 2007 and 2020, doubling every 20 months.
For development practioners, these data hold the potential—as yet largely untapped—to allow decision makers to track development progress, improve social protection, and understand where existing policies and programmes require adjustment. It has already been widely acknowledged that the recording, accessing, data mining and dissemination of information will affect in a crucial way the progress of knowledge of mankind in the next years.
According to a UN Global Pulse paper titled, “Big Data for Developement: Challenges and Opportunities,” data including call logs, mobile-banking transactions, online user-generated content such as blog posts and Tweets, online searches, satellite images, etc.— can be turned into actionable information through computational techniques that can unveil trends and patterns within and between these extremely large socioeconomic datasets.
“New insights gleaned from such data mining should complement official statistics, survey data, and information generated by Early Warning Systems, adding depth and nuances on human behaviours and experiences—and doing so in real time, thereby narrowing both information and time gaps,” states the paper.
Take, for example, across the developing world, mobile phones are routinely used not only for personal communications, but also to transfer money, to search for work, to buy and sell goods, or transfer data such as grades, test results, stock levels and prices of various commodities, medical information, etc. Tracking these information channels can provide information on emerging concerns and patterns at the local level which can be highly relevant to global development.
The big question though is whether such massive data has any analytical value or significance to the policy making process. Moreover, given the digital divide in many developing countries, there is also question marks over the representativeness, validity and realiability of such data – as well as the overarching privacy issues of utilising personal data.
While technology driven social change is increasingly becoming a mantra, what has to be also appreciated is that it cannot be a magical bullet that will resolve all the age-old development challenges.
However, there is no doubt that the vast dataset that have been enabled by new technologies certainly provide an opportunity that can be tapped into by development practitioners. Already, mapping and data visualization are becoming integral and powerful tools to respond to natural and man made disasters as well as poverty, hunger and disease.
At the most general level, properly analysed, these new data can provide snapshots of the well-being of populations at high frequency, high degrees of granularity, and from a wide range of angles, narrowing both time and knowledge gaps.
According to Global Pulse’s paper, ” … big data for development is about turning imperfect, complex, often unstructured data.”
“This implies leveraging advanced computational tools (such as machine learning), which have developed in other fields, to reveal trends and correlations within and across large data sets that would otherwise remain undiscovered,” states the paper.
In many developing countries, the data revolution may be especially relevant to supplement limited and often unreliable data. With intent and capacity to make sense of the data, new opportunities can be unveiled.
According to some experts, “… employing massive datamining, science can be pushed towards a new methodological paradigm which will transcend the boundaries between theory and experiment.”. Big data constitutes an historic opportunity to advance our common ability to support and protect human communities by understanding the information they increasingly produce in digital forms.
