Common Misconceptions Regarding Big Data, Analytics

Gov1 posted the following question on Quora:

“What are some common myths or misconceptions holding local governments back from taking full advantage of big data, analytics and other open data resources?”

The post generated some interesting responses:

Meta Brown, Author of Data Mining for Dummies:

The most common misconception in analytics is the belief that a large quantity of data is inherently valuable or a solution to any given problem. What’s really valuable is relevant data, and governments are collecting less of that, not more, than they dd just a few years ago.

Consider what may be the ultimate in Big Data enterprises: the NSA’s mass surveillance of telecommunications in the United States. Although many details of the program are still not public, we certainly know that it has been a tremendous undertaking with a multi-million dollar budget, a large computing facility and staff.

What would be the evidence if this undertaking were successful? Since the object is to detect terrorists and bring them to justice, success would imply convictions for terrorist crimes. Of course, successful detection would also imply prevention, so acts of terrorism would be prevented.

Alas, it hasn’t worked out. If you know of any convictions that can be credited to NSA surveillance, please let me know, because I don’t. Nor did the monitoring program prevent the Boston bombings.

That’s not to say that governments should not make use of data. On the contrary, the right data used effectively can help lawmakers protect the public, improve the economy and better the lives of citizens. But the right data isn’t necessarily Big Data or open data. (Open data is primarily records of government transactions. This information has been public for some time; open data initiatives may make it more conventient to obtain, but they don’t produce new data.)

Often, very often, what governments and cosntituents need is data collected specifically to inform lawmakers or government staff about the state of the economy or the public. This kind of data is often known as “statistical” data. It’s the kind of data gathered by the United States Census Bureau, for example, and by dozens of other statistical agencies. And the sad thing about the buzz over open data is that it’s masking a decline in funding for statistical data collection and analysis at every level of government.

Here are articles which discuss this in greater depth:

Open Data: What’s It Hiding?

Ending the American Community Survey: Privacy is not the issue http://smartdatacollective.com/m...

Why Business Needs Public Data http://smartdatacollective.com/c...

Protecting Public Data http://smartdatacollective.com/m...

Alan Morrison, Researcher on data and analytics technology:

Some myths and misconceptions that may be prevalent:

1. It’s all about the data. It’s nice to provide access to data, but much more important to help people solve the problems they have and find what they’re looking for. User-friendly site design and building sufficient context for users are the more difficult and essential things. The challenge is more along the lines of process optimization, curation and extreme simplication and clarification, which suggests an iterative process.
2. Doing more with data will require lots of resources. No, it will require a reallocation of resources, different ways of working and lots of iterations to get the new working models right.
3. We’ll have to build a new capability ourselves. If it’s online, it’s persistent and should be available. It’s important to survey first what’s available as a service, what’s been shared as open source that you can take advantage of, and find out who else has done what you’re trying to do so you can tap their brains.

Bart Rosseau, Works for the City of Ghent, Belgium:

In my experience there’s a lot of misconceptions what analytics and data can do. Here’s some elements I encounter:

1. There’s some work in educating people what data can mean for them; the perceived promise of Big Data isn’t always applicable.
2. There’s some reluctance in decision making circles to rely on data. The political reality does not always leave room for data analytics.
3. What’s the issue you want to solve by using data? Focus is key. Often the flow is reversed: we have data, let’s do some analytics, and see what comes up
4. We can’t afford the specialists, so it’s hard to build knowledge within the organisation.
5. Good examples: good use cases within local governments that address real challenges, real issues and contribute to working solutions that go beyond a cool data viz would help to cross the bridge.

Alket Cecaj, co-author of scientific paper “Re-identification of Anonymized CDR datasets Using Social Network Data":

I think privacy of citizens is one of those “common myths”. Another one is that governments fear to loose power of decision on important issues if they implement a smart government methodologhy. Other excuse is that the technology is not ready to really help the citizens in improving their lifes in a substantial way.