For all those who think collecting Metadata is no biggie, I urge those people to read this article from Wire magazine and rethink their position.
The Wired article is correct about the meta data, but they leave out some important information — meta data is more valuable than the actual content of phone calls and email. Yes. It’s true.
Meta data is structured information. This means field values across different datasets will either be identical, or so similar as to need only a minor adjustment to be normalized. Time. Location. Phone number. Phone owner. Length of call. Number frequency. All these and more are highly structured pieces of data, and the more you have from different sources, the possibility of deep and meaningful insights increase.
Call content, email content, tweets, text messages, web page content, etc. are all unstructured data — language. While we might have a billion phone call transcripts in english, each will be formatted differently, use different syntax/slang, have varying levels of literacy, and so on. Normalizing this unstructured data for big-data analysis requires expensive (in terms of processing power) and slow natural language processing. Keyword and entity extraction can be performed fairly well, but even then, the all-important context or meaning will be missing.
Pattern-matching meta data in trillion-record datasets is the big-data analyst’s wet-dream. It’s highly plausible that such a system, with all data from everywhere, could identify that a new burner phone is being used by criminals or terrorists within 2-3 days using just the meta data. The same pattern matching could also identify that a burner is being used by a journalist — or — using the journalist’s known number, determine what potentially damaging story a journalist is currently investigating. Once pattern-matching flags something, then the available unstructured data is examined.
Oh, and don’t presume that the typical methods of encrypting or protecting your privacy will thwart this. Even if you use TOR for browsing, and the NSA/FBI takes an interest in what sites you visit — if they have your ISP logs and that of upstream Internet providers (it has been reported that they do), it only makes it about 5% harder to nail down where you went online.
Is Mass Spying Being Used to Make Some People Rich?
The Wall Street Journal reported that the NSA spies on Americans’ credit card transactions. Many other agencies are doing the same. In fact, virtually all U.S. intelligence agencies – including the CIA and NSA – are going to spy on Americans’ finances. The IRS will also be spying on Americans’shopping records, travel, social interactions, health records and files from other government investigators.
Not only can the NSA intercept and store virtually all digital communications on the Internet, butprivate contractors can also view all of your data (and the government isn’t keeping track of which contractors see your data and which don’t). And because background checks regarding some contractors are falsified, it is hard to know the types of people that might have your information.
In an articled entitled “How Much Are the NSA and CIA Front Running Markets?”, Naked Capitalismreports:
A 2008 paper [which was subsequently published in the prestigious Oxford Quarterly Journal of Economics] by Arindrajit Dube, Ethan Kaplan, and Suresh Naidu … found evidence that the CIA and/or members of the Executive branch either disclosed or acted on information about top-secret authorizations of coups. Stocks in “highly exposed” firms rose more in the pre-coup authorization phase than they did when the coup was actually launched.
Here’s how the dataset was developed:
We selected our sample of coups on the following basis: (1.) a CIA timeline of events or a secondary timeline based upon an original CIA document existed, (2.) the coup contained secret planning events including at least one covert authorization of a coup attempt by a national intelligence agency and/or a head of state, and (3.) the coup authorization was against a government which nationalized property of at least one sufficiently exposed multinational firm with publicly traded shares.
Out of this, the authors found four coup attempts that met their criteria: the ouster of Muhammed Mossadegh in Iran in 1953, two programs in Guatemala in 1952 and 1954 that eventually removed Jacobo Arbenz Guzman; the unsuccessful effort to topple Castro in 1961, and an operation that began in Chile in 1970 and culminated in overthrow of Salvador Allende. Then they chose companies:
We apply 3 criteria to select our sample of companies. First, a company must be publicly traded, so that we can observe a stock price. Secondly, the company must be “well-connected”, in terms of being linked to the CIA. Finally, the company should be highly exposed to political changes in the affected country, in the sense that a large fraction of a company’s assets are in that country.
They used these criteria to devise two samples (based on different definitions of “highly exposed”) and tested both.
Covert operations organized and abetted by foreign governments have played a sub- stantial role in the political and economic development of poorer countries around the world. We look at CIA-backed coups against governments which had nationalized a considerable amount of foreign investment. Using an event-study methodology, we find that private information regarding coup authorizations and planning by the U.S. government increased the stock prices of expropriated multinationals that stood to benefit from the regime change. The presence of these abnormal returns suggests that there were leaks from the CIA or others in the executive branch of government to asset traders or that government officials with access to this information themselves traded upon it. Consistent with theories of asset price determination under private information, this information took some time to be fully reflected in the stock price. Moreover, the evidence we find suggests that coup authorization information was only present in large, politically connected companies which were also highly exposed.
We find that coup authorizations, on net, contributed more to stock price rises of highly exposed and well connected companies than the coup events themselves. These price changes reflect sizeable shifts in beliefs about the probability of coup occurrence.
Our results are robust across countries, except Cuba, as well as to a variety of controls for alternate sources of information, including public events and newspaper articles. The anomalous results for Cuba are consistent with the information leaks and inad- equate organization that surrounded that particular coup attempt.
Now sports fans, given the fact that there’s reason to believe that people in the intelligence with access to privileged information weren’t above leaking it to people who could take advantage of it, why should we expect things to be different now? And given what has already been revealed about the NSA’s data gathering, if you were a clever trader and had access to this information, how would you mine it? How would you go about finding patterns or events to exploit?
Given that the U.S. has been recently been involved in military actions and/or training exercises in Iraq, Afghanistan, Libya, Syria, Yemen, Pakistan and up to 35 African nations (and see this), there is opportunity for people who know what military forces will be deployed where to make money.
And since data on consumers is worth a lot of money, someone who had access to consumers’ spending patterns, habits, preferences and other private data could sell the information.
And due to the fact that large companies spend huge sums of money to conduct industrial espionage on their rivals, information gained through spying could be invaluable. Especially since the NSA closely spies on the European Union, the European Parliament, Germany, the G20 summit andChinese universities.
And several financial and economic experts – such as Jim Rickards, Max Keiser, German central bank president Ernst Welteke, Swiss economists Remo Crameri, Marc Chesney, Loriano Mancini and Bill Bergman (senior financial markets policy analyst for the Federal Reserve Bank of Chicago for 13 years) – also say that there were insider trades right before 9/11 by people who knew the attacks were coming … people with “no conceivable ties to al-Qaeda” according to the 9/11 Commission.
You don’t have to believe that 9/11 was an inside job to believe that this theory is at least possible. After all, 9/11 was foreseeable to people in intelligence services worldwide … as was Al Qaeda flying planes into the World Trade Center and Pentagon.