Terror futures and “open source intelligence”

Ooooh, cool… I get to write a long post about a topic I really love that I don’t think I’ve ever discussed with any of you! (I talk so much that it’s rare to come across a secret passion of mine.)

While the DoD’s insta-shitcanned “terror futures” market was pretty stupid and incredibly poorly handled, it stems from a generally reputable idea. Here’s hoping the Feds won’t throw the baby out with the bathwater.

So here’s the baby, as it were: Markets do more than just assign value to goods. Markets represent a form of knowledge, and futures-style trading does consolidate and maximize that knowledge. The Hollywood Stock Exchange is a great example of this. HSX is often far more accurate in predicting opening-weekend grosses than Variety is. Basically, research into this field has proven that people in groups are smarter than any single person in that group; an efficient market (based on accurate and transparent underlying data) magnifies this effect. So as offensive and unsettling as its ramifications may be, a worldwide network of well-read amatuer and professional predictors might be more accurate at assessing and predicting threats than even the sharpest spook in Langley. (This Slate article is a decent half-defense of the Policy Analysis Market, or PAM, that just went down in flames, though it misses the most obvious way to ensure that terrorist don’t profit by manipulating the results: make it a “closed-end fund” open only to thoroughly vetted individuals– a few thousand to start with, sprinkled around the globe.)

The realization that bountiful intelligence information is available in the public sphere but is nonetheless undervalued and underanalyzed has given birth to the idea of “open source intelligence.”

And yes, Virginia, people were using this term before Mr. Torvalds unleashed the Penguin Horde upon us. Though its funding by the U.S. government has been fairly miniscule, “OSINT” is recognized as an equal partner to “HUMINT” (human intelligence, from real live spies in other countries) and “SIGINT” (signals intelligence, based on intercepted transmissions). The CIA is focused on HUMINT, the NSA on SIGINT, and to date OSINT has been the bastard child– in no small part because it intrinsically argues against the need for a massive intelligence bureaucracy. But wow, could we really use its insights now.

The Internet has accelerated interest in this trend, and as an underlying meme it has gained currency in situations like the “yellowcake scandal” with the NYT reporting that the Niger documents could have been revealed as a fraud by “anyone with Google.” But it has been in the culture for a while. One of my favorite ’70s-paranoia-thrillers is Three Days of the Condor, in which Robert Redford plays a secret government operative tasked with, get this, reading everything printed around the world. (This speaks volumes about the explosion of information, even in our lifetimes–this premise was a stretch even then, but believable.) Anyway, our man “Condor” has his boring-museum-functionary cover blown and someone is out to kill him. The underlying idea is that someone who reads all the world’s papers is at least as dangerous as any spy.

Australia’s National Open Source Intelligence Centre is a quasi-governmental group (with a fittingly scary logo) that advances OSI. NOSIC defines Open Source Information as “publicly available information (ie any person could lawfully obtain the information by request or observation), as well as other unclassified information that has limited public distribution or access. This latter is referred to as “grey” literature and includes non-proprietary information from companies and other organisations.” Open Source Intelligence goes further and “results from the integration of legally and ethically available sources, which require: analysis, collection management, source validation, multi-source fusion and compelling presentation.”

If you’re really interested, someone has posted the “Open Source Intelligence Professional Handbook 1.0” as presented at the Fifth International Symposium on Global Security and Global Competitiveness (1996).

I ran across this document online a few years ago, and it was kind of a revelation to me. I supposed I had always wanted to be a spy; I will write a long post someday on how this has to do with repressed attraction, and why it is so unsurprising that closeted gays from Yale in the ’30s and ’40s created the CIA. “But not this year,” as Monica’s Peter would say.

No, it was a revelation because I realized that a huge part of my job is OSI for my clients: sorting out relevant and actionable competitive information from the Web (and many sales brochures on enterprise solutions ordered surreptitiously), tracking emerging trends, and projecting these trends forward to imagine the market conditions my clients will be competing in five years down the road. So while I may not have the perfectly groomed five-o’clock-shadow of Colin Farrell in that abysmal Al Pacino trainwreck The Recruit, I can claim a legitimate relation to all that sexy “tradecraft.”

Anyway, we can all laugh at the Pentagon’s terrible PR, but some of the underlying ideas are sound. I, for one, would much prefer a mindset that looks to existing data for intelligence to one that thinks only of more spies, more wiretaps, and more email-reading minions to defend the realm.