Agile Zone is brought to you in partnership with:

Jason Hull is a principal at OpenSource Connections, a Solr search engine consultancy that works with clients across all three layers of the search experience to improve the search user experience in a cost-effective and revenue-driving way. Jason leads client engagements with marketing functions, helping them to think about the business case behind search. Jason holds a BSc from the United States Military Academy at West Point and a MBA from the University of Virginia’s Darden Graduate School of Business. Prior to founding OpenSource Connections, Jason led an analytics team at Capital One responsible for call center demand forecasting and managed their internal call center IT infrastructure investment budget. Jason is a DZone MVB and is not an employee of DZone and has posted 14 posts at DZone. You can read more from them at their website. View Full User Profile

Lessons from DI2E: Intelligence is About Search, Not Gathering

  • submit to reddit

I just came back from the 2011 DI2E conference in Dallas, Texas and certainly left with a lot of impressions. I want to state up front that my impressions were very positive about the state of U.S. intelligence activities. We’ve come a long way since we first started inserting Green Berets into Afghanistan. Whereas we once faced an intelligence gathering problem, we no longer do. Instead, we have trouble indexing, sorting, filtering, and analyzing the glut of intelligence that we have gathered. I did notice that Solr and Lucene are in use pretty robustly in the intelligence community, with Solr a part of the ICDL and JDL (intelligence community data layer and joint data layer respectively). That’s fantastic news, but there’s a long way to go.

  • Going to the cloud helps, but is not the panacea. One of the big issues that the intelligence community faces is the ability to index what it calls “big data.” Granted, it was a public conference, so they weren’t going to reveal all of the secrets, but I didn’t hear anything which led me to believe that the data indexing issue was insurmountable. Certainly throwing enough hardware at the problem can help solve it, but whose cloud rules? So, instead of waiting for the cloud to deliver a magical elixir, the intelligence community can also look at improving indexing, caching strategies, and leveraging other systems. One common complaint that I heard was that Hadoop causes read time isues – it’s too slow. To answer that, look at Katta or Cassandra, or be willing to crack into Hadoop’s code and do modifications. The government can’t just rely on COTS to solve its problems.
  • A glut of money led to inefficient development and stovepipes. Because each service got a pile of money, they each developed their own solutions. Most of them don’t integrate well with other systems and solutions. They also use proprietary systems and protocols. At least the military recognizes the problem, as one speaker said that they can no longer add twenty different license costs for software that all does essentially the same thing.
  • Even with all of that money, search user experience got left behind. The military employs people, called Knowledge Managers, to teach people how to use the tools that are there. However, the tools are, at best, clunky, and at worst, a terrible user experience. As a result, as one of the speakers noted, over 90% of the tools are not used. I am certain that Netflix does not have a cadre of trainers to teach its users how to find movies. Zappos does not have its payroll stacked with people who can tell you how to find shoes on the Zappos website. Why? Because the user experience is well conceived and well executed. The military should look at innovators in industry and adapt their models. The money saved spending a little on vastly improved user experience will be more than made up for in the reduction of knowledge management requirements. Improving search and the search user experience has a HIGHLY positive ROI in this environment. It begs for attention and improvement.
  • You cannot just Google for intelligence information. Google’s strength is that it has the wisdom of the masses. It has algorithms designed over the years to identify what the average person wants to see and when a website has legitimate information rather than being a spam or link farm. The intelligence community is vastly different. First, how many regular Google users are trying to do an IPB for a commander via Google? The sample size from which to draw the algorithms is incredibly small. Secondly, while analysts are interested in knowing the provenance of intelligence information, there is no spam in the intelligence community dataset. Thus, the real need is to tune for relevancy – get relevant results for the specific user set that the search identifies. When the intelligence community buys a license for a search engine which doesn’t make relevancy tuning easy, intuitive, and comprehensive, it is wasting taxpayer money.
  • Acquisition hinders innovation. Because requirements and contracting have to be so specific with most government contracts, the cycle time from need to fielding is often over two years. That does not keep up with the pace of change. This is not a new jet fighter. New, category killer technology is unlikely to pop up overnight. However, in a theater where information dominance is paramount, that lack of speed is going to kill soldiers. Risk aversion and an acquisition strategy shackled by a 1970s Cold War mindset means that we don’t get the best information technology available out to the soldiers who need it most in a timely fashion.

As GEN Odierno stated, the intelligence community is going to need to get more bang for the buck. Defense budgets have been reduced and will continue to see declines in the intermediate future. This funding reduction means that intelligence community technology users will need to demand increased interoperability, open standards, and control. They will need to be willing to look at new, innovative, and alternative solutions, not the same old ones trotted out by the same old integrators, and they will need to demand alacrity and agility from both providers and the acquisition community, because the rate of growth of the amount of collected data will only continue to increase. The tooling and capability to search, retrieve, and do something with that information needs to catch up quickly.

Published at DZone with permission of Jason Hull, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)