Beyond Search: A New Study

For the last two years, it’s become increasingly apparent to me that enterprise search has been marginalized. The Big Five in 2003 when I started work on The Enteprise Search Report were Autonomy, Convera, Endeca, Fast Search & Transfer, and Verity.

As I write this, Autonomy acquired Verity. Convera sold its behind-the-firewall search business to Fast Search & Transfer while Autonomy hired some Convera sales professionals. Microsoft is paying $1.2 billion for Fast Search & Transfer. Endeca received an injection of cash from Intel and SAP. The blue-chip vendors are in transition. With many Fortune 1000 companies have five or more behind-the-firewall search systems, Web site search, database search, and goodness knows how many departmental search systems – the traditional search environment has its own “global warming” underway.

As I worked on Google Version 2.0 (Infonortics, 2007), I had solid open source data in front of me that Google was investing significant resources in search technology that was designed to break free of the confines of the “search box.” My own research and studies conducted for various clients made it clear that in organizations, search was a less-than-gratifying service. In one study, conducted for a large U.S. entity, about two-thirds of the search system’s users were somewhat dissatisified or dissatisfied with search.

What took shape in my mind as I was finishing Google Version 2.0 was a Cliff’s Notes or a Baedeker to help a person with a broken search system to fix that system. I set out to boil down the information I had gathered on dozens of systems and numerous search-related projects into a “how to” with a minimum of technical jargon. I wanted to present information that was muscle, not fat.

Beyond Search is about 250 pages in length, contains dozens of illustrations and screen shots, and contains information not available in any other single publication at this time. Tony Safina (an ace researcher), Don Anderson (code surfer), Stuart Schram IV (tech wizard), Jessica Bratcher (another ace researcher), and I have squeezed my files, search gurus, and vendors for information.

Stephen Arnold boxes up some of the paper information collected in the course of the research for Beyond Search. Despite ArnoldIT’s effort to go paperless, work on a study of the complexity and scope of this project involves hard copies and old-fashioned cardboard storage boxes.

A public Web log is available here. I will post information that supplements the distilled information in this new study. There’s no charge for the information on the Web log, and you can visit the Web log without any obligation to buy the study. My goal in creating a Web log called Beyond Search was to make certain that details and ancillary information are not “lost.” Some of the information I track, develop, and catalog have not previously been available unless someone hired ArnoldIT on a for-fee basis. Now, I’m shifting gears and want to help you get the maximum mileage out of your investment in search and retrieval.

What’s In Beyond Search?

The book has an easy-to-follow structure. There’s a preface that explains more about my conclusion that search is broken. I introduce a new term to replace the now shop worn phrase enterprise search. I use the phrase “behind-the-firewall search.” There’s no implication in behind-the-firewall search of the foolish assertion that a vendor can index “all” of an organization’s content. Behind-the-firewall search means that a user can access certain information to which that person has access from a desktop server, a mobile phone, or a wireless notebook., Implicit in the phrase is the notion that a firewall delimits those who can access information and those who may not. Forget the categorical affirmatives, Beyond Search focuses on real-life situations, not marketing hype puffed like a hot air balloon.

You will find the following information in this new study, which will be published by Frank Gilbane, The Gilbane Group:

1. Setting the Stage – Seven known issues with basic search and retrieval in organizations
2. Fixing a Search System – The three basic remediation options are presented and discussed
3. Avoiding Pitfalls – More than a dozen problems are identified and solutions outlined
4. Moving beyond Key Words – An overview of the next-generation technologies and selected vendors’ use of these technologies so you have a bird’s-eye view of what’s available. I try to make clear what’s meaningful in semantic techniques, taxonomies, and other jargon.
5. Google and Dataspaces – A look at Google’s leap-frogging approach to behind-the-firewall search
6. The Beyond Search Market Space – A discussion of the new search-and-retrieval initiatives, including a market map to help you sort out the confusing terminology and a merging of these sectors with key players in each sector. This section includes a bird’s-eye review of the 24 vendors’, their strengths, weaknesses, and an Arnold “net-net” one-line assessment of each company’s technology
7. Profiles – Beyond Search includes short, technically-informed, and direct profiles of two dozen vendors. These vendors range from innovators in St. Louis, Missouri with a system able to process hundreds of megabytes of content every two or three minutes to a cutting-edge semantics engine developed in Spain. You will learn which of these companies’ can integrate with an existing system and which can make a replacement fast, economical, and hassle-free. I’ve looked at more than 50 systems as well as reviewing those in the third edition of my Enterprise Search Report. In this new study, I’ve chosen to profile vendors who point the direction to the world outside of the prison of the search box. You will find very brief passing references to the now-familiar search systems, but the effort has gone into screening, analyzing, and profiling the most innovative and technically-interesting systems now in the commercial channel. You will need to know about these companies if you expect to remediate or rip and replace your existing search system.
8. Glossary – The terminology for next-generation search is muddled, confusing, and fluid. The glossary in this study provides the common industry jargon and a plain English explanation. You can de-fog almost any vendor’s marketing collateral with this word list.

Who Should Buy This Study?

This study has been written for a procurement team trying to figure out what options are available to get an ailing search system back on track. The study provides useful insight into an important change taking place in search. Consultants, analysts, entrepreneurs interested in search, and intelligence professionals will find the study useful as well. Academics will find this study long on “real world” information and very short on the sort of information that Ph.D.. candidates find thrilling. I’m no scholar. I’m a person who power levels. (If you don’t know what that means, check out my Web log for an explanation and a picture.)

Where Can I Get This Study?

The study is available from The Gilbane Group. You can get additional information here. Alternatively, you can write The Gilbane Group at beyondsearch at gilbane.com.

When Will It Be Available?

I’ve finished the research and the writing. The Gilbane Group has the study in production at this time. I anticipate that you will be able to get a copy in April 2008.

Stephen Arnold, February 21, 2008