An Enterprise Search Mini Case: LucidWorks and Its Accelerating Mission
June 11, 2015
I read “Lucidworks Accelerates Product Focused Mission with Major Fusion Upgrades.” LucidWorks (Really?)—né Lucid Imagination—appears to be working on products. (Note that the company names appears in different ways: “Lucidworks” with variants “LucidWorks”, “Lucid Works,” and “lucidworks”.)
Lucidworks wants to accelerate its mission. Will this be a quick and easy task?
Flashback in time. Lucid Imagination was founded in 2007. You can read about the vision of the company in interviews with these Lucid (no pun intended) executives:
- Marc Krellenstein, formerly Northern Light and one of the founders of Lucid Imagination, March 17, 2009
- Brian Pinkerton, formerly, December 21, 2010, possibly Amazon?
- Paul Doscher, formerly with Exalead, April 16, 2012
- Miles Kehoe, formerly New Idea Engineering, January 29, 2013, now a consultant
- Mark Bennett, formerly New Idea Engineering, March 4, 2014
These interviews make clear the difficult journey that Lucid Imagination took. (What is interesting is that Lucid’s principal competitor was Elasticsearch, now Elastic. That company came from obscurity to the go-to provider of open source search. To be fair, Shay Bannon, founder of Elastic, had compiled considerable experience with the Compass open source search system.)
Why did I cover Lucid in five interviews?
The reason is that open source search appeared to be the salve to soothe the wounds inflicted by proprietary search system vendors. Satisfaction with search was declining. Users were disaffected with high profile proprietary brands. The community approach addressed, in part, the brutal research, development, and customer support costs which search drags to each meeting with stakeholders.
Lucid had a lead; Elastic benefited. Lucid seeks a focus; Elastic is serving customers. Lucid would be an excellent business school case study, ranking at the top along with the Hewlett Packard Autonomy search situation and the Fast Search criminal charges matter. That is rarified case study company.
In the interviews cited above, it is clear that Lucid embraced Solr and made an attempt to emulate the full featured approach to content processing exemplified by Autonomy and Fast Search & Transfer. Elastic, on the other hand, took a more direct approach, relying on Lucene for the heavy lifting, and narrowing its focus to tools which were almost utilitarian. Want to search a log file? Go with Elastic.
The other key difference is the lack of managerial drama at Elastic. Elastic’s management team appears, at least to this observer in Kentucky, as stable. Lucid, on the other hand, has seen the departure of founders early in the company’s history. Presidents arrived and departed. Marketers appeared and disappeared. Major committers joined and then jumped ship; for example, Brian Pinkerton ended up at Amazon, working on its search product. Yonik Seeley also left to start his own search company Heliosearch. Dr. Krellenstein went from strong supported of Lucid to a disaffected founded. He quit.
As recently as September 2014, Lucid Works featured in “Trouble at LucidWorks: Lawsuits, Lost Deals, and Layoffs Plague the Search Startup Despite Funding.” The headline makes several points. First, LucidWorks has ingested more than $40 million, which puts it on a par with Attivio and Coveo in the money department. But Elastic garnered about $70 million at about the same time. The headline also reveals the disjunctions among managers, regardless of which president was on watch. And, the headline focuses on the point that it is a search vendor, which is not in my opinion a particularly magnetic positioning for software.
According to the “Trouble at LucidWorks” article The Guardian and Nordstrom’s abandoned Lucid’s software. The less than flattering Venture Beat story added:
The situation seems to have worsened following shakeups in the sales team, leaving young salespeople inexperienced in the enterprise-software game trying to win deals. “I don’t think any of the sales team hits (their) number except one guy,” said a former employee. And that one guy has resorted to “dropping his pants,” as the sales expression goes, promising to significantly chop the price of a service if his lead commits to buying right away, a different former employee said. The sales goals aren’t increasing. The revenue target for the year is $12 million, right in line with last year, that former employee said. And it doesn’t help that LucidWorks has fumbled with partnerships it was trying to get in place. It was working on alliances with Amazon Web Services, Intel, and Splunk, one former employee told VentureBeat. “Will [Hayes] imploded that with comments he made in the final agreement,” that former employee said of one partnership. And after Hayes stepped up as chief executive in June, he’s laid off people in marketing, sales, and business development. On the technology side of the company, meanwhile, employees have missed deadlines for shipping software to customers, month after month, another former employee said. Outside the office, the company has other distractions — in court, to be exact. Mike Moody, a former senior vice president of engineering at LucidWorks who was terminated in December, sued LucidWorks and certain executives in February for unlawful termination, according to documents submitted to the U.S. District Court for the Northern District of California. LucidWorks is also ensnared in a case it filed against Seeley, one of its founders, in the Superior Court of California, San Mateo County. “This is a case about double-dealing on an employer, which arises from the secretive founding and launching of the company Heliosearch by Yonik Seeley before his resignation from his former employer LucidWorks in October 2013,” the complaint begins. “Unknown to LucidWorks, while Seeley was still employed by LucidWorks, he simultaneously was working directly against LucidWorks’ interests by developing and promoting his new venture Heliosearch as a competing alternative to LucidWorks.”
Jury Is Still Out on Microsoft Delve
June 11, 2015
Sometimes hailed as Pinterest for the enterprise, Microsoft Delve is a combination of search, social, and machine learning, which produces an information hub of sorts. Delve is also becoming a test subject, as enterprise experts decide whether such offerings intrude into users’ workflow, or enhance productivity. Read more in the Search Content Management article, “Microsoft Delve May Drive Demand for Office365.”
The article summarizes the issue:
“As Microsoft advances further in its mobile-first, cloud-first strategy, new offerings such as Microsoft Delve are piquing companies’ curiosity but also raising eyebrows. Many companies will have to gauge whether services like Delve can enhance worker productivity or run the risk of being overly intrusive.”
As SharePoint unveils more about its SharePoint Server 2016, more will become known about how it functions along with all of its parts, including Delve. It will be up to the users to determine how efficient the new offerings will be, and whether they help or hinder a regular workflow. Until the latest versions become available for public release, stay tuned to ArnoldIT.com for the latest news regarding SharePoint and how it may affect your organization. Stephen E. Arnold is a longtime leader in search and his work on SharePoint is a great go-to resource for users and managers alike.
Emily Rae Aldridge, June 11, 2015
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Enterprise Search Excitement: HP Autonomy Litigation News
June 10, 2015
It is not the weekend and there is some minor Hewlett Packard Autonomy litigation news. I read “HP to Pay $100 Million to Settle Case Tied to Autonomy Deal.” The write up reports that HP will pay $100 million to a settlement fund. In the words of the write up, the money is
to resolve a lawsuit stemming from an impairment charge HP took after paying $10 billion for the British company. The money will go to people who bought HP shares between Aug. 19, 2011 and Nov. 20, 2012.
According to write up, the litigation “has no merit.” If so, $100 million is a hefty chunk for something that is fairy dust.
One consequence of the HP Autonomy dust up is that enterprise search vendors are using some artful metaphors to describe their systems’ capabilities. With the Fast Search problem in Norway and the HP Autonomy issue in the US and the UK, enterprise search vendors have certain made me aware of the consequences of having a disagreement over business models and accounting.
Which enterprise search vendor is next in line to make headlines. I have heard that the Lexmark search push has resulted in some frowns and indigestion. Vivisimo has disappeared into the maelstrom of IBM and its software. Oracle is sending what I interpret as mixed signals about the benefits of its hat trick in search: Endeca, InQuira, and RightNow.
My view is that it is tough be a search vendor looking for traction using words like customer support and business intelligence. Worth watching the HP Autonomy imbroglio and the wordsmithing of vendors trying to sidestep the shockwave of high profile search vendor legal activities.
Stephen E Arnold, June 9, 2015
SharePoint: Enterprise Search Which Will Never Ever Let You Lose Anything Again
May 30, 2015
Bold assertion. I read “Why Using Microsoft SharePoint Will Improve Your Business Performance with a Simple Search Feature.” Memorable for several reasons:
- SharePoint has “amazing search capabilities.” (I mistakenly understood that the “new” SharePoint search was not yet available. Oh, well, I am in Harrod’s Creek, not a “nice venue in London.” Search is better when viewed from a “nice venue” I assume.
- I will never lose anything again. I assume, perhaps incorrectly, that the “anything” refers to a document I created and either parked intentionally or had parked for me by Microsoft’s “amazing” SharePoint. I note that the statement is a categorical, and then often present logical challenges to someone who asks, “Really? What’s the evidence you have to back up this wild and frisky claim?”
- I note that I can type a word or phrase to “surface every relevant document across all of the sites I have access to.” The author adds, “It’s brilliant.” Okay, got it, but I don’t believe it based on observation, our own hands on experiences, and the weed pile of third party vendors who insist their software actually makes SharePoint usable. I would list them, but you probably have these outfits’ burned into your memory.
What is interesting is that the focus of the write up seems to be Microsoft Dynamics GP. It is mentioned a couple of time. There are also references to Delve, another Microsoft search system.
Frankly I am not sure if the cheerleading for “brilliant” search is credible. We have worked on projects in organizations where SharePoint is the “pluming.” In a conference call last week, the client, a relatively large outfit in the Fortune 100, reported these “issues” with SharePoint:
- Users cannot locate documents created within 24 hours and written to the designated SharePoint device
- Documents in a results list do not include the version of the document for which the user searches
- Images of purchase orders for a company issued with a unique code cannot be retrieved
- Queries take more time than a Google query to complete
- The information about employees with specific expertise is not complete; that is, there will be no data about education or certain projects
- Collaboration is flakey
- The system crashes.
I could work through the list, but the point is that SharePoint is big business for those who get a job to maintain it and, in theory, make it work. SharePoint is the fertile field in which third party vendors plant applications to improve on what Microsoft offers. There are integrators who have specialized skills and want SharePoint to remain the money tree plantation the consultants have come to call home.
In short, what can one believe about Microsoft search? Delve into that.
Stephen E Arnold, May 30, 2015
Stephen E Arnold, June 2, 2015
X1 Search: A Unified Single Pane of Glass
May 26, 2015
I read “X1’s Microsoft Enterprise Search Strategy: Better Than Microsoft’s?”
Here’s the passage I noted:
Providing one single pane of glass to a business worker’s most critical information assets is key. Requiring end-users to search Outlook for email in one interface, then log into another to search SharePoint, and then another to search for document and OneDrive is a non-starter. A single interface to search for information, no matter where it lives fits the workflow that business workers require.
The write up points out that X1 starts with an “end user’s email and files.” That’s fine, but there are other data types to which an end user requires access.
My reaction was these questions and the answers thereto:
- What about video?
- What about drafts of financial data or patent applications and other content centric documents in perpetual draft form?
- What about images?
- What about third party content downloaded by a user to a local or shared drive?
- What about Excel files used as text documents and Excel documents with data and generic column names?
- What about versions?
- What about time and data flags versus the time and date information within a content object?
- What about SMS messages?
- What information is related to other information; for example, an offer of employment to a former employee?
- What about employee health, salary, and performance information?
- What about intercepted data from watched insiders using NGIA tools?
- What about geo-plotted results based on inputs from the organization’s tracking devices on delivery vans and similar geo systems?
My point is that SharePoint represents a huge market to search and content processing vendors. The generalizations about what a third party system can do boggle my mind. Vendors as a rule do not focus on the content issues my questions probe. There are good reasons for the emphasis on email and experiences. Tackling substantive findability issues within an organization is just not what most SharePoint search alternatives do.
Not surprisingly, for certain types of use cases, SharePoint search remains a bit of a challenge regardless what system is deployed into a somewhat chaotic sea of code, functions, and components.
A unified single pane of glass is redundant. Solutions to the challenges of SharePoint may deserve this type of remediation because vendors have been tilting at the SharePoint windmill in a highly repetitive way for more than a decade. And to what end? For many, SharePoint information access remains opaque, cloudy, and dark.
Stephen E Arnold, May 26, 2015
Welcome YottaSearch
May 26, 2015
There is another game player in the world of enterprise search: Yotta Data Technologies announced their newest product: “Yotta Data Technologies Announces Enterprise Search And Big Data Analytics Platform.” Yotta Data Technologies is known for its affordable and easy to use information management solutions. Yotta has increased its solutions by creating YottaSearch, a data analytics and search platform designed to be a data hub for organizations.
“YottaSearch brings together the most powerful and agile open source technologies available to enable today’s demanding users to easily collect data, search it, analyze it and create rich visualizations in real time. From social media and email for Information Governance and eDiscovery to web and network server logs for Information Technology Operations Analytics (ITOA), YottaSearch™ provides the Big Data Analytics for users to derive information intelligence that may be critical to a project, case, business unit or market.”
YottaSearch uses the popular SaaS model and offers users not only data analytics and search, but also knowledge management, information governance, eDiscovery, and IT operations analytics. Yotta decided to create YottaSearch to earn revenue from the burgeoning big data market, especially the enterprise search end.
The market is worth $1.7 billion, so Yotta has a lot of competition, but if they offer something different and better than their rivals they stand a chance to rise to the top.
Whitney Grace, May 26, 2015
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Search 2020: Peering into the Future of Information Access
May 22, 2015
The shift in search, user behaviors, and marketing are transforming bread-and-butter keyword search. Quite to my surprise, one of my two or three readers wrote to one of the goslings with a request. In a nutshell, the reader wanted my view of a write up which appeared in the TDWI online publication. TDWI, according to the Web site, is “your source for in depth education and research on all things data.” Okay, I can related to a categorical affirmative, education, research, and data.
The article has a title which tickles my poobah bone: “The Future of Search.” The poobah bone is the part of the anatomy which emits signals about the future. I look at a new search system based on Lucene and other open source technology. My poobah bone tingles. Lots of folks have poobah bones, but these constructs of nerves and tissues are most highly developed in entrepreneurs who invent new ways to locate information, venture capitalists who seek the next Google, and managers who are hired to convert information access into billions and billions of dollars in organic revenue.
The write up identifies three predictions about drivers on the information retrieval utility access road:
- Big Data
- Cloud infrastructure
- Analytics.
Nothing unfamiliar in these three items. Each shares a common characteristic: None has a definition which can be explained in a clear concise way. These are the coat hooks in the search marketers’ cloakroom. Arguments and sales pitches are placed on these hooks because each connotes a “new” way to perform certain enterprise computer processes.
But what about these drivers: Mobile access, just-in-time temporary/contract workers, short attention spans of many “workers”, video, images, and real time information requirements? Perhaps these are subsets of the Big Data, cloud, and analytics generalities, but maybe, just maybe, could these realities be depleted uranium warheads when it comes to information access?
These are the present. What is the future? Here’s a passage I highlighted:
Enterprise search in 2020 will work much differently than it does today. Apple’s Siri, IBM’s Watson, and Microsoft’s Cortana have shown the world how enterprise search and text analytics can combine to serve as a personal assistant. Enterprise search will continue to evolve from being your personal assistant to being your personal advisor.
How are these systems actually working in noisy automobiles or in the kitchen?
I know that the vendors I profiled in CyberOSINT: Next Generation Information Access are installing systems which perform this type of content processing. The problem is that search, as I point out in CyberOSINT, is that the function is, at best, a utility. The heavy lifting comes from collection, automated content processing, and various output options. One of the most promising is to deliver specific types of outputs to both humans and to other systems.
The future does tailor information to a person or to a unit. Organizations are composed of teams of teams, a concept now getting a bit more attention. The idea is not a new one. What is important is that next generation information access systems operate in a more nuanced manner than a list of results from a Lucene based search query.
The article veers into a interesting high school teacher type application of Microsoft’s spelling and grammar checker. The article suggests that the future of search will be to alert the system user his or her “tone” is inappropriate. Well, maybe. I turn off these inputs from software.
The future of search involves privacy issues which have to be “worked out.” No, privacy issues have been worked out via comprehensive, automated collection. The issue is how quickly organizations will make use of the features automated collection and real time processing deliver. Want to eliminate the risk of insider trading? Want to identify bad actors in an organization? One can, but this is not a search function. This is an NGIA function.
The write up touches on a few of the dozens of issues implicit in the emergence of next generation information access systems. But NGIA is not search. NGIA systems are a logical consequence of the failures of enterprise search. These failures are not addressed with generalizations. NGIA systems, while not perfect, move beyond the failures, disappointments, and constant legal hassles search vendors have created in the last 40 years.
My question, “What is taking so long?”
Stephen E Arnold, May 22, 2015
Yotta Search: A Full Service Solution
May 17, 2015
I spoke to a colleague who asked me about Yotta Search. I dug through my Overflight files and located a write up about the new enterprise search system from Yotta Data Technologies and a company called Yotta Customer Analytics. One Yotta is in Cleveland. The other is in Silicon Valley. Both are in the analytics game.
A “yotta” is a whole lotta data, the biggest unit of data. I wonder if the company has a comment on a set of yottas?
I checked my files for the company offering Yotta search, based in Cleveland, home of EPI Thunderstone, another enterprise search vendor. The company behind Yotta Search is Yotta Data Technologies.
According the firm’s Web site at www.yottadatatechnologies.com:
Yotta Data Technologies (YDT) is a technology company built on a foundation of deep industry experience and driven by a passion for innovative excellence. We provide data management and information governance solutions to corporations, firms and agencies, whether they be a small local firm or a multinational corporation with offices around the globe. Each of our platforms maintains the high levels of quality, performance and security that are critical within information governance initiatives and any data management project.
The search system appears to be based on open source technology if I understand this Web site information:
Yotta Search is a versatile enterprise search solution being developed by Yotta Data Technologies (YDT) for teams, small to medium sized businesses and large corporations. Yotta Search provides powerful, fast and flexible technology that is not only well beyond full text search, but also powers the search and analysis features of many of the world’s largest internet sites and data platforms.
The operative phrase is “being developed.” The company asserts capabilities in these functions:
- Business intelligence
- Discovery
- Information governance
- Virtual data rooms.
I noticed a news item called “Yotta Data Technologies Announces Enterprise Search and Big Data Analytics Platform.” If the information is correct, Yotta is no longer “being developed,” one can license the system. The url provided is www.yottasearch.info. The story describes the Yotta search system in this way.
YottaSearch is easy – and budget friendly – to implement with a cloud-based, Software-as-a-Service (SaaS) delivery model and a disruptive, subscription-based pricing model.
Key Functionality of the YottaSearch
- Data Point Connectors – Local, Network, Email, Enterprise Systems, Databases, Social Media
- File Crawlers – Detects & Parses over 1,000 file types
- File Indexer – Language Detection, Deduplication, Near Real Time, Distributed, Scalable
- Advanced Search Engines – Based on the high performance Apache Lucene library
- Data Analytics – Intelligent analysis of structured and unstructured data
- Dynamic Dashboards – Explore, analyze, navigate and define large volumes of complex data.
The system can be used for a number of applications, according to the write up:
- Enterprise Search and Analytics
- Information Governance
- IT Operations Analytics (ITOA)
- Investigations & eDiscovery
- Knowledge Management (KM)
- Internet of Things (IoT), Event & Log Data Analysis
Also, Yotta offers global data services and global electronic discovery services. The company’s tag line is “Information intelligence for corporations, firms, and agencies.”
Like I said, a lotta yottas and a robust line up of functionality which some more established search and content processing systems do not possess. Is Yotta competing with Elastic or is Yotta competing with the ABC vendors: Attivio, BA Insight, or Coveo? Worth watching.
Stephen E Arnold, May 17, 2015
Quote to Note: How to Make Search Relevant
May 16, 2015
Short honk: I read “Intranet Search? Sssh! Don’t Speak of It.” It seems that enterprise search is struggling and sweeping generalizations about information governance and knowledge management are not helping the situation. But that’s just my opinion.
But set that “issue” aside. Here’s the quote I noted:
The only way this situation [search is a problem’] will change is with intranet managers stepping up to the challenge and telling stories internally. The problem with search analytics (even if you do everything that Lou Rosenfeld [search wizard] recommends) is that there is no direct evidence of the day-to-day impact of search.
Will accountants respond to search stories? Why is there no direct evident of the day to day impact of search? Perhaps search, along with some other hoo hah endeavors, is simply not relevant in today’s business environment? Won’t more hyperbole filled marketing solve the problem? Another conference?
The wet blanket on enterprise search remains “there is no direct evidence of the day to day impact of search.” After 30 or 40 years of implementations and hundreds of millions in search development, why not? Er, what about this thought:
Search is a low value utility which has been over hyped.
Stephen E Arnold, May 17, 2015
Elasticsearch Transparent about Failed Jepsen Tests
May 11, 2015
The article on Aphyr titled Call Me Maybe: Elasticsearch 1.5.0 demonstrates the ongoing tendency for Elasticsearch to lose data during network partitions. The author goes through several scenarios and found that users can lose documents if nodes crash, a primary pauses, a network partitions into two intersecting components or into two discrete components. The article explains,
“My recommendations for Elasticsearch users are unchanged: store your data in a database with better safety guarantees, and continuously upsert every document from that database into Elasticsearch. If your search engine is missing a few documents for a day, it’s not a big deal; they’ll be reinserted on the next run and appear in subsequent searches. Not using Elasticsearch as a system of record also insulates you from having to worry about ES downtime during elections.”
The article praises Elasticsearch for their internal approach to documenting the problems, and especially the page they opened in September going into detail on resiliency. The page clarifies the question among users as to what it meant that the ticket closed. The page states pretty clearly that ES failed their Jepsen tests. The article exhorts other vendors to follow a similar regimen of supplying such information to users.
Chelsea Kerwin, May 11, 2014
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
-
- Subscribe to Beyond Search
Feature archive
News archive
-
