Elastic and Its Approach to Its Search Business
February 16, 2021
This blog post is about Elastic, the Shay Banon information retrieval company, not Amazon AWS Elastic services. Confused yet? The confusion will only increase over time because the “name” Elastic is going to be difficult to keep intact due to Amazon’s ability to erode brand names.
But that’s just one challenge the Elastic search company founded by the magic behind Compass Search. An excellent analysis of Elastic search’s challenges appears in “Elastic Has Stretched the Patience of Many in Open Source. But Is There Room for a Third Way?”
The write up quotes an open source expert as saying:
Let’s be really clear – it’s a move from open to proprietary as a consequence of a failed business model decision…. Elastic should have though their revenue model through up front. By the time the team made the decision to open source their code, the platform economy existed and their decisions to open source ought to
have been aligned to an appropriate business model.
I circled this statement in the article:
Sympathy for Elastic’s position comes from a perhaps unexpected source. Matt Assay, principal at Elastic’s bête noire AWS, believes it’s time to revisit the idea of “shared source”, a licensing scheme originally dreamed up by Microsoft two decades ago as an answer to the then-novel open source concept. In shared source, code is open – as in visible – but its uses are restricted… The heart of the problem is about who gets to profit from open source software. To help resolve that problem, we just might need new licensing.
Information retrieval is not about precision and recall, providing answers to users, or removing confusion about terms and product names — search is about money. Making big bucks from a utility service continues to lure some and smack down others. Now it is time to be squishy and bouncy I suppose.
Stephen E Arnold, February 16, 2021
Open Source Software: The Community Model in 2021
January 25, 2021
I read “Why I Wouldn’t Invest in Open-Source Companies, Even Though I Ran One.” I became interested in open source search when I was assembling the first of three editions of Enterprise Search Report in the early 2000s. I debated whether to include Compass Search, the precursor to Shay Branon’s Elasticsearch reprise. Over the years, I have kept my eye on open source search and retrieval. I prepared a report for an the outfit IDC, which happily published sections of the document and offering my write ups for $3,000 on Amazon. Too bad IDC had no agreement with me, managers who made Daffy Duck look like a model for MBAs, and a keen desire to find a buyer. Ah, the book still resides on one of my back of drives, and it contains a run down of where open source was getting traction. I wrote the report in 2011 before getting the shaft-a-rama from a mid tier consulting firm. Great experience!
The report included a few nuggets which in 2011 not many experts in enterprise search recognized; for instance:
- Large companies were early and enthusiastic adopters of open source search; for example Lucene. Why? Reduce costs and get out of the crazy environment which put Fast Search & Transfer-type executives in prison for violating some rules and regulations. The phrase I heard in some of my interviews was, “We want to get out of the proprietary software handcuffs.” Plus big outfits had plenty of information technology resources to throw at balky open source software.
- Developers saw open source in general and contributing to open source information retrieval projects as a really super duper way to get hired. For example, IBM — an early enthusiast for a search system which mostly worked — used the committers as feedstock. The practice became popular among other outfits as well.
- Venture outfits stuffed with oh-so-technical MBAs realized that consulting services could be wrapped around free software. Sure, there were legal niceties in the open source licenses, but these were not a big deal when Silicon Valley super lawyers were just a text message away.
There were other findings as well, including the initiatives underway to embed open source search, content processing, and related functions into commercial products. Attivio (formed by former super star managers from Fast Search & Transfer), Lucid Works, IBM, and other bright lights adopted open source software to [a] reduce costs, [b] eliminate the R&D required to implement certain new features, and [c] develop expensive, proprietary components, training, and services.
Enterprise Search: Flexible and Stretchy. Er, No.
January 21, 2021
Enterprise search, the utility service, thrills users and information technology professionals alike. There are quite a few search and retrieval vendors chasing revenue. Frankly I have given up trying to keep track of outfits like Luigi’s Box, Yext (yes, enterprise search!), and quite a few repackagers of Lucene; e.g., IBM, Attivio, Voyager Search, and more. There are some proprietary outfits as well.
Then there is the Compass Search sibling Elastic and its Elasticsearch. Open source search makes a great deal of sense to:
- Companies wanting a no cost or low cost way to provide search and retrieval-type functionality to an application
- Penny pinchers who want “the community” to fix bugs so that cash is freed up to lease fancy cars, receive bonuses, and focus on more important software features which can be offered for a fee and a license handcuff
- Competitors who want to provide a familiar environment to those with cash to spend and wave the magic wand of open source in front of young believers who think proprietary software is a crime against humanity.
The history of Elasticsearch and Amazon reaches back to the era when Lucid Works (né Lucid Imagination) lost some staff to Amazon’s Burlingame, California, office. That was the bell which sounded when the Bezos bulldozer decided A9 was not enough. Sure, A9 works but the folks from the Lucene/Solr outfit would map the route from A9 to a more open, folksy world of open source search.
The open source version of Lucene was the beating heart of Elastic, the now public company.
Then Amazon does what Amazon does: The company shifted the bulldozer into gear and went for open source search developers who could seamlessly (sort of) move into the newly blazed path to AWS. Once inside, the fruits of the thousand plus services, features, and functions were just a click away. Policeware vendors, start ups, and some big outfits followed the Bezos bulldozer. The updated IBM slogan reads, “Nobody gets fired for buying AWS.”
Elastic was upset.
“Amazon: NOT OK – Why We Had to Change Elastic Licensing” picks up this story and explains where Elastic fits into the world crafted by the Bezos bulldozer.
The write up explains:
Our license change is aimed at preventing companies from taking our Elasticsearch and Kibana products and providing them directly as a service without collaborating with us.
Elastic’s essay notes:
We think that Amazon’s behavior is inconsistent with the norms and values that are especially important in the open source ecosystem. Our hope is to take our presence in the market and use it to stand up to this now so others don’t face these same issues in the future.
The essay concludes:
I believe in the core values of the Open Source Community: transparency, collaboration, openness. Building great products to the benefit of users across the world. Amazing things have been built and will continue to be built using Elasticsearch and Kibana. And to be clear, this change most likely has zero effect on you, our users. And no effect on our customers that engage with us either in cloud or on premises.
Several observations:
- Commercial behemoths like Amazon use open source the way my neighbor burns firewood, old carpets, and newspapers. The goal is to optimize available cash.
- Amazon’s move into Elastic’s territory began more than five years ago. Amazon does kill off loser products like health and food delivery but it keeps others in tall cotton when it pays off.
- Those completing [a] Amazon certification, [b] partner indoctrination, or [c] inputs from free or low cost Amazon training arrive ready to do the search thing Amazon’s way.
Net net: Beyond Search understands Elastic’s anguish and actions. Perhaps the license shift and the assumptions about open source are unlikely to stand up to the Bezos bulldozer? Open source Elasticsearch is a bargain. It may be tough to compete with free plus discounts for AWS goodies and other Amazon benefits. Legal or illegal, fair or unfair, open source or closed source — the bulldozer grinds forward.
Stephen E Arnold, January 21, 2021
Mobile and Social Media Users: Check Out the Utility of Metadata
January 15, 2021
Policeware vendors once commanded big, big bucks to match a person of interest to a location. Over the last decade prices have come down. Some useful products cost a fraction of the industrial strength, incredibly clumsy tools. If you are thinking about the hassle of manipulating data in IBM or Palantir products, you are in the murky field of prediction. I have not named the products which I think are the winners of this particular race.
Source: https://thepatr10t.github.io/yall-Qaeda/
The focus of this write up is the useful information derived from the deplatformed Parler social media outfit. An enterprising individual named Patri10tic performed the sort of trick which Geofeedia made semi famous. You can check the map placing specific Parler uses in particular locations based on their messages at this link. What’s the time frame? The unusual protest at the US Capitol.
The point of this short post is different. I want to highlight several points:
- Metadata can be more useful than the content of a particular message or voice call
- Metadata can be mapped through time creating a nifty path of an individual’s movements
- Metadata can be cross correlated with other data. (If you attended one of my Amazon policeware lectures, the cross correlation figures prominently.)
- Metadata can be analyzed in more than two dimensions.
To sum up, I want to remind journalists that this type of data detritus has enormous value. That is the reason third parties attempt to bundle data together and provide authorized users with access to them.
What’s this have to do with policeware? From my point of view, almost anyone can replicate what systems costing as much as seven figures a year or more from their laptop at an outdoor table near a coffee shop.
Policeware vendors want to charge a lot. The Parler analysis demonstrates that there are many uses for low or zero cost geo manipulations.
Stephen E Arnold, January 15, 2021
Open Source: Does It Mean What You Think It Means?
January 15, 2021
I spotted an article on Newswire called “Tech Giant Technology Is Open Source for the Pandemic, So Why Does It Feel So Closed?” The awkward title intrigued me. Open means, according to Dictionary.com:
not closed or barred at the time, as a doorway by a door, a window by a sash, or a gateway by a gate:to leave the windows open at night.
(of a door, gate, window sash, or the like) set so as to permit passage through the opening it can be used to close.
Pretty obvious. But open appears to mean closed. The “source” refers to software I assumed.
The write up sets me straight:
“The term ‘open source’ is being applied to the final design of an instrument – and I’m pleased to say there has been a willingness during the pandemic to share these final designs – but the design process itself also needs to be open, something it isn’t now,” explains physics researcher Dr Julian Stirling.
Okay, the “design process” has to be available. To get more insight into this open is closed issue, navigate to the original technical paper at this link. So far the paper is open, but as I have learned, open can be closed and often locked up behind a paywall.
Stephen E Arnold, January 15, 2021
Does Open Source Create Open Doors?
December 21, 2020
Here’s an interesting question I asked on a phone call on Sunday, December 20, 2020: “How many cyber security firms rely on open source software?”
Give up?
As far as my research team has been able to determine, no study is available to us to answer the question. I told the team that based on comments made in presentations, at lectures, and in booth demonstrations at law enforcement and intelligence conferences, most of the firms do. Whether it is a utility function like Elasticsearch or a component (code or library) that detects malicious traffic, open source is the go-to source.
The reasons are not far to seek and include:
- Grabbing open source code is easy
- Open source software is usually less costly than a proprietary commercial tool
- Licensing allows some fancy dancing
- Using what’s readily available and maintained by a magical community of one, two or three people is quick
- Assuming that the open source code is “safe”; that is, not malicious.
My question was prompted after I read “How US Agencies’ Trust in Untested Software Opened the Door to Hackers.” The write up states:
The federal government conducts only cursory security inspections of the software it buys from private companies for a wide range of activities, from managing databases to operating internal chat applications.
That write up ignores the open source components commercial cyber security firms use. The reason many of the services look and function in a similar manner is due to a reliance on open source methods as well as the nine or 10 work horse algorithms taught in university engineering programs.
What’s the result? A SolarWinds type of challenge. No one knows the scope, no one knows the optimal remediation path, and no one knows how many vulnerabilities exist and are actively being exploited.
Here’s another question, “How many of the whiz kids working in US government agencies communicate the exact process for selecting, vetting, and implementing open source components directly (via 18f type projects) or from vendors of proprietary cyber security software?”
Stephen E Arnold, December 21, 2020
Fess Up: Elasticsearch Is a Threat to Proprietary Search and Retrieval
December 1, 2020
We have been poking around the world of Elasticsearch-based information retrieval systems. There are some interesting plays; that is, entrepreneurs use Elasticsearch (Shay Banon’s open source system) as a platform.
Fess provides Elasticsearch for personal use, although one can employ the system for an organization. The system is:
Fess is Elasticsearch-based search server, but knowledge/experience about Elasticsearch is NOT needed because of All-in-One Enterprise Search Server. Fess provides Administration GUI to configure the system on your browser. Fess also contains a crawler, which can crawl documents on Web/File System/DB and support many file formats, such as MS Office, pdf and zip.
Fess became available in 2019. The CEO of the N2SM, Inc. company is Masaharu Manabe. Demonstrations and links to the code are available at this link. A fee-based version of the software is provided under the name N2 Search. More information about the for fee version is here. A discussion forum is available at this link.
Observation: The Elasticsearch ecosystem is providing alternatives to the proprietary search systems. Beyond Search thinks that some vendors of proprietary search software are likely to be see Elasticsearch as digital kudzu. Good news or bad news for the Coveos, Fabasofts, and Microsoft Fast type folks? That’s a question some of these types of vendors stakeholders may be asking as they beat the bushes for deals in customer service, chatbots, business intelligence, and smart software services.
Stephen E Arnold, December 1, 2020
Elastic: The Add Value to Open Source Outfit Bounces Along
November 25, 2020
Elastic Adds New Features to Enterprise Search, Observability, and Security Solutions
Search and data-management firm Elastic has some new features to crow about. BusinessWire posts “Elastic Announces Innovations Across its Solutions to Optimize Search and Enhance Performance and Monitoring Capabilities.” One new tool is Kibana Lens, a visual data analysis tool with a drag-and-drop interface described as intuitive. There is also a beta launch of the searchable snapshots, an efficient way to manage data storage tiers with searchable snapshots. The press release tells us:
“New expanded Elastic Observability features, including user experience monitoring and synthetics, give developers new tools to test, measure, and optimize end-user website experiences. The launch of a new dedicated User Experience app in Kibana provides Elastic customers with an enhanced view and understanding of how end users experience their websites. In addition, Elastic customers can use the new user experience monitoring feature to review Core Web Vitals, helping website developers interpret digital experience signals. Elastic users can also leverage a dev preview release of synthetic monitoring in Elastic Uptime to simulate complex user flows, measure performance, and optimize new interaction paths without impact to a website’s end users. The combination of these two new observability features gives Elastic customers a deeper view of their customers’ digital experience before and after a site update is deployed.”
See the write-up for its list of specific updates and features to Elastic’s Enterprise Search, Observability, Security, Stack, and Cloud products. Built around open source software, the company prides itself on its user-friendly products that have been adopted by major organizations around the world, from Cisco to Verizon. Elastic began as Elasticsearch Inc. in 2012, simplified its name in 2015, and went public in 2018. The company is based in Mountain View, California, and maintains offices around the world.
Cynthia Murrell, November 25, 2020
Court Case Hunger? Judyrecords Is Available
November 24, 2020
Unable to pay the fee for LexisNexis-type commercial search systems? You are not alone. If you want information from court records, navigate to Judyrecords. Within the last couple of months, the system has added more than 35 million cases. Aren’t these data available for free elsewhere? Sure, if you like going through hoops like verification procedures. Judyrecords lets a user plug in the names of entities and view results. I ran one of my go to queries: “Palantir IBM.” Here are the results:
This may not be important to you, but for those who have to wade through for fee legal search systems, Judyrecords is helpful. But for how long? Yes, that is a good question. For now, however, give it a whirl. Keep in mind that US court systems without online technology or special arrangements for document access prevent the system from being comprehensive. Lawyers enjoy results which must be checked by billable professionals, however.
Stephen E Arnold, November 23, 2020
Open Source Kumbaya in 2020: Pay Me for Support
November 16, 2020
I read “No, Open Source Does Not Mean Includes Free Support.” The write up illustrates one small change in the open source community in the last five or six years. With more and more organizations using open source software as the engine for their “platform” or “system”, individuals who create open source software are shifting. For example, when we worked on the Lucene Revolution conference years ago, there was a lot of talk about the community, the FOSS spirit, the desire to break free of the chains proprietary software vendors locked to licensees, etc.
Compare that kumbaya approach to this statement in the write up:
Don’t get me wrong. I’m happy to help. Selling support is what keeps the lights on here (did I mention the cost of running a web server?). But coming to me under false pretense and/or expecting that I must provide free service on top of a software I gave away without charge is not going to win you any favors. It stops being free, when it starts costing me! My time is valuable. If you want a piece of it, I want money in return. Period.
This is an excellent point. In my own experience, we know that some high profile products would not exist without open source software. In fact, some vendors do not reveal the extent of their dependence on software which can be downloaded and used without providing so much as an email address, let alone a credit card.
Net net: 2020 may become the year in which open source kumbaya is replaced with a different ethos. Come by here but bring a way to pay.
Stephen E Arnold, November 16, 2020
-
- Subscribe to Beyond Search
Feature archive
News archive
-
