Semantic Search Craziness Makes Search Increasingly Difficult
April 3, 2016
How is that for a statement? Search is getting hard. No, search is becoming impossible.
For evidence, I point to the Search Today and Beyond: Optimizing for the Semantic Web Wired Magazine article “Search Today and Beyond: Optimizing for the Semantic Web.”
Here’s a passage I noted:
Despite the billions and billions of searches, Google reports that 20 percent of all searches in 2012 were new. It seems quite staggering, but it’s a product of the semantic search rather than the simple keyword search.
Wow, unique queries. How annoying? Isn’t it better for people to just run queries which Google has seen and cached the results?
I have been poking around for information about a US government program called “DCGS.” Enter the query and what do you get? A number of results unrelated to the terms in my query; for example, US Army. Toss in quotes to “tell” Google to focus only on the string DCGS. Nah, does not do the job. Add the filetype:ppt operator and what do you get, documents in other formats too.
Semantic search is now a buzzword which is designed to obfuscate one important point: Methods for providing on point information are less important than assertions about what jargon can deliver.
For me, when I enter a query, I want the search system to deliver documents in which the key words appear. I want an option to see related documents. I do not want the search system doing the ad thing, the cheapest and most economical query, and I don’t want unexpected behaviors from a search and retrieval system.
Unfortunately lots of folks, including Wired Magazine, this that semantic search optimizes. Wonderful. With baloney like this I am not sure about the future of search; to wit:
…the future possibilities are endless for those who are studious enough to keep pace and agile enough to adjust.
Yeah, agile. What happened to the craziness that search is the new interface to Big Data? Right, agile.
Stephen E Arnold, April 3, 2016
Secure Email on the Dark Web
April 1, 2016
Venturing safely onto the Dark Web can require some planning. To that end, FreedomHacker shares a “List of Secure Dark Web Email Providers in 2016.” The danger with Tor-accessible email providers, explains reporter Brandon Stosh, lies in shady third parties. He writes:
“It’s not that finding secure communications on Tor is a struggle, but it’s hard to find private lines not run by a rogue entity. Below we have organized a list of secure dark web email providers. Please remember that no email provider should ever be deemed secure, meaning always use encryption and keep your opsec to its highest level….
“Below we have listed emails that are not only secure but utilize no type of third-party services, including any type of hidden Google scripts, fonts or trackers. In the list below we have gone ahead and pasted the full .onion domain for verification and added a link to any services who also offer a clearweb portal. However, all communications sent through clearweb domains should be presumed insecure unless properly encrypted, then still it’s questionable.”
The list of providers includes 10 entries, and Stosh supplies a description of each of the top five: Sigaint, Rugged Inbox, Torbox, Bitmessage, and Mail2Tor; see the article for these details, and to view the other five contenders. Stosh wraps up by emphasizing how important email security is, considering all the sensitive stuff most of us have in our inboxes. Good point.
Cynthia Murrell, April 1, 2016
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Netflix Algorithm Defaults To “White” Content, Sweeps Diversity Under the Rug
April 1, 2016
The article Marie Claire titled Blackflix; How Netflix’s Algorithm Exposes Technology’s Racial Bias, delves into the racial ramifications of Netflix’s much-lauded content recommendation algorithm. Many users may have had strange realizations about themselves or their preferences due to collisions with the system that the article calls “uncannily spot-on.” To sum it up: Netflix is really good at showing us what we want to watch, but only based on what we have already watched. When it comes to race, sexuality, even feminism (how many movies have I watched in the category “Movies With a Strong Female Lead?”), Netflix stays on course by only showing you similarly diverse films to what you have already selected. The article states,
“Or perhaps I could see the underlying problem, not in what we’re being shown, but in what we’re not being shown. I could see the fact that it’s not until you express specific interest in “black” content that you see how much of it Netflix has to offer. I could see the fact that to the new viewer, whose preferences aren’t yet logged and tracked by Netflix’s algorithm, “black” movies and shows are, for the most part, hidden from view.”
This sort of “default” suggests quite a lot about what Netflix has decided to put forward as normal or inoffensive content. To be fair, they do stress the importance of logging preferences from the initial sign up, but there is something annoying about the idea that there are people who can live in a bubble of straight, white, (or black and white) content. There are among those people some who might really enjoy and appreciate a powerful and relevant film like Fruitvale Station. If it wants to stay current, Netflix needs to show more appreciation or even awareness of its technical bias.
Chelsea Kerwin, April 1, 2016
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Patents and Semantic Search: No Good, No Good
March 31, 2016
I have been working on a profile of Palantir (open source information only, however) for my forthcoming Dark Web Notebook. I bumbled into a video from an outfit called ClearstoneIP. I noted that ClearstoneIP’s video showed how one could select from a classification system. With every click,the result set changed. For some types of searching, a user may find the point-and-click approach helpful. However, there are other ways to root through what appears to be patent applications. There are the very expensive methods happily provided by Reed Elsevier and Thomson Reuters, two find outfits. And then there are less expensive methods like Alphabet Google’s odd ball patent search system or the quite functional FreePatentsOnline service. In between, you and I have many options.
None of them is a slam dunk. When I was working through the publicly accessible Palantir Technologies’ patents, I had to fall back on my very old-fashioned method. I tracked down a PDF, printed it out, and read it. Believe me, gentle reader, this is not the most fun I have ever had. In contrast to the early Google patents, Palantir’s documents lack the detailed “background of the invention” information which the salad days’ Googlers cheerfully presented. Palantir’s write ups are slogs. Perhaps the firm’s attorneys were born with dour brain circuitry.
I did a side jaunt and came across a white paper from ClearstoneIP called “Why Semantic Searching Fails for Freedom-to-Operate (FTO).”i The 12 page write up is from a company called ClearstoneIP, which is a patent analysis company. The firm’s 12 pager is about patent searching. The company, according to its Web site is a “paradigm shifter.” The company describes itself this way:
ClearstoneIP is a California-based company built to provide industry leaders and innovators with a truly revolutionary platform for conducting product clearance, freedom to operate, and patent infringement-based analyses. ClearstoneIP was founded by a team of forward-thinking patent attorneys and software developers who believe that barriers to innovation can be overcome with innovation itself.
The “freedom to operate” phrase is a bit of legal jargon which I don’t understand. I am, thank goodness, not an attorney.
The firm’s search method makes much of the ontology, taxonomy, classification approach to information access. Hence, the reason my exploration of Palantir’s dynamic ontology with objects tossed ClearstoneIP into one of my search result sets.
The white paper is interesting if one works around the legal mumbo jumbo. The company’s approach is remarkable and invokes some of my caution light words; for example:
- “Not all patent searches are the same.”, page two
- “This all leads to the question…”, page seven
- “…there is never a single “right” way to do so.”, page eight
- “And if an analyst were to try to capture all of the ways…”, page eight
- “to capture all potentially relevant patents…”, page nine.
The absolutist approach to argument is fascinating.
Okay, what’s the ClearstoneIP search system doing? Well, it seems to me that it is taking a path to consider some of the subtlties in patent claims’ statements. The approach is very different from that taken by Brainware and its tri-gram technology. Now that Lexmark owns Brainware, the application of the Brainware system to patent searching has fallen off my radar. Brainware relied on patterns; ClearstoneIP uses the ontology-classification approach.
Both are useful in identifying patents related to a particular subject.
What is interesting in the write up is its approach to “semantics.” I highlighted in billable hour green:
Anticipating all the ways in which a product can be described is serious guesswork.
Yep, but isn’t that the role of a human with relevant training and expertise becomes important? The white paper takes the approach that semantic search fails for the ClearstoneIP method dubbed FTO or freedom to operate information access.
The white paper asserted:
Semantic
Semantic searching is the primary focus of this discussion, as it is the most evolved.
ClearstoneIP defines semantic search in this way:
Semantic patent searching generally refers to automatically enhancing a text -based query to better represent its underlying meaning, thereby better identifying conceptually related references.
I think the definition of semantic is designed to strike directly at the heart of the methods offered to lawyers with paying customers by Lexis-type and Westlaw-type systems. Lawyers to be usually have access to the commercial-type services when in law school. In the legal market, there are quite a few outfits trying to provide better, faster, and sometimes less expensive ways to make sense of the Miltonesque prose popular among the patent crowd.
The white paper, in a lawyerly way, the approach of semantic search systems. Note that the “narrowing” to the concerns of attorneys engaged in patent work is in the background even though the description seems to be painted in broad strokes:
This process generally includes: (1) supplementing terms of a text-based query with their synonyms; and (2) assessing the proximity of resulting patents to the determined underlying meaning of the text – based query. Semantic platforms are often touted as critical add-ons to natural language searching. They are said to account for discrepancies in word form and lexicography between the text of queries and patent disclosure.
The white paper offers this conclusion about semantic search:
it [semantic search] is surprisingly ineffective for FTO.
Seems reasonable, right? Semantic search assumes a “paradigm.” In my experience, taxonomies, classification schema, and ontologies perform the same intellectual trick. The idea is to put something into a cubby. Organizing information makes manifest what something is and where it fits in a mental construct.
But these semantic systems do a lousy job figuring out what’s in the Claims section of a patent. That’s a flaw which is a direct consequence of the lingo lawyers use to frame the claims themselves.
Search systems use many different methods to pigeonhole a statement. The “aboutness” of a statement or a claim is a sticky wicket. As I have written in many articles, books, and blog posts, finding on point information is very difficult. Progress has been made when one wants a pizza. Less progress has been made in finding the colleagues of the bad actors in Brussels.
Palantir requires that those adding content to the Gotham data management system add tags from a “dynamic ontology.” In addition to what the human has to do, the Gotham system generates additional metadata automatically. Other systems use mostly automatic systems which are dependent on a traditional controlled term list. Others just use algorithms to do the trick. The systems which are making friends with users strike a balance; that is, using human input directly or indirectly and some administrator only knowledgebases, dictionaries, synonym lists, etc.
ClearstoneIP keeps its eye on its FTO ball, which is understandable. The white paper asserts:
The point here is that semantic platforms can deliver effective results for patentability searches at a reasonable cost but, when it comes to FTO searching, the effectiveness of the platforms is limited even at great cost.
Okay, I understand. ClearstoneIP includes a diagram which drives home how its FTO approach soars over the competitors’ systems:
ClearstoneIP, © 2016
My reaction to the white paper is that for decades I have evaluated and used information access systems. None of the systems is without serious flaws. That includes the clever n gram-based systems, the smart systems from dozens of outfits, the constantly reinvented keyword centric systems from the Lexis-type and Westlaw-type vendor, even the simplistic methods offered by free online patent search systems like Pat2PDF.org.
What seems to be reality of the legal landscape is:
- Patent experts use a range of systems. With lots of budget, many fee and for fee systems will be used. The name of the game is meeting the client needs and obviously billing the client for time.
- No patent search system to which I have been exposed does an effective job of thinking like an very good patent attorney. I know that the notion of artificial intelligence is the hot trend, but the reality is that seemingly smart software usually cheats by formulating queries based on analysis of user behavior, facts like geographic location, and who pays to get their pizza joint “found.”
- A patent search system, in order to be useful for the type of work I do, has to index germane content generated in the course of the patent process. Comprehensiveness is simply not part of the patent search systems’ modus operandi. If there’s a B, where’s the A? If there is a germane letter about a patent, where the heck is it?
I am not on the “side” of the taxonomy-centric approach. I am not on the side of the crazy semantic methods. I am not on the side of the keyword approach when inventors use different names on different patents, Babak Parviz aliases included. I am not in favor of any one system.
How do I think patent search is evolving? ClearstoneIP has it sort of right. Attorneys have to tag what is needed. The hitch in the git along has been partially resolved by Palantir’’-type systems; that is, the ontology has to be dynamic and available to anyone authorized to use a collection in real time.
But for lawyers there is one added necessity which will not leave us any time soon. Lawyers bill; hence, whatever is output from an information access system has to be read, annotated, and considered by a semi-capable human.
What’s the future of patent search? My view is that there will be new systems. The one constant is that, by definition, a lawyer cannot trust the outputs. The way to deal with this is to pay a patent attorney to read patent documents.
In short, like the person looking for information in the scriptoria at the Alexandria Library, the task ends up as a manual one. Perhaps there will be a friendly Boston Dynamics librarian available to do the work some day. For now, search systems won’t do the job because attorneys cannot trust an algorithm when the likelihood of missing something exists.
Oh, I almost forget. Attorneys have to get paid via that billable time thing.
Stephen E Arnold, March 30, 2016
Microsoft and the Open Source Trojan Horse
March 30, 2016
Quite a few outfits embrace open source. There are a number of reasons:
- It is cheaper than writing original code
- It is less expensive than writing original code
- It is more economical than writing original code.
The article “Microsoft is Pretending to be a FOSS Company in Order to Secure Government Contracts With Proprietary Software in ‘Open’ Clothing” reminded me that there is another reason.
No kidding.
I know that IBM has snagged Lucene and waved its once magical wand over the information access system and pronounced, “Watson.” I know that deep inside the kind, gentle heart of Palantir Technologies, there are open source bits. And there are others.
The write up asserted:
For those who missed it, Microsoft is trying to EEE GNU/Linux servers amid Microsoft layoffs; selfish interests of profit, as noted by some writers [1,2] this morning, nothing whatsoever to do with FOSS (there’s no FOSS aspect to it at all!) are driving these moves. It’s about proprietary software lock-in that won’t be available for another year anyway. It’s a good way to distract the public and suppress criticism with some corny images of red hearts.
The other interesting point I highlighted was:
reject the idea that Microsoft is somehow “open” now. The European Union, the Indian government and even the White House now warm up to FOSS, so Microsoft is pretending to be FOSS. This is protectionism by deception from Microsoft and those who play along with the PR campaign (or lobbying) are hurting genuine/legitimate FOSS.
With some government statements of work requiring “open” technologies, Microsoft may be doing what other firms have been doing for a while. See points one to three above. Microsoft is just late to the accountants’ party.
Why not replace the SharePoint search thing with an open source solution? What’s the $1.2 billion MSFT paid for the fascinating Fast Search & Transfer technology in 2008? It works just really well, right?
Stephen E Arnold, March 30, 2016
Google Reveals Personal Data in Search Results
March 30, 2016
Our lives are already all over the Internet, but Google recently unleashed a new feature that takes it to a new level. Search Engine Watch tells us about, “Google Shows Personal Data Within Search Results, Tests ‘Recent Purchases’ Feature” and the new way to see your Internet purchases.
Google pulls the purchase information most likely from Gmail or Chrome. The official explanation is that Google search is now more personalized, because it does pull information from Google apps:
“You can search for information from other Google products you use, like Gmail, Google Calendar, and Google+. For example, you can search for information about your upcoming flights, restaurant reservations, or appointments.”
Personalized Google search can display results now only from purchases but also bills, flights, reservations, packages, events, and Google Photos. It is part of Google’s mission to not only organize the world, but also be a personal assistant, part of the new Google Now.
While it is a useful tool to understand your personal habits, organize information, and interact with data like in a science-fiction show, at the same time it is creepy being able to search your life with Google. Some will relish in the idea of having their lives organized at their fingertips, but others will feel like the NSA or even Dark Web predators will hack into their lives.
Whitney Grace, March 30, 2016
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Google Dorking: It Is Search, Folks
March 29, 2016
I received a call from a former client this morning (March 28, 2016). The question? Google dorking. Relax. Google dorking is another way to say advanced search. In those How to Search with Google seminars I used to do for an outfit where the metros are unreliable and trust is a weird concept, I covered a number of Google dorking methods.
I don’t make those lectures’ content available for free in this blog, but you can round up some basic info at these links:
- The Science of Google Dorking
- How to Use Google to Hack(Googledorks)
- Untangling the Web which maybe is available at this link. No promises, gentle reader.
The dear, dear Alphabet Google thing kills or breaks useful search functions. This weekend, the FILETYPE: instruction performed like the University of Virginia men’s basketball team. You will have to do some thinking.
By the way, as Google shifts to its magical artificial intelligence methods, finding information via Google is getting more and more difficult.
We do webinars on how to deal with the Alphabet Google thing. Write seaky2000 at yahoo dot com and inquire about a 75 minute webinar. Yep, the same one I do for government types.
Stephen E Arnold, March 29, 2016
Retraining the Librarian for the Future
March 28, 2016
The Internet is often described as the world’s biggest library containing all the world’s knowledge that someone dumped on the floor. The Internet is the world’s biggest information database as well as the world’s biggest data mess. In the olden days, librarians used to be the gateway to knowledge management but they need to vamp up their skills beyond the Dewey Decimal System and database searching. Librarians need to do more and Christian Lauersen’s personal blog explains how in, “Data Scientist Training For Librarians-Re-Skilling Libraries For The Future.”
DST4L is a boot camp for librarians and other information professionals to learn new skills to maintain relevancy. Last year DST4L was held as:
“DST4L has been held three times in The States and was to be set for the first time in Europe at Library of Technical University of Denmark just outside of Copenhagen. 40 participants from all across Europe were ready to get there hands dirty over three days marathon of relevant tools within data archiving, handling, sharing and analyzing. See the full program here and check the #DST4L hashtag at Twitter.”
Over the course of three days, the participants learned about OpenRefine, a spreadsheet-like application that cane be used for data cleanup and transformation. They also learned about the benefits of GitHub and how to program using Python. These skills are well beyond the classed they teach in library graduate programs, but it is a good sign that the profession is evolving even if the academia aspects lag behind.
Whitney Grace, March 28, 2016
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Search as a Framework
March 26, 2016
A number of search and content processing vendors suggest their information access system can function as a framework. The idea is that search is more than a utility function.
If the information in the article “Abusing Elasticsearch as a Framework” is spot on, a non search vendor may have taken an important step to making an assertion into a reality.
The article states:
Crate is a distributed SQL database that leverages Elasticsearch and Lucene. In it’s infant days it parsed SQL statements and translated them into Elasticsearch queries. It was basically a layer on top of Elasticsearch.
The idea is that the framework uses discovery, master election, replication, etc along with the Lucene search and indexing operations.
Crate, the framework, is a distributed SQL database “that leverages Elasticsearch and Lucene.”
Stephen E Arnold, March 26, 2016
Ixquick and StartPage Become One
March 25, 2016
Ixquick was created by a person in Manhattan. Then the system shifted from the USA to Europe. I lost track. I read “Ixquick Merges with StartPage Search Engine.” Web search is a hideously expensive activity to fund. Costs can be suppressed if one just passes the user’s query to Bing, Google, or some other Web indexing search system. The approach delivers what is called a value-added opportunity. Vivisimo used the approach before it morphed into a unit of IBM and emerged not as a search federation system but a Big Data system. Most search traffic flows to the Alphabet Google advertising system. Those who use federated search systems often don’t know the difference and, based on my observations, don’t care.
According to the write up:
The main difference between StartPage and the current version of Ixquick is that the former is powered exclusively by Google search results while the latter aggregates data from multiple search engines to rank them based on factors such as prominence and quantity. Both search engines are privacy orientated, and the merging won’t change the fact. IP addresses are not recorded for instance, and data is not shared with third-parties.
Like DuckDuckGo.com, Ixquick.com and StartPage.com “protect the user’s privacy. My thought is that I am not confident Tor sessions are able to protect a user’s privacy. A general interest search engine which delivers on this assertion is interesting indeed.
If you want to use the Ixquick function that presents only Google results, navigate to www.ixquick.eu. There are other privacy oriented systems; for example, Gibiru and Unbubble.
Sorry, I won’t/can’t go into the privacy angle. You may want to poke around how secure a VPN session, Tails, and Tor are. The exploration may yield some useful information. Make sure your computing device does not have malware installed, please. Otherwise, the “privacy” issue is off the table.
Stephen E Arnold, March 25, 2016
-
- Subscribe to Beyond Search
Feature archive
News archive
-
