Information: Dark Sides and Bright Sides
April 9, 2013
I find the information revolution semi-bright or semi-dark. I read “Are We Paying Enough Attention to Information Technology’s Dark Side?” My first reaction was, “Nah.” Most outfits are worrying about revenues. Google has to deal with the shift from the money Gold Rush of the desktop era to the lower revenue per click of the mobile world. Microsoft has to worry about the economic impact of its initiatives to nowhere. Smaller outfits in search have either been crushed like Convera or squished like Dieselpoint, mired in controversy like Autonomy and Fast Search, or just unable to make ends meet, deliver a product which works, or get their act together long enough to close a deal.
Paradise Lost may help illuminate the dark sides and bright sides of information. A happy quack to Lapidary Apothegms for reminding me of this phrase.
The concern of the “Dark Side” write up is broader. The big issue is Big Ideas. With references to high profile information luminaries like James Clapper, the director of national intelligence, and governmental issues. Here’s the quote I find interesting:
While the idea of lumbering bureaucracies adapting quickly may seem unlikely; it’s entirely possible they’ll adapt just fast enough to remain in place for awhile yet. And instead of quick change, the classic definition of the state will twist and wither. Whether its successor proves good or ill remains to be seen—but if history (and Marc Goodman) is any guide, it’ll be some of each.
The future is the semi-bright and semi-dark situation.
With regard to information, flows of information, data, and knowledge can erode certain structures. In an organization, as information moves more freely, the old chokepoints are bypassed. The notion which has gripped managers and bureaucrats is that flowing information has more of luminescence than cutting off that flow thus casting shadows.
In my experience, information is not neutral. Digitization has its own motive power. In one talk I gave years ago, I pointed out that information breeds more of itself. The image I used in my lecture was a sci-fi decision maker surrounded by Tribbles. Tribbles just kept on making more Tribbles. Bad news were Tribbles in the confines of a starship.
Even though I have worked in information centric businesses and government agencies for decades, I am not sure I understand information. I do not have a clear grasp of its behaviors. Over the years, I have formulated some “laws”, which I describe in some of my writings and talks. A recent example is Arnold’s Law of Vulnerability. In a nutshell, the “law” reports data from our research that says, “As the volume of information increases attack surfaces expand.”
The implication of this “law” is that digital information disconnects from the factual and becomes the propaganda described by Jacques Ellul. A software program which crashes a system or more importantly modifies it in a manner unknown to the system developers is a growing problem.
Conflating political movements, digital data, and next generation systems increases complexity. In short, as informationizing operates, clear thinking becomes more and more difficult. Thus, we now have to navigate in a datascape in which:
- Facts are not facts, even the results of a scientific experiment can be falsified or, more troubling, placed in an “objective journal” as an advertorial
- Systems have minimal ability to detect falsified data from sensors, SMS messages, or data streams which contain signals to which the smart software responds in a Pavlovian way
- Humans accept outputs of systems as though those outputs were a reality which corresponds to the actuality of a single individual.
Work needs to be done in the space between the bright and dark of information. Much remains to be done and not by failed webmasters, azure chip consultants, search engine optimization experts, and unemployed journalists. Perhaps Google’s smart software can just take on the job
Social Media Strategies
April 9, 2013
Social media is not just for personal use anymore it has expanded into the business world. The Expert System Cogito Blog piece “Understanding the Strategic Value of Social Media Analysis” talks about how many companies are selling themselves short when it comes to using social media.
“I have often said that companies are missing out on the real value of social media analysis. More often than not, even the big players don’t have the processes or models in place to really make use of the data gained from the analysis. As a result, social media analysis has a limited impact on the business, not to mention the budgets assigned to such projects.”
However, despite the usual oversights the author talks about a recent encounter with the head of customer experience at a well-known bank. They were going to discuss the tools they would need to support social media analysis but instead of going through the usual song and dance the manager was actually prepared to discuss exactly what they needed from them. Even more surprising the customer was actually able to provide specific examples of quantitative as well as qualitative data that she wanted to be able to extract from the streams of data. This made it easier to talk about semantics and how it can bring value to their company. Strategies such as focusing on extracting relationships between monitored entities and relieving some of the social media noise through deep analysis and contextualization can help to improve product visibility as well as market trends. The author ends by nothing that they are sure that they haven’t seen the last of their “usual pitch” because many organizations do not have a clear and concise strategy when it comes to social media projects. However, as the trend changes and more and more companies are realizing the importance of social media semantic technology vendors better strike fast and learn how to “grab the bull by its horns.”
April Holmes, April 09, 2013
Sponsored by ArnoldIT.com, developer of Augmentext
Pay to Play Content: Now Even the New York Times Knows
April 8, 2013
Mondays usually start in a predictable way. I walk the dogs. I eat a cardiologist-approved breakfast. I find out what my wife has on her list for me to do. But this morning I flipped through the New York Times, environmentally unfriendly version, and burst out laughing.
My wife asked, “What’s so funny?”
I replied, “The New York Times describes pay to play with more crazy synonyms than I thought possible.”
She asked, “And that’s humorous?”
To me it was. Navigate to these two articles. The first is on the front page of Harrod’s Creek edition and Google-crafted this way: “Scientific Articles Accepted (Personal Checks, Too).” The story appears in the April 8, 2013, edition which you will find in the dead tree version. My link points to a short lived version of the file on another newspaper. After a rousing quote “the dark side of open access” the story jumps to section B, page 8.
The second story appears in the business section of the same issue. Its title is “Sponsoring Articles, Not Just Ads. Branded Content on the Web Mingles with Regular Coverage.” The story features a creative graphic showing pencils held in a roll of money. (You remember. Printed money just like the early newspaper moguls collected by the horse drawn cart in the good old days of publishing.)
The point of both articles is that there are people who will pay to get their content published in a form which has some respectability. Academics pay to play in the academic journals. Companies pay to get their ideas published in a wide range of channels. The New York Times mentions Mashable, but there are many other outfits who charge money to run content. My Augmentext operation is in this business too. I suppose I could trot out the names of big publishers who offer college guides with inflated “inclusions” describing the wonders of certain college campuses. The write ups are compelling and once produced money for those who operated these quasi-reference services.
What words does the New York Times use to describe these pay to play operations? Here’s a list of some of the terms from the write up:
- Advertising
- Advertorials
- Branded content
- Campaign
- Content
- Corporate propaganda
- Native advertising
- Pure editorial
- Sponsored content
Here in Harrod’s Creek, we call content someone wants published for money:
- An inclusion
- A “pay to play” story
- POP or Plain old propaganda as defined by Jacques Ellul. If the name does not ring a bell, you can find the information in his decades old study in Propaganda: The Formation of Men’s Attitudes.
The professional publishing sector has been charging academics for page proofs and other services for many years. Now the practice has diffused to conferences. In my view, the use of “pay to play” methods is now part of the atmosphere and has been for decades.
I find it fascinating that the topics are now front page news from the New York Times. Perhaps “real” journalists are learning more about how the information world works.
What troubles me is that none of these questions is addressed:
- Do modern systems identify pay to play content?
- Are automated content processing systems giving equal weight to shaped content and objective content?
- Are the outputs from analytics systems manipulable?
In my proprietary report on this subject, the surprising answer is, “We just process data.”
In short, despite the huff and puff of next generation content processing system cheerleaders, the systems have what William James called “a certain blindness.” In the quest for revenues, many organizations are unwittingly conspiring to deliver information which at best is semantically swizzled and at worst weaponized. Oh, the phrase “weaponized information” does not appear in the New York Times’ stories nor in the gigabytes of words explaining the wonders of next generation analytics. Like the New York Times, the present is too much with us.
Stephen E Arnold, April 8, 2013
Elasticsearch Joins Fog Creek
April 8, 2013
Elasticsearch is trying to expand its reach by partnering with other trendy tech services. It is definitely getting some headlines. The most recent headline is detailed by Market Watch in their article, “Fog Creek Selects Elasticsearch to Search and Analyze Terabytes of Data.”
“Elasticsearch, the company behind the popular real-time search and analytics open source project, today announced that Fog Creek has selected Elasticsearch to provide instant search capabilities within Kiln, its software development product. Kiln is designed to support and simplify development workflow for users searching more than 100,000 source code repositories. Elasticsearch is now a critical ingredient of Kiln, providing instant search for 300,000,000 requests across 40 billion lines of code to improve overall performance, reliability and user experience.”
Elasticsearch is known for collaboration with leading edge products, but it is not without its controversies as well. GitHub recently reached out to Elasticsearch to develop its new search infrastructure, but the service quickly exposed security concerns and then crashed. So when it comes to a search infrastructure that goes beyond trends, trust an industry standard. Do not assume that every search application will be safe enough for the enterprise. For instance, consider LucidWorks. They are built on open source Lucene/Solr, employ one quarter of the Core Committers on that project, and are optimized for the enterprise. Choose industry confidence, not trends.
Emily Rae Aldridge, April 8, 2013
Sponsored by ArnoldIT.com, developer of Beyond Search
InTrade: A Harbinger of Prediction Woes to Come?
April 7, 2013
Key word search is not to useful when there are trillions of content objects. Clustering trillions of objects is not economically feasible, so the sets are trimmed. Who’s to know? Predictive analytics sounds so darned promising because “real time processing” is cheap, plentiful, and trivial to boot.
What can go wrong with text processing, text analytics, social crowdsourcing data, and the other Lone Ranger silver bullets? How can predictive systems come back and bite a user, an investor, or an employee who loses her job?
I suppose that the article “InTrade Announces $700,000 Cash Shortfall And Risk Of Imminent Liquidation” describes an anomaly. Here’s the key point in my opinion:
..the company has posted the following message on its site, which says that it has discovered a $700,000 cash shortfall that must be rectified immediately in order to avoid liquidation.
InTrade, is or maybe was, a prediction market. The company says:
It’s a market that allows you to make predictions on the outcome of hundreds of real-world events. Stock exchanges find the price of stocks, and futures markets find the price of commodities. Prediction markets find the probability of something happening – a predefined, uncertain future event.
InTrade is more than voting. The company uses a range of methods to answer yes or no. Life should be so simple. The company even posted some Golden Rules to make the system almost foolproof; for example, “If you sell shares you profit if the market value of the shares goes down. Your profit is maximized if the market settled at $0.00.”
Eel bites can be painful due to “alien style jaws.” Investors in some predictive outfits may experience similar bites.
There are many meanings for the word “prediction.” I don’t want to get into a squabble that InTrade is one type of prediction and an outfit like Digital Reasoning or Agilex is another type of prediction. I want to capture several thoughts so I can include them in my text analytics lecture later this month, chance willing, of course:
First, predictions are slippery eels. I once offered predictions to my clients. Now I offer clients. I learned that regardless of methods predictions jump into a murky pool and get lost. Stick your hand in the pool and one can come up with nothing or an eel clamping on the extremity. Ouch.
Second, predictions and various methods and the companies built upon them can simply fail. Why not predict that? I think that getting hoisted by one’s petard is part of life.
Third, InTrade may be one example of what can happen when hyperbole outraces the capabilities of the numerical recipe crowd. Will other companies in the fancy math business suffer similar fates? I don’t know. I won’t predict.
If you are into fancy math, why not plug your retirement nest egg into one of the analytics outfits and let me know how that works out for you. Azure chip consultants, feel free to weigh in and explain to me and my two to three readers how such a clever idea could end up in a pickle of reality.
Stephen E Arnold, April 7, 2013
Business Structures Revealed through New Analysis Technique
April 7, 2013
Now here is an interesting implication of social-graph analysis in business. The MIT Technology Review reports, “Social Networks Reveal Structure (And Weaknesses) of Business.” We’ve known for some time that, through the analysis of connections, social networks can reveal even more about us than is obvious to most users. Now, researchers at Israel’s Ben Gurion University used this concept to derive an impressive amount of information about businesses. The article reveals that the team begins:
“. . . by using a search engine to find the Facebook pages of a number of individuals who work for a specific company.
“Using these individuals as seeds, they then begin crawling the social networks, sometimes jumping from one network to another, looking for other individuals at the same company. These in turn become seeds to find more employees and so on.
“They end up with a basic network of links between employees within the company. It’s then that the fun begins.
“Using standard measures of connectedness, Fire and co then identified people in positions of leadership and by adding in details such as location, mined from the Facebook pages, they reconstructed the international structure of these organisations. They also used community detection algorithms to reconstruct the organisational structure of the company.”
Wow. The researchers used their method on several “well known hi-tech companies” and found startling details. For example, they found a cluster of comparatively disconnected folks at a large organization, and discerned they belonged to an acquired startup that had yet to be well-integrated into the company. This sort of information can be used by companies to monitor themselves, but it could also be used by potential investors (for good or ill for the business, I suppose, depending on what turned up.)
More ominously, competitors could use the information to their advantage. Now that this technology is in the news, many companies will want to prevent such details from emerging, but how? Researcher Michael Fire advises them to “enforce strict policies which control the use of social media by their employees.” Immediately, I might add. And, I suspect that whatever was previously considered a “strict policy” must become even more strict in order to avoid exposure from this technique.
Won’t employees be thrilled?
Cynthia Murrell, April 07, 2013
Sponsored by ArnoldIT.com, developer of Augmentext
Consultants Take It in the Snoot: IDC Criticizes Itself and a Competitor
April 5, 2013
Have you ever been hammering a nail and smacked your finger. Imagine what it feels like to whack your nose. Ouch.
Navigate to to “Gartner and IDC predictions: Oops, Forget What We Said Last Time.” The main point of the write up it seems to me is that azure chip consulting firms just crank out predictions. When the spirit moves the azure chip crowd, new predictions are generated using fancy math, a dash of spreadsheet fever, and inputs from other azure chip consultants.
With new outputs, new predictions are possible. The predictions, if I read the IDC write up correctly, have little connection with previous predictions. Here’s what IDC said about itself:
IDC is no better: It predicted last month that total PC shipments in 2013 would fall 1.3 percent from 2012. Of course, before that, it predicted an increase of 2.8 percent. It too predicted that Windows Phone would be a major hit, based apparently solely on the fact it came from Microsoft.
What was the benchmark against which IDC’s work was measured? IDC points to the estimable Gartner Group which warranted several paragraphs of commentary. I won’t repeat the IDC summary of another firm’s predictions. Please, read them in a “real” news publication, not a free blog produced by a 70 year old in rural Kentucky.
But per my want I will set forth several observations:
First, for many years I worked at a couple of above average outfits very much into the prediction game. After moving to other tasks, I have watched the rise of the prediction industry. At Halliburton NUS, fancy math was needed to figure out which fuel rods should be moved. Moving nuclear fuel rods is a useful function but I know that most of my two or three readers are fully conversant with nuclear engineering’s use of statistical methods. At Booz, Allen & Hamilton in the days when the firm competed fiercely with McKinsey, Bain, and Boston Consulting, most of the technology group’s units relieved on a wide range of fancy math to perform tasks which most “real” journalists and “experts” have a comfortable familiarity. I won’t enumerate the methods to perform nuclear plume modeling, a topic which I am confident most “experts” discuss at lunch each day. Now every “expert” is a master of advanced math. How comforting it is to know that technical expertise is widely available to perform a range of data centric tasks. Obviously talk about skills shortages in math and related disciplines is wrong.
It is easy to make predictions using back of the envelope methods like Briggs’s Equations. Azure chip outfits can work these sequences in their head, whilst texting, and writing client reports I surmise.
Second, the notion that predictions exist without context is an interesting one. What the IDC criticism of azure chip consultancies suggests is that just about any old estimate is good enough to today’s busy, super sophisticated professional. However, is it possible that azure chip firms and their number crunchers are edging into the murky swamp of marketing. Creating a number and providing an insight based on that number is easy to do. Without critical thinking about a prediction, is it possible that fiction takes the place of fact? Just a question to consider. You may want to take a look at The Cheating culture: Why More Americans Are Doing Wrong to Get Ahead.
Third, are predictions part of the shift from doing work the way my grandfather did it to the thumb typing methods of today? My view is that the shift to knowledge work allows almost everyone hang out the “expert” shingle. Take me. I studied medieval poetry in Latin and bumbled into computing by accident. The journey from that first programming job in 1963 to the present has been a long, arduous journey. Is it possible today that individuals better suited to running a lathe or making cookies find themselves working at azure chip consulting firms cranking out predictions?
Click the image to learn about expert marketing. It is easy to create a video to communicate one’s capabilities.
Fourth, the information technology world is in flux. Those running technology units are not sure what to do. For guidance, these folks want to have back up, data, and case studies. The information flows from azure chip consultants and “experts”. What happens if the inputs are not on point? Perhaps the niggling glitches like security issues, flawed software, ever present cost overruns, and stuff that just doesn’t work as advertised provide some indication? Glitches are standard operating procedures in many “smart” systems. What if the experts’ advice is not accurate? Won’t problems escalate and cascade? Are the present economic challenges attributable to some degree to expert inputs which were incorrect? Interesting question in my opinion.
The IDC critique of itself and Gartner Group seems like an isolated and infrequent incident. But even the search engine optimization crowd. Now this group has almost as many unemployed “real” journalists and failed middle school teachers as the azure chip consulting industry, published “Big Data—Has It Blinded Us with Science?” When folks who intentionally distort the results of an “objective” search engine, maybe there is something troubling even today’s always-on, always-busy, always-selling crowd. Is marketing more important than anything except generating revenue?
I wonder how that self inflicted poke to the nose feels. At my age, my most forceful blow is as gentle as a summer breeze.
Stephen E Arnold, April 5, 2013
Sponsored by Augmentext
Predictive Analysis Research in the UK
April 3, 2013
The field of predictive analysis is proceeding apace, we learn from Science Daily‘s “Predictive Analysis: New Generation of Computational Intelligence Systems.” Predictive analysis, as the name suggests, is the art/science of making predictions (about the past, present, or future) from heaps of data. This article looks at progress being made at the Smart Technology Research Centre, a part of Bournemouth University in Dorset, U.K. The article explains that the researchers:
“. . . are developing computer programs capable of learning. With this intelligent software, computers can make judgments about the quality and reliability of the data they gather. They look for patterns and adapt according to what the information will be used for.”
That would be the work of an “adaptive algorithm” that effectively learns as it encounters data. The Centre has already put some of its findings to work helping companies in tourism and communications, but are looking to build a more general-use system. The write-up relates:
“Building a learning computer system capable of adapting according to the information that is fed into it is no easy task. Most prediction software until recently has been tailor made to solve specific problems. This can make them expensive to maintain and hard to adapt.
“For this, Professor Gabrys and his team have turned to one of the most successful problem solvers on the planet for inspiration — Mother Nature herself. They are building systems which process information in a similar way to the human brain, with its networks of neurons that constantly rewire themselves as we learn.”
Other natural inspirations include genetics and certain animal behaviors, like those of social insects (think ants and bees) or of flocking and swarming creatures (birds and fish.) However, Professor Bogdan Gabrys, the research center’s chair of computational intelligence, emphasizes the limits of the technology—once the system you are studying becomes as complex as, say, financial markets, accurate predictions become unrealistic. The key, he says, is knowing where to draw that line.
Cynthia Murrell, April 03, 2013
Sponsored by ArnoldIT.com, developer of Augmentext
Recorded Future Sheds Light on Sony
April 2, 2013
One of the best ways to find the correlation between data from various sources it by comparison. Recorded Future is an online business intelligence analysis tool designed for open web sources. Recorded future scans thousands of different type such as blogs and online news and analyzes the text to identify certain references entities or events. The GamaGanda article “Analyzing Sony with Recorded Future #2” is a great example of how the intelligence tool works. The author does an analysis on Sony’s position in the video gaming market. Recently, Sony had been discussing innovation and was due to make a big announcement. The author used Recorded Future to see if Sony was really making any innovative strides and more specifically how they shaped up when compared to Microsoft and Apple.
“Patent filling is something you can track and it gives you a pretty good idea on a company’s disposition to invest in R&D and hence design innovative products and services.”
As shown on the online screen shots Sony is definitely not keeping up with the innovative minds of Microsoft and Apple and the number of patents clearly demonstrate that. Also more alarming is that Sony has seen an increasing number of layoffs over the past several years. Though patent filing seems to be increasing they are still behind the other big two and have lost a lot of ground in the race. Only time will tell if the upcoming announcement and future projects will help Sony to bounce back. With just a few screenshots Recorded Future broke the bad news to Sony shareholders and now all they can do is watch, wait and hope for the best.
April Holmes, April 02, 2013
Sponsored by ArnoldIT.com, developer of Augmentext
Actian and Big Data Get Fast Answers
April 2, 2013
Actian is already a leader when it comes to big data management thanks to its analytics database Vectorwise Action Apps and it seems that they are focused on making a good thing even better. The Fort Mills Times article “Actian Announces Availability of Vectorwise 3.0 for Getting Fast Answers from Big Data” talks about Actian Corp and the introduction of its new Vectorwise 3.0 analytics database with advanced Hadoop integration. The Vectorwise 3.0 analytic database allows companies to integrate Hadoop using the Vectorwise Hadoop Connector and it helps companies to get important business information form their big data quickly and with ease. Fred Gallagher, general manger of Vectorwise stated
“We are talking about moving data at a rate of up to 3 Terabytes per hour onto a modest x86 server costing less than $15,000. That’s like downloading 20 movies in two minutes. Vectorwise 3.0 is now even faster, with a more efficient storage engine, supporting more data types and analytical SQL functions, and enhanced DDL features. It also has improved monitoring and profiling accessibility.”
Many companies rely highly on data to shape their business and Actian studied companies Hadoop usage to develop the innovative Vectorwise 3.0 analytics database. In the past Hadoop data stores were slow when it came to batch processing of analytics but the Vectorwise Hadoop Connector helps to alleviate this problem by delivering not only timely but also responsive multi-user querying and analytics processing. Industry users are already expressing their eagerness for the Hadoop Connector’s release in April 2013 and more importantly its positive impact on data analytics. As Hadoop integration and the benefits continue to make the news it looks like Hadoop’s got the data game down packed.
April Holmes, April 02, 2013
Sponsored by ArnoldIT.com, developer of Augmentext
-
- Subscribe to Beyond Search
Feature archive
News archive
-
