Endeca Explanation
November 19, 2012
We’ve turned up a useful summary of Endeca’s Information Discovery system; the description occurs within a post about using integration platform CloverETL with the Endeca product. “Oracle Endeca Information Discovery—CloverETL” is posted at Saichand Varanasi’s OBIEE, Endeca and ODI Blog. After referring readers to his Endeca overview, the blogger dives into the Clover. He writes:
“Today we will see how to create Clover ETL graph and populating data which will be used by MDEX engine for reporting (Studio). Endeca Information discovery helps organization to answer quickly on relevant data of both structured and Un structured. It helps to search and discover and analysis. Information is loaded from multiple data source systems and stored in a faceted data model that dynamically supports changing data. Information discovery enables an iterative approach. Integration features a new ETL tool, The integrator (Clover ETL) that lets you extract source records from a variety of source types flat files to databases.”
Next, Varanasi walks us through an example project. Along the way, he also explains how Endeca Information Discovery functions. A happy side effect, if you will. See the post for details.
Founded in 1999 and based in Cambridge, MA, Endeca was acquired by Oracle just over a year ago. The company has been at the forefront of faceted search technology, particularly for large e-commerce and online library systems.
Cynthia Murrell, November 19, 2012
Sponsored by ArnoldIT.com, developer of Augmentext
IntelTrax Summary: November 9 to November 15
November 19, 2012
This week, the IntelTrax advanced intelligence blog published some important information regarding the state of big data and its impact on some of the world’s most up and coming industries.
“The Ethics of Big Data” examines the possible ethical quandries that develop from big data analysis. However, despite the potential ethical challenges that face the industry in the end the pros, outweigh the cons.
The article states:
“Yet it cuts both ways: Consumers also can take advantage of the democratizing effects of big data. In fact, there’s an app for that: RateDriverenables users to quickly determine the appropriate rate they should expect to pay for attorney’s fees in 51 U.S. markets.
Big data holds promise to improve the legal profession and the quality of service that we deliver to clients, says Carolyn Elefant, a Washington, D.C., attorney and technology evangelist. “Significantly, big data would inject a strong dose of transparency into lawyer marketing and assist consumers in hiring lawyers. How so? Because big data can be used to show the likelihood of winning a case and the true cost.”
An article that shows the way that big data is transforming the healthcare industry is, “Big Data is the New Anti-Virus.” However, it looks at it from the angle of computer health and how to better detect viruses.
The article states:
“With Seculert Sense, customers can now upload log files using a Secure FTPS tunnel, or upstream logs through Syslog directly from a secure web gateway or web proxy devices, or log aggregation solution for real-time detection and forensics investigation. Built on Amazon Elastic MapReduce, Seculert Sense launches a “big data analysis cloud” that rapidly analyzes an organization’s vast amount of log data, going back months or even years and comparing it against the thousands of unique malware samples collected by Seculert. Over time, Seculert Sense continues to digest huge amounts of data in order to identify persistent attacks that are going undetected by next generation IPs, Anti-Bot and Secure Web Gateways.”
Big data analytics is not only taking off in America, it is becoming a world-wide phenomenon. “Asian Analytics on the Verge of a Boom” describes the potential for big data analytics success in Asia.
According to the article,
“Two different consumer analytics platforms from Singapore Management University (SMU) and StarHub respectively aim to provide insights into consumer behavior, so companies can develop and tailor initiatives that will be more relevant to and better received by customers.
Rajesh Balan, director of LiveLabs Urban Lifestyle Innovation Platform at SMU, said the platform will enable organizations to utilize real-time insights, helping their campaigns go to market and assess the outcome faster. On the consumer end, it will turn what most users perceive as intrusive spam messages on their phones into something useful.”
It does not matter what country you live in or what industry you work in. Big Data analytics technology is becoming too important to overlook. Digital Reasoning has been using automated understanding of big data for nearly a decade.
Jasmine Ashton, November 19, 2012
Sponsored by ArnoldIT.com, developer of Augmentext
Inxight Moving to EC Wise
November 13, 2012
If true, this is an interesting development. The site Dvd to Ipad Converter Reviews announces the acquisition of data-management-developer resource Inxight Software by enterprise software company EC Wise in “BO M & Inxight Software.” The headline’s a little confusing; whither “business objects” in text mining we wonder?
The brief write up states:
“EC Wise said the company plans to Inxight’s unstructured information that companies added to the EC Wise business intelligence products to help customers take full advantage of all the data to make the right decisions. . . . EC Wise said, Inxight’s text analytics, federated search and visualization applications, will become part of EC Wise XI platform.”
Unfortunately, we have zero information about the value of this deal, which is expected to be completed in July.
The piece also gives us this observation:
“The acquisition reflects the consolidation trend in the software industry. Last Tuesday, the German software company AG said the United States have been given approval to $ 546,000,000 of its acquisition of webMethods.(AG is a building SOA-based software vendor), last week, Microsoft also announced that six billion U.S. dollars acquisition of online advertising company aQuantive.Earlier this month, Oracle acquired the company to 495 million Agile Software Corporation.”
EC Wise U.S. is based in San Rafael, California, while EC Wise Sichuan makes its home in Chengdu, China. The company focuses on business intelligence, big data, business process optimization/ automation, and, interestingly, gaming and entertainment.
Inxight Software‘s impressive customer roster includes Morgan Stanley and Yahoo. It emerged in 1997 from the Xerox Palo Alto Research Center. The company has changed hands a couple of times already, having been bought by Business Objects in 2007, which was in turn absorbed by SAP in 2008. Let’s hope the company finds a happy home at EC Wise.
Cynthia Murrell, November 13, 2012
Sponsored by ArnoldIT.com, developer of Augmentext
TEMIS on Project ROBUST
November 12, 2012
Europe’s ROBUST project seeks to help online business communities with several issues, including large-scale data management and analysis. Now, TEMIS’ Tagline says the project has received positive feedback from scientists and potential clients in “Project ROBUST: Combining Scientific Excellence and Industry Relevance in EU Research Project.” TEMIS is developing infrastructure it says will bring the benefits of ROBUST’s findings to the rest of the world.
ROBUST, which stands for “Risk and Opportunity management of huge-scale BUSiness communiTy cooperation,” emerged as a response to the growing importance of online business communities. The onslaught of participants and posts has grown faster than the tools to manage them. Besides addressing that pressing issue, EU researchers also wondered which factors in these communities boost participation and engagement. The organization tapped experts from companies large and small, as well as some academics, to discover answers. The article tells us:
“As diverse as these challenges are, so are the solutions that start to take shape in the ROBUST project: IT Innovation progresses with an application allowing to model and monitor the various risks a community owner might be facing (experts leaving or exchanges becoming unproductive), Polecat is working on visual metaphors to provide a quick insight into the health of a community while the University of Koblenz is investigating whether and how patterns of agreement and disagreement among participants correlate with their observed participation.”
We agree that this project represents an important step forward. Now, though, we wonder: will European support catch up with companies wrapping more value-adds around basic text processing?
Cynthia Murrell, November 12, 2012
Sponsored by ArnoldIT.com, developer of Augmentext
Exclusive Interview: Chris Westphal, Visual Analytics
November 12, 2012
With the hype surrounding analytics, I have difficulty separating the wheat from the Wheaties when it comes to companies offering next-generation . Since the election, some individuals have been positioned as superstars of analytics. (The full text of my interview with Mr. Westphal appears in the ArnoldIT Search Wizards Speak series at this link.)
I am not comfortable with political predictions nor the notion of superstars. I did some checking and got solid referrals to Chris Westphal, one of the founders of Visual Analytics. The company has a solid client base in the world where analytics are essential to security and risk mitigation.
Mr. Westphal was gracious with his time, and I was able to speak with him in Washington, DC and then continue our discussion via email. In the course of my conversations with Mr. Westphal, he provided a different perspective on the fast-growing analytics sector.
His company, Visual Analytics or VAI is a privately-held company based in the Washington, DC metropolitan area providing proactive analytical, decision support, and information sharing solutions in commercial and government marketplaces throughout the world for investigating money laundering, financial crimes, narcotics, terrorism, border security, embezzlement, and fraud.
He told me that his firm’s approach and its success are a result of focusing on client problems, not imposing a textbook solution on a situation. He said:
We are problem driven. One of the most important areas we have found that separates us from much of our competition is the ability to deliver actual “analytics” to our end-user clients. It is not simply running a query looking for a specific value or graphically showing the contents of a spreadsheet. Our approach is to exploit the patterns and behaviors that are hidden with in the structures and volumes of the data we are processing. Our system effectively taps multiple, disparate sources to deliver one of the industry’s only federated data access platforms. We continue to focus on how to create algorithms (in a generic fashion), that can detect temporal sequences repeating activities, commonality, high-velocity connections, pathways, and complex aggregations.
One of the keys to Visual Analytics success is the company’s distinction between analytics and monitoring. Mr. Westphal pointed out:
The world is full of very good data management systems. There are databases, crawlers, indexers, etc. Our approach is to provide a layer on top of these existing sources and provide “interface-compliant-queries” to pull out relevant content. In about 90 percent of our engagements, we take advantage of the existing infrastructure with little to no impact on the client’s information technology processes, networks, or hardware footprint. If special processing is required, we tune the data management application to best meet the structure of the data so it can be processed/queried to maximize the analytical results. One other discussion is to differentiate “analytics” from “monitoring.” Much of our capability is to expose new patterns and trends, define the parameters, and verify data structures, content, and other key factors. Once we’ve locked in on a valuable pattern, we can continue to look for the pattern or it can be recoded into another system/approach (e.g., like is typically done with inline transactional systems) for real-time detection. The hard-issue is detecting the pattern in the first place.
The technical approach of Visual Analytics relies on open source and proprietary systems and methods. Mr. Westphal noted:
We have a very robust data connection framework consisting of different methods for different purposes. The core “connectors” are for relational databases and are based on standard database connector protocols. Our system also has drivers to other platforms such as information retrieval systems, various enterprise systems, plus the ability to create custom web services to expand, where necessary, to handle new sources or systems (including proprietary formats – assuming there is a Web-service interface available. We also have Apache Lucene built into our application at the data-layer so it can crawl and index content as needed. We try to make options available along with guidance about each approach. We offer a collection of methods to deliver the right-content for meeting a wide range of client needs. We always reference “contextual analytics” which basically means providing the actual content or pointers to content for any data entity – regardless of where it resides.
The full text of the interview is available at http://goo.gl/2y6T8. After my discussions with Mr. Westphal I remain convinced that the notion of next generation analytics is more rich and mature than some applications of next generation analytics.
Stephen E Arnold, November 12, 2012
IBM Focuses on Big Data Analysis
November 12, 2012
In their special Big Data section, InformationWeek tells us that “IBM Accelerates Big Data Analysis.” Big Blue made a rash of big data-related announcements at their 2012 Information On Demand conference in Las Vegas. We find it interesting; IBM owns proprietary analytics companies Cognos, SPSS, and now Vivisimo. Is IBM now making a too-little-too-late play to catch up with other firms in big data analytics? Perhaps they should have asked Watson what to do.
Whatever the case, IBM is now forging ahead in this brave new realm. The article states:
“What the announcements have in common is that they’re all about parts of IBM’s Big Data Platform, which is Big Blue’s umbrella term for a diverse collection of data-management and analysis technology bridging the relational database and Hadoop worlds.”
Just what we need—another bridge between relational databases and the venerable, open-source Hadoop. Ah, well; IBM knows it doesn’t matter if you don’t do it first if you can do it better.
The first set of announcements involve IBM’s BigInsights platform, which links proprietary tools like BigSheets to a distribution of Hadoop. The new twist here is the inclusion of analytics accelerators for text and social media data. Of course. Another addition to BigInsights addition, InfoSphere Data Explorer, sounds useful. The write up explains:
“Unlike BigSheets, which lets you explore Hadoop data in a spreadsheet-like interface, InfoSphere Data Explorer can look across multiple data sources using data-federation and analysis technology from IBM’s Vivisimo acquisition. With access to Hadoop as well as data warehouses, data marts, and possibly other sources, the software can automatically find correlations in data across these platforms.”
Other big data developments from IBM include InfoSphere Streams for processing in real-time environments; the latest version of PureData System for Analytics (formerly known as the IBM Netezza appliance); the Big Data Solution For CMOs which combines PureData with IBM’s Unica Web analytics software; and, finally, a bundle of predictive analytics services called Analytic Answers, which runs inside IBM’s Smart Cloud. See the article for details on all these offerings.
Cynthia Murrell, November 12, 2012
Sponsored by ArnoldIT.com, developer of Augmentext
IntelTrax Summary: November 2 to November 8
November 12, 2012
This week the IntelTrax advanced intelligence blog published some excellent article summaries regarding big data’s growing impact on the globalized workplace.
“Big Data Talent Pool Grows” explains how job seekers are embracing the big data analytics profession due to the fact that it welcomes new talent.
The article states:
“The just-released InformationWeek 2012 State of IT Staffing Survey reveals that 40% of those who cite big data and analytics as a top hiring priority say they’ll increase staffing in these areas by 11% or more during the next two years. At the same time, 53% of these companies say it will be hard to find big-data-savvy analytics experts. Respondents expect to try a mix of retraining of existing people, hiring of new employees and contracting of consultants and temporary employees to fill the gap.
Practitioners, vendors, and educators we spoke to for our Big Data IT Staffing report offer seven tips for finding the right talent.”
The article, “The Healthcare Analytics Trickle Down” shows how the pairing of healthcare and data analytics is starting to pay off for many companies and its starting to trickle down.
The article states:
“If you’re old enough to remember the Reagan administration, you remember the politically charged expression “trickle-down economics,” which referred to the theory that if you provide benefits and incentives to businesses and the wealthy, those benefits would trickle down to wage earners at lower socioeconomic levels.
In some ways, big data analytics is like trickle-down economics. Only the biggest healthcare providers with the deepest pockets can afford the kind of analytics platforms required to get useful intelligence from tens of thousands of patient records. But in theory, those benefits will trickle down to smaller providers that either don’t have the financial support or the large patient populations to do this type of data crunching on their own.”
We all knew that big data was something worth investing in, but save the world? that seems to be a little bit much. “50 Ways Big Data Can Save the World” showcases the new startup Bidgely, which aims to turn every appliance in your home into a data scientist, providing you with real time results on your energy usage.
The article states:
“Utilities worldwide are installing smart meters on homes and businesses, which means there could be as much as 50 terabytes of energy data that can emerge from a million or so homes in a year. The problem has been that there haven’t been very many ways to make good use of all this data to benefit the average consumer. But a startup called Bidgely, which raised a series A round from Khosla Ventures, says it has created algorithms that can dig into real-time smart meter energy-consumption data, can reduce consumers’ home energy use by between 4 percent to 12 percent, and can also deliver other beneficial home services to consumers.”
Whether you are looking to utilize big data to protect the environment, save lives, or boost business for your company, there are solutions available that can be very beneficial. Thanks to companies like Digital Reasoning, this technology is more affordable, accessible and customizable than ever.
Jasmine Ashton, November 12, 2012
Sponsored by ArnoldIT.com, developer of Augmentext
Big Data Analytics Boosted by IBM PureSystems
November 8, 2012
Big Data analytics is receiving a much-needed overhaul with a series of platforms from IBM. IBM released a new expert integrated system last month, called PureSystems, which allows users to connect to the system and receive specific pieces of computing power to monitor business operations. The article “IBM Launches New PureSystems for Transactions and Big Data Analytics” on Forbes shares the details of the new platform.
We learn:
“PureData System for Analytics builds on the Netezza acquisition to deliver results in seconds or minutes rather than hours. Francisco said it has the largest library of database analytical functions in the data warehouse market, said Francisco, and can scale across the terabyte or petabytes running on the system. It can support extremely high volume high speed analytics for clients like mobile phone carriers who want to identify potential churn and provide offers to retain customers.”
The article also states that one client has reported the ability to build and deliver more applications in the last three months than in the last three years using the targeted, high-volume, high-speed analytics.
However, we believe that attacking analytics in Big Data is only one piece of the enterprise data management solution. Using integrators such as Intrafind adds another layer of robust and reliable solutions so that enterprises find the data they need to analyze.
Andrea Hayden, November 08, 2012
Sponsored by ArnoldIT.com, developer of Augmentext
A Different Kind of Debate Analysis
November 6, 2012
Expert System is weighing in on the US presidential debates, linguistics style. The company supplies highlights of its findings in “Obama vs. Romney on Language: the Three Debates.” The semantic and linguistic analysis of the three presidential debates was performed using Expert System’s Cogito semantic platform.
The study found Governor Romney to be the more wordy of the two, by 14%, while President Obama favored the action verbs “do” and “make.” The press release also highlights some topical differences:
“Among the people cited, Obama spoke most often of Osama Bin Laden, followed by Gaddafi (reflecting on his foreign policy achievements). Former U.S. presidents were also regularly mentioned, George W. Bush most frequently in general, while presidents Lincoln and Eisenhower were most often cited by Obama, with Romney recalling Roosevelt and Reagan. Behind the United States, China was the second-most mentioned nation by both candidates, with Iraq (Obama) and Iran (Romney) in third place. Libya got more attention in the last two debates, but was more often cited by Obama (ranking behind Massachusetts) than by Romney.”
Though it is interesting, I’m not sure how helpful this data is. Expert System’s Luca Scagliarini admits that the analysis is “by no means a predictor” of the race’s outcome, but asserts confidence that semantic analysis can help anyone better understand any information. I suppose that is an important tenant of faith in the data analysis business.
Business and government organizations in several fields turn to Expert System for data management, collaboration, and customer relationship management. The company is based in Modena, Italy, and has satellite offices in Europe and the US.
Cynthia Murrell, November 06, 2012
Sponsored by ArnoldIT.com, developer of Augmentext
Crowdsourcing Intelligence Forecasts
November 5, 2012
It seems that spies, too, can jump upon the crowdsourcing bandwagon. Will they do so with a Bond-like flourish?
The BBC reports that “Intelligence Agencies Turn to Crowdsourcing.” The Intelligence Advanced Research Projects Activity (Iarpa) is searching for ways to tap into the much-lauded “wisdom of the crowds” to boost agencies’ ability to predict the future. Toward that end, they have helped research firm Applied Research Associates launch a Web site on which anyone can submit their predictions. The site aims to make the process fun, by awarding “reputation” points to those whose predictions come true, allowing them to advance through “missions.” Just what we need– a reason to root for that violent event we perceive simmering on the horizon! Wait. . . is this really such a good idea? I suppose I should just be glad it isn’t in the form of a video game. Yet.
Reporter Sharon Weinberger writes:
“The intelligence community has often been blasted for its failure to forecast critical world events, from the fall of the Soviet Union to the Arab Spring that swept across North Africa and the Middle East. It was also heavily criticized for its National Intelligence Estimate in 2002, which supported claims that Iraq had weapons of mass destruction.
“Those failures raised larger questions about how the intelligence agencies come up with forecasts, which is usually a deliberative process involving a large number of analysts. The Iarpa project, known officially as Aggregative Contingent Estimation, is looking at whether crowdsourcing can result in more accurate forecasts about future events than those traditional forms of intelligence estimation.”
Well, I suppose it couldn’t hurt. Right?
See the article for more details, like the controls against skewed results and efforts to identify the most effective forecasters. I, like Weinberger, think the site’s disclaimer is worth noting: “Forecast topics are not related to actual intelligence operations.” Sure, we believe that.
Cynthia Murrell, November 05, 2012
Sponsored by ArnoldIT.com, developer of Augmentext
-
- Subscribe to Beyond Search
Feature archive
News archive
-
