Researchers Glean Audio from Video
July 10, 2015
Now, this is fascinating. Scary, but fascinating. MIT News explains how a team of researchers from MIT, Microsoft, and Adobe are “Extracting Audio from Visual Information.” The article includes a video in which one can clearly hear the poem “Mary Had a Little Lamb” as extrapolated from video of a potato chip bag’s vibrations filmed through soundproof glass, among other amazing feats. I highly recommend you take four-and-a-half minutes to watch the video.
Writer Larry Hardesty lists some other surfaces from which the team was able reproduce audio by filming vibrations: aluminum foil, water, and plant leaves. The researchers plan to present a paper on their results at this year’s Siggraph computer graphics conference. See the article for some details on the research, including camera specs and algorithm development.
So, will this tech have any non-spying related applications? Hardesty cites MIT grad student, and first writer on the team’s paper, Abe Davis as he writes:
“The researchers’ technique has obvious applications in law enforcement and forensics, but Davis is more enthusiastic about the possibility of what he describes as a ‘new kind of imaging.’
“‘We’re recovering sounds from objects,’ he says. ‘That gives us a lot of information about the sound that’s going on around the object, but it also gives us a lot of information about the object itself, because different objects are going to respond to sound in different ways.’ In ongoing work, the researchers have begun trying to determine material and structural properties of objects from their visible response to short bursts of sound.”
That’s one idea. Researchers are confident other uses will emerge, ones no one has thought of yet. This is a technology to keep tabs on, and not just to decide when to start holding all private conversations in windowless rooms.
Cynthia Murrell, July 10, 2015
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Deep Learning System Surprises Researchers
June 24, 2015
Researchers were surprised when their scene-classification AI performed some independent study, we learn from Kurzweil’s article, “MIT Deep-Learning System Autonomously Learns to Identify Objects.”
At last December’s International Conference on Learning Representations, a research team from MIT demonstrated that their scene-recognition software was 25-33 percent more accurate than its leading predecessor. They also presented a paper describing the object-identification tactic their software chose to adopt; perhaps this is what gave it the edge. The paper’s lead author, and MIT computer science/ engineering associate professor, Antonio Torralba ponders the development:
“Deep learning works very well, but it’s very hard to understand why it works — what is the internal representation that the network is building. It could be that the representations for scenes are parts of scenes that don’t make any sense, like corners or pieces of objects. But it could be that it’s objects: To know that something is a bedroom, you need to see the bed; to know that something is a conference room, you need to see a table and chairs. That’s what we found, that the network is really finding these objects.”
Researchers being researchers, the team is investigating their own software’s initiative. The article tells us:
“In ongoing work, the researchers are starting from scratch and retraining their network on the same data sets, to see if it consistently converges on the same objects, or whether it can randomly evolve in different directions that still produce good predictions. They’re also exploring whether object detection and scene detection can feed back into each other, to improve the performance of both. ‘But we want to do that in a way that doesn’t force the network to do something that it doesn’t want to do,’ Torralba says.”
Very respectful. See the article for a few more details on this ambitious AI, or check out the researchers’ open-access paper here.
Cynthia Murrell, June 24, 2015
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
MIT Discover Object Recognition
June 23, 2015
MIT did not discover object recognition, but researchers did teach a deep-learning system designed to recognize and classify scenes can also be used to recognize individual objects. Kurzweil describes the exciting development in the article, “MIT Deep-Learning System Autonomously Learns To Identify Objects.” The MIT researchers realized that deep-learning could be used for object identification, when they were training a machine to identify scenes. They complied a library of seven million entries categorized by scenes, when they learned that object recognition and scene-recognition had the possibility of working in tandem.
“ ‘Deep learning works very well, but it’s very hard to understand why it works — what is the internal representation that the network is building,’ says Antonio Torralba, an associate professor of computer science and engineering at MIT and a senior author on the new paper.”
When the deep-learning network was processing scenes, it was fifty percent accurate compared to a human’s eighty percent accuracy. While the network was busy identifying scenes, at the same time it was learning how to recognize objects as well. The researchers are still trying to work out the kinks in the deep-learning process and have decided to start over. They are retraining their networks on the same data sets, but taking a new approach to see how scene and object recognition tie in together or if they go in different directions.
Deep-leaning networks have major ramifications, including the improvement for many industries. However, will deep-learning be applied to basic search? Image search still does not work well when you search by an actual image.
Whitney Grace, June 23, 2015
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Video and Image Search In the News
June 17, 2015
There’s been much activity around video and image search lately. Is it all public-relations hype, or is there really progress to celebrate? Here are a few examples that we’ve noticed recently.
Fast Company reports on real-time video-stream search service Dextro in, “This Startup’s Side Project Scans Every Periscope Video to Help You Find the Best Streams.” Writer Rose Pastore tells us:
“Dextro’s new tool, called Stream, launches today as a mobile-optimized site that sorts Periscope videos by their content: Cats, computers, swimming pools, and talking heads, to name a few popular categories. The system does not analyze stream text titles, which are often non-descriptive; instead, it groups videos based only on how its algorithms interpret the visual scene being filmed. Dextro already uses this technology to analyze pre-recorded videos for companies … but this is the first time the two-year-old startup has applied its algorithms to live streams.”
Meanwhile, ScienceDaily reveals an interesting development in, “System Designed to Label Visual Scenes Turns Out to Detect Particular Objects Too.” While working on their very successful scene-classification tool, researchers at MIT discovered a side effect. The article explains that, at an upcoming conference:
“The researchers will present a new paper demonstrating that, en route to learning how to recognize scenes, their system also learned how to recognize objects. The work implies that at the very least, scene-recognition and object-recognition systems could work in concert. But it also holds out the possibility that they could prove to be mutually reinforcing.”
Then we have an article from MIT’s Technology Review, “The Machine Vision Algorithm Beating Art Historians at Their Own Game.” Yes, even in the highly-nuanced field of art history, the AI seems to have become the master. We learn:
“The challenge of analyzing paintings, recognizing their artists, and identifying their style and content has always been beyond the capability of even the most advanced algorithms. That is now changing thanks to recent advances in machine learning based on approaches such as deep convolutional neural networks. In just a few years, computer scientists have created machines capable of matching and sometimes outperforming humans in all kinds of pattern recognition tasks.”
Each of these articles is an interesting read, so check them out for more information. It may be a good time to work in the area of image and video search.
Cynthia Murrell, June 17, 2015
Sponsored by ArnoldIT.com, publisher of the CyberOSINT monograph
Tweets Reveal Patterns of Support or Opposition for ISIL
March 31, 2015
Once again, data analysis is being put to good use. MIT Technology Review describes how “Twitter Data Mining Reveals the Origins of Support for the Islamic State.” A research team lead by one WalidMagdy at the Qatar Computing Research Institute studied tweets regarding the “Islamic State” (also known as ISIS, ISIL, or just IS) to discern any patterns that tell us which people choose to join such an organization and why.
See the article for a detailed description of the researchers’ methodology. Interesting observations involve use of the group’s name and tweet timing. Supporters tended to use the whole, official name (the “Islamic State in Iraq and the Levant” is perhaps the most accurate translation), while most opposing tweets didn’t bother, using the abbreviation. They also found that tweets criticizing ISIS surge right after the group has done something terrible, while supporters tended to tweet after a propaganda video was released or the group achieved a major military victory. Other indicators of sentiment were identified, and an algorithm created. The article reveals:
“Magdy and co trained a machine learning algorithm to spot users of both types and said it was able to classify other users as likely to become pro- or anti-ISIS with high accuracy. ‘We train a classifier that can predict future support or opposition of ISIS with 87 percent accuracy,’ they say….
“That is interesting research that reveals the complexity of the forces at work in determining support or opposition to movements like ISIS—why people like [Egypt’s] Ahmed Al-Darawy end up dying on the battlefield. A better understanding of these forces is surely a step forward in finding solutions to the tangled web that exists in this part of the world.
“However, it is worth ending on a note of caution. The ability to classify people as potential supporters of ISIS raises the dangerous prospect of a kind of thought police, like that depicted in films like Minority Report. Clearly, much thought must be given to the way this kind of information should be used.”
Clearly. (Though the writer seems unaware that the term “thought police” originated with Orwell’s Nineteen Eighty-Four, the reference to Minority Report shows he or she understands the concept. But I digress.) Still, trying to understand why people turn to violence and helping to mitigate their circumstances before they get there seems worth a try. Better than bombs, in my humble opinion, and perhaps longer-lasting.
Cynthia Murrell, March 31, 2015
Stephen E Arnold, Publisher of CyberOSINT at www.xenky.com
-
- Subscribe to Beyond Search
Feature archive
News archive
-
