Full Motion Progress
Written by Peter Buxbaum
GIF 2010 Volume: 8 Issue: 8 (December)
During several intelligence postings with the Army and Department of Defense, Brigadier General Brian Keller (Ret.), saw firsthand how the deployment and analysis of full motion video dramatically aided the U.S. war effort in Iraq.
“One factor that made the surge in Iraq successful,” he related, “was our ability to successfully target al-Qaeda leadership as well as their media centers and bomb making facilities. These successes were related to the increased availability of full motion video.
“Video allows you to discern patterns of life and behaviors associated with people and the networks they operate in,” explained Keller, now a senior ISR strategist at Science Applications International Corp. (SAIC). “It leads us to other locations and people and provides an opportunity to surveille them. Observing patterns of people coming and going often leads to an opportunity to conduct a raid. These operations often result in the capture of people and equipment, and they provide further information that allows analysts and operators to move to the next objective. Instead of finding a needle in a haystack, we’re finding needles within needles.”
There is no question about the increased demand in the military and intelligence communities for access to and analysis and exploitation of full motion video. Experts say this is driven by the explosion in the number of available sensors and platforms that provide FMV: A few dozen assets 10 years ago have become thousands today. Estimates indicate that the equivalent of 17 years of video was taken in theater in 2009. The demand for and access to video also drives the need for more sophisticated analysis and exploitation tools, which industry is in the process of developing and introducing.
A number of technology developments contribute to the increased availability of and demand for full motion video by the military and intelligence communities. Besides the numbers of platforms and sensors able to provide FMV, the trend has also been driven by the increase in the available bandwidth, noted Commander Joe Smith, technical executive for the sensor assimilation division at the National Geospatial-Intelligence Agency, allowing video to be pushed and pulled to and from the edge of the network to its core.
“There has also been a massive increase in the use and availability of video technology,” said Smith. “Young soldiers and sailors are used to sending and receiving video. There has also been an increase in processing capabilities and in the ability to store data.”
Beyond Snapshots
The key added value that video brings over still imagery intelligence is the ability to observe targets over time. “One of challenges of still imagery is that it only provides snapshot in time,” said Charlie Morrison, director of spatial solutions business development at Lockheed Martin. “With video you can learn more about enemy activity.”
“Video allows persistence,” added Jon Armstrong, director of FMV solutions at Lockheed. “It allows for activity- based decision making. U.S. forces can target a vehicle, for example, by understanding where it is coming from or where it is going.”
In other words, FMV provides a capability to understand human activity over and above the insights to be derived from still imagery. “Fifty percent of human intelligence involves the observation of activity,” said Jane Bernat, director of integrated solutions at Overwatch Systems, a unit of Textron Inc. “If we wanted to take down an enemy compound, for example, a day of video surveillance will provide the understanding we need that would take a lot longer through other means. This represents a whole new way of doing business.”
As the functional manager for intelligence imagery, including video imagery, NGA plays an important role in the storage, standardization, analysis and dissemination of video data. “We are responsible for setting standards for processing and formatting this type of data,” said Smith.
The key data standard is the Motion Imagery Standards Profile, which outlines how data is to be formatted and handled and what metadata is to be included with it. Metadata is key to searching for relevant video data in a database, as well as synchronizing video data with other forms of intelligence data.
NGA also partners with the services and defense agencies that collect, analyze and disseminate video intelligence. “We don’t own or operate any of the FMV collectors,” said Smith. “We partner with the armed services and other government agencies to integrate video into the National System for Geospatial Intelligence.” NGA is a partner in the ground segments of intelligence collections systems like the Distributed Common Ground System (DCGS) that are fed by assets like Predator and Shadow.
NGA is also active in the Defense Intelligence Information Enterprise. “This is a new and growing capability that allows us to link together the information technology architectures of the military services and intelligence communities so that we can share information and move information back and forth,” said Smith. “In many cases we can also share computational and storage resources. NGA is a lead in both of those efforts, especially for motion imagery.”
Video Architecture
The tools used by NGA analysts are rudimentary, according to Smith, allowing analysts to play the video backwards and forwards and to add metadata tags. A flurry of home-grown and incompatible analysis tools around the military and intelligence communities led NGA to develop the National System for Geospatial Intelligence Objective Video Architecture (NOVA), which will be rolled out in the next few months.
NOVA will allow analysts located at diverse locations who are working on the same video stream to share their annotations and their tagging of objects and events, and to do so in a standardized way. “That way, someone who looks at the product sometime later can receive the benefit of those who looked at it earlier,” said Smith.
NOVA combines the geospatial information systems that NGA already provides with commercial broadcast capabilities. “It is similar to the system a television producer covering a football game or a news story has in the production van,” said Smith. “It allows you to rapidly search large volumes of motion imagery and video based on a specific context. We’re applying those same principles to Predator video. We have also made some improvements to the storage of the massive volumes of video data.”
Lockheed Martin, which is one of the two leads, along with Harris, on NOVA, is in the process of delivering similar capabilities to the Afghanistan theater under the auspices of Joint Forces Command. Developed by Lockheed, Harris and NetApp under a $29 million contract, several nodes of the Valiant Angel system were deployed to Afghanistan earlier in 2010. The system is designed to take control of the flood of video being generated by using broadcast television technology to help commanders collect, archive, search, analyze and share full motion video.
The Valiant Angel system incorporates tools and technologies from Lockheed Martin’s Audacity video analysis system, Harris’ Full-Motion Video Asset Management System (FAME), and NetApp’s Data ONTAP high-performance storage technology.
Each Valiant Angel node consists of a suite of high-capacity servers that store and archive video footage from multiple sensors and UAVs, together with a software system that allows users to catalog, tag, search and analyze video clips. “Users can search the archive for a specific person or vehicle, or they can fuse mapping, geospatial and multi-source intelligence data with video feeds to conduct in-depth analysis,” said Armstrong. Valiant Angel works with both archived video and with live, incoming video streams.
Tagging people and objects found in video feeds enables users to search the system for otherwise hardto- find footage. “For example, a red truck in front a building may be of interest,” said Armstrong. “The system allows users to query the system for the last time within 30 days the same vehicle was located at the same location.”
From an analysis and exploitation standpoint, full-motion video does not exist in a vacuum. FMV is another geospatial data type that is to be integrated with other data to provide richer and more useful intelligence products. The integration of multi-source intelligence represents a major theme in the enhancements being introduced to exploitation kits.
“The more appropriate analysis of video isn’t restricted to video on its own,” said Rob Mott, vice president for military and intelligence solutions at Intergraph. “It needs to be done integrated manner.” “The addition of different types of information such as human intelligence and signal intelligence provides a richer situational concept of what the warfighter can expect to see and experience when entering an area,” said Mark Wolsky, director of marketing at Overwatch. Georeferenced Intelligence
Key to performing analysis with the Valiant Angel system is the capability to overlay other georeferenced intelligence onto the video. “The overlays can be from any georeferenced source,” said Armstrong. “They are limited only by the imagination.”
Recent enhancements to Intergraph’s geospatial exploitation tool kit include capabilities to fuse full motion video, geospatial elevation data and satellite imagery into a single view. The tool allows analysts to overlay street names on video images to help “get the video oriented to the surrounding area,” said Mott. “Analysts can also compare archive video with real-time video to do change detection which is a more advanced type of analysis. Through the integration of all that data you can create a combined analytical environment.”
SAIC recently launched its Advanced Intelligence Multimedia Exploitation Suite (AIMES), a motion imagery exploitation system that enables intelligence analysts to fuse, exploit and report on motion imagery data. AIMES builds on SAIC’s existing Video Processing Capability line of products to include new advanced analytic functions.
AIMES is built on an open architecture that enables the rapid integration of third-party tools, algorithms and services. “This enables the retention of a consistent FMV infrastructure, while keeping pace with the new technologies,” said Keller. AIMES is compatible with DCGS services to enable the integration of all-source spatial and geospatial information “AIMES enables near real-time and forensic fusion of full motion video, all-source intelligence information, as well as synchronized visualization of raw data, chat and processed intelligence,” said Keller.
Overwatch combines a number of its different products to provide warfighters with FMV intelligence in near real-time. The company’s Tactical Remote Exploitation (T-REx) product, packaged in a ruggedized tablet computer or a small workstation, enables allsource intelligence analysis from an array of sources.
“T-REx enables analysts to review video and identify an area of interest and then realize multi-source forensic data based on the georeferenced coordinates,” Wolsky said. The analyst can then transmit intelligence products—such as three-dimensional renderings, or short video clips—to warfighters over handheld devices.
As FMV assumes ever-greater importance among intelligence analysts, collaboration tools that streamline the process of sharing and discussing video become increasingly important. A solution being developed by Merlin International includes a tool from a company called cut2it that allows users to share, engage with and collaborate within a video.
“What we have found in the ISR market is that users don’t want create multiple copies of the same video,” said J.P. Morgenthal, Merlin International’s chief architect. “With this tool, analysts identify the critical portion of a video and send a link to their colleagues to discuss and add knowledge to that particular clip.’
Merlin’s solution also includes real-time image processing software for FMV from MotionDSP that enhances the images viewed by analysts. The video feeds transmitted by UAVs can suffer from a number of deficiencies deriving from the shakiness of the platform, the standard resolution of its sensor and the dust and haze that are often present.
“Algorithms analyze how the neighboring frames in your clip are related to one another in a process called motion estimation,” said Sean Varah, chief executive officer of MotionDSP. “We recover the best available image information from multiple frames and reconstruct each frame of video with this information to enhance the appearance and quality of the video.”
Intergraph has also incorporated video enhancement and stabilization into its exploitation kit. “Video shot over the desert form a Predator following a convoy, for example, will usually include distortion due to heat haze,” said Mott. “This capability stabilizes, enhances and brightens the raw video and displays it in the analytical environment.”
Storage and Automation
For all of the progress that has been so far in the analysis and exploitation of FMV, the intelligence community and its vendors have further challenges to overcome. One involves the storage of the staggering volumes of video data intake.
“It is a tremendous issue from a policy perspective,” said Morrison. “How long must it be stored before it is discarded? Some users may want the data to be stored for a certain number of days and thrown out. Other may want to prioritize certain packages of data over others based on different criteria such as how often it is accessed. They need tools in order to be able to do this.”
“One thing that we have to get better at is understanding how to capture and retain the contextual data that goes along with the video. This becomes important when you are thinking about forensics. Data on why this video data was collected, who collected it, what was going on at the time, and what the results were are not associated with the video archives in place today. We have the ability to capture some of this data on our T-REx and Shadow platforms. Now we need to figure how to that make this information available to the community as a whole,” said Bernat.
Another capability that is on the drawing boards is the automation of some of the analytics processes. “Instead of having individuals staring nonstop at video, what is needed is more automated initial processing of video using different technologies such as pattern recognition and change detection,” said Armstrong. “We also need some additional indicators and alerts so that we can focus the attention of the human operator where there is higher probability of it being needed and where it can generate an actual outcome.”
Armstrong also expects that some of the video processing will be done onboard the platforms and sensors so that the data does not have to be downloaded and processed. “It’s a good idea to move some of these pattern recognition and change detection algorithms to the platform, and then transmit only those portions of the video deemed necessary off the platform,” he said.
Eventually, the quality of enhanced video will also be improved and there will be an ability to automatically classify objects seen by the camera. “We need to make software so that the sensor sees the way the human eye sees,” said Varah. “You can drive in a bouncy pickup truck and still see people and objects around you. The human brain has the capability of ignoring the shaking of the head and still be able to understand your surroundings.” ♦
