Standards for Motion
Written by Peter Buxbaum
GIF 2011 Volume: 9 Issue: 5 (July/August)
The demand in the military and intelligence communities for access to and analysis and exploitation of motion imagery and full motion video—together with the availability of new and improved technologies—has yielded ever-greater sources of video in recent years.
The number of available sensors and platforms that provide full motion video has exploded from a few dozen assets 10 years ago to thousands today. The volume of video generated annually by the U.S. military in the Southwest Asia theater can be measured in dozens of years.
All of this has led to a requirement that video segments from different sources be compatible with one another, so that they can be collected, read, analyzed, exploited and disseminated in an interoperable fashion. And this, in turn, has led the Department of Defense to promulgate standards for the video data, metadata, transport, and other features and functions associated with video.
DoD did not have to work in a vacuum. In the commercial world, the Society of Motion Picture and Television Engineers (SMPTE) some years ago adopted standards for the use of equipment in those industries. The adoption of the SMPTE standards has allowed studios to mix and match equipment from different manufacturers without fear that they will be incompatible. DoD, through the Motion Industry Standards Board (MISB) of the National Geospatial- Intelligence Agency, the functional manager for intelligence imagery, has adapted the SMPTE standards to military and intelligence requirements.
“We believe standards are absolutely critical,” said Jon Armstrong, director of full motion video solutions at Lockheed Martin. “The processing, exploitation and dissemination of video need to be sensor- and platform-agnostic.”
“The bottom line is that you can’t build software if you don’t have standards to build them to,” said Jane Bernat, director of integrated solutions at Overwatch, an operating unit of Textron Systems, a Textron Inc. company, which offers software and hardware for the capture and analysis of motion imagery. “Data exported in a proprietary format is difficult or impossible to use.”
“The MISB has done a good job at not reinventing the wheel,” said Sean Varah, chief executive officer of MotionDSP. “For example, the video transport formats that MISB has adopted are based on the commercial MPEG format. They have used the commercial specifications and have found places where special data required by the military can be inserted.”
Guidelines and Practices
The MISB was established under a DoD directive “to formulate, review, and recommend standards for motion imagery, associated metadata, audio and other related systems” for DoD, the intelligence community and National System for Geospatial-Intelligence. Motion imagery systems and components for use within DoD and the intelligence community are subject to MISB requirements. MISB releases include engineering guidelines, recommended practices and standards.
Engineering guidelines provide guidance that implementers should follow whenever possible, and often evolve into standard practices or standards. Recommended practices represent implementation practices that the MISB strongly recommends all implementers and programs should follow, and that may be ignored only with the consent of the MISB. A standard represents a requirement for all compliant systems.
To be MISB compliant, any new motion imagery system must be digital, produce a MPEG-2 transport stream, use one of three standards for image compression and produce non-destructive metadata.
The key standard is the Motion Imagery Standards Profile (MISP), which outlines how data is to be formatted and handled and what metadata is to be included with it. Metadata is key to searching for relevant video data in a database as well as synchronizing video data with other forms of intelligence data.
A MISP-compliant file or stream must include three components: motion imagery; metadata in prescribed format; and a media container— the package that carries the motion imagery, metadata or both. An MPEG-2 transport stream containing both imagery and metadata must be present. Video transported via the Real-Time Transport Protocol carries motion imagery and metadata as separate streams. Time stamping and synchronization must always be included in the metadata and/or media container.
The metadata associated with the imagery must describe the platform and sensor context of the imaging collection, the geospatial environment of the imagery and the security level of the imagery. The Key Length Value metadata format prescribed by the MISB is an adaptation of the SMPTE standard.
“The primary thing that standards do is to provide for data interoperability,” said Varah. “Video applications in the commercial space have been standard-centric for a long time. That means that you can buy a box from Sony and plug it into one from Panasonic or Harris, and all are running on the same standards. Video on the military and intelligence side has traditionally been more ‘siloed’ or proprietary, until recently. The MISB has set up working groups to discuss the intricacies of standards to the nth detail, to adopt those standards and to force vendors to support them.”
The more widespread adoption of the MISB standards has led to much higher levels of interoperability, according to Varah. At a military event in 2009, he recalled, video vendors were still having trouble getting hardware and data to connect with one another. By 2010, six or seven more vendors had adopted the MISB standards. “As a result, we were quickly able to connect up and get our products to work together,” said Varah. “It sounds basic, but is very useful for the customer.”
Given these facts, it is not surprising that more and more vendors are adopting the MISB standards for their devices. But in some cases, legacy formats must also be supported.
“To date, the majority of what we have built and produced to support government motion imagery and full motion video from a systems perspective is all MISB compliant,” said Armstrong. “But further work led to the realization that not all platforms produce MISB-compliant data, so we are producing products that support both.”
Audacity, developed by Lockheed Martin, is a video analytics engine that tags, sorts and catalogs digital footage. Audacity is a component of Valiant Angel, an integrated video analysis system employed by the U.S. military.
Generation of MISB-non-compliant data appears to happen most often with tactical UAVs, according to Bernat. “There is no true enforcement of the standards for tactical UAVs. They are not deployed as part of a national ISR strategy, but as an organic asset of a particular service unit.
“It is still possible to view the video,” Bernat added, “but problems arise if you want to transmit the data to a central location and use it centrally.”
Metadata Issues
Many of the problems associated with non-compliant data arise from garbled metadata, according to Bernat. This may lead images to be directionally disoriented, or may prevent users from speeding up or slowing the video because it is difficult to understand the original data frame rate.
“Non-compliant metadata can cause analysts to misunderstand the geospatial position of the image, leading to an inappropriate interpretation of what the analyst saw,” said Bernat. “An experienced analyst may be able to adapt to the situation, but there are many junior analysts who won't properly understand the video if the data is rendered incorrectly.”
“We would all be better served if everything were MISB compliant,” said Armstrong. “But some platforms are not there yet.”
Overwatch’s MISB-compliant V-TRAC Basic is a full motion video exploitation tool that provides analysts with the ability to receive and analyze full motion video. V-TRAC Basic enables analysts to locate video data in an area or time of interest, correlate it with existing still imagery, perform mensuration and integrate it with exploitation tool sets.
Using V-TRAC Basic, analysts are able to quickly link video areas of interest with georeferenced data in order to perform deeper image exploitation. V-TRAC Basic can also be used to enable federated discovery of motion imagery, still imagery and other GEOINT data sources. Motion imagery clips can be logged based upon mission, platform and sensor metadata.
“Without compliance to the MISB standards, we would not be able to do that, and it would render the tool useless,” said Bernat.
V-TRAC Pro, scheduled for release in the third quarter of 2011, will provide additional full motion video capabilities, said Bernat, and will also be MISB compliant.
The MISB-compliant software from MotionDSP enhances the images viewed by analysts. The video feeds transmitted by UAVs can suffer from a number of deficiencies deriving from the shakiness of the platform, the standard resolution of its sensor, or dust and haze. Algorithms analyze how the neighboring frames in a clip are related to one another in a process called motion estimation. The software discovers the best available image information from multiple frames and reconstructs each frame of video with this information to enhance the appearance and quality of the video.
Companies like MotionDSP comply with a “hard requirement” from customers to provide MISB-compliant products, said Varah. “The fact that we are fully compliant with MISB is an important feature of our product. The biggest impact of standards compliance on our software is that it makes it much easier for us to integrate into existing large systems. Our customers need to plug our software into their video systems. It’s like buying a TV with a Blu-ray; you want a guarantee that the two will work together.”
The MISB standards also provide smaller companies with a level playing field to build software and integrate into large systems. “They also force those large systems to develop an open architecture that can accommodate compliant software,” said Varah. “Every DoD request for proposals for video products and services includes MISB compliant data support.”
The MISB standards are not a panacea. They do not solve every potential problem, but are continuing to be developed to become more comprehensive. “Standards are not a perfect science,” said Bernat. “They are subject to interpretation, but given a standardized model to build software, you can avoid most major problems with interoperability.”
“There can be some differences of opinion as to what is MISB compliant,” added Armstrong. “Some video technologies claim to be compliant, but when they are tested, they may or may not be. That is always the standards challenge. You have to narrow it down to get it right.”
The development of new technologies can also challenge a standards regime. “When someone invents a new sensor with new capabilities, it often runs ahead of the standards,” said Bernat. “Then it has to be rolled back in.”
Expanding to Content
The MISB standards as they stand now are not comprehensive. But they are likely to be expanded to include descriptions of video content as well as the quality of the feed.
“We don't really follow MISB standards as much as the requirements of our customers,” said Sadiye Guler, chief executive officer of IntuVision Inc. The company provides tools that facilitate real-time monitoring event detection for security and surveillance applications and post-event video analysis for the Navy Criminal Investigative Services, among other DoD and intelligence customers.
“What we do is look into the content of the video to understand what is going on— whether, for example, you are looking at vehicles or people, and what direction they are moving in. Most of the video we see is MISB compliant,” Guler said.
A potential next step for the MISB, according to Guler, will be standards that allow for a description of objects found in the video within the metadata. Today’s metadata standards include information on the geolocation of the scene and the sensor and platform that generated the video.
“Content information is what we add with our application,” she explained. “But there is no standard yet for adding content information into that metadata stream. There is not a tag for identifying specific objects of interest that may be found in the video. Adding content information to the metadata will encourage the development of open source tools that people can build on, rather than everyone developing their own proprietary solution.”
Adding content information will also make searching for and interpreting video that much easier, Guler contended. “It will enable the analysis of a video stream or file in real time. An analyst won't have to watch two hours of video to find events or objects of interest. Instead, the metadata can be queried to look, for example, for large vehicles present at a given location within a specific time range. That will be huge in terms of developing better analytics, detecting patterns and being able to respond to things more quickly.”
Armstrong expects upcoming MISB standards to tackle the challenging and growing field of wide area motion and image sensors. “The MISB is working to get a coalition to coalesce around what data formats these types of sensors will handle, how the data will be handled, and what the expectations are of the metadata associated with this type of video,” he said. “There is a lot of work to be done to support wide-area motion imagery.”
Varah expects the MISB to go beyond data interoperability standards to tackle video quality standards. “They will be coming up with better ways to describe video qualities,” he explained. “One of the challenges with video is that it is hard to describe the quality of the video you need. Some applications might require the video to be able to distinguish a person from an animal. Some might need to be able to identify a vehicle, while others want to be able to read the license plate.”
Varah expects upcoming MISB standards to develop a numeric system for describing video quality. He also expects future MISB standards to govern video analytics. “They are really just getting started,” he said. ♦
Keeping an Eye on Imagery Standards
David L. Irvin, an NGA employee and the chairman of the Motion Imagery Standards Board, recently responded to questions about the work of the board.
Q: What do MISB standards cover?
A: To adequately answer this question, a clear definition of motion imagery/full motion video is essential.
Motion imagery is defined as imagery [a likeness or representation of any natural or man-made feature or related object or activity] utilizing sequential or continuous streams of images that enable observation of the dynamic temporal behavior of objects within the scene. Motion imagery temporal rates—nominally expressed in frames per second—must be sufficient to characterize the desired dynamic phenomena. Motion imagery is further defined as including metadata and nominally beginning at frame rates of 1 Hz (1 frame per second) within a common field of regard. Full motion video (FMV) falls within the context of these standards.
With this baseline knowledge, the MISB’s role is to ensure the development, application and implementation of standards that maintain interoperability and quality for FMV, associated metadata, audio and other related motion imagery systems in the Department of Defense, the intelligence community and the National System for Geospatial Intelligence (NSG). The standards vetted through the MISB are subject to the DoD IT Standards Registry and the Intelligence Community Standards Registry for community recognition and use. In addition, the MISB supports the standardization efforts for motion imagery in NATO, through its role as custodian of Standardization Agreement 4609 Digital Motion Imagery. The NATO standards are in lockstep with the MISB standards, ensuring international interoperability. MISB standards do not apply to video teleconference, video telemedicine and video support services applications, although many of these applications use MISB standards.
Q: How are MISB standards enforced?
A: MISB standards are codified in the Motion Imagery Standards Profile (MISP) and mandated in the DoD and intelligence community standards registries. The MISP is the compilation of technical guidance for motion imagery across DoD, the IC and NSG. The Joint Interoperability Test Command provides certification through validation and conformance testing based on the MISP standards. Successful testing provides the certification to warrant that the systems are compliant with applicable standards and will perform as advertised.
Q: Why are MISB standards important?
A: No single commercial motion imagery standard will provide all the guidance necessary to build interoperable systems for use across the diverse missions of DoD, the IC and NSG. The MISB standards are the foundation for ensuring this interoperability across and within DoD, the IC and NSG. Interoperability through standards results in cost savings and increased opportunities for the application of multiple sources for operations. MISB formats and standards also help to simplify and streamline the acquisition, operation and maintenance of ISR systems across DoD, the IC and NSG.
Q: How are MISB standards being continually developed?
A: MISB standards are based largely on open, international, commercial standards enabling the leveraging of large commercial R&D investments. Additionally, the MISB continually interacts with subject-matter experts and stakeholders from the government community, industry, national labs and academia to discover, research and develop significant opportunities to advance standards in the interest of DoD, the IC and NSG. Engagement with the motion imagery community through technical exchange meetings, conferences, and workshops builds productive standards and other guidance for community concurrence and adoption.
Q: What sorts of classes of metadata are covered by MISB standards?
A: The classes of metadata covered by the MISB standards include the following: platform/sensor; calibration; encoder; and processing.
One of the MISB’s recent success stories was the approval of MISB Standard 0902.1, “Motion Imagery Sensor Minimum Metadata Set.” This standard prescribes the absolute minimum metadata required to be MISP-compliant. This minimum set includes the following information: time and place; platform attributes; sensor attributes; and security and classification information.
These metadata classes describe the environment from which the imagery is collected allowing for the spatial, temporal and spectral accuracy required to perform detailed analysis.
Q: What is not covered by these classes of metadata at this point?
A: The complete defining of content metadata, describing a full range of activities for analysis, to include the where and when but more importantly expanding the information to tell the story of the activity of individuals or groups, how they are moving, direction of movement, or even the activity being performed.
Q: What are MISB's plans for the future in this area?
A: The MISB will continue to ensure that standards for DoD, the IC and NSG will remain current with commercial standards, technology and operational requirements. A focus on further development in support of content metadata will form an increasingly important and large part of the MISB’s development of guidance. This work will much more directly support the processing, exploitation and dissemination process in that it will help enable the rapid and effective interpretation of motion imagery with respect to mission requirements. ♦
