Fusion Comes of Age
Written by Karen E. Thuermer
GIF 2011 Volume 9: Issue: 8 (November/December)
With their dramatically enhanced ability to detect, locate, and track objects, advancements in technology for fusing data from multiple geospatial and other intelligence sensors are offering the military a host of new analytical and operational advantages.
The technology being developed is able not only to fuse information from different sensors of the same modality and imagery sensors of different modalities, but also to fuse information from different intelligence disciplines, develop new ways to reason and make decisions from fused information, and provide fusion-based solutions to hard problems in a net-centric environment.
Multi-source and multi-INT fusion is also examining measurements and databases for fused and composite signatures of targets of interest, conflation of multi-sensor and multi-modality data, and development of automated fusion exploitation algorithms for handling particularly difficult situations.
Companies such as Northrop Grumman, BAE Systems, Harris, Merrick & Co. and ERDAS/Intergraph are developing certain uses for multi-source and multi-INT fusion that benefit the military and intelligence gathering.
Northrop Grumman, for example, is making significant investments in multi-modality sensing technologies for a single aircraft. These technologies would leverage multiple sensor types such as electro-optical (EO), light detection and ranging (LiDAR), hyperspectral/multi-spectral (HSI/MSI), and synthetic aperture radar (SAR) on a single platform working in concert to collect data.
“Additionally, we are working to move processing and dissemination closer to the collection point instead of having to do all that work on the ground,” reported Sean Love, geospatial business development director at Northrop Grumman.
Such measures would cut costs and the time needed to bring timely and mission critical information to the user.
“Very simply, this would offer the military a better, more timely intelligence picture,” Love explained. “Multi-modality sensing provides a faster response with more detail as well as mission specific collection. All this without needing to introduce another aircraft to get the data needed.” Without multi-modality sensing, the only way to get all the different types of imagery data is to fly multiple aircraft, he explained.
For example, collecting EO and LiDAR information would be done with one aircraft where another aircraft would be dispatched to collect HSI/MSI data and still a third aircraft would be flown to collect SAR data. “The new technology would enable all of these missions to be flown as a single mission,” he said.
Sensor Payloads
Merrick has been integrating multi-source airborne sensors for close to a decade. The typical sensor payload consists of a high resolution, true color EO digital camera and a specialized LiDAR system containing a high grade position orientation system (POS) consisting of an IMU and survey grade GPS system.
“Depending on client’s needs, we often integrate other airborne sensors such as color infrared (CIR) cameras, video cameras, thermal cameras, and hyperspectral cameras,” outlined Matthew Bethel, manager of systems engineering at Merrick. Among hyperspectral cameras, Merrill has past experience utilizing VNIR, SWIR, and LWIR sensors.
“With the use of the POS, all integrated sensor data can be fully orthorectified at the final deliverable stage through direct georeferencing processes combined with a very detailed LiDAR surface model,” Bethel explained. “This allows Merrick to produce the highest accuracy of final products possible.”
Where most integration companies might stop here, Merrick adds extensive integration and total production workflow development and processing experience.
“We have working knowledge of all major airborne mapping systems and close professional relationships with their vendors,” Bethel remarked.
The company has developed its own multi-sensor flight planning extension for Esri’s ArcGIS. This allows Merrick to plan for optimal acquisition and minimize reflights for any and all sensors.
“The products of this flight plan software are used during acquisition with the onboard flight management system,” Bethel explained.
Merrick has developed a very powerful data management system to organize, archive, transfer and process all multi-sourced sensor data.
“This foundation is vital when processing multi-terabyte data missions flown day after day,” he added.
Merrick has detailed workflows for final processing of all sensor level data products. These workflows have been optimized with high performance computing techniques to produce end products as quickly as possible within a scalable processing environment.
At the core of its LiDAR processing is Merrick’s Advanced Remote Sensing (MARS) software. “The primary purpose of MARS is to provide an end-to-end LiDAR processing software suite of tools,” Bethel explained.
The second strength of MARS is its powerful data fusion engine, which provides the user true multi-sensor data analysis and fusion between all raster/vector geospatial products and the actual LiDAR point clouds.
In Bethel’s opinion, what separates MARS from all other data fusion packages is its strength to operate on the actual full resolution LiDAR point cloud data, not derivative products of them, which are often rasterized and/or severely decimated. “Within the MARS suite, there are many filter and analysis tools that are completely automated and batchable,” he said.
Examples of these include classifying and extracting buildings under vegetation and helicopter landing zone analysis.
Using these powerful tools and extensive workflows, Merrick is unique from the aspect that all data fusion processes preserve, respect and track the accuracy of all input datasets throughout the entire process so that the end products have a true and measureable level of accuracy to reduce uncertainties, Bethel maintained.
“The same workflow and tools preserve temporal information throughout the data fusion process,” he said.
The system offers the military many benefits, such as simultaneous acquisition with multiple sensor payloads.
“This lowers the cost of acquisition, keeps the temporal state of all collected datasets consistent, and speeds up the processing throughput duration to end products,” Bethel reported.
Another advantage is the reduced uncertainty of locating and positioning targets through the process of propagating the different accuracies of each sensor used in the fusion engine.
“Finally, huge savings are found in the ability to minimize the amount of data that needs to be moved downstream by processing multi-source sensor data in an efficient manner and generate usable end products,” he added.
The technology solves a host of challenges, particularly the barriers to collecting as much information as possible to make confident decisions for target selection.
An example of this may be collecting HSI derived targets but not having adequate locational information for them, extracting a helicopter landing zone but not having three-dimensional power lines above detected by LiDAR, or being able to detect hot spot activity in a thick canopy environment but not having the LiDAR vegetation penetration capabilities to find the ground footpaths leading to the targets.
“These are only some of real world examples where proper multi-source sensor fusion collection systems, processing and final fusion derived data products can provide the most exploitable information possible,” Bethel pointed out. “When utilized properly, this technology truly lives up to the goal of the sum being greater than the parts.”
Visual and Algorithmic Environments
Products from Intergraph and ERDAS, which were recently united under common ownership, have a long history of enabling multi-source fusion. Both product lines support the use of a vast array of varying data formats and sensor types, and enable users to combine these sources in both visual and algorithmic environments to derive information not attainable from any one source in isolation.
According to Tom Lobonc, executive product director, Intergraph Government Solutions, Intergraph’s GeoMedia provides a unique ability to access geospatial data in almost any form to bring an integrated geospatial view together.
“Along with a broad set of powerful analytic and editing tools, GeoMedia enables customers in several industries to efficiently manage and understand their investments in geospatial assets,” Lobonc pointed out.
ERDAS provides a spatial modeling environment that allows users to create custom processing models that fuse various data sources to create derived information products.
“Both ERDAS and Intergraph’s GeoMedia suite provide a strong set of interfaces for data and metadata exchange that fully align with global spatial data infrastructure standards such as those specified by OGC and the INSPIRE Directive,” Lobonc explained.
Key features resulting from the integration of Intergraph and ERDAS products will be the addition of numerous vector processing capabilities, the ability for a user to create customized processing operators, an expansion of the existing operators to include many more ERDAS legacy functions, and the ability to use Python scripting to create sophisticated processing chains out of both supplied operators and third-party processing functions.
These processing chains can then be uploaded to the ERDAS Apollo server and made available via an OGC Web Processing Service (WPS) to enable users connected via thin clients to execute a wide variety of geospatial processing remotely without requiring detailed knowledge of the algorithms themselves.
“The algorithms actually have built in intelligence that helps guide the user to data inputs specifically required for the algorithms while simultaneously excluding all unusable data from the available selections,” he said.
These processing models provide the military with multiple benefits. Among them are processing on demand on the server side (no local data required), no requirement for the user to have expertise in processing steps, a fully interoperable access and delivery mechanism using OGC services, and the ability to request and receive derived information products on mobile devices.
The processing models are also in high demand. “In fact, the ability for remote users to execute custom geospatial processing on demand over a thin client is in extremely high demand as we move from an environment consisting largely of existing stock products to an environment that demands highly flexible information products tailored to specific applications,” Lobonc stressed.
The models also solve a number of challenges, as exemplified in the following case study example:
- An operator is responsible for determining potential egress routes and helicopter landing zones to use at the conclusion of a mission.
- The information was prepared at the forward operating base with multiple data sources before the mission was initiated. At the conclusion of mission activities, the operator has determined that the planned egress routes and extraction sites are no longer usable.
- The operator uses his mobile device to contact the server and request WPS models that can generate a new mobility analysis and potential landing zones. The models are selected and they provide the operator with the best possible data sources for the analysis.
- The operator makes the final data source selection and initiates the processing. Within minutes a new mobility analysis and set of landing zones are sent back to the operator’s mobile device and they are immediately used to begin the egress and call in air support.
Multiple Initiatives
Harris Corporation has many initiatives ongoing to address multiple aspects of multi source/sensor fusion. These projects range from one-stop shop multi-INT data access systems to multi intelligence data correlation in space-and-time with its Full Motion Video Asset Management Engine (FAME).
“We are also providing multi-source product generation, including tactical imagery and LIDAR fusion,” commented Sleighton Meyer, spokesperson for Harris Government Communications Systems.
These initiatives provide multiple benefits to the military. For one, they provide access to multiple forms of intelligence via a single source and relate them in space and time.
“This enables the military to solve more complicated intelligence problems by having the bigger, more integrated picture,” Meyer said.
Relating multi-INT in space and time allows reconstruction of events for forensics as well as activity modeling. It also enables missions that were previously impossible by combining the information from multiple sensors.
Normally, timely tactical imagery provides tremendous situational awareness. Unfortunately, the inaccuracy of imagery prevents it from being used for other purposes. However, because Harris’ multi-sensor activities involve fusion of tactical imagery with LIDAR data, they enable targeting through very timely tactical imagery.
“Our multi-sensor solutions fuse the LiDAR and tactical imagery to provide tremendous accuracy, enabling users to select points from 3-D scenes that can be used in conducting precision missions,” Meyer added.
BAE Systems, meanwhile, has been collaborating with government, industry and academia partners to advance, develop, and integrate multi-INT ISR processing, exploitation and dissemination (PED).
“Using a standards-driven approach enables interoperability between PED systems, algorithms and multi-INT fusion,” remarked Jason Latonio, BAE engineer.
The goal is to provide answers to the end user, without necessarily focusing on what sensor, platform or data phenomenology the intelligence data is derived from, Latonio explained. “These answers can then be used and fused with other information, and visualized and utilized in any mobile, Web or workstation client that supports the standards landscape.
Capabilities are delivered into a virtual machine-based application infrastructure at multiple security levels as operational prototypes.
“Selected operational users can reach into the environment to access the prototype multi-INT systems and services for operational testing and evaluation,” Latonio continued. “The warfighter, therefore, gains access to leading-edge technology early, helping refine usability and requirements before it is made available to the larger intelligence community.”
Capabilities to date involve data management for IMINT, SIGINT, LiDAR, HSI, IR, SAR and ground moving target indication. Search and discovery tools allow correlation across the data phenomenologies, where the data can then be processed by a suite of tools and services, such as data fusion algorithms, automated change detection, extraction algorithms, normalcy and anomaly production and a full spectrum of analytical clients.
“The results are published to fused track, detection, extraction and reporting repositories where the intelligence information is than discoverable and retrievable,” Latonio said. It can also be used by referenced tools, thus enabling a continuous intelligence production cycle, so intelligence information is improved and enriched continuously over time.
Future Directions
Without a doubt, multi-modality sensing technology is the future. From Northrop Grumman’s perspective, as budgets continue to be stretched, the ability to fly one mission that would accomplish what three or four missions today can accomplish will become more and more important.
“This technology also enables us to move toward a true multi-INT picture,” said Love. “The information gathered in a single multi-modality mission will be much more uniform than the way information is gathered today, since there is a time lapse between different types of data collection.
Latonio of BAE Systems sees future activities including the integration of WAMI, WAPS and FMV tools and services. “These capabilities will benefit from existing applications already in place,” he said. “Capabilities are developed in rapid development cycles, with continuous user feedback, ensuring the highest quality product is delivered to users.
Additionally, using interface and data standards within the environment enables a great deal of re-use across capabilities and tools, eliminating stovepipes and proprietary implementations. “Together with reduced cycle times to operations and reduced risk with user acceptance, this generates cost savings for our military, enabling GEOINT in the hands of users,” he concluded.
Multi source access and correlation in space and time enables activity modeling, Meyer noted. “This sets the stage for activity-based, predictive intelligence. This will allow analysts to focus more on the intent of our enemies.”
Bethel sees its biggest area of growth in more standardization of sensor product levels. “This refers to file types, calibration processes and overall workflows,” he said. “This will promote more streamlined processing and improved consistency of varying sensors’ data, yielding better accuracies of derivative products.”
From this breakthrough will come more automation and faster processing aiming for near- or real-time data products, he maintained.
Lastly, in the future, the sensor fusion industry will be able to utilize modularized sensor payloads that have interchangeable components. The conversion form a LiDAR/SWIR/RGB collection to a LiDAR/LWIR collection will be effortless.
Meanwhile, the industry will continue on the trend of fusing more and more disparate data sources to provide complex analyses on demand.
“There will be a rapidly growing percentage of the overall information product requests that will be handled dynamically as needed rather than relying upon pre-generated products carried into the battlefield,” remarked Lobonc.
As the adoption of Open Geospatial Consortium and other standards increases, he added, users will receive the benefit of greatly increased interoperability among different software systems--which will also provide a much greater variety of possible information products that can be provisioned to a single receiving client. ♦
