.jpg)
Seeing With Your Brain
MGT 2008 Volume: 6 Issue: 3 (May/June)
DARPA and NGA are looking at merging neuroscience with computers to create
technology that could speed the delivery and improve the quality of situational awareness.
By Cheryl Gerber
Imagine a computer that can read human brain waves to assess the lay of the land. It might seem futuristic, but that’s what the Defense Advanced Research Projects Agency (DARPA) and the National Geospatial-Intelligence Agency had partially in mind when they awarded contracts under DARPA’s Urban Reasoning and Geospatial Exploitation Technology (URGENT) program.
Brain-inspired software is modeled after how the human brain works with the human visual system. Merging neuroscience with computers creates technology that could speed the delivery and improve the quality of situational awareness, both for warfighters operating in urban environments and intelligence analysts processing satellite images. Ultimately, the technology could allow both groups to make better decisions faster and more objectively.
DARPA gave the same criteria to three prime contractors, all of whom are approaching URGENT from different angles. The result will be three URGENT data collections during the first phase of the program, from which to build a database consisting of about 150 objects of interest with various conditions and view angles. One of the prime contractors, Lockheed Martin, and its partners call their approach to the program Object Recognition via Brain-Inspired Technology (ORBIT).
To build 2-D/3-D data about many urban objects, Lockheed Martin’s object recognition technology works in concert with SPADAC’s contextual spatial analysis capability to achieve a complete picture of objects and their context for the ORBIT approach. Data about objects is collected both by airborne and terrestrial electro-optical (EO) and light detection and ranging (LIDAR) sensors.
EO sensors capture photons of light emitted by objects and illuminate them. LIDAR sensors shine laser light onto objects and wait for reflections of the laser light to determine the how far away the various parts of the object are located. The aggregation of these distance reflection points can be analyzed to help determine the shape and thus the nature of an object.
“An EO sensor gives you high-resolution color information, while a LIDAR sensor provides high-resolution information about the shape of an object. When you put them together, you get the best of both,” said John Darvill, principal investigator for ORBIT at Lockheed Martin.
“LIDAR has not been a common sensor, but it’s now becoming more common and higher resolution, meeting the level of EO standards. The fact that we can use both EO and LIDAR together is one of the enabling technologies in the DARPA URGENT program,” he said.
LIDAR ACCURACY
SPADAC has seen improved accuracy resulting from the incorporation of LIDAR technology. “What makes this program different from other object recognition programs that came before is that we are focusing so much now on LIDAR. Advances in LIDAR technology have raised the accuracy level and lowered the cost of sensors to collect both 3-D and color data in the same sensor. Before, the sensors were only able to collect 3-D shape information, but no color. The fusion of the color information with 3-D improves the accuracy,” said Jason Dalton, SPADAC vice president of engineering.
SPADAC specializes in the fusion of geospatial intelligence with predictive analysis by developing various versions of its technology, Signature Analyst. Traditionally, core geographic information systems did not contain predictive analysis and modeling. In recent years, SPADAC has been building these capabilities into its products, first for the Departments of Defense and Homeland Security and now by leveraging the patented Signature Analyst technology for DARPA in ORBIT. In addition, SPADAC is launching a commercial product version with decision support capability.
“DoD and DHS have been using SPADAC’s Signature Analyst for four years now, and we will release Desktop Version 3.0 of the product for the commercial market” this spring, said Dalton.
Signature Analyst enables clients to make higher-confidence predictions about where, and often when, an anticipated event will take place. For example, an analyst can determine where a border tunnel might be located, which types of infrastructure is most vulnerable to various kinds of threats, where suicide bombers are most likely to strike or how resources should be best allocated. It does this by finding commonalities and relationships in the different data sources, including past events, their relationship to their environment, and other factors such as human terrain and social networking information.
Part of the goal is to improve and hasten situational awareness— that is, to help the warfighter determine better and faster what is and isn’t a threat. The other part is to ease a burden on analysts. The sheer volume of data that needs to be processed is a chief impediment to faster object recognition. Automating the burden would lift a load off analysts’ backs.
“DARPA’s object analysts have to spend many days and weeks viewing imagery and 3-D data and manually annotating all of it. Meanwhile, DARPA’s objective is to achieve object recognition 100 times faster. There’s so much data being collected and not enough analysts to extract all the objects, so the technology could become a force multiplier, in which one analyst can do the work of many,” said Dalton. “We look forward to implementing an automated process to empower the analysts to do more analysis rather than all that data preparation,” he said.
HIERARCHICAL MEMORY
Lockheed is also working with Numenta, a California-based provider that is conducting advanced technology research in the course of developing a software platform called the Numenta Platform for Intelligent Computing.
“Lockheed has been involved with Numenta technology for two years and is a member of the Numenta Partner Program for technical interchange. “We have a collaborative technical relationship with Numenta. We use their technology, modify it and apply it,” explained Darvill.
Numenta’s core technology is based on an invention by Jeff Hawkins called hierarchical temporal memory (HTM), a new computing paradigm that replicates the structure and function of the human neo-cortex, the largest area of the brain and the one that handles most of our high-level thought.
According to a research document entitled, “Learn Like a Human,” by Jeff Hawkins, the neocortex contains about 30 billion neurons, and its structure is quite uniform. Neuroscientists have long suspected that all parts of the neo-cortex work on a common algorithm. Much experimental evidence supports the idea that the neo-cortex is like a single, flexible learning machine.
Although it is uniform, the neo-cortex is divided into dozens of areas, responsible for functions such as language or vision. They are connected by bundles of nerve fibers in a hierarchal design. The senses feed input directly to some regions, which feed information to other regions. But the hierarchal arrangement is clear and well-documented. Neurons at low levels of the hierarchy represent simple structure in the input while neurons at higher levels represent more complex structure of the input.
Hawkins’ concept of using brain-inspired, hierarchal representations in computing could solve problems that have plagued artificial intelligence and neural networks, most notably their inability to handle large complex problems, for which it took too long to train the system or ate up too much memory. However, a hierarchy allows for the reuse of knowledge, requiring less training and better allocated memory. As an HTM is trained (through data input from sensors), the low-level network nodes learn first, and high-level nodes share what was previously learned in low-level nodes.
For instance, a system might initially take time and memory to learn what a dog looks like, but once it has, it can learn what cats look like in a shorter time, since dogs and cats share many low-level features such as fur, paws and tails.
Like the human brain, HTM’s are dynamic memory systems built around a hierarchy of nodes. Their purpose is to learn and train computers to perform tasks that have, up to now, been easy for people but difficult for computers.
“Our software platform is designed to be good at what the human brain can do—inference and pattern recognition even in the presence of noise,” said Donna Dubinsky, Numenta’s chief executive officer. Noise in the field of neurotechnology is defined as irrelevant data that needs to be discarded.
“Jeff Hawkins developed a theory about how the brain works. It learns a model of the world by exposure through its senses. In the same way, our software is self-learning and has to be exposed to the material that it has to learn. So we train the software. For example, we expose it to a lot of tanks so it learns tank-ness,” she said.
And HTM is not programmed the way computers normally are. Rather, it is configured with software tools and then trained by exposing it to sensory data, the same way the human neocortex learns new things. The Numenta software platform was created using the Python high-level language and C++ object oriented programming. It was designed on an open sensor platform, enabling it to accept input from all existing sensors.
“We have an application programming interface (API) that lets programmers create any kind of sensor they want to link the sensor to their platform,” she said. “We’re in the software platform business. We’re focused on building an application community so this is a tool for programmers to apply to their problem and their data.
“We provide the core technology to do object recognition and then Lockheed trains our technologists on the specific objects they want to recognize,” she added.
The company has also built an HTM network structure that reflects the nested hierarchical structure of objects in the temporal and spatial world. A spatial example is a neighborhood, consisting of houses, roads and schools. A temporal example is human language, comprised of consonants and vowels to make syllables into words.
The two key functions of an HTM network are: to discover causes by examining input data from sensors and computer files with recurring patterns and to infer novel causes in an environment by determining the most-likely hierarchy of subcauses. Every network node establishes and shares knowledge from new input data. The technology is evolving toward the ability to make predictions and direct behavior.
Alpha release of the Numenta platform is slated for this summer, followed by the beta release at the end of this year.
URBAN OBJECTS
Lockheed also works with the University of Pennsylvania and the University of California, Berkeley (UCB). UCB takes raw data, called point clouds, which represent the thousands of points comprising the shape of an object, and turns it into high resolution surfaces. “UCB takes the points as input and outputs a reconstructed surface for object recognition. They do the data massaging or data processing of point clouds or raw collections of data,” Darvill said.
Avideh Zakhor, a professor of electrical engineering and computer sciences at UCB has been conducting research for ORBIT since the start of the first phase last year. She specializes in 3-D modeling of urban environments.
“We are trying to recognize urban objects automatically using data supplied for that purpose. When a LIDAR laser scanner bounces off a point of an object, it returns a point cloud, which is series of points. We record the return of the series of points in 3-D space that define the shape of the object,” she said.
These point clouds are made into pictures of houses, trees and cars, for example, by placing the LIDAR data underneath and EO sensor data over it, Zakhor explained.
Zakhor has been conducting her own research in this area for more than a decade. “We built our own scanning system and hardware in 2000 to generate models based on a wide variety of data acquisition systems. We wanted a fast, automated way to generate photo-realistic models of urban environments on a wide variety of acquisition platforms.”
UCB’s goal to achieve speed syncs up with DARPA’s and ORBIT’s objectives. “You need speed to achieve scalability and scalability is the big challenge. It’s very compute-intensive. We want it to take 10 seconds or less to generate a 10-square kilometer area in the model, but once you put the model on the Web, there’s finite bandwidth when it’s interacting with other 3-D models and with 3-D rendering and visualization streaming through networks,” she said.
“We want users to be able to interact with complicated models on remotely located servers as if they were local,” Zakhor noted. “So we asked: How do you streamline the acquisition of data, the computation of it and the visualization and rendering of it?”
Answering those questions is what UCB continues to work on in its research and development efforts for ORBIT.
Phase I of URGENT began in May 2007 and will last for 18 months. Phase II will then build upon and enhance the capabilities established in Phase I to help DARPA to achieve its mission to understand urban environments quickly and objectively.
The other two prime contractors are BAE Systems and Hughes Research Lab. ♦