Ontology-based fusion of sensor data and natural language

We describe a prototype ontology-driven information system (ODIS) that exploits what we call Portion of Reality (POR) representations. The system takes both sensor data and natural language text as inputs and composes on this basis logically structured POR assertions. The goal of our prototype is to represent both natural language and sensor data within a single framework that is able to support both axiomatic reasoning and computation. In addition, the framework should be capable of discovering and representing new kinds of situations and thematic roles, (e.g., roles such as agent, patient and instrument), based on new compositions of existing representations. We applied our prototype in an intelligence analysis use case to test the hypothesis that a framework of this sort can produce usefully structured information from combined natural language and sensor data inputs. We further tested our hypothesis by adding an enhanced US Air Force ontology framework to our ODIS in order to (1) process a collection of sensor data, intel reports, and mission plans; (2) build composite POR representations from these data; and (3) machine analyze the fused results to infer mission threats.