NAVAL POSTGRADUATE SCHOOL

Monterey, California


POPULATION AND MAINTENANCE OF AN INTELLIGENCE DATABASE UTILIZING INTELLIGENT AGENT TECHNOLOGIES

 

by

 

Charles M. Carroll

 

March 2002

 

   Thesis Advisors:                                            Mark E. Nissen

                                                                        Neil Rowe


THESIS-- Edited for Web distribution

 

 




REPORT DOCUMENTATION PAGE

Form Approved OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188) Washington DC 20503.

1. AGENCY USE ONLY (Leave blank)

 

2. REPORT DATE 

March 2002

3. REPORT TYPE AND DATES COVERED

Master’s Thesis

4. TITLE AND SUBTITLE:  Title (Mix case letters)

Population And Maintenance Of An Intelligence Database Utilizing Intelligent Agent Technologies

5. FUNDING NUMBERS

 

6. AUTHOR(S)   Carroll, Charles M.

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

Naval Postgraduate School

Monterey, CA  93943-5000

8. PERFORMING ORGANIZATION  REPORT NUMBER    

9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES)

N/A

10. SPONSORING / MONITORING

      AGENCY REPORT NUMBER

11. SUPPLEMENTARY NOTES  The views expressed in this thesis are those of the author and do not reflect the official policy or position of the Department of Defense or the U.S. Government.

12a. DISTRIBUTION / AVAILABILITY STATEMENT 

Distribution authorized to U.S. Government Agencies and their Contractors; Administrative/Operational Use; March 2002.  Other requests for this document must be referred to Superintendent, Code 0052, Naval Postgraduate School, Monterey, CA  93943-5000 via the Defense Technical Information Center, 8725 John J. Kingman Rd., STE 0944, Ft. Belvoir, VA  22060-6218

12b. DISTRIBUTION CODE

 

13.    ABSTRACT (maximum 200 words)

This thesis addresses the innovative gains of employing revolutionary software-agent technology for the automated construction and maintenance of databases.  The intelligence process involves huge amounts of data and information that have varying degrees of assurance and processed intelligence must freely flow throughout the intelligence community to be effective.  Many analysts maintain a database to track and collate specific data; making these databases public requires additional validation and maintenance rigor.  Prior thesis work led to the design of a database to collect, organize and distribute key intelligence data, but man-hour expenditure for manual database construction and maintenance is often difficult to justify in the face of customer requests for finished intelligence products.  We use a socio-technical systems approach to generate a high-level process redesign alternative.  We also evaluate the feasibility of two key components within the new design using existing agent technologies: a simple program to rank document relevance based on word clues learned from training, and a commercial product for automated entry of structured data.  The results outline a clear plan and feasible technological approach for effecting dramatic improvements in this process.

14. SUBJECT TERMS  Intelligent Agent, Database Construction, Database Population, Multi-agent Systems

15. NUMBER OF PAGES

117

16. PRICE CODE

17. SECURITY CLASSIFICATION OF REPORT

Unclassified

18. SECURITY CLASSIFICATION OF THIS PAGE

Unclassified

19. SECURITY CLASSIFICATION OF ABSTRACT

Unclassified

20. LIMITATION OF ABSTRACT

 

UL

NSN 7540-01-280-5500                                                                                                                      Standard Form 298 (Rev. 2-89)

                                                                                                                                                            Prescribed by ANSI Std. 239-18

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

THIS PAGE INTENTIONALLY LEFT BLANK

 

 

 

 

 

 


 

POPULATION AND MAINTENANCE OF AN INTELLIGENCE DATABASE UTILIZING INTELLIGENT AGENT TECHNOLOGIES

 

Charles M. Carroll

Lieutenant Commander, United States Navy

B.S., University of North Carolina – Chapel Hill, 1987

 

 

Submitted in partial fulfillment of the

requirements for the degree of

 

 

MASTER OF SCIENCE IN INFORMATION SYSTEMS AND OPERATIONS

 

 

 

from the

 

 

NAVAL POSTGRADUATE SCHOOL

March 2002

 

 

 

Author:             Charles M. Carroll

 

 

Approved by:               Mark E. Nissen, Thesis Advisor

 

 

 

Neil Rowe, Thesis Advisor

 

 

                                   Dan C. Boger, Chairman

                                   Information Sciences Department

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

THIS PAGE INTENTIONALLY LEFT BLANK


ABSTRACT

 

 

 

This thesis addresses the innovative gains of employing revolutionary software-agent technology for the automated construction and maintenance of databases.  The intelligence process involves huge amounts of data and information that have varying degrees of assurance and processed intelligence must freely flow throughout the intelligence community to be effective.  Many analysts maintain a database to track and collate specific data; making these databases public requires additional validation and maintenance rigor.  Prior thesis work led to the design of a database to collect, organize and distribute key intelligence data, but man-hour expenditure for manual database construction and maintenance is often difficult to justify in the face of customer requests for finished intelligence products.  We use a socio-technical systems approach to generate a high-level process redesign alternative.  We also evaluate the feasibility of two key components within the new design using existing agent technologies: a simple program to rank document relevance based on word clues learned from training, and a commercial product for automated entry of structured data.  The results outline a clear plan and feasible technological approach for effecting dramatic improvements in this process.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

THIS PAGE INTENTIONALLY LEFT BLANK

 


TABLE OF CONTENTS

 

 

 

I.          introduction........................................................................................................ 1

A.        The foundation of understanding............................................. 1

B.        RESEARCH QUESTIONS.............................................................................. 2

C.        Chapter Outline...................................................................................... 3

d.        terms............................................................................................................. 4

II.        Process Evaluation Methodology and Intelligent Agents defined     5

A.        Process innovation.............................................................................. 5

B.        Process Evaluation methodology............................................. 5

1.         Identify Process for Innovation............................................................ 6

2.         Identify Change Levers........................................................................ 7

3.         Develop Process Visions...................................................................... 7

4.         Understand Existing Processes............................................................ 7

5.         Design and Prototype the New Processes........................................... 8

C.        intelligent agents defined and categorized................... 10

1.         What are Intelligent Agents?............................................................. 10

2.         Information Retrieval, Information Extraction, and Data Mining... 11

3.         Machine Learning.............................................................................. 12

4.         Machine Learning Techniques.......................................................... 13

III.       Apply Process Evaluation Methodology to Database Construction and Maintenance....................................................................................................... 21

A.        Adms............................................................................................................. 21

B.        Apply Process innovation methodology.............................. 29

1.         Identify Process for Innovation.......................................................... 29

a.  Enumerate Major Processes and Determine Process Boundaries 29

b.  Qualify Culture and Politics of the Process................................ 31

c.  Assess Strategic Relevance and Health of Database Construction and Maintenance 34

2.         Identify Change Levers...................................................................... 35

3.         The Process Vision............................................................................. 35

a.  Process Objectives.......................................................................... 35

b.  Critical Success Factors and Potential Barriers......................... 38

4.         Evaluate Existing Process.................................................................. 39

a.  Construction.................................................................................. 40

b.  Maintenance.................................................................................. 43

c.  KOPeR Analysis of the Current Process...................................... 45

5.         Design The New Process................................................................... 46

a.  Re-Designed Construction Sub-Process Attributes..................... 47

b.  Re-Designed Maintenance Sub-Process Attributes..................... 52

c.  KOPeR Analysis of the Re-designed Process............................... 54

IV.       Experiments........................................................................................................ 57

a.        Creation of the Datasets............................................................... 57

b.        A program for automatic document relevancy calculation     60

1.         The Learning and Rating Algorithms................................................ 60

2.         Results and Discussions..................................................................... 61

c.        a structured-data agent............................................................... 64

V.        conclusions........................................................................................................ 67

A.        APplication.............................................................................................. 67

b.        OTHER related work.......................................................................... 68

c.        rECOMMENDATIONS for further research.......................... 69

appendix A – amplifiing notes on process flow diagrams and KOPer results         71

A.        Amplifiing notes on process flow diagrams (figure 3.10-3.12, and figure 3.14)................................................................................................................. 71

B.        Koper analysis results.................................................................... 73

1.         Current Database Construction Sub-Process................................... 73

2.         Current Database Maintenance Sub-Process................................... 74

3.         Re-Designed Database Construction Sub-Process.......................... 75

4.         Re-Designed Database Maintenance Sub-Process.......................... 75

appendix B – recall-precision data and statistical calculations 77

A.        recall-precision data and graphs........................................... 77

b.        average distance and overall performance calculations          85

C.        ADJUSTED Baseline Calculations................................................ 87

D.        Null hypothesis test.......................................................................... 93

List of references....................................................................................................... 97

Bibliography................................................................................................................... 99

INITIAL DISTRIBUTION LIST........................................................................................ 101

 

 

 

 

 

 


LIST OF FIGURES

 

 

 

Figure 1.1:        Research Questions............................................................................................. 2

Figure 3.1:        ADMS Relationships From Ward (2001)........................................................... 22

Figure 3.2:        Country Table From Ward (2001)..................................................................... 23

Figure 3.3:        Decision-Maker Table From Ward  (2001)....................................................... 24

Figure 3.4:        Media Source Table From Ward (2001)............................................................ 25

Figure 3.5:        Alliance Table From Ward (2001)..................................................................... 26

Figure 3.6:        Affiliation Group Table From Ward (2001)........................................................ 26

Figure 3.7:        Decision-Maker/Media Source Table Modified From Ward (2001)................... 27

Figure 3.8:        Country/Affiliations Table From Ward (2001).................................................... 28

Figure 3.9:        Sub-Processes and Segments............................................................................ 30

Figure 3.10:      Current Database Construction Sub-Process...................................................... 41

Figure 3.11:      Current Database Maintenance Sub-Process...................................................... 44

Figure 3.12:      Re-Designed Database Construction Sub-Process............................................. 48

Figure 3.13:      A Close Look at the Search Agent..................................................................... 49

Figure 3.14:      Re-Designed Database Maintenance Sub-Process............................................. 53

Figure 4.1:        Re-Designed Construction Sub-Process Simulated Activities.............................. 59

Figure 4.2:        Illustration of Dij Measurement for Document 8 of Dataset 1............................... 62

Figure 4.3:        Fetch AgentBuilder™ From Fetch Technologies (2001)..................................... 65

Figure B.1:       Recall-Precision Graph (Dataset 1 as Test Dataset)............................................ 83

Figure B.2:       Recall-Precision Graph (Dataset 2 as Test Dataset)............................................ 83

Figure B.3:       Recall-Precision Graph (Dataset 3 as Test Dataset)............................................ 84

Figure B.4:       Recall-Precision Graph (Dataset 4 as Test Dataset)............................................ 84

Figure B.5:       Recall-Precision Graph (Dataset 5 as Test Dataset)............................................ 85

 

 

 

 

 

 

 


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

THIS PAGE INTENTIONALLY LEFT BLANK


LIST OF TABLES

 

 

 

Table 2.1:         Davenport’s Process Innovation Steps From Davenport (1993)........................... 6

Table 2.2:         Davenport’s Key Activities in Identifying Processes for Innovation Modified From Davenport (1993).   6

Table 2.3:         Fowler’s Agent Classification After Fowler (1999)............................................. 10

Table 2.4:         Characteristics of Machine Learning................................................................... 14

Table 3.1:         Process Objectives............................................................................................ 36

Table 3.2:         KOPeR Design Comparisons............................................................................ 54

Table B.1:        Dataset 1 Data.................................................................................................. 78

Table B.2:        Dataset 2 Data.................................................................................................. 79

Table B.3:        Dataset 3 Data.................................................................................................. 80

Table B.4:        Dataset 4 Data.................................................................................................. 81

Table B.5:        Dataset 5 Data.................................................................................................. 82

Table B.6:        Overall Performance Calculations Compared to 38.9% Baseline......................... 86

Table B.7:        Overall Performance Calculations Compared to 38.9% Baseline for Recall 0.5 86

Table B.8:        Dataset 1 Random Document Selection for Adjusted Baseline Calculations......... 88

Table B.9:        Dataset 2 Random Document Selection for Adjusted Baseline Calculations......... 89

Table B.10:      Dataset 3 Random Document Selection for Adjusted Baseline Calculations......... 90

Table B.11:      Dataset 4 Random Document Selection for Adjusted Baseline Calculations......... 91

Table B.12:      Dataset 5 Random Document Selection for Adjusted Baseline Calculations......... 92

Table B.13:      Adjusted Baseline Calculations........................................................................... 93

 


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

THIS PAGE INTENTIONALLY LEFT BLANK


ACKNOWLEDGMENTS

 

In a work such as this there are many contributors, but I would like to specifically recognize the intelligence professionals at the Joint Intelligence Center, US Pacific Command (JICPAC).  From my three year disassociated tour at JICPAC I learned a lot about intelligence processes and am proud to have served with such great Americans.  I hope these pages contribute to your mission.

For Dr. Nissen and Dr. Rowe, I appreciate the hours you spent with me helping to shape this product, wading through my sometimes-verbose prose and helping me to get back to center.   

This work is dedicated to my beautiful wife, Alice, who still believes I am the cog that holds the submarine force together.  Our equally yoked life is made possible through the love of our Lord and Savior, Jesus Christ, without whom we would not have accepted so readily the trials of this life.

 

 

 

 

 


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

THIS PAGE INTENTIONALLY LEFT BLANK


I.       introduction

 

 

A.        The foundation of understanding

 

Data represents the foundation from which intelligence-community analysts build information, knowledge, and finally understanding.  Internet technologies have enabled data and information availability that was not possible thirty years ago.  Intelligence analysts, however, expend an extraordinary amount of analytical time searching for and collating data.  Often flat or relational database tables are used to structure found data.  In this research, we investigate an innovative process for database construction and maintenance. 

We use a socio-technical systems approach.  Understanding how the intelligence community functions and the people who perform those functions are as important to system design as the underlying agent technologies (Nissen, 2001).  As such, a differentiation between the process customer and the process design customer should be made clear.  The process customer requests specific data to be gathered for the construction of a database; to some extent that customer dictates the data presentation and accuracy, but is not as concerned with the process as much as the process output.  The process design customers are the intelligence analysts at the theater intelligence centers.  Our process design strives to answer the question, “What is the gain to the analyst?” 


B.        RESEARCH QUESTIONS

 

This thesis explores how intelligent agents (computer software) can be used to automate the intelligence database construction and maintenance process.  Subsidiary questions and the chapters in which they are addressed are shown in Figure 1.1.

 

Figure 1.1:        Research Questions

 

We conduct a detailed evaluation of the intelligence process associated with database construction and maintenance to determine candidate elements that may benefit from implementation of intelligent agents.  We apply current capabilities and future probabilities of intelligent-agent software as it relates to the primary research question (Figure 1.1).  The research addresses implementation concerns and identifies shortcomings in evaluated agents for future technical improvements.

 

C.        Chapter Outline

 

Chapter II contains a description of the process evaluation methodology we have chosen and defines intelligent-agent capabilities and categories.  We also compare the capabilities of different learning algorithms.

Chapter III begins by defining the purpose and major attributes of the Adversary Decision Makers System (ADMS) database (Ward, 2001).  This chapter also discusses the current intelligence environment at a typical theater joint intelligence center.  Once the reader has a working knowledge of ADMS and the environment, process innovation methodology generates redesign alternatives.  Intelligence community issues such as database validation and database usefulness are discussed here. 

In Chapter IV we conduct a proof of concept analysis for a learning agent in this environment.  We identify potential agents for implementation within our redesigned system.

Chapter V outlines efforts in progress that may contribute to our re-designed database construction and maintenance process.  This chapter also includes recommendations for future research. 

 

d.        terms

 

Within these pages intelligence analyst or analyst refers to a person.  The analyst is an intelligence-community professional (also referred to as a domain expert).  An intelligent agent or agent is computer software that conducts some level of intelligent analysis to assist the intelligence analyst. 

Database construction and maintenance is used in the context of construction and maintenance of a database like ADMS.  By construction we mean the act of acquiring data and placing it in the database.  (This process is called database population in some domains).  Although the results of this research apply in part to the other databases, the focus is on a single example database -- ADMS.

The Joint Intelligence Center Pacific (JICPAC) is Pacific Command’s (PACOM’s) theater intelligence center.  It provides primarily operational level intelligence to the theater Commander In Chief (CINC) and his subordinate commands.  JICPAC consists of military and civilian intelligence analysts with varying degrees of experience.

 

 

II.      Process Evaluation Methodology and Intelligent Agents defined

 

A.        Process innovation

 

Large organizations like the U.S. military tend to resist change.  Change when it occurs is often gradual.  Even organizations that are open to change like the military intelligence community often approach change looking for process improvement, which studies the steps in the current process and evaluates ways to improve those steps.  Process improvement can result in an increase in productivity, a decrease in cost, and/or an increase in quality.  Process innovation attempts to step back from the process and focus on the process objective (Davenport, 1993).  It tries to effect significant change and offers hope for larger magnitude improvement.  The idea is that one is not tied to the current process during the initial stages of development.

 

B.        Process Evaluation methodology

 

A structured approach to evaluation of proposed processes is necessary to provide completeness to a study; “Simply inserting IT into a process in no way guarantees performance improvement” (Nissen, et al., 1999).  Though there are numerous evaluation methodologies defined in literature, the researcher has chosen the methodology of (Davenport, 1993).  It defines five steps to process innovation (see Table 2.1). Davenport’s intention is to apply his methodology to an entire company or organization.  We have limited our analysis to the process of database construction and maintenance, and concentrated on a particular database, ADMS.  The results of this research, however, are applicable to a large spectrum of intelligence databases. 

 

1.      Identify the Process for Innovation

2.      Identify Change Levers

3.      Develop Process Visions

4.      Understand Existing Processes

5.      Design and Prototype the New Processes

Table 2.1:         Davenport’s Process Innovation Steps From Davenport (1993)

  

 

1.         Identify Process for Innovation

We apply Davenport’s key activities in identifying the process in Chapter III (Table 2.2).       

·        Enumerate major processes

·        Determine Process Boundaries

·        Qualify the culture and politics of each process

·        Assess strategic relevance of each process

·        Render high-level judgments of the “health” of each process

Table 2.2:         Davenport’s Key Activities in Identifying Processes for Innovation Modified From Davenport (1993).

 

2.         Identify Change Levers

Davenport identifies information technology, the role of information, and the organization and human resources as levers for process change.  He argues that these levers should enable process change, not drive them.  Davenport separates information technology from information since information technology is merely a tool for managing and manipulating information.    In the corresponding section in Chapter III, we identify the intelligence analyst’s socio-technical environment. 

3.         Develop Process Visions

Process visions consist of “specific, measurable objectives and attributes of the future process state”(Davenport, 1993).  Process visions should focus on the concepts that are necessary to obtain the desired output and not be limited by what seems reasonable.  An initial vision statement formulates the foundation of this effort.  This statement answers the question, “How could we do things differently?”  From the initial statement one identifies key process characteristics that describe how the process will function.  This step identifies flow, output, organization, and technology requirements.  Performance objectives are generated for each.   Next we evaluate critical success factors to determine the critical components of the process that would have to work for the process to achieve the performance objectives.  Finally, we identify potential barriers to implementation of the process.

4.         Understand Existing Processes

The application of the methodology continues with a description of the current database population and maintenance process.  We use KOPeR, a high-level process redesign tool, to identify redesign alternatives.  KOPeR, which stands for Knowledge-Based Organizational Process Redesign, uses process redesign knowledge from reengineering literature and practice.  KOPeR requires user input of process size, process length, handoffs, feedback loops, information technology support (IT-S), information technology communication (IT-C), and information technology automation (IT-A).  Process size is the number of steps in the process.  Process length is the largest number of steps from process initiation to completion.  A process step conducted by a different individual than the previous step represents a handoff.  IT-S is defined as the number of steps in which computers contribute to the execution of the step, but the human user remains a part of the step.  IT-C are those steps which have computer assisted communication (e-mail, informational web pages, etc) between process design customers.  IT-A is the number of steps performed entirely by computers, without human assistance.

KOPeR evaluates these inputs and returns redesign advice based on process parallelism, handoffs fraction, feedback fraction, IT support fraction, IT communications fraction, and IT automation fraction.  Parallelism is a measure of steps that can be conducted simultaneously (i.e. in parallel).  If Parallelism is less than two, that is if less than half of the steps can be accomplished in parallel with other steps, KOPeR evaluates the process as sequential.  The handoffs (feedback, It support, IT communications, and IT automation) fraction is the number of handoff (feedback, It support, IT communications, and IT automation) steps divided by the total number of steps.  KOPeR’s redesign alternatives provide feedback to the process visions generated previously.  (Nissen, 2000)     

5.         Design and Prototype the New Processes

We conclude Chapter III with high-level design generation of the new process.  This high-level design exploits the methods for improvement suggested within the previous four steps.  We use KOPeR to evaluate the designed process.   

To evaluate learning agent capabilities (in Chapter IV) we use a stratified five-fold cross-validation process to generate recall-precision values for each agent tested.  This method is stratified in that instead of random sampling to generate the training and test datasets, sampling is done to guarantee each class is represented in both training and test sets.  The training dataset is used to teach the agent the rules required for proper classification; in it, the correct classification of each item is given.  The test dataset is used to evaluate the final optimized scheme.  In five-fold cross-validation, the data is split into five equal partitions, where each partition in turn is used for the test set while remainder is used for training set; the testing set is rotated until all partitions are used. 

Finding everything relevant needs to be weighed against presenting the user with the task of wading through excessive false positives.  To capture this tradeoff, recall-precision evaluation is often used.  Recall is the number of documents retrieved that are relevant (also called true positives) divided by the total number of documents that are relevant (the sum of true positives and false negatives); recall measures how well the agent finds relevant documents.   Precision is the number of documents retrieved that are relevant (true positives) divided by the total number of documents that are retrieved (sum of true and false positives); precision measures how well the agent avoids irrelevant documents.  Values for recall and precision are plotted on a recall-precision graph with recall on the x-axis and precision on the y-axis.  (Witten/Frank, 2000)

 

 


C.        intelligent agents defined and categorized

 

1.      What are Intelligent Agents?

Intelligent agents are merely software that performs a task normally requiring human cognitive skills.  They are used in numerous applications in everyday computing.  Many Internet search engines use intelligent agents to find relevant URLs (Schwartz, 1998).  Agents categorize the content of web pages for more rapid return on user searches.  The Microsoft Office™ Paperclip is an intelligent agent; it monitors user actions and predicts when a user requires special assistance in formatting, printing, etc.  In simulation, intelligent agents are given rule sets to react to the environment and their performance is not scripted.  Fowler (1999) classified agents in the four groups shown in Table 2.3.

 


Class of Agents

Description

Information Filtering

Select relevant information from a flow of filtering email, network news groups, or FAQ’s.

Information Retrieval [and Data Mining]

Locate specific information within networks and/or databases.

Advisory

Provide intelligent advice.

Performative

Perform special tasks like business transactions, negotiations, scheduling, cooperative learning, etc.

 Table 2.3:        Fowler’s Agent Classification After Fowler (1999)

 

2.      Information Retrieval, Information Extraction, and Data Mining

Information retrieval and data mining use similar techniques with slightly different meaning.  Information retrieval attempts to locate useful text documents from a large collection in response to a user query (Salton, et al., 1997).  The text documents may contain structured, semi-structured, or natural language (free text) data.  Structured data resides in tables, where certain attributes are already categorized.  Semi-structured data then is the bridge between natural language and tabled data.  It may have free text writing, but elements of the text are standardized based on content requirements.  Natural language is the common format of journals, newspaper and magazine articles, e-mail, and this thesis; it is plain text writing.    

Information retrieval has special constraints, because to truly understand and interpret a written document requires reading comprehension.  This is a difficult task for machines.  For example, in “President Bush was sworn into office today ending eight years of service in the oval office by President Clinton,” it is easy for a human to determine the current President but not so trivial for a machine.

Information extraction, a subset of information retrieval, uses predefined types of information (sometimes called signatures) from text.  In our case, information extraction techniques could search for known word combinations or sentence patterns of intelligence significance to identify the values for fields in the ADMS database (Riloff, 1996).

Data mining is the extraction of potentially useful information from data stored in databases.  Data mining has special constraints because of the media on which it operates.  It often deals with databases of different structures created by different people with different objectives and varying validity.  Data mining must recognize similar data under different field names.  Data miners speak of data warehouses, enterprise-wide integrated databases.  They are conglomerations of all the databases within the organization.  Although a data warehouse is not essential to data mining, it simplifies the process. (Witten/Frank, 2000)

Both data mining and information retrieval face a challenge that intelligence analysts face: discovering data is not necessarily discovering information.  Data is often flawed, inexact, discontinuous, and contingent on accidental coincidences.  To program a machine to discern “good” data from “bad” data is challenging.  

3.      Machine Learning

Machine learning is “the acquisition of structural descriptions from examples” (Witten/Frank, 2000).  Presenting the machine with a list of examples of some concept and having it determine the principles behind them to locate similar examples outside of the training set offers potential for generality of a given intelligent agent and significant time savings.  An intelligence analyst is much too busy to expend efforts in understanding, designing, and testing a tailored algorithm for just one database.  Machine learning enables an algorithm (the intelligent agent) to automatically evaluate the data and go in search of similar data.  The agent can take its learned rules and provide the analyst with new candidate examples from new data; the analyst would review the new data and provide feedback to the agent to make it more precise in classification of data.  

In this research an instance is one user-classified document within the training dataset.  Recall and precision can measure the performance of learning methods on a set of instances comprising the training set.  Real world applications typically demonstrate that as recall increases precision decreases. (Craven et al., 2000)

Training data can generate incorrect rules by overfitting.  For example, a small dataset showing the sporting preferences of American teenage boys may generate a rule that no American boy likes football if the dataset is too small.  Missing values represent another obstacle in machine learning.  They will appear often in our dataset, as few documents will contain all the clues we seek.  A missing value can be given an average or normal value, a value in a range of values, or made an attribute in its own right.  (Witten/Frank, 2000)

Witten and Frank identify four kinds of machine learning: classification, association, clustering, and numeric prediction.  Classification takes classified examples and infers how to classify unseen examples; this is a supervised scheme because a domain expert provides the training examples.  Association is not just concerned with classification, but predicts related combinations of attributes and instances according to commonalities; it normally involves non-numeric data where most attribute values are missing.  Clustering groups examples that belong together; success is measured by how useful the result is to the user.  Finally, numeric prediction predicts an outcome that is a continuous numeric quantity.

4.         Machine Learning Techniques

Researchers have developed numerous learning algorithms optimized in various domains over the last few years.  We generated Table 2.4 to show some common learning methods and their advantages and disadvantages.  We now discuss these methods in order.

Table 2.4:         Characteristics of Machine Learning

 

A common learning method learns a list of rules with exceptions.  The algorithm generates a rule set that applies generally to the entire dataset.  The algorithm or a domain expert then generates exceptions to the rules for specific instances.  Users are more likely to understand the agent decision process because this method is consistent with how humans think, but in some cases the exceptions become voluminous.  (Witten/Frank, 2000) 

The 1R (1 Rule) technique is a simple method that tests a single attribute and branches accordingly.  Missing attribute values are easily handled, as they become another value for the attribute.  The algorithm chooses the rule for each attribute with the lowest error rate.  This technique is surprisingly successful in many cases and is often within a few percentage points in accuracy when compared to more complex schemes.  1R is susceptible to overfitting if there are a large number of possible values for each attribute.  Placing several values that have a high statistical probability of predicting the outcome into one category can minimize overfitting.  (Witten/Frank, 2000)

One common type of statistical modeling is Naïve Bayes, which assumes independence between attributes.  This method is often successful despite the often-false assumption of independence.  Dependent attributes, however, do tend to skew the data.  As an example, suppose in our database we have decision-maker X from country Y whose title is “Maximum Leader” but also holds the title of “Commander In Chief”.  The two titles are not independent.  In fact many articles may contain only one of the two titles thus reducing the weighted value of a single title.  Bayes Networks are more advanced models using Bayesian analysis that do not assume independence.  Both Bayes techniques effectively handle missing values by omitting them.   (Witten/Frank, 2000)

In the statistical “bag of words” method each word in a classified document within the training dataset is evaluated.  The number of times a word appears in a given class is compared to the number of times it occurs in the other document classes and the word is assigned an appropriate statistical weight.  The algorithm then generates a classification probability for each document in the test dataset based on the re-occurrence and weighting of the words that were in the training dataset.  This method is simple for the user, who only has to classify the documents in the training dataset.  Overfitting can occur when classified documents contain large amounts of text not relevant to the document’s classification.  This is countered by increasing the size of the training dataset and/or with domain expert review of the word weights.     

Statistical distribution models are common when numerical values dominate a database.  Numerical values are often handled assuming a Gaussian, or normal, probability distribution.  Any appropriate distribution, however, can be used.  Errors are introduced when chosen distribution functions do not accurately predict the numerical outcome. (Witten/Frank, 2000)

Neural Networks are also a form of statistical model.  Neurons, similar to nodes in a decision tree, form the basic element.  Neurons are connected by synapses.  A neural network agent usually learns by modifying connection weights (also called synaptic weights).  Neurons are organized in layers, usually an input and output layer, and often one or more intermediate (or hidden) layers.  Each neuron in the input layer is given an attribute to monitor.  In the simplest case this attribute would have a condition of on or off.  If the attribute is present (e.g. the existence of a particular word or phrase in a document) the neuron fires along the synapses.  The input neurons feed to the hidden layers (or directly to output layers in some cases) that deal with combinations of input neurons.  These hidden layers then feed to the output layer.  The output layer neurons produce an output such as “this is a relevant document.”  The network learns by a domain expert identifying when it is wrong.  The network then lowers the weighting for synapses that provided a wrong response and raises the weighting of the synapses that promoted a correct response.  The “Perceptron” neural network has been around the longest (circa, 1950’s) and is widely used in pattern recognition.  This network only uses input and output neurons and can only learn linear functions.  The backpropogation network uses hidden layers in addition to the input and output layers, allowing evaluation of non-linear functions.  Backpropogation requires a large number of iterations to achieve reliable learning.  Because of the hidden layers and complex perturbations in synaptic weighting it is often difficult to demonstrate to a domain expert how the agent is learning.    (Abdi et al., 1999)      

Decision Trees are a visual representation of a learning agent’s decision process.  Nodes in a decision tree represent the testing of a particular attribute.  In reality a decision tree’s nodes correspond to a set of rules.  The advantage of decision trees is in presentation of the learning process.  Decision trees face a drawback called the replicated sub-tree problem: parts of the decision tree are reproduced each time a given conclusion is activated.  The Divide and Conquer Technique (also called top-down induction of decision trees) is the most popular use of decision trees. This technique conducts an analysis of each node to determine the expected amount of information required to be able to classify an event.  From this method best decision nodes can be generated given the calculated information gain from each node.  Intuitively this is consistent with weighting based on correlation to the classification.  Information gain tends to prefer attributes with large numbers of possible values.  There is a technique called “gain ratio” used to mitigate this.  Essentially it divides the value of each daughter node by the total number of daughter nodes. This can also have an overcompensating effect in that the machine-learning program could prefer an attribute just because it has fewer possible values than the other attributes.  This is usually corrected by choosing an attribute that maximizes the gain ratio if the information gain for a given attribute is at least the average of the information gain of all attributes examined.  (Witten/Frank, 2000)

Covering (also called separate-and-conquer) algorithms take each class and search for a way to cover all (or most) instances within the class while excluding all (or most) instances not in the class.  It uses a set of rules vice a decision tree, but often has similar results as divide and conquer algorithms and does not have the replicated sub-tree problem.  In the multi-class case a decision tree node attempts to take into account all classes.  The covering algorithm maximizes only a single class at a time and thus could be more accurate for that single case.  Each separating step reduces the instance set thus increasing the efficiency of the next separating step.  The drawback is that a single test example may receive multiple classifications.  This is countered by further testing of multiple classification examples. (Witten/Frank, 2000)

Instance-based learning (also called the nearest-neighbor classification) uses each new instance to develop further learning.  The algorithm compares the new instance to the closest existing instance using a distance metric to define the classification of the new instance.  Suitable attribute weights are often difficult to derive.  It is also difficult to show an explicit structure to the user.  Computationally this method is time consuming for large datasets because the entire training set must be examined to classify each new instance.  The nearest-neighbor method is widely used in pattern recognition. (Witten/Frank, 2000)

Genetic Algorithms are based on genetic concepts of crossover and mutation.  The Darwinian concept of survival of the fittest prevails.  Conceptually the algorithm chooses a number of functions (that is, a relation between inputs and outputs) that describe the training dataset.  It then tests those functions against the known truth.  The functions that best represent the user defined outcomes become prominent within the population of functions.  Parents (functions) are selected for crossover using a weighted process.  Stronger parents are more likely to be chosen.  Randomly selected parts of each parent are combined to form offspring functions. These functions are again evaluated against the known truth.  Mutation takes place by randomly selecting some offspring (normally less than 10%) and changing a random part of its function.  Mutation allows the algorithm a chance to break away from local optimization in search of global optimization.  This method works with linear and non-linear functions.  Rule sets and rule generations are very difficult to demonstrate.  Because the algorithm is randomly searching through a vast number of functional components, convergence to a good function can be lengthy.  (Da Ruan, 1997)


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

THIS PAGE INTENTIONALLY LEFT BLANK

III.           Apply Process Evaluation Methodology to Database Construction and Maintenance

 

In this chapter, we briefly describe ADMS and then step through the Davenport method for process innovation.

 

A.        Adms

 

Ward (2001) designed a relational database, ADMS, for the Information Operations (J39) shop at USPACOM “that would track the sources of information input available to adversary decision makers in the Pacific theater Area of Responsibility (AOR).”  J39 thought that the information was known already, but not tracked in an efficient manner.  In his recommendations LT Ward states, “USPACOM should begin populating the ADMS database with real data and begin using it as a functional resource to prove its value to other interested parties.”  Unfortunately, ADMS construction is not complete.  Construction and maintenance of a database of this sort is labor intensive.  Man-hours expended searching the Internet, the Secret Internet Protocol Router Network (SIPRNET), and the Joint Deployable Intelligence Support System (JDISS) are significant.  Data validation also requires time-consuming cognitive analysis. 

Figure 3.1 shows the relationships between each of the seven ADMS tables; primary keys are in bold.  Primary keys are one or more fields in a table whose value uniquely identifies each record (row) in the table.  Relational databases provide a means to access and reassemble data in many ways without having to reformat or reorganize the tables.  Multiple related tables also enhance the ability to update a single piece of data once for the entire database.  For example, If ADMS were a flat table a change in the government type (communist, socialist, democracy, etc) of a particular country would require updating the entry for every decision-maker in the database from that country; as a relational database one only has to update a single field in the Country Table (Figure 3.2).

 

Figure 3.1:        ADMS Relationships From Ward (2001)

 

The Country Table (Figure 3.2) relates to the Decision-Maker Table, the Alliance Table, and Affiliation Group Table through the country name field.  In addition to the government type, this table tracks the country leader for each country and the country’s theater priority.  A given theater contains numerous countries; more than forty for US Pacific Command.  As such, a formal process has been established to order intelligence priorities by country.     

 

Figure 3.2:        Country Table From Ward (2001)1

 

The Decision-Maker Table (Figure 3.3) contains a record for each key decision-maker within a country.  The individual’s title and country are included in this table.  The Decision-Maker Table is related to the Media Source Table through the decision-maker’s name field, and to the Country and Alliance Tables through the country name field.

 

 

Figure 3.3:        Decision-Maker Table From Ward  (2001)1

 


Details regarding each media source are given in the Media Source Table (Figure 3.4).  A media source is a newspaper or magazine publication or a television or radio broadcast.  The Media Source Table is related to the Decision-Maker Table through the media name field.

 

Figure 3.4:        Media Source Table From Ward (2001)1

 

The Alliance Table and the Affiliation Group Table list influential organizations that effect government decision-makers.  The Alliance Table (Figure 3.5) shows formal alliances between nation-states and is related to the Decision-Maker Table and Country Table through the alliance country name fields.  The Affiliation Group Table (Figure 3.6) shows influential groups and organizations that should be recognized when building a decision-maker influence model and is related to the Country Table through the affiliation name field.

Figure 3.5:        Alliance Table From Ward (2001)1

 

Figure 3.6:        Affiliation Group Table From Ward (2001)1

 

The Decision-Maker/Media Source Table (Figure 3.7) and the Country/Affiliations Table (Figure 3.8) establish many-to-many relationships between other tables.  The Decision-Maker/Media Source Table links media sources to decision-makers.  Likewise, the Country/Affiliations Table links affiliations to a nation-state.

 

Figure 3.7:        Decision-Maker/Media Source Table Modified From Ward (2001)1

 

  

Figure 3.8:        Country/Affiliations Table From Ward (2001)1

 

The data to construct a database such as ADMS is obtained largely from free-text documents.  The process customer will likely require at least some of this source information at some point.  As J39, for example, begins to apply ADMS to their influence models, they may need to know more about what influences the media sources that influence the decision-maker.  One might attempt to represent this in an added data field in the Media Source Table, but rarely is the answer to such a question a single datum: Media sources are influenced by many, often conflicting, factors.  A good intelligence answer to this kind of question would discuss the different factors and assess the most influential.  This requires researching more detail than is available in a database.  The files that contain the data used to construct the database, called source documents, provide much of this needed detail.  Source documents, such as newspaper articles and intelligence reports, contain more information than the data that was generated from the source documents and entered into the database.  A customer tasking of this sort is routine if the customer only wants detailed analysis on a few media sources.  As the number of sources increase, the task increases more than linearly because it begins to impact other high-priority tasks.  Additionally, if significant time has passed since the database was created (or there has been analyst turnover), the source documents that built the database may be harder to locate.      

 

B.        Apply Process innovation methodology

 

In this section we step through the Davenport method for process innovation, applying it to the database construction and maintenance process.

1.      Identify Process for Innovation

The first step in Davenports’ innovation method is to identify a process for innovation.  This involves enumerating the major processes, determining process boundaries, qualifying the culture and politics of each process, assessing strategic relevance of each process, and rendering high-level judgments of the “health” of each process. 

a.  Enumerate Major Processes and Determine Process Boundaries

 Our process of interest in this analysis involves database construction and maintenance, which can be divided into the major sub-processes of database construction and database maintenance (Figure 3.9).  Within database construction data acquisition locates applicable data; data analysis classifies the data; data entry transforms the analyzed data into a useful format; and data validation assesses the accuracy of the data.  Intelligence analysts seek validation by attempting to locate multiple independent credible sources containing the same data.  For example, four newspaper articles dated within a few days of each other, which indicate that a certain decision-maker has been replaced, does not necessarily validate the fact the decision-maker has been replaced.  The newspaper reporters are likely getting their information from the same media source (e.g. associated press).  A couple of newspaper articles backed up by a reliable human intelligence report and/or some other independent credible source would cause an analyst to assess the data as validated.

 

Figure 3.9:        Sub-Processes and Segments

The database maintenance segments of Figure 3.9 are: an implementation trigger, new data acquisition, data analysis, data validation, and data entry.  An implementation trigger is an event that makes it necessary to update the data:  A process customer could request that the data be updated at a given periodicity (continuously, quarterly, annually, etc.).  Also, maintenance implementation could be triggered by an external event (real-world crisis or military exercise).  New data acquisition seeks data that has been generated since database construction was completed.  New data that represents potential changes to the database is most significant; new data that confirms the fields of the existing database has value, but is of less immediate concern to the analyst unless it counters other new data.   Data analysis, data validation, and data entry have the same meaning in the maintenance sub-process as they had in the construction sub-process.  

b.  Qualify Culture and Politics of the Process      

Intelligence analysts are assigned specific areas of responsibility within the joint intelligence center’s overall area of responsibility.  Analysts work in country or regional teams (called divisions) and are usually responsible for one aspect of that country or region (e.g. political-military affairs).  There is overlap of analyst responsibility within a division as awareness of all intelligence aspects of the target country helps in understanding an assigned segment.   

An analyst spends roughly half the time studying current reports on the analyst’s area of responsibility and preparing formal reports and briefs on significant changes within their purview.  The analyst builds a knowledge base through this daily review of current intelligence.  Intelligence analysts work regularly with data management systems and are adept at web searching and surfing.  Every analyst has at least one computer for data management and communication (most have several to support multiple security classification levels).  Although comfortable with information technology developments, IT training is not consistent among analysts.  Analysts are, in general, skilled web page designers and are comfortable with the most prolific commercial office software products, but are usually not computer programmers.  Analysts usually have sufficient storage space on their computer systems to accumulate large amounts of data. 

Though many senior analysts perform many of the roles of management, they still conduct considerable intelligence analysis.  Intelligence community executives (Admirals/Generals and Captains/Colonels or equivalent) as well as mid-level management (Commanders/Lieutenant Colonels and Lieutenant Commanders/Majors or equivalent) are also skilled computer users.  Though the community is open to changes in information technology, sweeping changes often lead to analytical setbacks as old database systems and file archiving systems have to be updated or dropped.  The community (like most of the Department of Defense) suffers from implementation of stove piped and sometimes-dated systems that often do not communicate gracefully with each other.   

The analyst’s world revolves around mounds of data and information with varying degrees of assurance.  An intelligence source may produce 100% accurate information on one occasion and completely inaccurate information on another.  Analysts continuously strive for data validation, which is hampered by circular reporting (multiple sources providing the same data from the same primary source).  The analyst also must manage gaps in knowledge.  The analyst searches for data and information to fill those gaps, but often has to proceed with analysis despite the gaps.  Intelligence community executives and analysts feel that they must remain credible to operators if they hope to contribute.  Data complexity and the issues of credibility create reluctance to trust a fully automated database construction system.  

Analysts gain information from HUMINT (reports from human intelligence), SIGINT (intercepted signals intelligence), IMINT (imagery intelligence), MASINT (Measurement and Signature Intelligence), and OSINT (Open Source Intelligence—newspapers, Internet websites, journals, etc). Analysts communicate regularly with other analysts through secure e-mail, secure telephone, secure video teleconferences, and face-to-face conferences. 

Several archive systems contain archived message reports.  These are explored using simple keyword-search techniques.  Additionally, most analysts maintain their own archive on their local servers or hard drives.  These archived files, which we refer to as stored files in this work, contain information that is pertinent to the analyst’s area of responsibility.  Stored files may or may not be related to a particular ongoing project, but represent potentially useful data for future analysis.  A few analysts maintain a hardcopy filing system. 

Web pages that reside on JDISS and SIPRNET often provide significant processed intelligence.  Like the Internet, some web pages are not current.  Numerous intelligence organizations post message traffic, intelligence reports, and intelligence briefs there.  This media has become the primary means of intelligence sharing.   

The analyst’s memory represents a good starting point for structuring a search.  The ability of the human mind to recall data or to at least be able to associate types of data with locations (to remember that one saw something about that somewhere) is a substantial part of intelligence gathering.  We call this known data in later sections.  The person is exposed to many more sources of information than are available from computer-based research.  A person’s experience in society, military operations, and human interaction provide insights difficult to gain from computer analysis alone. 

c.  Assess Strategic Relevance and Health of Database Construction and Maintenance

True databases (those containing tables, records and fields) are fairly common within the intelligence community.  There are standardized databases that track common intelligence needs (e.g. Order of Battle).  These are shared among intelligence agencies and customers.  But many analysts maintain a database of some kind to track and collate specific data, and are reluctant to share these with other intelligence agencies or customers.  Analysts create these databases without outside tasking to assist in performing their duties.  Making the database “public” requires much more validation rigor and database maintenance than is required to meet the needs of the originating analyst.  Many of these informal databases do not survive analyst turnover.  Occasionally they are posted to a secure web site usually for consumption by a specific customer, but available to all who have access (if they can find it among all the information on the network).  Man-hour expenditure for meticulous database construction and maintenance is often difficult to justify in the face of customer requests for finished intelligence products.  Current database construction and maintenance processes are manpower-intensive with much of the man-hour expenditure on non-analytical tasks.

2.      Identify Change Levers

E-business is actively pursuing ways to automate business processes.  Intelligent agents technology is part of this automation.  Multi-agent systems are being used to manage inventories, track consumer preferences, negotiate contracts, and manage schedules.  Agents are used in various decision-support systems including financial-portfolio advice, buyer/seller matchmaking, supply chain decision support, and computer-interface support (Gates/Nissen, 2001).  Many database construction and maintenance tasks are similar to e-commerce tasks.    

3.      The Process Vision

We desire to establish a less manpower-intensive means to construct and maintain the ADMS database by automating as much of the process as possible while minimizing quality degradation.  Toward this end, we discuss process objectives, critical success factors, and potential barriers.  Development of these points is guided by the author’s professional experience in this area.

a.  Process Objectives

Based on our discussion of the major sub-processes and determination of process boundaries, our qualification of the culture and politics of the process, the strategic relevance and health of the process, the change levers, and our process vision; we develop a set of process objectives (Table 3.1).  We discuss these in turn in this section.


 




PROCESS OBJECTIVES

·        The process execution should be within the capability of intelligence analysts. 

·        User interface functionality should be similar to Microsoft Officeä products. 

·        Documents presented by the system should not duplicate documents the analyst has recently reviewed.   

·        The new system should preferentially present information that contains new data or validates data that has not been validated. 

·        Precision/Recall default should favor precision.

·        The analyst should be able to turn on or off automated functions.

·        Structured and semi-structured data entry should be automated. 

·        Reference source files should be easy to archive and should automatically link to appropriate database fields.

·        The system should accept newly developed tools as agent technology advances. 

·        The system should be able to process common text documents and interface with common database files.  

·        The system should operate across classification levels.

Table 3.1:         Process Objectives

 

Our process design customer is the intelligence analyst, who should be able to execute the system applications from a desktop without significant technical assistance.  To minimize training requirements and to achieve rapid analyst familiarity, the system should have a good user interface with functionality similar to the interface in Microsoft Officeä products, which are familiar to analysts.

The new system should make every effort to save analyst time.  Data and information presented by the system should not duplicate documents the analyst has recently reviewed but the system should preferentially present new data (fields not yet filled) or validating data for data that has not previously been validated.  Because of the concern for analysis time, default precision/recall settings should reflect a higher weight on precision, but the analyst should have the ability to adjust precision/recall settings.  Though some may argue that we want the analyst to know everything about everything to do within his or her area of responsibility, significant information availability prevents this in most cases.  To set the default to 100% recall forces the analyst to sift through all possible relevant documents and is not an improvement over the current process.

The system should have interactive tools that allow the analyst to control database construction and maintenance to the extent the analyst deems necessary based on the judged quality of source documents and agent capability.  Where a trusted agency has made data available that is suitable for the database (usually structured or semi-structured text), the system should automate data entry.  In situations where that agency maintains data current through a web site or similar product, the system should be able to automatically update itself when the trusted agency changes its data. 

Reference-source files should be easy to archive and should automatically be linked to appropriate database fields so that analysts and customers can easily evaluate data validity by “drilling” down to (i.e. looking up) the information that supports the data.  This function should present source documents that support and contradict database data.

Information retrieval is a rapidly evolving field.  The innovated system architecture must accept newly developed tools as agent technology advances.  This allows new algorithms to be added to our “open” system.  Although primarily used in an Internet-like environment, the system should gracefully handle text documents and interface with database files used by analysts.  The system will need to be able to operate in classification domains up to Top Secret Sensitive Compartmented Information.  Automatic multi-level operation would greatly improve the process.

b.  Critical Success Factors and Potential Barriers

The intelligence community appears open to technological change.  Its organization is comfortable with information technology at all levels.  Though community-wide innovation will advance best with a top-down approach, the lower levels (the analysts) will realize the gains of an innovated database construction and maintenance process.  Some analysts may resist change because they may perceive that the process improvement might lead to some job elimination.  Our current doctrine requiring information superiority limits any significant reduction in analysts.  Nevertheless, those who lead the new process implementation should consider this potential resistance and be prepared to counter it.  Excellent user interfaces and promises of analytical perfection will not replace the critical evaluation of actual analysts.  Success only occurs with the production of a system that genuinely improves the analyst’s ability to conduct analysis.

  Technology represents the primary success factor and therefore the largest potential barrier.  Though much research is being conducted in information retrieval, systems are not yet mature.  Many existing algorithms require extensive domain-expert involvement to realize good results even in a laboratory environment.  If the analyst is required to expend as much time setting up the system, monitoring its actions, and making corrections as would have spent constructing and maintaining the database manually, we are no better off.  Tasking priority for database maintenance and construction is rarely high enough to justify higher man-hour cost even if the innovated system finds more or better information than the current methods.

A system that crosses classification domains will have to overcome significant security concerns.  The intelligence community operates with classified networks largely disconnected from each other and from unclassified networks.  Naturally, any software that operates across those boundaries would undergo intense certification.  The early versions of the innovated system should therefore operate in any classification domain, but not across domains.  Cross-domain transfer of data should be handled manually initially.

4.      Evaluate Existing Process

In general the analyst receives tasking to construct a database like ADMS would come to the Directorate of Operations (DO) within the theater joint intelligence center.  In order to increase the efficiency of the overall intelligence community, each intelligence command has been assigned specific intelligence responsibilities.  One of the functions within DO is to determine if the joint intelligence center is responsible for the specific intelligence requested in the tasking.  If not, the joint intelligence center forwards the tasking to the appropriate agency.  Often portions of products like ADMS are actually compiled by several intelligence agencies.  We will assume the theater joint intelligence center has overall responsibility for this product.  ADMS is designed for multiple-country analysis and would therefore be partitioned by country across several divisions within the joint intelligence center.  We focus here on the process that a single country analyst employs for the construction and maintenance of one portion of the database and do not address cross-division coordination.   

a.  Construction

Figure 3.10 illustrates the flow of activities associated with the current construction sub-process.  Each activity shown in this Figure is discussed in turn.

 Analysts have quite a bit of autonomy in how they approach database construction and the actual process used varies.  In general the analyst begins by identifying data that they already know from professional experience or that is easily accessible from the analyst’s stored files.  After this, the analyst determines if each datum of the identified data is useful to the construction of the ADMS database (i.e. if it fills a field in the database).  We label the stored files and known data acquisition and analysis as analysis of immediate data in Figure 3.10. 

The analyst next validates immediate data by searching for new data that confirms the immediate data.  (Some situations may allow validation to occur after data entry, but data validation must occur before the database is released back to the process customer.)  The analyst then manually enters the validated data into the database.  Many analysts develop a way to track source documents (defined on page 29) to assist in answering process customer detailed questions.  Process customers and future analysts often need to know the source of the data within the database.  Source documents are usually stored in electronic sub-directories designed by the analyst.   

With analysis and entry of immediate data complete, the analyst moves to research.  The primary acquisition tools of research are keyword web search, other domain experts, daily review of current intelligence, and filed hardcopy documents.  As in the analysis of immediate data the analyst evaluates each datum as useful to ADMS


Figure 3.10:      Current Database Construction Sub-Process


construction or not, conducts validation, manually enters data into ADMS, and saves the source document to an appropriate sub-directory. 

The analyst uses data derived from the analysis of immediate data to assist in identifying potential keywords for a web search on JDISS.  After adjusting the search parameters, the analyst evaluates the numerous documents located by the search engine.  As new data is located, the analyst may realize new keywords for further searching on JDISS.  Data can also be located on SIPRNET and the Internet but must be manually transferred to JDISS.  If the database is to reside on a lower-classification system, all data from higher-classification systems must be downgraded, a labor-intensive manual process.  Thus, analysts seek to maintain databases on the highest classification level acceptable to the process customer.

Analysts can also coordinate with other domain experts that may have data useful in constructing the database.  This is done by face-to-face communications, by phone, video-teleconference, and by e-mail.  E-mail provides the best means for communication as the message gets through despite time-zone differences and meetings.  E-mail also provides an electronic record of source documents for the analyst’s files.  If the other domain expert must expend significant time to produce the required data, the analyst or the process customer submits a formal tasking to justify the man-hour requirement.

As the analyst conducts daily review of current intelligence, potential data for ADMS are identified.  The analyst can influence other agency intelligence reporting by submitting system tasking to alert other agencies of new intelligence requirements related to ADMS.  These tasking messages are routed differently depending on the targeted collection method.  The joint intelligence center has a division that brokers these requests. 

When ADMS is sufficiently populated with validated data, the analyst sends the database to the process customer.  This may include a review by management personnel within the joint intelligence center.  The database could be sent by e-mail or be distributed through a web page.  The chosen method depends on the desires of the process customer, the joint intelligence center analyst, and joint intelligence center management.  The advantage of an e-mailed product is that it is made available only to the process customer and the customer can make alterations to the database to meet evolving requirements.  The web page method limits the occurrence of multiple versions of the database since the process customers would access the information directly from the web page.  This method also makes the data available to any customer with network access (an advantage and disadvantage).

When construction is complete, residual data is usually retained in the analyst’s files.  Residual data is source and un-validated data.  Though the original analyst can quickly locate this data if it is needed, follow-on analysts (there is a high turnover of military analysts) are less likely to locate the appropriate sub-directories or grasp the logic of non-standardized sub-directories.  In general, residual data is not readily available to process customers or other analysts.

b.  Maintenance

Figure 3.11 illustrates the flow of activities associated with the current maintenance sub-process.

 


Figure 3.11:      Current Database Maintenance Sub-Process


Like construction, there is not a set procedure for database maintenance.  The maintenance trigger is usually initiated manually.  Having maintenance triggered continuously severely affects daily tasking priorities.  In practice it is very difficult to maintain the level of effort used in construction for extended periods due to the daily vigor required to maintain currency and the ever-changing tasking priorities.  A continuous maintenance trigger, therefore, is only effective if the tasking priority is kept high.

In maintenance the database must be reviewed to re-familiarize (or familiarize if an analyst turnover has occurred) the analyst with its components and to identify known or suspected superceded data in the database.  From residual data, if available, the analyst identifies previously un-validated data and database data whose corresponding source documents is old as defined by the analyst. 

From the database review the analyst generates possible areas to explore to update the database.  The analyst returns to stored files to search for immediate data.  Any data found that is contrary to the database must be validated before the database is changed.  Once validated, data entry is performed manually.  Further research is performed as in construction.  If required, the analyst may update the system tasking developed in the construction process.  When maintenance is complete, the process customer receives the updated database as in the construction process.  

c.  KOPeR Analysis of the Current Process

As indicated in Figure 3.10 and amplified in Appendix A, the current construction sub-process contains fifteen steps, has a maximum process length of eleven, has three IT support steps, and has three IT communication steps.  There is one feedback loop; not shown in the Figure: At any time the analyst can seek feedback from the process customer for tasking amplification or to demonstrate preliminary results.  One handoff occurs when construction is complete.  From this input KOPeR identifies the sub-process as sequential with low IT support, IT communication, and IT automation fractions.  Full detail of KOPeR diagnoses and recommendations for all sub-processes discussed in this chapter are provided in Appendix A. 

The current maintenance sub-process as indicated in Figure 3.11 and amplified in Appendix A contains seventeen steps, has a maximum process length of thirteen, has three IT support steps, and has three IT communication steps.  There is one feedback loop.  One handoff occurs when maintenance is complete.  KOPeR also diagnoses this as a sequential process with low IT support, IT communication, and IT automation fractions. 

5.      Design The New Process           

The current construction and maintenance sub-processes are sequential largely due to the analyst’s inability to simultaneously perform tasks.  Additionally, data acquisition segments in the research phase (keyword web search, use of other domain experts, daily review of current intelligence, and searching filed hardcopy documents) could be considered sequential rather than parallel, since the analyst can only perform one of these at a time.  Intelligent agent technology could increase system parallelism.  Another option is to place additional analysts on the task, but this would increase the number of handoffs within the system, which would increase process friction.  Synthesizing data from all sources also becomes more difficult with addition of analysts.

 The analyst’s keyword searches on JDISS and SIPRNET and the review of secure intelligence reports are as much IT communication steps as IT support.  Certainly the computer enhances those processes, but the documents are made available on these secure networks as a form of IT communication between intelligence analysts.  To improve IT communication we focus our efforts on the overall database construction and maintenance process.    If database construction and maintenance were a more efficient use of analyst time, more databases would be shared on secure networks, thus improving IT communication.

a.  Re-Designed Construction Sub-Process Attributes

Our proposed re-designed construction sub-process (Figure 3.12) begins with analysis of immediate data as in the current process.  The analyst enters data into ADMS through a database agent, which automatically generates hypertext links and stores source files based on user “drag and drop” identification of source documents associated with manually entered data.  This assisted data entry provides an automated means of storing and referencing source documents that is intuitive to process customers and analysts.  The database agent tracks contradictory data and alerts the analyst when a user-set threshold ratio of contradicting to confirming data is exceeded.  The database agent also tracks source documents, validated and un-validated data, and notes missing data to support user analysis and provide information for other agents.  It is helpful to the analyst to retain un-validated data in the database if it is flagged so that it can be removed or hidden prior to sending the database to the process customer.

 


Figure 3.12:      Re-Designed Database Construction Sub-Process


The search agent, magnified in Figure 3.13, receives source documents and ADMS data from the database agent (source documents and ADMS data were created from the analysis of immediate data discussed above).  It uses ADMS primary key fields that have been filled to construct a keyword search.  The agent searches the network (including the analyst’s stored files) for documents that contain the keywords.  It ignores

Figure 3.13:      A Close Look at the Search Agent

 

documents that are duplicates of previously saved source documents.  The search agent uses a learning algorithm and existing source documents to generate rules for classification of the documents located during the keyword search.  Using the learned rules, a rating algorithm classifies each document located by the keyword search.  When the search agent has identified a set number of potentially useful documents (based on precision/recall requirements), it presents the user with those documents.  The analyst reviews those documents and classifies them (“relevant to ADMS construction” or “not relevant to ADMS construction”) while updating ADMS and adding new source documents using assisted data entry discussed previously.  The updated entries in ADMS add keywords (from ADMS primary keys) for further search by the search agent.  The user-classified documents from the search agent’s previous presentation of documents provide more documents for the search agent to refine its classification rules.  As this process continues the search agent continues to refine its rule set and continues to present to the user with improving precision. 

As discussed in Chapter II, numerous learning algorithms might prove effective for a given dataset.  The search agent can run several learning algorithms against a training dataset and choose the most efficient algorithm or combinations of algorithms.  This “algorithm maximization scheme” continues through all iterations of the learning process.

When the analyst concludes that a record in ADMS is validated, the search agent removes that record’s primary key from its keyword search and removes its supporting source documents from the training dataset.  This action prevents the search agent from presenting documents to the analyst, which contain data that has already been validated. 

Once construction has begun the search agent monitors the user’s daily review of current intelligence.  The search agent categorizes all documents based on user action.  If the analyst finds data during daily review that fills or validates a field in the database, the analyst interfaces with the database agent through assisted data entry to enter the new data into ADMS and to save the source documents.  The source documents are also fed to the search agent’s training dataset as a positive example.  If the analyst takes no action relative to ADMS with a document, the search agent assumes the document is not relevant and adds it to the training dataset as a negative example.  This technique causes the training dataset to build while preventing the search agent from presenting a document recently reviewed by the analyst.  The search agent provides a reminder periodically to the analyst that the agent is monitoring the review of traffic and offers the opportunity to turn this function off in busy situations. 

If during the search process the user finds structured or semi-structured data, they interface with the structured-data agent to automatically enter it into ADMS.  If the analyst so designates, the structured-data agent can also automatically update ADMS when the structured source document changes.  When multiple structured documents are located, the agent compares the data and alerts the analyst of conflicts.  Data entered into ADMS in this manner is automatically linked by hypertext to source documents.

Data found by analyst review of hardcopy documents and coordination with other system experts is entered through assisted data entry as with the analysis of immediate data.  The database agent can log the location of hardcopy source documents rather than force the analyst to electronically reproduce the data for the source files.

   At any point in construction, the analyst can interface with the database agent to view data that has not been validated.  If the analyst directs, the search agent can focus its efforts only on validation of existing, un-validated data.  When the analyst indicates database construction is complete, the system conducts a final check by interfacing with the analyst to determine an action for un-validated data.  It can be forwarded to the process customer or held back in the primary ADMS database (with a flag).

b.  Re-Designed Maintenance Sub-Process Attributes

The re-designed maintenance sub-process flow is illustrated in Figure 3.14.  The analyst sets the maintenance trigger during the final check of the construction process.  Periodic and continuous triggers launch maintenance automatically; maintenance can also be initiated manually.  Then the structured-data agent presents the analyst with data that has changed from structured text documents assigned during construction that were not identified as automatic update sites.  The analyst then may accept these changes or initiate search for validating data. 

Also upon initiation of the maintenance trigger, the database agent presents the analyst with a list of old and un-validated data.  The presented data allows the user to be aware of potential holes in ADMS during manual research.  Data is determined to be old based on a user-set age.

The database agent sends its old and un-validated data to the search agent that begins a web-search based on the filled ADMS primary key fields that contain old and un-validated data and rules learned from construction.  For old data the search agent re-introduces the corresponding source documents files into the training dataset to assist in rule generation.  The search agent and structured-data agent then function in a similar manner as in re-designed database construction.


Figure 3.14:      Re-Designed Database Maintenance Sub-Process


c.  KOPeR Analysis of the Re-designed Process

The re-designed construction sub-process (Figure 3.12) has 16 steps, with six IT support, three IT automation, and three IT communication steps.  The process length is nine, with one handoff and one feedback loop.  KOPeR evaluates this as a sequential process with low IT support, IT communication, and IT automation fractions.

The re-designed maintenance sub-process (Figure 3.14) has fifteen steps, with five IT support, six IT automation, and three IT communication steps.  It has a process length of seven.  There is one handoff, one feedback loop.  KOPeR assesses sub-process parallelism as OK (not a sequential process), but notes low IT support, IT communication, and IT automation fractions.   

 


Table 3.2:         KOPeR Design Comparisons

 

Table 3.2 compares the KOPeR process measurements of the current and new processes.  The re-designed construction and maintenance sub-processes have improved parallelism over the current sub-processes, but less so for the construction sub-process due to the analyst’s inability to perform simultaneous tasks.  The handoff, feedback, and IT communication nodes are identical in all four sub-processes.  Changes in these corresponding fractions are a result of fluctuations in process size.  Though KOPeR evaluates IT communication as low, we note again that improved database construction and maintenance may generate more database availability on secure networks and thus improve analyst-to-analyst communication. 

Though KOPeR evaluates the new sub-processes with low IT support and automation, we note improvement over the current processes.  Data complexity and issues of intelligence analyst credibility require analyst involvement in key points in the process.


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

THIS PAGE INTENTIONALLY LEFT BLANK

IV.            Experiments

 

 

As a proof of concept for one function of the search agent described in the last chapter, we now study a learning algorithm against Internet documents retrieved for ADMS construction.  We also present a potential structured-data agent for incorporation into the re-designed process. 

 

 

a.        Creation of the Datasets             

 

In the first experiment we test a learning algorithm’s ability to fill the Decision-maker Table (Figure 3.3) of the ADMS database.  We identify known decision-makers (six in our experiment) for which we conduct a keyword web search by manual entry of data on an Internet web-crawler and a search engine.  We also use the keywords in a search of Foreign Broadcast Information Service (FBIS) articles.  FBIS, an administrative part of CIA, translates radio and television stations, newspapers, and Internet sources in countries throughout the world (Naval War College, 2000). 

Because documents that discussed the original six decision-makers often introduced additional leaders, this initial search identified more decision-makers within our target country.  In addition, many of these web-based documents contained hypertext links to other potentially relevant documents that were not found by the keyword search; we also downloaded and inspected these documents.   Intelligence analysts use this technique to locate database construction data in web environments (Section III.B.4). 

To evaluate the subsequently described learning algorithm, we manually classified each downloaded document.  If the document mentioned at least one person’s name and title from the country of interest and that person was a military flag officer, provincial governor or of higher organizational rank, we classified it as “relevant.”  If the document did not meet these criteria we classified it as “irrelevant.”  We assumed the data was not validated and therefore also counted documents that confirmed data from previous documents as relevant.  Using these criteria we identified 105 relevant documents and 165 irrelevant documents.  We then divided the dataset into five subsets each containing 21 relevant and 33 irrelevant documents to achieve stratified five-fold datasets for cross-validation.


Figure 4.1:        Re-Designed Construction Sub-Process Simulated Activities

 

Though our dataset collection was conducted manually, it simulates analysis of immediate data, assisted data entry, and the search agent’s first keyword search in the re-designed database construction sub-process (figure 4.1).  The known leaders (six) in our initial search function as the output of analysis of immediate data, which becomes the initial keywords for the search agent through ADMS primary key fields.  The training datasets we created simulate classified documents from analysis of immediate data, daily review of traffic, and the search agent’s first presentation of documents.  The test dataset simulates documents retrieved by the search agent’s keyword web search that are presented to the rating agent for classification (Section III.B.5).  We are now prepared to evaluate a learning algorithm’s ability to increase document precision for the user. 

 

b.        A program for automatic document relevancy calculation

 

Dr. Neil Rowe of the Naval Postgraduate School designed a learning and rating program that establishes the probability that a document is relevant or not based on the occurrence of particular words in the training dataset.  We first discuss the program and then report experiments with it when applied to our dataset. 

1.      The Learning and Rating Algorithms

Prior to running the program, a user classifies the training dataset by placing relevant documents into a “yes” directory and irrelevant documents into a “no” directory.  The program then tries to remove English suffixes (“destems”) from each word in each document that is not in a dictionary of valid words.  When the program has completed counting occurrences of each valid word in the “yes” directory, it counts occurrences of each valid word in the “no” directory.  The program next calculates a relevance probability for each word as the ratio of the count of the word in the “yes” directory to the count of the word in both the “yes” and “no” directories provided that there are at least ten occurrences of the word. The relevance probability is considered statistically significant if it is outside of two standard deviations from the ratio of the total number of words in the “yes” directory to the total number of words in both directories.   

Once these relevance probabilities are learned, rating can be applied to an unclassified set of documents.  The rating algorithm destems words in the unclassified documents and compares them to the list of words with relevance probabilities.  The program calculates a weight for each, which is the word’s relevance probability minus the probability that a random word occurs in a “yes” document.  The weights for all words are averaged to get a total strength for the document. 

The strength for a document varies from a minimum of negative one to a maximum of positive one (the values we experienced were between –0.3 and +0.3).  The larger the strength, the more certain we are that a document is relevant.  Additional code could be written to determine the minimum strength value to achieve a desired precision based on the training dataset.  This value represents the cut-off for documents the search agent presents to the user. 

We evaluate the performance of the program using recall-precision curves generated by setting the minimum strength value to the strength of one of the fifty-four documents in the test dataset.  We then compare the assessment to the user classification and determine the number of true positives, false positives, true negatives, and false negatives.  From this we generate a precision and recall value.  We repeat this process for each of the fifty-four documents in the test dataset to generate a recall-precision graph.                    

2.      Results and Discussions         

The document strength values and the recall-precision curves generated from each of the five test subsets are recorded in Appendix B.  We now examine whether the algorithm is statistically better than the current manual search process.       

To locate and download the dataset’s 270 documents we used methods like the current web retrieval methods discussed in Chapter III.  This manual method yielded 38.9% precision (105 relevant documents ¸ 270 total documents = 0.389).  We compare the precision-recall data from each of the stratified five-fold cross-validations to the 38.9% precision.  Figure 4.2 illustrates this for Dataset 1.  The mean distance () from the 38.9% precision curve for each of the five test subsets is given in Appendix B as well as overall performance.  The program performs 18.8% better than the 38.9% baseline with a standard error (Equation 4.1) of +/- 4.8%. 

 

Figure 4.2:        Illustration of Dij Measurement1 for Document 8 of Dataset 1

Equation 4.1:   

S is the sample standard deviation

N is the sample size (five in our case)

 

Our 38.9% precision baseline was determined after one analyst conducted an Internet search; results by different analysts with different search engines would vary.  To quantify this variability, one would need to task several analysts to conduct a manual search for relevant documents and statistically compare precision results, a difficult experiment.  To study the variability in our baseline analysis we randomly select twenty-one documents from one of the five subsets and determine the resulting recall-precision point.  We repeat this random selection fifty-four times to generate the same number of points that were generated in each recall-precision graph of the stratified five-fold cross-validation. In Appendix B we determine the mean distance from the adjusted baseline is 1.1%, with standard error of +/- 0.5%.  So the program averaged 17.7% better than the adjusted baseline.  We use a statistical null hypothesis test in Appendix B to show that this is statistically significant.  

We have shown that even with a small dataset of documents, the program improves our search process.  It also requires less analytical time to achieve the higher precision.  Additionally, our re-designed process envisions a search agent with user-set recall-precision values, which the program permits.  When we determine  for only those points where recall is less than 50%, the resulting performance increases to an average precision of 74.6%, which is 35.7% above the 38.9% baseline (calculations in Appendix B).  Note the program was not specifically tailored for ADMS or international intelligence gathering.  The dictionary of valid words used does not contain foreign names, foreign governmental acronyms, or many foreign cities.  Additionally, a domain expert review of the learning algorithm’s identified words and relevance probabilities could benefit performance.   

      

c.        a structured-data agent

 

Of the 105 relevant documents located during our manual search, three are structured or semi-structured documents.  All three contained substantial data for construction of the Decision-Maker Table (Figure 3.3).  An analyst would need hours to manually enter each datum from these three documents.  The automated entry envisioned for the structured-data agent in Chapter III would greatly improve this process, and tools are becoming available like those based on the language XML. 

Fetch TechnologiesSM designed the Fetch Agent Platform™, which provides an example.  It conducts automated data extraction from structured and semi-structured HTML documents.  AgentBuilder™, part of the Fetch Agent Platform™, has a machine-learning algorithm that does not rely on HTML structure.  The user first teaches AgentBuilder™ the relationship between the semi-structured web data and the target database by classifying a handful of examples from the semi-structured document.  A machine-learning algorithm then generates rules that are used by AgentRunner™ to automatically transfer the semi-structured data to the target database.  An automated maintenance feature can detect a change in the semi-structured document and automatically or through user interaction incorporate the changes into the target database. (Fetch Technologies, 2001)

On 7 November 2001 Steve Minton of Fetch Technologies conducted a demonstration of the Fetch Agent Platform™ capabilities.  The system has an excellent graphical user interface (Figure 4.3) with intuitive functionality.  Using a semi-structured document similar to those in our dataset, he taught AgentBuilder™ appropriate classifications for a simple database in less than five minutes.  AgentRunner™ then automatically entered the data into the database.  The results of this demonstration are very promising.     

 

Figure 4.3:        Fetch AgentBuilder™ From Fetch Technologies (2001)

 

Further research is still required to verify interoperability with a database like ADMS and to evaluate fully the required attributes in Chapter III (automated update of the database when changes occur in the target document, comparison of structured and semi-structured documents to alert user of conflicts, and automated generation of hypertext links from the database to the source files).  Fetch Technologies is in the final stages of development of the Fetch Agent Platform™ and appears willing to support further research.  They require the researcher to complete a two-day training seminar prior to granting full access to their software.

 

V.               conclusions

 

 

A.        APplication

 

We have developed a high-level process design to “innovate” the intelligence-community database construction and maintenance process.  We obtained respectable results with an automated implementation of the search agent defined within our process.  The structured agent also appears well within reach of existing commercial products.   Other aspects of the high-level design are probably within reach of current technologies, but we have not validated those technologies in an intelligence environment.   

Though we used the ADMS database to focus our research, our methods can be applied to many other databases.  Much information and data available to intelligence analysts is constrained to the same electronic networks, stored files, current intelligence reports, other domain-expert assistance, and hard-copy files outlined in Chapter III.  Some databases designed for legacy systems may not support automated link generation between database data and source documents nor automated input from the structured-data agent, both discussed in Section III.B.5.  Nevertheless, even these systems could benefit from improved web searching of our search-agent design.

Aspects of our re-designed database construction and maintenance process can benefit other research.  Intelligence analysts, for example, could use the search agent to locate information for a specific intelligence question in the generation of an intelligence product.  The search agent could learn from the analyst’s initial search results and provide better precision in document retrieval.  The structured-data agent could automatically generate daily reports for intelligence customers.  As a customer became interested in a different region, the agent would retrieve weather forecasts, topographical information, threat contacts, political analysis, and other current intelligence about the region.   The agent could then display this multiple source data on one web page. 

      

b.        OTHER related work

 

Other work related to our database construction and maintenance process may provide a good starting point for follow-on research.  The Defense Advanced Research Projects Agency (DARPA) is conducting extensive research on foreign language translation under their Translingual Information Detection, Extraction and Summarization (TIDES) programTIDES is focused on the automated processing and understanding of foreign language data with the goal to find and interpret needed information regardless of language (Wayne, 2002).  If it is successful, TIDES will greatly increase the amount of information available to intelligence analysts.  Though this certainly has tremendous advantages, it risks information overload in analysis.  Integration of our database construction and maintenance design with TIDES may enhance the user’s ability to access only the required information.

ClearForest™ advertises a platform and accompanying products that review large amounts of text, locate relevant information, and produce interactive executive summaries.  The ClearReasearch™ and ClearSight™ products appear to automatically build entity relationships with drill-down capability to source documents.  The online demonstration focuses on corporation data, but has definite parallels to the decision-maker influence modeling requirements of Pacific Command’s Information Operations Cell (Section III.A).  This information retrieval and summarization scheme could enhance our search agent.  (ClearForest, 2002) 

Oak Ridge National Laboratory has developed a multi-agent platform for US Pacific Command, called VIPAR, which automatically collects, organizes, and summarizes thousands of newspaper articles.  The system is tailored for a specific Pacific Command task, but probably could be re-tailored for ADMS (Potok, 2002).  

Scheffer et al. (2001) investigates web mining using active hidden Markov models with user-labeled tokens for the training dataset.  They developed a web mining tool, “SemanticEdge”, with a user interface to allow domain experts to assist the learning algorithm.  This method offers a potential improvement over full document classification since the user identifies the portion of the document that is relevant. 

 

c.        rECOMMENDATIONS for further research

 

We have executed the first four steps of Davenport’s process innovation methodology (identify the process for innovation, identify change levers, develop process visions, and understand the existing process).  We have laid the foundation for step five, design and prototyping of the new process, with our high-level process design (Section III.B.5) and our proof of concept experiments (Chapter IV).  Further application design and prototype generation is required to complete the methodology.   

Future researchers could build on this work by continuing to develop and test components of our design.  It appears that Fetch Technologies™ (Section IV.C) has a solution to the structured agent component, but further testing is required to validate their product.  Other companies, ClearForrest™ for example, may have competitive products for comparison. 

A programmer could develop code to handle “assisted data entry” that automatically stores and links data in the database to appropriate source documents (Section III.B.5).    Additionally, the learning and rating program we evaluated (Section IV.B) requires a graphical user interface, an intelligence domain dictionary, and an automatic interface with a web crawler or search engine.  Other learning algorithms could also be evaluated against the search agent attributes and incorporated into an algorithm maximization scheme (Section III.B.5).  Further work could develop a prototype of the database agent and the search agent (Section III.B.5) or locate and test commercially available agents.  As research develops these components, further work could investigate the best method for multi-agent interaction in this environment.

appendix A – amplifiing notes on process flow diagrams and KOPer results

 

 

A.        Amplifiing notes on process flow diagrams (figure 3.10-3.12, and figure 3.14)

                                                                                                   

Figure 3.10 through 3.12, and Figure 3.14 are flow diagrams for the current and redesigned maintenance and construction sub-processes.  This section identifies those steps counted for KOPeR analysis.  Steps in dashed lines in these figures were not included in KOPeR analysis; they represent trivial analytical tasks, but are displayed to show actual process flow.  Each counted step is identified within these figures as IT support (IT-S), IT communication (IT-C), IT automation (IT-A), or manual.  The handoff in all sub-processes occurs at process completion.  Customer feedback can occur at any time (relations between customers and intelligence analysts are generally open), but is not shown on the diagrams.     

 The fifteen counted steps of the current construction sub-process in Figure 3.10 are (1) stored file data acquisition, (2) stored-file and known-data analysis, (3) data validation of immediate data, (4) data entry of immediate data, (5) saving the source documents for immediate data, (6) system tasking, (7) daily review of current intelligence, (8) filed hardcopy document acquisition, (9) data acquisition from other domain experts, (10) keyword web search, (11) analysis of acquired data, (12) data validation of research and analysis data, (13) data entry of research and analysis data, (14) saving the source documents for research and analysis data, and (15) delivering ADMS to the process customer.

The seventeen counted steps of the current maintenance sub-process in Figure 3.11 are (1) the maintenance trigger, (2) reviewing database for known or suspected changes, (3) stored-file data acquisition, (4) stored-file analysis, (5) data validation of immediate data, (6) data entry of immediate data, (7) saving the source documents for immediate data, (8) update system tasking, (9) daily review of current intelligence, (10) filed hardcopy document acquisition, (11) data acquisition from other domain experts, (12) keyword web search, (13) analysis of acquired data, (14) data validation of research and analysis data, (15) data entry of research and analysis data, (16) saving the source documents for research and analysis data, and (17) delivering ADMS to the process customer.

The sixteen counted steps of the re-designed construction sub-process in Figure 3.12 are (1) stored-file data acquisition, (2) stored-file and known-data analysis, (3) assisted data entry of immediate data, (4) data acquisition from other domain experts, (5) filed hardcopy document acquisition, (6) daily review of current intelligence, (7) analysis of manually acquired data, (8) system tasking, (9) assisted data entry of research and analysis (other than electronic search) data, (10) database-agent activities, (11) search-agent activities, (12) analysis of search-agent documents, (13) assisted data entry for natural-language documents, (14) structured-data agent activities, (15) final check, and (16) Delivery of ADMS to the process customer.

The fifteen counted steps of the new maintenance sub-process in Figure 3.14 are (1) the maintenance trigger, (2) structured-data agent activities, (3) analysis of structured-data agent changes since construction, (4) database-agent activities, (5) data acquisition from other domain experts, (6) filed hardcopy document acquisition, (7) daily review of current intelligence, (8) analysis of manually acquired data, (9) update system tasking, (10) assisted data entry of research and analysis (other than electronic search) data, (11) search-agent activities, (12) analysis of search agent documents, (13) assisted data entry for natural-language documents, (14) final check, and (15) delivery of ADMS to the process customer.

 

B.        Koper analysis results

 

Background for KOPeR analysis is provided in Section II.B.5.  The results shown below are the direct output of KOPeR analysis.

1.      Current Database Construction Sub-Process

Measurements suggest the Current Database Construction sub-process suffers from the following pathologies:

Parallelism (1.364) - sequential process.

Handoffs fraction (0.067) - handoffs look OK.

Feedback fraction (0.067) - feedback looks OK.

IT support fraction (0.2) - inadequate IT support.

IT communication fraction (0.2) - inadequate IT communications.

IT automation fraction (0.0) - IT automation first requires substantial infrastructure in terms of support and communication.

For redesign, KOPeR recommends considering the following:

Delinearize process activities to increase parallelism; such activities must be sequentially-independent (e.g., have mutually-exclusive inputs and outputs).

Look to information technology to increase support to process activities; decision support systems and desktop office tools generally have good payoffs and intelligent systems can greatly enhance knowledge work; be sure to address personnel training and maintenance of the IT.

Look to information technology to increase support to process communications; e-mail and shared databases through local/wide area networks generally have good payoffs and workflow systems can greatly expedite process flows; be sure to address personnel training and maintenance of the IT.

Look to information technology to automate process activities, but note that substantial IT infrastructure is first required, particularly in terms of process support and communication; try workflow systems for support and communication, and then look to intelligent agents, which can enable many electronic commerce opportunities.

2.      Current Database Maintenance Sub-Process

Measurements suggest the Current Database Maintenance sub-process suffers from the following pathologies:

Parallelism (1.308) - sequential process.

Handoffs fraction (0.059) - handoffs look OK.

Feedback fraction (0.059) - feedback looks OK.

IT support fraction (0.176) - inadequate IT support.

IT communication fraction (0.176) - inadequate IT communications.

IT automation fraction (0.0) - IT automation first requires substantial infrastructure in terms of support and communication.

For redesign, KOPeR makes the same recommendations as those for the Current Construction Sub-Process.

3.      Re-Designed Database Construction Sub-Process

Measurements suggest the Re-designed Database Construction sub-process suffers from the following pathologies:

·        Parallelism (1.778) - sequential process.

·        Handoffs fraction (0.063) - handoffs look OK.

·        Feedback fraction (0.063) - feedback looks OK.

·        IT support fraction (0.375) - inadequate IT support.

·        IT communication fraction (0.188) - inadequate IT communications.

·        IT automation fraction (0.188) - IT automation first requires substantial infrastructure in terms of support and communication.

For redesign, KOPeR has the same recommendations stated previously.

4.      Re-Designed Database Maintenance Sub-Process

Measurements suggest the Re-designed Database Maintenance sub-process suffers from the following pathologies:

·        Parallelism (2.143) - parallelism looks OK for this class of process.

·        Handoffs fraction (0.067) - handoffs look OK.

·        Feedback fraction (0.067) - feedback looks OK.

·        IT support fraction (0.333) - inadequate IT support.

·        IT communication fraction (0.2) - inadequate IT communications.

·        IT automation fraction (0.2) - IT automation first requires substantial infrastructure in terms of support and communication.

Since parallelism has improved above KOPeR’s threshold, it does not recommend efforts to delinearize this sub-process.  All other recommendations are the same as the previous analysis.

appendix B – recall-precision data and statistical calculations

                                                                                                   

A.        recall-precision data and graphs

 

Table B.1 through B.5 contains data from which the recall-precision graphs (Figure B.1 through B.5) are derived.  The first column of each table, “Dataset Document #”, contains the unique identifier for each of the fifty-four documents within the particular dataset.  The second column, “Document Strength”, gives the average strength of the corresponding document as calculated by the Rowe rating algorithm.  The “True Value” column corresponds to the domain expert classification (“Positive” corresponds to a documents classified as relevant).  The fourth through seventh columns are the number of documents in the dataset that are in each category (“True Pos”, “False Neg”, “True Neg”, and “False Pos”) when the row’s document strength is assigned the cut-off for presentation to the user.  We calculate Recall directly from the true positive and false negatives.  Precision, likewise, is calculated directly from the true positives and false positives.  In this way, each document strength generates one recall-precision point for the dataset’s recall-precision graph.  The last two columns in these tables will be discussed in the next section.    


Table B.1:        Dataset 1 Data

 


Table B.2:        Dataset 2 Data

 


Table B.3:        Dataset 3 Data

 


Table B.4:        Dataset 4 Data

 


Table B.5:        Dataset 5 Data

 

Figure B.1:       Recall-Precision Graph (Dataset 1 as Test Dataset)

 

Figure B.2:       Recall-Precision Graph (Dataset 2 as Test Dataset)

Figure B.3:       Recall-Precision Graph (Dataset 3 as Test Dataset)

 

Figure B.4:       Recall-Precision Graph (Dataset 4 as Test Dataset)

Figure B.5:       Recall-Precision Graph (Dataset 5 as Test Dataset)

 

 

b.        average distance and overall performance calculations

 

The Dij column in Tables B.1 through B.5 is the result of the 38.9% precision baseline subtracted from the corresponding Precision value.  From this we determine the mean distance () for each dataset.  These values are displayed in Table B.6 along with the overall performance measure versus the 38.9% precision line.

 


Table B.6:        Overall Performance Calculations Compared to 38.9% Baseline

 

The last column in Tables B.1 through B.5 is used in Table B.7 to calculate overall performance of the Rowe program versus the 38.9% baseline for lower recall (recall  0.5).  This calculation is conducted to show the advantage of adjustable recall-precision settings in the search agent.

 


Table B.7:        Overall Performance Calculations Compared to 38.9% Baseline for Recall 0.5


C.        ADJUSTED Baseline Calculations

 

Using the Microsoft Excelä random number generator we randomly selected 21 documents from dataset one.  We repeat the random selection fifty-four times.  The resultant Precision and B1j calculations are shown in the last two columns of Tables B.8.  We repeat this process for dataset two through five and record the results in Tables B.9 through B.12.

 


Table B.8:        Dataset 1 Random Document Selection for Adjusted Baseline Calculations

 


Table B.9:        Dataset 2 Random Document Selection for Adjusted Baseline Calculations

 


Table B.10:      Dataset 3 Random Document Selection for Adjusted Baseline Calculations

 


Table B.11:      Dataset 4 Random Document Selection for Adjusted Baseline Calculations

 


Table B.12:      Dataset 5 Random Document Selection for Adjusted Baseline Calculations

 


We next calculated the mean distance from the 38.9% precision line for each dataset and the resulting total mean and standard error for all five datasets (Table B.13).  As we are only concerned with variation above the 38.9% baseline, we used the absolute value of each dataset mean distance () to determine the overall average.  This overestimates the adjusted baseline, which requires the algorithm performance to be higher before it is evaluated as statistically different from the adjusted baseline.   

 


Table B.13:      Adjusted Baseline Calculations

 

 

D.        Null hypothesis test

 

We choose as a null hypothesis, which is assumed true unless statistically rejected, that the adjusted baseline is as good as the algorithm performance.  Statistically, this means that the population mean of all possible adjusted baseline samples, μb, is equal the population mean of all random samples of the Rowe performance measure, μd.  

We do not know the true population means, but we do have the results of one random sample of each total population.  Given this we establish a test statistic, T, to determine with a reasonable degree of certainly if our null hypothesis is true.  Equation C.1 defines T, where Sd is the standard deviation for our sample of algorithm performance (), Sb is the standard deviation for our sample of the adjusted baseline overall average (), and Nd  and Nb represent the size of each sample; five in both cases (Bailey, 1971).  

 

 Equation B.1:  

 

Given our null hypothesis, equation C.1 can be simplified as in Equation C.2.

 

Equation B.2: 

 

The statistical distribution of numerous random sample means of the same population is a normal distribution.  Since we have a small number of samples, we use Gossett’s t-distribution to analyze our sample variation.  A t-distribution is a bell-shaped, symmetric distribution that is flatter than a normal distribution.  The t-distribution is defined for different degrees of freedom.  For our case the degree of freedom is defined by Bailey (1971) in equation C.3.

Equation B.3:   

 

Using Equation C.3 and the results from Table B.6 and Table B.13, we determine we have four degrees of freedom ().  Since the null hypothesis is rejected only if the algorithm proves to perform statistically better than the adjusted baseline, we use a one-sided test with 97.5% confidence that compares the upper tail of the adjusted baseline (plus its error) to the lower tail of the algorithm performance (minus its error).  Thus we reject the null hypothesis only if T is greater than the critical value on the t-distribution curve corresponding to 0.975 (t(0.975,4)=2.77645). 

Using the results from Table B.6 and Table B.13, we determine from Equation C.2 that T=3.67 and reject the null hypothesis (T>t0.975,4).  Thus the algorithm performance is statistically better than the adjusted baseline.


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

THIS PAGE INTENTIONALLY LEFT BLANK

List of references

 

 

Hervé Abdi, Dominique Valentin, and Betty Edelman, Neural Networks (Thousand Oaks, CA: Sage Publications, 1999).

 

Daniel E. Bailey, Probability and Statistics: Models for Research (New York: John Wiley and Sons, 1971).

 

“ClearForest Knowledge Your Way.” <http://www.clearforest.com/products/products.asp> (21 February 2002).

 

M. Craven, D. Dipasquo, D. Freitag, A. McCallum, T. Mitchell, K. Nigam, and S. Slattery, “Learning to construct knowledge bases from the World Wide Web” in Artificial Intelligence, Volume 118, April 2000, 69-113.

 

Thomas H. Davenport, Process Innovation (Boston: Harvard Business School Press, 1993).

 

“Fetch Technologies: Overview.” Lkd.  Fetch Technologies at “Fetch Technologies.” 2001. <http:/www.fetch.com.default.asp/> (25 February 2002).

 

D. N. Fowler, “Innovating the Standard Procurement System Utilizing Intelligent Agent Technologies,” (Monterey, CA: Naval Postgraduate School, M. S. Thesis, 1999).

 

William R. Gates and Mark E. Nissen, “Designing Agent-based Electronic Employment Markets” in Electronic Commerce Research Journal 1:3, Special Issue on Theory and Application of Electronic Market Design, 2001, 239-263.

 

Naval War College, “Intelligence/C4ISR and The Operational Commander” in Joint Maritime Operations: Block 3.7 (Newport, RI: Naval War College, 2000).

 

Mark E. Nissen, “Agent Based Supply Chain Integration” in Journal of Information Technology Management, Volume 2:3, 2001.

 

Mark E. Nissen, “Redesigning Reengineering through Measurement Driven Inference” in MIS Quarterly, Volume 22:4, December 1998, 509-534.

 

Mark Nissen, Magdi Kamel, and Kishore Sengupta, “Integrated Analysis and Design of Knowledge Systems and Processes” Information Resources Management Journal, August 1999.

Thomas E. Potok, “Intelligent Software Agent Technology” (a VIPAR presentation) Lkd. VIPAR Multi-Agent Intelligence Analysis System at “VIPAR.”  2001.  <http:www.csm.ornl.gov/~v8q/Homepage/Projects/vipar.htm > (25 February 2002).

 

Da Ruan, ed., Intelligent Hybrid Systems: Fuzzy Logic, Neural Networks, and Genetic Algorithms (Norwell, Massachusetts: Kluwer Academic Publishers, 1997).

 

Gerard Salton, Amit Singhal, Manar Mitra, and Chris Buckley, “Automatic Text Structuring and Summarization” in Information Processing and Management: An International Journal, Volume 33: 2, March 1997, 193-207.

 

Tobias Scheffer, Christian Decomain, and Stefan Wrobel, “Mining the Web with Active Hidden Markov Models,” in 2001 IEEE International Conference on Data Mining: 29 November-2 December 2001 eds. Nick Cercone, T.Y. Lin, and Xindong Wu (Washington: IEEE Computer Society 2001), 645-646.

 

C. Schwartz, “Web Search Engines” in Journal of American Society for Information Science, Volume 49:11, September 1998, 973-982.

 

T. L. Ward, “A Database Of Adversary Decision Makers,” (Monterey, CA: Naval Postgraduate School, M. S. Thesis, 2001).

 

Charles Wayne, “Translingual Information Detection, Extraction and Summarization (TIDES).”  <http://www.darpa.mil/ito/research/tides/mission.html> (19 February 2002).

 

Ian H. Witten and Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations (San Francisco: Morgan Kaufmann Publishers, 2000).

 

 

Bibliography

 

Ellen Riloff, “Using Learned Extraction Patterns for Text Classification,” in Connectionist, Statistical and Symbolic Approaches to Learning for Natural Lanquage Processing (Springer Verlag: 1996), 275-279.

 

Jiawei Han, Yue Huang, Nick Cercone, and Yongjian Fu, “Intelligent Query Answering by Knowledge Discovery Techniques” in IEEE Transactions on Knowledge and Data Engineering, Volume 8:3, June 1996, 373-390.

 

Karen Sparck Jones and Julia R. Galliers, Evaluating Natural Language Processing Systems: An Analysis and Review (Berlin, Germany: Springer, 1995).

 

Elias Oxendine IV, “Managing knowledge in the Battle Group Theater Transition Process (BGTTP)” (Monterey: Naval Postgraduate School Thesis, 2000).

 

S. K. Michael Wong and Cory J. Butz, “Constructing the Dependency Structure of a Multiagent Probabilistic Network” in IEEE Transactions on Knowledge and Data Engineering, Volume 13:3, May/June 2001, 395-415.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

THIS PAGE INTENTIONALLY LEFT BLANK

INITIAL DISTRIBUTION LIST

 

 

 

1.                  Defense Technical Information Center

Ft. Belvoir, VA

 

2.                  Dudley Knox Library

Naval Postgraduate School

Monterey, CA

 

3.                  Department of the Navy

Office of the Chief of Naval Operations (N6109)

Washington DC

 

4.                  Fleet Information Warfare Center

NAB Little Creek

Norfolk, VA

 

5.                  LCDR Steven J. Iatrou

Naval Postgraduate School

Monterey, CA

 

6.                  Mark E. Nissen

Naval Postgraduate School

Monterey, CA

 

7.                  Neil Rowe

Naval Postgraduate School

Monterey, CA 

 

8.                  Dave Olwell

Naval Postgraduate School

Monterey, CA

 

9.                  CDR Stephen Recca

Naval Postgraduate School

Monterey, CA

 

10.              D. C. Boger

Naval Postgraduate School

Monterey, CA



1 The use of any country or decision-maker in these figures is for example purposes only and does not necessarily represent real data nor indicate any position of the United States Government regarding status of the country or the decision-maker.

1 The use of any country or decision-maker in these figures is for example purposes only and does not necessarily represent real data nor indicate any position of the United States Government regarding status of the country or the decision-maker.

        1 The use of any country or decision-maker in these figures is for example purposes only and does not necessarily represent real data nor indicate any position of the United States Government regarding status of the country or the decision-maker.

1 The use of any country or decision-maker in these figures is for example purposes only and does not necessarily represent real data nor indicate any position of the United States Government regarding status of the country or the decision-maker.

 

        1 The use of any country or decision-maker in these figures is for example purposes only and does not necessarily represent real data nor indicate any position of the United States Government regarding status of the country or the decision-maker.

1 The use of any country or decision-maker in these figures is for example purposes only and does not necessarily represent real data nor indicate any position of the United States Government regarding status of the country or the decision-maker.

1 The subscript i represents the 1st, 2nd, 3rd, 4th, or 5th fold of the cross-validation process; the subscript j represents each of the fifty-four points on the recall-precision curve