Knowledge Acquisition

Information presented in this module is largely summarized from:
Jones, P.H. 1989. Knowledge Acquisition. In: Barrett, J.R. and D.D. Jones. Knowledge Engineering in Agriculture. ASAE Monograph No. 8, ASAE, St. Joseph, MI.

Introduction

Knowledge acquisition is the process of extracting, structuring and organizing knowledge from one source, usually human experts, so it can be used in software such as an ES. This is often the major obstacle in building an ES.

There are three main topic areas central to knowledge acquisition that require consideration in all ES projects. First, the domain must be evaluated to determine if the type of knowledge in the domain is suitable for an ES. Second, the source of expertise must be identified and evaluated to ensure that the specific level of knowledge required by the project is provided. Third, if the major source of expertise is a person, the specific knowledge acquisition techniques and participants need to be identified.

Theoretical Considerations

An ES attempts to replicate in software the reasoning/pattern-recognition abilities of human experts who are distinctive because of their particular knowledge and specialized intelligence. ES should be heuristic and readily distinguishable from algorithmic programs and databases. Further, ES should be based on expert knowledge, not just competent or skillful behavior.

Domains

Several domain features are frequently listed for consideration in determining whether an ES is appropriate for a particular problem domain. Several of these caveats relate directly to knowledge acquisition. First, bona fide experts, people with generally acknowledge expertise in the domain, must exist. Second, there must be general consensus among experts about the accuracy of solutions in a domain. Third, experts in the domain must be able to communicate the details of their problem solving methods. Fourth, the domain should be narrow and well defined and solutions within the domain must not require common sense.

Experts

Although an ES knowledge base can be developed from a range of sources such as textbooks, manuals and simulation models, the knowledge at the core of a well developed ES comes from human experts. Although multiple experts can be used, the ideal ES should be based on the knowledge of a single expert. In light of the pivotal role of the expert, caveats for choosing a domain expert are not surprising. First, the expert should agree with the goals of the project. Second, the expert should be cooperative and easy to work with. Third, good verbal communication skills are needed. Fourth, the expert must be willing and able to make the required time commitment (there must also be adequate administrative/managerial support for this too).

Knowledge Acquisition Technique

At the heart of the process is the interview. The heuristic model of the domain is usually extracted through a series of intense, systematic interviews, usually extending over a period of many months. Note that this assumes the expert and the knowledge engineer are not the same person. It is generally best that the expert and the knowledge engineer not be the same person since the deeper the experts' knowledge, the less able they are in describing their logic. Furthermore, in their efforts to describe their procedures, experts tend to rationalize their knowledge and this can be misleading.

General suggestions about the knowledge acquisition process are summarized in rough chronological order below:

  1. Observe the person solving real problems.
  2. Through discussions, identify the kinds of data, knowledge and procedures required to solve different types of problems.
  3. Build scenarios with the expert that can be associated with different problem types.
  4. Have the expert solve a series of problems verbally and ask the rationale behind each step.
  5. Develop rules based on the interviews and solve the problems with them.
  6. Have the expert review the rules and the general problem solving procedure.
  7. Compare the responses of outside experts to a set of scenarios obtained from the project's expert and the ES.
Note that most of these procedures require a close working relationship between the knowledge engineer and the expert.

Practical Considerations

The preceding section provided an idealized version of how ES projects might be conducted. In most instances, the above suggestions are considered and modified to suit the particular project. The remainder of this section will describe a range of knowledge acquisition techniques that have been successfully used in the development of ES.

Operational Goals

After an evaluation of the problem domain shows that an ES solution is appropriate and feasible, then realistic goals for the project can be formulated. An ES's operational goals should define exactly what level of expertise its final product should be able to deliver, who the expected user is and how the product is to be delivered. If participants do not have a shared concept of the project's operational goals, knowledge acquisition is hampered.

Pre-training

Pre-training the knowledge engineer about the domain can be important. In the past, knowledge engineers have often been unfamiliar with the domain. As a result, the development process was greatly hindered. If a knowledge engineer has limited knowledge of the problem domain, then pre-training in the domain is very important and can significantly boost the early development of the ES.

Knowledge Document

Once development begins on the knowledge base, the process should be well documented. In addition to tutorial a document, a knowledge document that succinctly state the project's current knowledge base should be kept. Conventions should be established for the document such as keeping the rules in quasi-English format, using standard domain jargon, giving descriptive names to the rules and including supplementary, explanatory clauses with each rule. The rules should be grouped into natural subdivisions and the entire document should be kept current.

Scenarios

An early goal of knowledge acquisition should be the development of a series of well developed scenarios that fully describe the kinds of procedures that the expert goes through in arriving at different solutions. If reasonably complete case studies do not exist, then one goal of pre-training should be to become so familiar with the domain that the interviewer can compose realistic scenarios. Anecdotal stories that can be developed into scenarios are especially useful because they are often examples of unusual interactions at the edges of the domain. Familiarity with several realistic scenarios can be essential to understanding the expert in early interviews and the key to structuring later interviews. Finally, they are ultimately necessary for validation of the system.

Interviews

Experts are usually busy people and interviews held in the expert's work environment are likely to be interrupted. To maximize access to the expert and minimize interruptions it can be helpful to hold meetings away from the expert's workplace. Another possibility is to hold meetings after work hours and on weekends. At least initially, audiotape recordings ought to be made of the interviews because often times notes taken during an interview can be incomplete or suggest inconsistencies that can be clarified by listening to the tape. The knowledge engineer should also be alert to fatigue and limit interviews accordingly.

In early interviews, the format should be unstructured in the sense that discussion can take its own course. The knowledge engineer should resist the temptation to impose personal biases on what the expert is saying. During early discussions, experts are often asked to describe the tasks encountered in the domain and to go through example tasks explaining each step. An alternative or supplemental approach is simply to observe the expert on the job solving problems without interruption or to have the expert talk aloud during performance of a task with or without interruption. These procedures are variations of protocol analysis and are useful only with experts that primarily use verbal thought processes to solve domain problems.

For shorter term projects, initial interviews can be formalized to simplify rapid prototyping. One such technique is a structured interview in which the expert is asked to list the variables considered when making a decision. Next the expert is asked to list possible outcomes (solutions) from decision making. Finally, the expert is asked to connect variables to one another, solutions to one another and variables to solutions through rules.

A second technique is called twenty questions. With this technique, the knowledge engineer develops several scenarios typical of the domain before the interview. At the beginning of the interview, the expert asks whatever questions are necessary to understand the scenario well enough to determine the solution. Once the expert begins the questions, the expert is asked to explain why each question is asked. When the interviewer perceives a rule, he interrupts and restates the rule to ensure that it is correct.

A third technique is card sorting. In this procedure, the knowledge engineer prepares a stack of cards with typical solutions to problems in the domain. The expert is asked to sort the cards according to some characteristic important to finding solutions to the problem. After each sort, the expert is asked to identify the sorting variable. After each sort, the expert is asked to repeat the process based on another variable. Note that this technique is usually not as effective as the 2 previous.

In large projects, later interviews cannot be expected to be as productive as early interviews. Typically, later interviews should become increasingly structured and follow a cyclical pattern where bits of knowledge are elicited, documented and tested. During this phase of knowledge acquisition, the interviewer must begin methodically to uncover the more subtle aspects of the knowledge. Typically, this process is based on scenarios. By modifying the scenarios in different ways, the interviewer can probe the expert's sensitivity.

During interviews, it may be helpful to work at a whiteboard to flexibly record and order the exact phraseology of rules or other representations. It may also be helpful to establish recording conventions for use such as color coding different aspects of a rule and using flags to note and defer consideration of significant but peripheral details.

Structured interviews should direct the course of a meeting to accomplish specific goals defined in advance. For instance, once a prototypic knowledge base is developed, the expert can be asked to evaluate it line by line. Other less obvious structures can be imposed on interviews, such as asking the expert to perform a task with limited information or during a limited period of time. Even these structured interviews can deviate from the session's intended goals. Sometimes such deviations show subtleties in the expert's procedures and at other times the interview simply becomes sidetracked, requiring the knowledge engineer to redirect the session.

Questionnaires

When specific information is needed, a questionnaire can sometimes be used effectively. Questionnaires are generally used in combination with other techniques such as interviews.

Decision Trees

Decision trees are widely recognized to be useful tools for the knowledge engineer in prototyping knowledge representations. In addition, they can be useful in knowledge acquisition on several different levels. Some knowledge engineers have found that experts can more readily relate to decision trees than rules.

Rule Development

Although complex representation techniques might eventually be used, rules are generally easier to use for characterizing knowledge during knowledge acquisition. Prototypic rules should be developed as soon as possible to serve as a focal point for directing the course of the knowledge acquisition process. The initial knowledge base can be developed from written materials or from example cases described by the expert during early unstructured interviews. Initial rules should be treated as approximations and their wording should be general to avoid pressuring the expert. As additional cases are described during interviews, the rule base can be expanded. Once a stable rule base begins to develop, it can provide feedback for structuring interviews. Initially the rules and procedures can be traced through by hand with the expert considering each step. The same pattern of tracing through rules should continue once a version of the knowledge base is developed on a computer and it frequent use should become part of the process.

Conclusions

Recognition of the central role of knowledge acquisition in the development of ES is an unavoidable prerequisite for any knowledge engineering project. If domains are defined and experts chosen with this in mind, a project's chances for success will be greatly increased. Once an appropriate domain is identified and a cooperative, available expert with the necessary stamina is found, then the practical approaches to knowledge acquisition outlined should be of help.

There are several points that deserve to be emphasized:

  1. Projects need to be well planned. The knowledge engineer has the responsibility initially to define explicit operational goals that are consistent with the resources available to a project. Early in the interview process, the purpose of the project and the roles of the participants should be carefully discussed. The discussion should lead to a consensus opinion on what is expected in a final product, who its users should be and how it should be delivered. As the project develops, the operational goals should be consciously reconsidered on a regular basis.
  2. The knowledge acquisition process should be carefully planned to match the requirements of the project's domain expert. For example, time lines that allow for the necessary pre-training, unstructured interviewing and the structured interview phases can be developed. Documentation procedures to be used during the project should be defined.
  3. Specific interviews or knowledge gathering events should be planned to accomplish specific goals. In pre-training, the goal might be to identify several realistic scenarios and during the first unstructured interviews one goal might be to develop a glossary of the expert's terminology. Later, the goals might be to obtain specific bits of information to explain apparent discrepancies. In planning individual interviews, the knowledge engineer should try to get explicit feedback.

Regardless of its size or the intended application, the knowledge acquisition process cannot be avoided. Recognition of this is the first step toward the successful development of a functional ES.