Studies on the application of AI techniques to engineering design, analysis, and manufacturing (AI EDAM) problems have been expanding steadily over the past decade. These studies have led to the founding of many new journals and the initiation of series of conferences. If all these research efforts were successful, some of us might have been unemployed but living happily off the royalties from our successful research products that were deployed in practice.

The status of research methodology employed by studies on the application of AI techniques to solving problems in engineering design, analysis, and manufacturing is poor. There may be many reasons for this status, including: unfortunate heritage from AI, poor educational system, and researchers’ sloppiness. Understanding this status is a prerequisite for improvement. The study of research methodology can promote such understanding, but, most importantly, it can assist in improving the situation. Concepts from the philosophy of science are introduced, and models of world views of science are built on them. These world views are combined with research heuristics or research perspectives and criteria for evaluating research to create a layered model of research methodology. This layered model can serve to organize and facilitate a better understanding of future studies of research methodologies. Many of the issues involved in the study of AI and AIEDAM research methodology using this layered model are discussed.

Less than a decade ago it seemed that a new paradigm of engineering–called computer-aided engineering (CAE) – was emerging. This emergence was driven in part by the success of computer support for the tasks of engineering analysis and in part by a new understanding of how computational ideas largely rooted in artificial intelligence (AI) could perhaps improve the practice of engineering, especially in the area of design synthesis. However, while this “revolution” has failed to take root or flourish as a separate discipline, it has spawned research that is very different from traditional engineering research. To the extent that such CAE research is different in style and paradigm, it must also be evaluated according to different metrics. Some of the metrics that can be used are suggested, and some of the evaluation issues that remain as open questions are pointed out.

Perspectives on research methodology in the field of “AI and design” are discussed. This perspective is based on a view of a “science of design” focusing on methods of design and on characteristics of design tasks that affect what methods are relevant for a given task. Two methodological issues are discussed: the need to try applying a design method on multiple tasks and domains, and the need to work with collaborators who are experts in the task domain of each research system you build.

A multifaceted approach to evaluating expert systems is overviewed. This approach has three facets: a technical facet, for “looking inside the black box”; an empirical facet, for assessing the system’s impact on performance; and a subjective facet, for obtaining users’ judgments about the system. Such an approach is required to test the system against the different types of criteria of interest to sponsors and users and is consistent with evolving lifecycle paradigms. Moreover, such an approach leads to the application of different evaluation methods to answer different types of evaluation questions. Different evaluation methods for each facet are overviewed.

In cases where a domain theory can be successfully expressed in a logical formalism and can be used to formulate a task in that domain in mathematical terms, the task of building sound knowledge-based systems is greatly facilitated. However, it is not immediately obvious how the design aspects of such tasks, where these are an important feature of problem solving, can be incorporated in this approach. Design issues differ from search problems in that there may be several choices, each valid in some sense, but not (necessarily) equally good or equally appropriate in the current context. A case study is described in which a methodology is used based on the development of proof plans. The ability to conduct research according to the Popperian framework of hypothesis, validation, testing, and modification in response to empirical evidence – the hypothetico-deductive approach – seems essential to any rigorous scientific endeavor. It is believed that proof planning is a method which readily exploits this inherently incrementalist approach and could prove to be a powerful tool in designing AI systems.

Contributions of general design theory (GDT) proposed by Yoshikawa for the development of advanced CAD (computer-aided design) and for innovative design from the research results of a group at the University of Tokyo are illustrated. First, the GDT that formalizes design knowledge based on axiomatic set theory is reviewed. Second, this theoretical result is tested against experimental work on design processes. Although in principle the theoretical results agree with the experimental findings, some problems can be pointed out. From these problems a new design process model, called the refinement model, is established, which has better agreement with the experimental findings. This model implies three guiding principles in developing a future CAD system. One is that future CAD requires a mechanism for physics-centered modeling and multiple model management. Second, a mechanism for function modeling is also required, and the FBS (function-behavior-state) modeling is proposed. Third, intention modeling is also proposed for recording decision-making processes in design. These advanced modeling techniques enable creative, innovative designs. As an example, the design of self-maintenance machines is illustrated. This design example utilizes design knowledge intensively on a knowledge-intensive CAD. This is a new way of engineering and can be called knowledge-intensive engineering. The design of self-maintenance machines is, therefore, an example of knowledge-intensive design of knowledge-intensive products, which demonstrates the power of the design methodology derived from the GDT.

One of the products of engineering, besides constructed artifacts, is design documentation. To understand how design participants use documentation, designers and typical documentation users were interviewed and protocols were taken of them both creating and using design documentation. The protocols were taken from realistic projects of preliminary design for heating, ventilation, and air conditioning systems (HVAC). The studies of document creation and use revealed three important issues: (1) design participants not only look up design facts; they frequently access documents to obtain information about the rationale for design decisions; (2) the design rationale that they see often is missing from the documents; and (3) design requirements change frequently over a project life cycle so that design documents are often inconsistent and out of date. Recognizing these documentation issues in design practice, a new approach was developed in which documents are no longer static records, but rather interactive design models supporting a case. The feasibility of the approach was demonstrated by constructing a running system and testing it with designers on realistic problems. The costs and benefits of creating and using documentation of design rationale also were analyzed. In particular, the active documents approach was evaluated for a routine, preliminary design in domains where community practice is widely shared and largely standardized. The approach depends on the feasibility of creating a parametric design model for the design domain.

This annotated bibliography includes a small sample of sources on various aspects of research methodology from diverse disciplines that influence research on artificial intelligence techniques in engineering design analysis and manufacturing (AIEDAM). Some of these sources are extended edited volumes containing many relevant contributions and pointing to additional references. These volumes are marked by a preceding bullet (•). The bibliography is not comprehensive; it covers only several important subjects, and in each subject it lists several representative contributions ordered chronologically.