The evaluation of idea sets for design solutions using Shah et al.’s criteria of quality, quantity, novelty and variety can help design teams understand the thoroughness of their ideation work and can help design researchers compare the performance of different ideation methods. However, existing methods for aggregating these metrics to obtain total set scores for quality, quantity, novelty and variety are problematic. The present paper proposes axioms for the desired behavior of aggregation functions for quality, quantity, variety and novelty, then defines functions that meet the axioms. These axioms are intended to ensure that scoring methods reflect best practices in ideation and appropriately reward preferred ideation behavior, such as promoting the contribution of all ideas. Further, this paper provides operational definitions for quality, novelty and quantity evaluations of ideas and draws from previous methods to provide expedient scoring methods of individual ideas. Evaluation mechanics are presented that allow repeatable evaluation of idea sets containing thousands of ideas. Software tools are provided to automatically calculate the aggregation functions for ideas evaluated according to the mechanics of this paper. Finally, a method for evaluating both the variety of complete sets of ideas and the contributions of individual ideas to the overall set variety is proposed. The evaluation of variety is sufficiently defined that it can be automatically evaluated for any genealogy tree of ideas. The operational definitions for evaluating quality, novelty and quantity are suitable for adoption in artificial intelligence tools to allow automated evaluation of idea sets for these quantities.

Affective engineering is being increasingly used to describe a systematic approach to the analysis of consumer reactions to candidate designs. It has evolved from Kansei engineering, which has reported improvements in products such as cars, electronics, and food. The method includes a semantic differential experiment rating candidate designs against bipolar adjectives (e.g., attractive–not attractive, traditional–not traditional). The results are statistically analyzed to identify correlations between design features and consumer reactions to inform future product developments. A number of key challenges emerge from this process. Clearly, suitable designs must be available to cover all design possibilities. However, it is also paramount that the best adjectives are used to reflect the judgments that participants might want to make. The current adjective selection process is unsystematic, and could potentially miss key concepts. Poor adjective choices can result in problems such as misinterpretation of an experimental question, clustering of results around a particular response, and participants' confusion from unfamiliar adjectives that can be difficult to consider in the required context (e.g., is this wristwatch “oppressive”?). This paper describes an artificial intelligence supported process that ensures adjectives with appropriate levels of precision and recall are developed and presented to participants (and thus addressing problems above) in an affective engineering study in the context of branded consumer goods. We illustrate our description of the entire concept expansion and reduction process by means of an industrial case study in which participants were asked to evaluate different designs of packaging for a laundry product. The paper concludes by describing the important advantages that can be gained by the new approach in comparison with previous approaches to the selection of consumer focused adjectives.