Rabbi Eliezer said to Ben Azzai: ‘Such words sound well when they issue from the mouths of those who practice them. There are some who preach well and practice well; others practice well but do not preach well. You preach well but do not practice well’. Ben Azzai replied, ‘But what shall I do seeing that my soul yearns for Torah. The world can continue through others’.
Babylonian Talmud, Tractate Yebamot, page 63b
1 Introduction
There are no statistics available on the success or failure of research projects to meet their stated objectives. This seems not to even be a major issue for the research community or funding agencies.Footnote 1 Nevertheless, following more than 35 years of research experience, my estimation is that the success rate of research meeting its stated goals is low.Footnote 2 Another related issue that has received attention in general and in the design research community is the poor transfer of results from research to practice. This outcome is an instance of the theory-practice problem (Reich Reference Reich1992).
Improving the fate of research projects is definitely a desirable goal and improving the rate of research results that are used in practice is similarly sought. I propose a principle which I call – Principle of Reflexive Practice (PRP) – that if followed, could help in this direction, and present some evidence to support it. The principle relies on the idiom: ‘Practice what you preach’, which is a phrase occasionally used to refer to those not acting as they want others to act. There are countless opportunities in professional work and daily life to bring such criticism upon ourselves. In particular, disciplines such as engineering, public policy, psychology, sociology, or education, that deal with human activities such as design, decision-making, learning, research, or knowledge management present a challenge to their professionals. As these professionals develop principles, knowledge, methods or tools aimed at assisting practitioners in these disciplines, they are also practitioners that at the same time can use their own research results. Such use of results would be called reflexive practice.
For example, design researchers develop methods for selection between alternatives or creating robust designs and can use them for selecting between different research approaches or designing a robust research project. Similarly, researchers in public policy could develop methods for decision-making under uncertainty and use these methods to prioritize funding research projects. Finally, researchers in psychology could develop a new approach to couple or relationship therapy and use it if they encounter issues in their own relationships, or clinical psychologists that treat people (not develop methods or tools), are expected to go to their own treatment or supervision sessions to work on issues that arise in their own practice. Note that these examples refer not only to ‘researchers’, a term used subsequently throughout the paper, but also to any professional who use something that could apply to him or her.Footnote 3
But reflexive practice or the PRP also applies to professionals trying to understand design better, and not only to those developing tools or methods; for example, they apply also to practitioners studying conditions under which professionals operate better than others. An example would be studying which environments foster creativity (Amabile et al. Reference Amabile, Conti, Coon, Lazenby and Herron1996; McCoy & Evans Reference McCoy and Evans2002). The PRP challenges researchers trying to answer this question to use their hypotheses in their own environments to make their work more creative even before they test it elsewhere. The PRP claims that such use would help refine the hypotheses by serving as an initial pretest. It goes without saying that these researchers are expected to use their findings in their environments if they wish their team to be more creative.
To summarize, the PRP might be relevant to any study that has some prescriptive aspect. The PRP will not be relevant to purely descriptive studies whose sole goal is, for example, to understand how designers work. Such studies may raise interesting and controversial questions, for example, what is, and how does one determine, the value of such studies to design; but these topics are outside the scope of this study. To illustrate studies that the PRP would not apply even though they are prescriptive by virtue of their development of methods, consider: design optimization, automated synthesis, or specific disciplinary analysis tools such as finite elements; it is difficult to see how one uses these tools on his own practice.
With these clarifications about the PRP, and to simplify the text, I will use tools or methods in this paper but keep in mind the broader applicability of the PRP to research results in general.
In most cases, the aforementioned challenge is not met or not acknowledged; consider for example, two recent personal accounts of lifelong careers in design that do not mention this idea (Andreasen Reference Andreasen2011; Birkhofer Reference Birkhofer2011). Most researchers in the aforementioned fields do not engage in this type of reflexive practice. Researchers might view the use of these methods as extra work they cannot affordFootnote 4 or might view the use of structured methods in general, as antithetical to their academic freedom or to creativity. If pressed to use their own methods, or more broadly, the methods of their profession, researchers might even justify their reluctance by using Kuhn’s (Reference Kuhn1962) view of scientific revolution (see e.g., Gordon & Raffensperger Reference Gordon and Raffensperger1969); i.e., when normal science within a discipline’s paradigm get stuck in anomalies or contradictions, new ideas are required to cause revolutionary developments. Structuring scientific activities might hinder researchers chance to make a revolutionary and significant impact.
There are other types of reflexivity that are discussed in various disciplines such as sociology, management, education, and psychology. In fact, each discipline has its own variety of interpretations that often run parallel without crossing disciplinary boundaries (Woolgar Reference Woolgar1988; Ashmore Reference Ashmore1989; Holland Reference Holland1999; Lynch Reference Lynch2000; Pels Reference Pels2000; Foley Reference Foley2002; Johnson & Duberley Reference Johnson and Duberley2003; Cunliffe & Sun Reference Cunliffe and Sun2005). These interpretations include synonyms for reflection, methodological tools to improve objectivity, tools to improve the ethical status of decisions, tools for self-improvement, as well as an inherent property of any action. This variety of interpretations is not surprising. In fact, we should probably acknowledge that for any concept there are multiple competing interpretations that have value in different contexts. What is interesting is that in spite of this variety, all existing interpretations involve the purpose of making sense of phenomena or artifacts. They are interpretive. In a term known in the design literature, these interpretations of reflexivity serve to describe situations – they are descriptive. In design, Glanville (Reference Glanville, Jacques and Powell1981, Reference Glanville1999) is perhaps the first and maybe only former researcher that argued that science is a design discipline and hence design practices apply to science or research. But even he did not ascribe a meaning of reflexivity to a prescription in a way put forward in this study. Otherwise, Mead (Reference Mead, von Foerster, White, Peterson and Russell1968) was probably the first who observed reflexive practice when she proposed to use cybernetics ideas to develop the new society for studying cybernetics. Her idea was rejected once as being silly, but she proposed it again to the American Society for Cybernetics. The present paper is an elaboration of the idea that was first presented in (Reich Reference Reich1992). At that time, it could not have been presented as a principle with sufficient evidence of its practice so its publication would have been a violation of itself (i.e., I advocate to use the principle but do not use it myself), but now, with four case studies that used it, I can demonstrate that I am using it in my own research hence being consistent with the PRP.
In this paper, following a short description of the methodology of this paper (Section 2), I elaborate on the basis of the PRP in design research methodology (Section 3), and present the principle in Section 4 with its related hypotheses. Section 5 presents some examples of research that embraced the principle and Section 6 concludes with the future of the PRP. The discussion on the relation between the PRP and contemporary product design practices and the potential impact of the PRP on research practice are left to a separate paper. Similarly, a more in-depth analysis of the impact of different interpretations of reflexivity in design could be explored in a separate study.
2 Methodology of this research
This paper is conceptual and exploratory; it describes an idea developed in the context of numerous studies on research methodology and design theory over an extended time period. As such, the research behind the PRP could be classified as a long-term multi-case study. Case study research method (Yin Reference Yin1994) is a well-accepted approach that leads to valid knowledge without the need (often because it is impossible) to test hypotheses with statistical tests in controlled experiments.
To put the concept of the paper in perspective of relevant contemporary design research, the context is established by reviewing papers from design journals such as Research in Engineering Design, Design Studies, Journal of Engineering Design, and Journal of Mechanical Design. Additional references were collected by searching in databases such as ISI, Scirus, and scholar.google for the keywords: ‘reflection’, ‘reflexive’, ‘reflexivity design’, ‘design research validation’, ‘designing design methods’, and ‘designing design research’. Publications that were found, were examined and if found indeed relevant to the three topics mentioned in Section 3, their references were also inspected, as well as newer papers that cite them. This created a growing network of publications from diverse disciplines that were not found in the direct keyword search. These studies, summarized in Section 3, provide the state of the art thinking on design research methodology relevant to the PRP. They also establish that the PRP has not been discussed before as a research hypothesis with associated evidence.
The proposal of the principle is not new. Initially proposed in 1992 (Reich Reference Reich1992), it served as guidance for some previous research projects and mentioned by other scholars. However, in Section 4, it is the first time it is presented as a central idea, as a hypothesis that could be supported by evidence. Four cases that are described in Section 5 provide some supportive evidence to the hypotheses that are stated explicitly in Section 4. Other potential benefits are left for future detailed studies.
If the PRP is part of research methodology, a legitimate question is asking whether this study uses the PRP it is advocating. I am raising this question to give readers a clearer idea of what I am hinting at when I propose this principle. The answer is positive. This paper has been in a working state for over 20 years. It is now at a state where I am using the PRP on any research project to offer guidance. Examples are provided later in the paper as a way of sharing my experience using it.
3 Review of design research methodology related to the PRP
The state of the art of design research and its future progress has always occupied design researchers (Alexander Reference Alexander1971; Rittel Reference Rittel1973; Archer Reference Archer1981; Cross Reference Cross1984; Reid et al. Reference Reid, Cohen, Garrett, Rabins, Richardson and Winer1984; Hubka & Eder Reference Hubka and Eder1987; Dixon Reference Dixon1988; Finger & Dixon Reference Finger and Dixon1989a ,Reference Finger and Dixon b ; Arciszewski Reference Arciszewski1990; Hundal Reference Hundal1990; Roozenburg & Cross Reference Roozenburg and Cross1991; Ullman Reference Ullman1991; Konda et al. Reference Konda, Monarch, Sargent and Subrahmanian1992; Cross Reference Cross1993; Hubka & Eder Reference Hubka and Eder1996; Bayazit Reference Bayazit2004; Horvath Reference Horvath2004; McMahon Reference McMahon2014). Studies on the subject had historical focus (e.g., Rittel Reference Rittel1973; Cross Reference Cross1993; Bayazit Reference Bayazit2004); they defined and attempted to resolve confusions between terminologies such as science, design and design methodology, classified design research, and dealt with a variety of other related issues (e.g., Alexander Reference Alexander1971; Spillers Reference Spillers1977; Finger & Dixon Reference Finger and Dixon1989a ,Reference Finger and Dixon b ; Ullman Reference Ullman1991; Roozenburg & Cross Reference Roozenburg and Cross1991; Konda et al. Reference Konda, Monarch, Sargent and Subrahmanian1992; Cross Reference Cross1993; Cantamessa Reference Cantamessa2003; Friedman Reference Friedman2003). They also sought to determine future directions for design research and design research methodology (e.g., Antonsson Reference Antonsson1987; Dixon Reference Dixon1988; Konda et al. Reference Konda, Monarch, Sargent and Subrahmanian1992; Cross Reference Cross1993; Reich Reference Reich1995a , Reference Reich2010, Reference Reich2013; Friedman Reference Friedman2003; Reich & Subrahmanian Reference Reich and Subrahmanian2013; McMahon Reference McMahon2014).
A particular subject that enjoys recent growing interest is the methodology that guides design research (Konda et al. Reference Konda, Monarch, Sargent and Subrahmanian1992; Reich Reference Reich1994a ,Reference Reich b ,Reference Reich c , Reference Reich1995a ,Reference Reich, Tomiyama and Mantyla b ; Eder Reference Eder1998; Frost Reference Frost1999; Pedersen et al. Reference Pedersen, Emblemsvag, Bailey, Allen and Mistree2000; Friedman Reference Friedman2003; Olewnik & Lewis Reference Olewnik and Lewis2005; Blessing & Chakrabarti Reference Blessing and Chakrabarti2009). There are three topics in research methodology, from specific to more general, that are related to the PRP:
-
1. Validation of design methods in practice.
-
2. The reflexive use of design methods in designing design research.
-
3. The broad perspective of research methodology as an overall guidance for design research.
3.1 Validation of design methods in practice
Within the broad area of research methodology, special attention has been given to study the validation of design methods (e.g., Pedersen et al. Reference Pedersen, Emblemsvag, Bailey, Allen and Mistree2000; Shah, Kulkarni & Vargas-Hernandez Reference Shah, Kulkarni and Vargas-Hernandez2000; Hazelrigg Reference Hazelrigg2003; Malak & Paredis Reference Malak and Paredis2004; Olewnik & Lewis Reference Olewnik and Lewis2005; Frey & Dym Reference Frey and Dym2006; Le Dain, Blanco & Summers Reference Le Dain, Blanco and Summers2013) and the validation of design support tools (e.g., Rzevski, Woolman & Trafford Reference Rzevski, Woolman and Trafford1980; Reich Reference Reich1994a ; Reich & Barai Reference Reich and Barai1999; Opiyo, Horváth & Vergeest Reference Opiyo, Horváth and Vergeest2002). These studies draw upon the nature of design methods and tools, other disciplines (e.g., medical research, information systems development, and social science), as well as practical experience in transforming design methods successfully to practice.
The long-term goal of these validation techniques is to improve the use of design methods in practice. In general however, there is only a small fraction of research results in design that are adopted by industry. The mismatch between practice and research also goes in the opposite way. There are tools that are used in industry, such as quality function deployment (QFD), Pugh’s controlled convergence, analytic hierarchy process (AHP), or robust design that might be criticized by some researchers and debated with their proponents; for example, the debate over the mathematical precision of QFD or Pugh controlled convergence and its perceived undesired subjective nature (e.g., Bouchereau & Rowlands Reference Bouchereau and Rowlands1999; Frey et al. Reference Frey, Herder, Wijnia, Subramanian, Katsikopoulos and Clausing2009, Reference Frey, Herder, Wijnia, Katsikopoulos, de Neufville, Oye, Subrahmanian and Clausing2010; Hazelrigg Reference Hazelrigg2010);Footnote 5 the lack of mathematical optimality and unwarranted assumptions of robust design methods (e.g., Nair et al. Reference Nair, Abraham, MacKay, Nelder, Box, Phadke, Kacker, Sacks, Welch, Lorenzen, Shoemaker, Tsui, Lucas, Taguchi, Myers, Vining and Wu1992); or the proposal of new mathematical foundation for AHP to address some of its behaviors (e.g., Salo & Hämäläinen Reference Salo and Hämäläinen1997; Triantaphyllou Reference Triantaphyllou2001).
What seems to be the guiding principle in design is pragmatism and not necessarily mathematical aesthetics or correctness (Reich Reference Reich2010). That is, QFD (Cristiano, Liker & White Reference Cristiano, Liker and White2001; Govers Reference Govers2001; Chan & Wu Reference Chan and Wu2002), selection methods (Dym, Wood & Scott Reference Dym, Wood and Scott2002; Frey et al. Reference Frey, Herder, Wijnia, Katsikopoulos, de Neufville, Oye, Subrahmanian and Clausing2010; Reich Reference Reich2010), and robust design or more generally statistical methods (Box & Liu Reference Box and Liu1999a ,Reference Box and Liu b ) are used by practitioners when they derive practical value out of them. Similarly, AHP is used to derive practical value in addition to being resistant to various changes that do not match its underlying foundation (Saaty Reference Saaty1997). To make this position more extreme, designers will also use incorrect methods if they lead to practical benefits (Subrahmanian et al. Reference Subrahmanian, Konda, Levy, Reich, Westerberg and Monarch1993).Footnote 6 By analogy, researchers may use the PRP not because it is logical but because it is practical. The burden of demonstrating its practical usefulness is thus central.
Frost (Reference Frost1999) stated that for any method developed in design science to be practiced in industry, its pragmatic value needs to be clearly demonstrated. If this goal is taken seriously, then as in any other artifact that needs to be designed properly to satisfy its goals, so do design methods or tools are required to be designed to satisfy their goals – to be valuable to their intended users. The next section addresses this topic.Footnote 7
3.2 Designing design research
Practitioners in any field use reflection-in-action (Schön Reference Schön1983). Reflection-in-action means thinking about the activity while it is performed in order to stir it toward a desired direction. It is critical to exercise it given that the world is changing constantly and even without it, our understanding of problems coevolves with their solutions, hence requiring rethinking the process. Considering researchers, reflection-in-action would be a necessary skill to become successful. Reflection-on-action is different from reflection-in-action; it means the retrospective analysis of an activity in order to extract knowledge for future improvements. There is ample research on these activities. There is even ample research on reflexivity. It is easy to find papers demonstrating that reflexive practice, e.g., by teams, leads to more effective and efficient performance in new product development projects (Hoegl & Parboteeah Reference Hoegl and Parboteeah2006), but really, the meaning of reflexive practice discussed in this paper is just like reflective practice in another paper with similar results (Zika-Viktorsson & Ingelgård Reference Zika-Viktorsson and Ingelgård2006). Given the confusion in the terms reflection and reflexivity mentioned in the introduction with many references, it is clear that reflexivity needs to be defined precisely. For now, let us define reflexivity in design as the application of design X for the development of design X, where X could be method, tool, principle, or knowledge. This will be the ‘strong’ version. A ‘softer’ version would be: the application of design X for the development of design Y, where X or Y take the former interpretations. Clearly this interpretation of reflexivity is different from reflection.
The reflexive nature of design really forces us to look at research projects or proposals as products to be designed (Glanville Reference Glanville, Jacques and Powell1981, Reference Glanville and Robertson1998).Footnote 8 In the same vein, there are guidebooks for young researchers advocating for designing research projects and there are examples of researchers designing design methods. For example, Rzevski (Reference Rzevski, Jacques and Powell1981) described the design of an evolutionary design method; however, his report really described the method and its assumptions rather than its design process. While this careful articulated report is very important as it allows readers to appreciate the scope of the method, it nonetheless is not a record of designing. In another example, a design rationale (DR) method based on QFD is designed by QFD tools (Reich Reference Reich2000). However, the latter design was not used in real practice; Section 4.1 elaborates on this example. Also, Teegavarapu (Reference Teegavarapu2009) argued for the use of design methods in designing design methods and attempted to use them to develop a new selection method, but the actual demonstration has numerous difficulties.
Finally, in relation to designing science policies, Gordon & Raffensperger (Reference Gordon and Raffensperger1969) described the use of structured tools to provide support for policy making as well as personal prioritization of basic research. In order to develop their method, they tested three other tools in astronomy research policy making and came to the conclusion that another was necessary. A relevance tree they created (Churchman, Ackoff & Arnoff Reference Churchman, Ackoff and Arnoff1957) defined a breakdown of topics in astronomy in a way they termed object-oriented approach. This method was rejected due to its subjective nature. The second tool was a Delphi study aimed at creating a consensus among astronomers; it was not completed since the study team was not prepared to mediate between different scientists. The third method was a variant of a morphological approach (Zwicky Reference Zwicky1962); it did not work because it did not contrast the needs with the means in a clear way. The fourth tool is the authors own development, called cross impact analysis, which employs a needs tree and a means tree with their interrelations that proved useful in several disciplines for several important purposes including:
-
1. making explicit the contribution of individual research to the objectives of the discipline;
-
2. displaying the assumptions, models, and theories in the discipline;
-
3. providing a basis for assessing priorities with minimal bias; and
-
4. discovering unexplored yet potentially important research directions.
This cross impact analysis tool continues to enjoy development and use until today mainly in technology forecasting or long-term planning (e.g., Schlange & Jüttner Reference Schlange and Jüttner1997). However, the paper on using it to design basic research programs has been cited less than a handful, reinforcing implicitly, what the authors say about the reluctance of scientists to adopt structured methods in their own practice.
3.3 Providing overall guidance to design research practice
If we agree that design projects need to be designed, then, since we know that in design, the requirements and the design evolve simultaneously (Nidamarthi, Chakrabarti & Bligh Reference Nidamarthi, Chakrabarti and Bligh1997; Dorst & Cross Reference Dorst and Cross2001; Braha & Reich Reference Braha and Reich2003; Maher & Tang Reference Maher and Tang2003), we have to agree that as we design the research, the requirements and the research itself change. We also acknowledge that design processes for such real situations are not fixed and need to be designed as well (Guindon Reference Guindon1990; Whitney Reference Whitney1990; Eppinger et al. Reference Eppinger, Whitney, Smith and Gebbala1994; Westerberg et al. Reference Westerberg, Subrahmanian, Reich and Konda1997; Braha & Reich Reference Braha and Reich2003; Karniel & Reich Reference Karniel and Reich2011). In fact, such changes are fundamental to our daily and professional life (Schön Reference Schön1973) and managing them is central to engineering (Eckert, Clarkson & Zanker Reference Eckert, Clarkson and Zanker2004; Subrahmanian et al. Reference Subrahmanian, Lee and Granger2015; Jarratt et al. Reference Jarratt, Eckert, Caldwell and Clarkson2011). In order to be in control of this situation, this reflexive situation requires constant reflection. Moreover, if design is a mutual learning process between users and designers (Béguin Reference Béguin2003) and if to fully exploit this learning, design processes should continue after a product is delivered to its users, then it is better to be one’s own user to bootstrap this learning.
As researchers studying contemporary product design and trying to improve it, we are fully aware of these arguments. Nevertheless, there is a bifurcation between our practice of design research and our observation of the object of our study. We rarely consider the outcome of our research as a regular product and we are not eager to use these products ourselves. This split, whose origin is as old as design research and is an instance of the theory-practice problem (Reich Reference Reich1992) in many disciplines and in philosophy, leads to research results being mostly unused in practice.
In contrast, I propose to design researchers that we ‘practice what we preach (Babylonian Talmud)’. This is more than designing our own research but taking the whole enterprise of design research and applying it to our own practice. The benefits from this practice include guaranteeing that our ‘preach’ to others is truly valuable and not harmful; providing cheap testing and generating valuable feedback that is otherwise difficult to extract from users; and prioritizing our personal and community efforts to valuable problems. If we do not practice it, we might be developing the wrong method for even the right problem or the ‘right’ method for the wrong problem.
4 The principle of reflexive practice for design research
The PRP is quite simple; it arises from the observation that design research is just a product; it might have special characteristics or needs but it shares many attributes with other types of artifacts. Therefore, design research is subjected to the same environmental changes and influences on product design practice that design researchers study. Consequently, many tools, methods, principles or knowledge that design researchers develop for designers in the hope that the latter improve and excel in their practice should be useful to researchers’ own practice.
The PRP could be rephrased in two parts as:
-
I. Moral obligation: Designers use design methods to create products that satisfy the products’ goals. Similarly, design researchers should use design methods to create research projects and products that satisfy their goals. This part follows the moral reason of ‘practice what you preach’.
-
II. Practical benefit: The application of design methods in design research bootstraps the successful development and validation of design methods in two ways:(1) ‘strong’ form – when researchers use their own tools on their own research they will obtain early and cheap feedback, and (2) ‘weak’ form – when researchers use design knowledge, principles, tools, or methods to design their research, their results would benefit from it just like product development in general benefits from such use. Whichever form is used, the PRP would lead to improving the success rates of projects.
Part I does not require further elaboration but part II is a hypothesis that requires supporting evidence. Before I present some evidence in Section 5, let us elaborate more on the principle.
Since research is a product and research activity involves design and implementation together, there would be ample opportunities to exercise design methods or tools. In using the word ‘opportunity’, the principle leaves room for judgment about a method or tool usefulness for a particular situation as would be in design practice.
The PRP could also be illustrated through different roles that researchers may assume in knowledge generation. In the traditional role of researchers, they develop meansFootnote 9 that are considered for adoption by practitioners (thick red line in Figure 1). Practitioners in turn, provide feedback on the means that are then considered for inclusion into the new means. In the reflexive role, researchers create means, adopt them for their own practice, and subsequently, as users, they provide feedback on the means (dotted blue line in Figure 1).
Figure 1. Research practice is similar to other design practices.
Table 1 presents Figure 1 with more details and with a graded move from the traditional to the reflexive role manifested in four different roles that design researchers can play in the research and development process of knowledge and products:
-
I. In ‘basic research’, researchers could develop some knowledge, basic understanding about design or even methods (e.g., finite elements methods or knowledge of cognitive fixation). These are taken by tool or method (referred to generally as means) developers to turn them into commercial products (or means in general) that could be used by practitioners that, in turn, use them to produce quality products for customers.Footnote 10
-
II. In ‘applied research’, researchers develop basic understanding and tools (e.g., concept generation tools) that they try to transfer into industry. Once designers agree to use the tools, they use them to design products that consumer buy.
-
III. Researchers in the role of designers design products for end users; for example, they could develop DR methods and tools whose customers are designers. In this scenario, while designing the tools, researchers can use their own tools to record their decision-making rationale. This is also an example of the fourth role.
-
IV. As customers, researchers can develop tools for themselves; for example, configuration generation method and use it to configure their research programs. Practicing this role is clear to all experimental scientists in any discipline, who need to design their experimental apparatus. For example, in order to study particle physics, the Large Hadron Collider was developed through a major effort of thousands of scientists and engineers to become the largest single machine ever developed (CERN 2016). Clearly all design practices were relevant to support the needs of the researchers as end users.
Table 1. Perspectives on researchers’ role in product development
Moving from role I to IV requires researchers to become more active in making research more practical; as the mediators between them and potential users (e.g., tools developers and designers) disappear, the responsibility to generate usable and relevant tools becomes theirs. By denoting the first two roles as basic and applied research – the dominant and almost exclusive styles of contemporary research – it becomes clear that the use of principles such as the PRP is not necessary for many researchers although they may still find it useful. In addition, from the literature survey, the last two roles that represent reflexive research are seldom practiced.
4.1 Validation of design methods in practice
Bunge (Reference Bunge1967) discussed the use of two means to improve research results: internal and external consistency. Internal consistency means that the research hypotheses do not have contradictions. This could be ensured when they are created before any data is collected. External consistency means that the theory needs to be compatible with the observed data. Both means relate to the artifacts resulting from the research activity: the theory and the data. However these consistency measures are independent. The PRP adds a focus on the research process and suggests, as a hypothesis that refines part II, that if one adheres to the PRP, then ensuring better internal consistency of the theory may improve the chance that the product of one’s research would be more relevant to practice, or in other words, that internal consistency with respect to the PRP may lead to external consistency.
The difference between the PRP and the two previous consistency means is that in contrast to them, the PRP is not stated as a necessary or sufficient condition for quality research. There could be valid research results generated without adhering to the PRP similar to the ability to obtain quality product by using ad hoc design processes. When dealing with processes, the question becomes that of increasing the chances of completing research successfully, and minimizing the risk of failure and research cost and duration. This lack of ‘precision’ about the PRP cannot be overcome as there is no single correct design process in design practice (Reich, Kolberg & Levin Reference Reich, Kolberg and Levin2006) and the choice depends on diverse factors related to technology, and the social and institutional setting of the design (Subrahmanian et al. Reference Subrahmanian, Reich, Smulders and Meijer2011a ,Reference Subrahmanian, Reich, Smulders and Meijer b ; Meijer, Reich & Subrahmanian Reference Meijer, Reich, Subrahmanian, Duke and Kriz2014; Reich & Subrahmanian Reference Reich and Subrahmanian2015, Reference Reich and Subrahmanian2017). But the PRP is a powerful way to sort out the options and get feedback while advancing in this uncertain research terrain.
Interestingly, when the PRP is applied to companies developing commercial products (e.g., software), customers including design researchers would value a company that uses its own software to drive its business and would doubt the value of a tool if its developers do not use it. For example, if the enterprise resource planning (ERP) software development companies SAP or Oracle would not use their tools to manage their resources, it would create difficulty to their customers; or if Microsoft would not use its software products to develop its tools and drive its business, it would be difficult to convince customers to do so.Footnote 11 However, design researchers are not hard pressed to react similarly to their own reluctance to use their tools.
An example of a ‘strong’ form challenge driven by PRP could be described as follows (Reich & Shai Reference Reich and Shai2012):Footnote 12 C–K theory assumes the existence of two spaces: K describes knowledge whose logical status is known and C describes knowledge whose logical status is unknown. The interplay between them allows creating new concepts in C and through this supports creativity. The application of the PRP is as follows. If C–K theory is accepted as a design theory, then we may agree that its logical status is true. If so, it belongs in the K space. Consequently, there must be a C space corresponding to this K space, and we may now ask if we can use this C space to create new concept theories in C and develop them into new design theories in K or dismiss them? If this is impossible, C–K would not be reflexively consistent; however, if it is possible, it opens a path to develop new advanced design theories. Similar challenges could be formulated regarding other design topics such as robust design, change management, process planning with DSM, etc.
4.2 Designing design research
This aspect of the PRP is stated more prescriptively than before. Research in general and design research specifically is a complex endeavor. There are many stakeholders with many conflicting goals acting together in a quite turbulent environment. Executing research and leading it to successful results is nontrivial and prone to failures. For example, what would happen if we checked research proposals against the results they produce? or if we checked the long-term contribution of design research against its goals as represented in strategic public records (e.g., in the US: Reid et al. Reference Reid, Cohen, Garrett, Rabins, Richardson and Winer1984; Committee 1991; Shah & Hazelrigg Reference Shah and Hazelrigg1996)? would we find high success rate? I anticipate that the answer is negative. We do not expect to succeed in product development without careful attention to all issues, management of the process as it unfolds, and using a variety of methods and tools. Similarly, we should not expect to have high success rate when we do not practice the same in research which is more risky than ordinary product development. The only way to succeed in product development and stay competitive is through engineering design (e.g., Committee 1991; Shah & Hazelrigg Reference Shah and Hazelrigg1996); the same holds for design research.
4.3 Providing overall guidance to design research practice
Beyond serving as a new consistency measure and a recommendation to design design research, the PRP provides overall guidance to research practice. Equating research with other products really opens up the way for all product development practices and ideas to serve design research. For example, in today’s turbulent environment, survival and more over flourishing means the ability to quickly identify trends and adapt to exploit them before the competitors. Consequently, many organizations are improving their agility and moving toward becoming learning organizations. Many organizations achieve these goals by selling off their secondary activities, only maintaining and strengthening their core competencies. Other organizations grow and expand to become integrators or complete solution providers.Footnote 13
What do these practices mean for design research? What does it mean to become agile design researchers? What does it mean to develop ‘product architecture’ in research or support ‘sustainability’? There is clear benefit to interpreting the PRP even in this way as the practice of the
$n$
-dim project suggests (see next section), but the benefits are much more and beyond the scope of this paper.
In January 2006, IBM was awarded a ‘KM reality’ prize by KMworld Magazine (KMWorld 2006). The award was given not for IBM knowledge management products but for the way it used them itself to manage its activities and bootstrap its ability to derive value for its customers. It is precisely the use of the PRP principle to guide its own practice. By its significant size, IBM can derive significant value for its customers by developing tools that would help it to excel. If IBM can deploy tools that intervene in the work practices of its employees, achieve acceptance, and derive true value from them, so might other enterprises, provided they have the resources to deploy these tools.Footnote 14 This example is particularly relevant to design because similar to knowledge management practices, and in contrast to analysis methods and tools, design methods developed in research intervene at the heart of engineering work and are therefore hard to get adopted by industry (Reich Reference Reich1994a ). The in-house deployment of its KM tools as they evolved through development allowed IBM fast feedback on their usefulness as well as developing deployment procedures that would have to be developed if one would wish to deploy design tools. Using the PRP in its ‘strong’ form may provide guidance to research that improves the research results acceptance rate.
5 Examples
The use of the PRP in guiding research has been discussed in various forms or used before (Reich Reference Reich1992, Reference Reich1994a ; Reich et al. Reference Reich, Konda, Levy, Monarch and Subrahmanian1996, Reference Reich, Subrahmanian, Cunningham, Dutoit, Konda, Patrick and Westerberg1999). The fundamental question that every design researcher should ask is ‘am I using design tools to drive my research’ and more importantly, ‘am I using the method I am developing myself?’ This section provides four examples that correspond to the three topics discussed in Sections 3 and 4. For each of the examples I provide the following information:
-
1. Which form of PRP is used (‘strong’ or ‘weak’), and which role in table 1 I assumed in this study?
-
2. How was PRP used? E.g., which design methods, principles, or knowledge were used?
-
3. What were the benefits of using the PRP?
This information demonstrates the PRP use and potential benefits in design research.
5.1 Design rationale (DR) capture: Validation of design methods in practice
DR research attempts to create methods that would be used by designers to record their rationale as they design products. The benefits of such recording are significant yet only one research project result known to this author, DRed, is used in practice (Bracewell et al. Reference Bracewell, Wallace, Moss and Knott2009). When design researchers explain their DR methods in scholarly publications, a clear divergence is apparent between their proposed methods which are often graphic (e.g., gIBIS) and the way they explain their methods in linear text.
In a former paper I argued that to be consistent, any DR proposal should be explained and justified in its own terms to deliver its experience as close to reality as possible (Reich Reference Reich2000). In that paper, I used QFD tools to develop a method, based on QFD, for DR capture. To be consistent, I captured the design in QFD and presented parts of it in the text (following the ‘strong’ form and role III in table 1, rather than adopting the classic research roles I or II). This capture demonstrated clearly the difficulties in using such methods and provided me feedback before I attempted to claim that this method was ready for practical use.
The paper illustrated that the rationale of the method could be captured in the proposed approach. Had I not gone through the exercise or used a simplified design problem; it could have been easy not to become aware of its difficulties. In contrast, by using the proposed method in the research practice itself, it became clear that there are many issues that must be addressed before it can be proposed to practitioners. In addition, besides helping in the design, the use of the methods provided valuable insight regarding their further development. Altogether, given the issues raised, I decided to stop this research project.
5.2 Designing design curriculum: designing design research
Designing curriculum is not an uncommon concept. In education in general, it has been advanced by different system thinkers (Banathy Reference Banathy1991; Diamond Reference Diamond1998; Clark Reference Clark2002). This design is supposed to be carried out by following an orderly process. In design education the idea was stated as a conclusion of the 2nd Mudd Workshop (Dym, Sheppard & Wesner Reference Dym, Sheppard and Wesner2001) and later, Dym (Reference Dym2004) suggested that the answers to what to improve in engineering education and how ‘could be greatly improved if some basic percepts from design theory and from systems analysis are brought into play as answers are sought (p. 308)’. A major demonstration of using structured methods to develop engineering programs and courses has been realized through the CDIO initiative and organization (Crawley et al. Reference Crawley, Malmqvist, Östlund and Brodeur2007).
In 2001, following reflection on experience teaching mechatronic courses in high schools (an example of reflection-on-action); we observed that a significant impediment to improving the robots developed by students in these courses was the lack of structured design method used by the students. Following the PRP; we decided to use design methods to design such a design method for a high school mechatronics course (Kolberg, Reich & Levin Reference Kolberg, Reich and Levin2003, Reference Kolberg, Reich and Levin2005; Reich et al. Reference Reich, Kolberg and Levin2006; Kolberg, Reich & Levin Reference Kolberg, Reich and Levin2014). Obviously, in order to have students use such design method we had to design its associated curriculum. We refer to the method and its curriculum as the course. We ignore in this analysis the multidisciplinary knowledge the students had to learn such as mechanics, electronics, software, etc. as teaching such knowledge was easy and was already exercised well in previous courses. Figure 2 depicts the process we used to design the course. It consists of two main steps: the design of the course and its implementation. The course design was subdivided into four steps:
-
1. Requirements collection and analysis: The requirements were formulated from studying designers but also from anticipating the future needs in future design environments. Design techniques that were used included: task analysis, idea generation techniques such as brainstorming, QFD, and failure analysis of products created in previous years’ projects.
-
2. Goals setting: The requirements or needs of future designers were translated into course goals and learning activities that could support them. Supporting design techniques for this step included: QFD and influence graphs (Reich & Kapeliuk Reference Reich and Kapeliuk2005).
-
3. Design of the design method: The course goals were matched with specific design methods to address them. Supporting design techniques included: QFD, function-means trees (Hubka & Eder Reference Hubka and Eder1988) or graphs, AHP (Saaty Reference Saaty1980), influence graphs, Pugh controlled convergence (Pugh Reference Pugh1990) and failure mode and effect analysis (FMEA). We intended to use SOS (Ziv-Av & Reich Reference Ziv-Av and Reich2005) to help us configure the course but felt that for a single course it did not warrant the effort. In relation to SOS, we also considered it as a tool to teach students to help them configure robots but through tests we conducted ourselves, we realized it would be too difficult for students and dropped this method.
-
4. Means identification, selection, and generation: The course goals and the design method were matched with specific learning methods and activities and other means to address them.
Altogether, this design was interesting because, we used design methods to design a design method for the course that ended up including 6 other design methods. The teachers of the course taught these methods to students, who then used them to design a product. This follows role II in table 1, but from another perspective, through the use of our method SOS in our research practice, we helped develop SOS (following role IV), and in testing SOS for inclusion in the design method, we also exercised role III.
Figure 2. Being reflexive about designing contexts for learning design (Reich et al. Reference Reich, Kolberg and Levin2006; Kolberg et al. Reference Kolberg, Reich and Levin2014).
The design of the design method proved to be highly successful in a study involving 4 high schools over 3 years (Kolberg et al. Reference Kolberg, Reich and Levin2005, Reference Kolberg, Reich and Levin2014; Reich et al. Reference Reich, Kolberg and Levin2006). Students who studied the complete method, boys and even more so girls, compared to students who studied parts of it or none, demonstrated superior design abilities (including winning international robotics competitions), improved their science grades, and improved their perception of technology. The success of the robotics design method continued to be demonstrated since 2004 (Kolberg et al. Reference Kolberg, Reich and Levin2014).
In addition to supporting our careful course design, prototyping the use of one design method we developed, SOS, provided fast feedback that helped us improve SOS further. This demonstrated the value of using the PRP in this research for the SOS research project (‘strong’ form for the SOS project). In addition, this example provides a complete demonstration of the usefulness of the PRP in its ‘weak’ form: using carefully design methods to design a new design method composed of several tools was proved successful.
5.3
$n$
-dim: Providing overall guidance to design research practice
$n$
-dim was developed through in-depth theoretical conceptualization (e.g., Konda et al.
Reference Konda, Monarch, Sargent and Subrahmanian1992; Monarch et al.
Reference Monarch, Konda, Levy, Reich, Subrahmanian and Ulrich1996; Reich et al.
Reference Reich, Konda, Levy, Monarch and Subrahmanian1996, Reference Reich, Subrahmanian, Cunningham, Dutoit, Konda, Patrick and Westerberg1999; Subrahmanian et al.
Reference Subrahmanian, Reich, Konda, Dutoit, Cunningham, Patrick, Thomas and Westerberg1997, Reference Subrahmanian, Monarch, Konda, Granger, Collins, Milliken and Westerberg2003; Westerberg et al.
Reference Westerberg, Subrahmanian, Reich and Konda1997) and validated in numerous industrial studies (e.g., Finger, Subrahmanian & Gardner Reference Finger, Subrahmanian, Gardner and Roosenburg1993; Subrahmanian et al.
Reference Subrahmanian, Monarch, Konda, Granger, Collins, Milliken and Westerberg2003; Subrahmanian et al.
Reference Subrahmanian, Lee and Granger2015; Davis et al.
Reference Davis, Subrahmanian, Konda, Granger, Collins and Westerberg2001).
$n$
-dim was developed to support evolutionary design conducted by diverse multidisciplinary teams where requirements and design coevolve in response to dynamic situations but also to better understand the issues involved. Consequently, weFootnote
15
had to be flexible when conducting research projects with industry. The use of the PRP, following role IV in table 1, was that the same dynamics would occur when we develop our research tools and consequently, we have to develop the tools and their theoretical underpinning to support this flexibility. In addition, we had to bring diverse multidisciplinary people to support this research. This is an example of using the ‘strong’ form of the PRP that does not involve using design tools but design principles or insight.
Figure 3. The
$n$
-dim project philosophy: iterative study and development (Reich et al.
Reference Reich, Subrahmanian, Cunningham, Dutoit, Konda, Patrick and Westerberg1999).
To make this more concrete, Figure 3 shows the conceptual view of the
$n$
-dim project operation. At the foundation (1), there is a software infrastructure designed to address different design contexts and also designed to scale up to handle real applications. In line with the philosophy, this infrastructure was replaced several times by more advanced versions to allow addressing increasingly complex and different design contexts (Subrahmanian, Westerberg & Podnar Reference Subrahmanian, Westerberg, Podnar and Sriram1991; Levy et al.
Reference Levy, Subrahmanian, Konda, Coyne, Westerberg and Reich1993; Krogh et al.
Reference Krogh, Dutoit, Levy and Subrahmanian1996; Cunningham, Subrahmanian & Westerberg Reference Cunningham, Subrahmanian and Westerberg1997; Reich et al.
Reference Reich, Subrahmanian, Cunningham, Dutoit, Konda, Patrick and Westerberg1999). As additional applications were developed,
$n$
-dim would include repositories (2) of various blocks for building diverse applications. At the top level (3), our research and development followed the philosophical position and the theories we developed and evolved through empirical studies. These theories guided us in future studies and development projects, and were subject to constant reflection and potential revisions (4).
Any research project started as a collaboration with industrial or other partner(s). In order to support design and study it at the same time, we adopted participatory action research (PAR) as our development methodology (Reich et al.
Reference Reich, Konda, Levy, Monarch and Subrahmanian1996), an extension of the approach practiced within the group to develop
$n$
-dim. Together with our collaborators, we studied the present state of information management in the organization (5) (Davis et al.
Reference Davis, Subrahmanian, Konda, Granger, Collins and Westerberg2001; Subrahmanian et al.
Reference Subrahmanian, Lee, Thomas and Collins2006). The bottlenecks and their severity suggested priorities in setting goals for collaborative projects. We jointly defined the project goals (6). The development process (7) and the
$n$
-dim infrastructure (1) and reused the repositories of previous building blocks (2) for prototyping the application (8). This development, in turn, enriched (9) the repositories and the infrastructure. The application was deployed and tested by its end users (5) (Davis et al.
Reference Davis, Subrahmanian, Konda, Granger, Collins and Westerberg2001; Subrahmanian et al.
Reference Subrahmanian, Monarch, Konda, Granger, Collins, Milliken and Westerberg2003). This process iterated (10) until the goals, as understood at each iteration, were satisfied by the evolving application. During the evolution, parts of the system that became stable were re-written quickly in more efficient code. The collaborative project was studied and reflected upon continuously (11) to uncover potential improvements to all aspects of the methodology. Its results were used to refine our theories. During such projects, we also identified critical areas for basic research, prioritized and executed them.
We note that the core ideas have not changed much over the years. This was at least partly due to the use of the PRP. While we have not fully used
$n$
-dim to help us develop it, the principles we developed in the research were used by us to evolve it: equal participation of diverse perspectives, rapid prototyping, coevolution of problem and solution, understanding of the temporary nature of solutions until a better version appears, etc. Consequently, the
$n$
-dim approach approximates role IV. Our hypothesis was and still is that this reflexive process supports the development of design support systems in the best way we know (Subrahmanian et al.
Reference Subrahmanian, Reich, Konda, Dutoit, Cunningham, Patrick, Thomas and Westerberg1997; Reich et al.
Reference Reich, Subrahmanian, Cunningham, Dutoit, Konda, Patrick and Westerberg1999). Independent support for this claim can be found in the report of the development of DRed (Bracewell et al.
Reference Bracewell, Wallace, Moss and Knott2009) where they refer to the
$n$
-dim project for item (1) in Figure 3 but in fact, adopted most of the critical elements of the
$n$
-dim approach. This approach allowed the
$n$
-dim project to last close to two decades with considerable success in transferring knowledge and working tools to industry and generating a comprehensive body of knowledge related to collaborative work, development of support systems, and design theory; it equally helped the DRed project achieve its project goals.
5.4 PSI framework for characterizing design
The last example relates to a framework for describing designing situations that crystallized over many years of research that included the
$n$
-dim project: the PSI Framework. The framework locates any designing situation in 3 spaces simultaneously: the problem or product space (P), the social space (S) and the institutional space (I), where each space is further characterized with 3 dimensions shown in Figure 4. Simply described, the P space describes what is being designed, the S space describes who is participating in the designing act, and the I space describes how the design takes place and in which organizational context. Further details of these spaces and dimensions can be found elsewhere (Meijer et al.
Reference Meijer, Reich, Subrahmanian, Duke and Kriz2014; Reich & Subrahmanian Reference Reich and Subrahmanian2015, Reference Reich and Subrahmanian2017).
Figure 4. Locating designing of the PSI framework in the PSI framework (Reich & Subrahmanian Reference Reich and Subrahmanian2015).
Since the PSI framework is a product, an artificial conception, born out of designing, then that designing process of the PSI framework also could be described with the PSI framework; otherwise, it would not be reflexively consistent. It turns out that it is not difficult to characterize the designing of the PSI framework with the PSI framework as shown in Figure 4 (Reich & Subrahmanian Reference Reich and Subrahmanian2015). Figure 4(a) depicts the spaces, which comprise the PSI product in its current form, and Figure 4(b) depicts the development of this product as characterized by the 3 spaces.
For example, the framework is developed by researchers collaborating with practitioners in multiple disciplines reflected in the ‘perspective’ dimension in the social space. Similarly, as a scientific project, the knowledge access to all interested is open except proprietary data revealed in empirical studies with companies, reflected by the ‘knowledge access’ dimension of the institutional space.
An important aspect of PSI is to make sure that all the spaces are aligned, meaning that a complex product would probably require a complex social space makeup and carefully crafted institutional space position including rules of operating the development process. Failure to align the spaces is the cause of many failures in industry (Reich & Subrahmanian Reference Reich and Subrahmanian2015). The ‘strong’ form of the PRP challenges us (as role IV) to check whether the PSI spaces of the development of the PSI framework are aligned; it provides guidance for managing this project. Given the complexity of the topic, do we have the necessary capabilities or skills? Should we opt for an open project rather than a closed one? Should we collaborate while forming intimate relationships with other partners?
All these issues are carefully considered to make sure that as we develop the PSI further, the spaces are aligned, for example, by mobilizing additional perspectives including from social sciences and those from companies to make sure it is relevant to real practice. All the collaborators have to agree to the openness required in using the PRP. This means leaving ego aside and adopting a learner and collaborator position with respect to the research questions. Such openness means also the willingness to question even the PRP.
6 Discussion
The PRP suggests that given that research outcome and research itself are human-made artifacts, that everything related to design research could be subjected to design principles, knowledge, methods or tools, and that this is probably the most effective way to address it. This ‘weak’ form of the PRP: using design to design research does not need much defense as it should be clear to researchers and educators of design that design improves the quality of products. But in design, as in research, practice is critical and since the PRP intervenes in our own practice as researchers, it requires additional effort beyond our ad hoc research planning practices. It is therefore a challenge with a promise; not a free lunch. The question now is not whether the PRP is good in principle or whether it is a moral obligation to use but understanding when does the PRP work well, how should it be applied and how can we improve it further. All these questions require more data to create a form of best practices of using the PRP.
What I had demonstrated in the four cases is the following:
-
1. PRP provides early feedback in its ‘strong’ form – in the 1st case (DR capture), I stopped the research project given the anticipated issues and resources available; in the 3rd and 4th cases (
$n$
-dim and PSI), PRP helped develop the research ideas. In the 2nd case, it helped the SOS project by prototyping it on the course design. -
2. PRP in its strong form (3rd and 4th cases) and its weak form (2nd case) improved the success rate of research projects as I consider these 3 cases to be successful and further attribute this to the design knowledge and tools used in the research.
There is no way to directly compare the use of the PRP with other ways of doing research as no research project could be carried out twice and none works in a controlled environment. In order to better understand the value and effectiveness of the PRP, we need to rely on acceptable case study methodology and continue to collect more data in the future.
Is it always good to practice PRP? What could be bad in being reflexive? There does not seem to be a negative effect to practicing the PRP other than spending additional resources. Consequently, as in any other project, the selection of methods to use must be based on their cost effectiveness. If we consider the ‘strong’ form of the PRP then there are clearly research topics in engineering design that would not benefit from the PRP such automated synthesis or finite element analysis. I can hardly see how these methods could help in designing their own development, but would obviously be delighted to prove wrong here. But if we consider the ‘weak’ form of the PRP, then I argue that any research project would become better if it is designed properly.
It is not necessary to practice PRP in order to obtain good research results. For example, there is no indication that QFD was developed using QFD or other design tools. Also, there is no indication that decisions required in the development of AHP used an initial version of AHP or that the development of ANP used AHP or any other design tool.Footnote 16 But the same lack of necessity works for regular products also. There are many good products that were invented by chance and developed intuitively but there are many more who fail (Stevens & Burley Reference Stevens and Burley1997). I contend that while not necessary, a thoughtful use of the PRP can only be beneficial. Following the four examples in this paper, I anticipate that adopting the PRP will improve the state of design research.
This research requires more evidence to understand when the PRP works well, how it works and how can we improve it further. It is hoped that additional researchers will be able to relate their research to the principle and report their use and experience to build a body of best practice knowledge about it.
Acknowledgments
I would like to thank my lifelong colleague and friend Eswaran Subrahmanian for endless discussions over the years that contributed to the ideas in this paper and to the three anonymous reviewers whose thoughtful comments improved the paper.
Open access
