1. Introduction
Learners, in design thinking (DT) courses, are tasked with documenting diverse information, which will subsequently undergo utilisation, management, and transformation in later activities of the process. Attempting to assimilate task comprehension and execution as well as documentation and management together can be tiresome and frustrating for students. Furthermore, assessing students’ comprehension and implementation of instructions throughout the design process is essential, along with assessing the final solutions/ outcomes. Without such assessment, while students may produce acceptable outcomes, a lack of understanding, ambiguity, or misguided assumptions about the instructions can result in an inadequate grasp of the design process. Therefore, there is a need to minimise complexity and facilitate an effective learning experience in DT. The work done in this paper aims to enhance documentation, management and assessment by developing documentation templates to support learners in the DT process.
The significance of design documentation in the design process can be attributed to several factors. Design documentation involves activities such as capturing, retrieving, compiling, and communicating design requirements (Reference Menning, Beyhl, Giese, Weinberg and NicolaiMenning et al., 2014). It serves as a reservoir, storing non-immediate information for future retrieval and therefore, it is a crucial component of both the design and handover processes (Reference Eris, Mabogunje, Jung, Leifer, Khandelwal, Hutterer, Hessling and NeeleyEris etal., 2005; Reference Menning, Beyhl, Giese, Weinberg and NicolaiMenning etal., 2014). In design education, documentation becomes a crucial element in the learning as well as the assessment process. An empirical study with pre-engineering students solving complex engineering design problems showed that individuals who devoted more time to documenting and revisiting their research and ideas tended to develop better-performing solutions (Reference Törlind and Wikberg-NilssonStrimel, 2014). It can also serve as an effective assessment tool, such as for assessing high school students’ DT skills, including human-centeredness, problem-solving, and collaboration (Reference Aflatoony, Wakkary and NeustaedterAflatoony et al., 2018).
The primary objectives of this paper are to assist learners in effectively documenting, transforming, and managing natural language-based information during the early stage of the DT process and to aid evaluators in ensuring traceability and capturing design rationale when assessing the learning of DT. To achieve this, the paper presents the development of templates that facilitate efficient documentation and assessment activities for the early stages of the DT process.
2. Information: its forms, transformation, management and assessment
DT process includes activities such as identifying problems, user needs and insights; formulating requirements; and generating innovative ideas and concepts. Furthermore, this information gets transformed from one form into another during the process. For instance, problems and user statements are transformed into needs, which are then translated into requirements, ideas are generated and combined into concepts to meet those requirements, etc. As noted by Gotel and Morris (Reference Gotel and Morris2009), requirements often originate from information content that undergoes various transformations and possesses different representations.
Along with this transformation, each description of information may contain associated information. The raw user statement, for example, can include associated information related to associated users, activities, or objects. The problem description might include more detailed information, such as its causes and consequences, that helps determine the problem's criticality. Similarly, a generated idea might be coupled with additional information, such as the source of inspiration. Figure 1 shows various types of information across the DT stages and activities. The stages and activities presented here are derived from the IISC DT model developed by Bhaumik et al. (Reference Bhaumik, Bhatt, Kumari, Menon and Chakrabarti2019).
Information flow and association across design thinking stages and activities
Finally, in DT, multiple activities necessitate the effective management of information. It includes adding new information, annotating existing information, performing calculations, classifying (e.g., needs), grouping and selecting (e.g., problems), ordering (e.g., requirement), modifying and combining (e.g., ideas), comparing, rating or ranking (e.g., concepts), removing irrelevant information, and conducting voting or decision-making processes. Operations such as comparing, combining, calculating, noting, annotating, ordering, classifying, and interpreting are also considered as some of the elementary operations used in the design process (Reference Hubka and EderHubka and Eder, 2002).
During design, the extracted information can be stored, processed, and communicated in various forms, such as natural language, mathematical models, charts, diagrams, schematics, sketches, projections and 3D models (Reference Rößler and GerickeSimon, 1996; Owen, Reference Owen2007; Auernhammer & Roth, Reference Auernhammer, Roth, Meinel and Leifer2023). These varieties of forms might be used to illustrate different aspects of design outcomes, manifested at various points in the design process, and vary in complexity and abstraction (Reference Zeng and GuZeng and Gu, 1999). Despite the design being related to real-world subjects and the final solution being a tangible product or system, natural language, as an abstract form of information, serves as the foundation for the design process, helping the more tangible aspects of the design to take shape. It would be a crucial component of future computer-aided design systems (Reference ZengZeng, 2008). As Owen (Reference Owen2007) noted, verbal language is used in description where explanation goes hand in hand with the creative process, forcing invention where detail is lacking and expressing relationships not apparent visually. As a result, DT benefits from structured natural language processing, facilitating the transformation of information.
Along with managing and transforming information, assessment plays a crucial role as it offers feedback and allows educators to measure learning outcomes. Two aspects of assessment are considered to comprehensively understand the effectiveness, progress, and impact of teaching. The first aspect relates to process and outcome assessment. Process assessment focuses on students’ comprehension and execution of various design activities, while outcome assessment evaluates the final results of these activities. The second aspect pertains to completeness and goodness. Completeness measures the fulfilment of desired instructions and outcomes for a given activity, while goodness evaluates the quality of activity execution and outcomes. Thus, both aspects are associated with the activities that learners learn and perform throughout the DT process.
However, while students make progress through DT stages, not all these transformations retain the original content and meaning, and some may be irreversible since the translation transformations may not work effectively in a backward direction, posing challenges for traceability in subsequent processes (Reference Gotel and MorrisGotel and Morris, 2009). During the assessment, since the experts mainly engage during the summative assessment, they lack the direct capability to assess the goodness of the process itself. For instance, the final outcomes do not offer insights into process-related questions like ‘How was the team's approach to generating solutions?’ or ‘What assumptions underpinned their work?’ or ‘How did they select particular problems to address?’ or ‘What criteria guided their choice of ideas and concepts?’. This leads to the requirement of a tool that supports the assessment of not only outcomes (both completeness and goodness) but also the goodness of the process by enabling traceability, an essential tool to know and validate the design rationale.
3. Documentation challenges in educational Design Thinking environments
One of the ways of documenting natural language-based information is method-specific documentation, which involves documenting the information pertaining to methods used during the DT process, such as empathy maps, design briefs, mind maps, value proposition canvases, and customer journey maps. In the literature, researchers have developed various tools to support the method-specific documentation process (e.g., Matz and Germanakos, Reference Matz and Germanakos2016; Brisco et al., Reference Brisco, Grierson, Lynn, Grierson, Bohemia and Buck2021).
Apart from this, various commercial tools exist for digital real-time collaboration and visualisation, such as ‘Miro’, ‘Viima’, ‘Jamboard’, and ‘MURAL’. These tools enable teams to work together and can be used for various purposes, such as brainstorming, planning, remote collaboration (Reference Loh, Ogo, Zhang and Takanoe.g., Loh et al., 2021), and creative activities. However, due to their general-purpose collaborative nature, these tools aren't specialised for supporting various operations required in the design process. For instance, sometimes, educators are constrained to selecting only those methods that were compatible with the functionalities of the virtual tools at hand (Reference Bader, Kruse, Dreßler, Müller and HenningerBader et al., 2020). As pointed out by Gottschalk et al. (Reference Gottschalk, Yigitbas, Nowosad and Engels2022), the practical application of DT is greatly influenced by the specific context, while online whiteboard tools come with predefined visual frameworks and do not adapt to the unique circumstances in which DT is employed.
In contrast to method-specific documentation, process-based documentation focuses on capturing and supporting the DT process. A digital documentation framework for educational DT projects was developed by Beyhl et al. (Reference Beyhl, Berg and Giese2013), offering centralized access to information through tool and service connectors, an active repository, and content organizers. The framework was designed to enable the aggregation of information from tools like Dropbox and Wiki systems. In another study, Beyhl et al. (Reference Beyhl, Berg, Giese, Leifer, Plattner and Meinel2014) explained the challenges concerning documenting during DT and proposed a method for organizing and retrieving design artefacts. Later, Beyhl and Giese (2016) proposed a recovery approach using knowledge extraction and recovery algorithms to reconstruct design paths, which was evaluated through educational DT documentation. In recent work, Schoormann et al. (Reference Schoormann, Hofer and Knackstedt2020) created a prototype software tool designed to assist collaborative reflection in DT projects. This tool was developed for various purposes, including visualizing project-related information, providing guidance, enabling collaborative evaluation of the process and results, and allowing for the storage and utilization of prior solutions and prototypes.
In addition to the above contributions, there are Learning Management Systems (LMS) such as Moodle and Blackboard, which are widely used by educational institutions to create online learning environments. They allow educators to create and manage courses, upload resources, engage in discussions, conduct quizzes and assignments, and track student progress (Reference Davis, Little, Stewart and AndersonDavis et al., 2008). Further, project management tools such as ‘Asana’ help students during the design projects by supporting task and subtask creation, scheduling dates and deadlines, and providing options for marking tasks (Reference Rößler and GerickeRößler & Gericke, 2023).
Apart from digital tools, the literature has been found where a physical documentation tool was developed to support the documentation process of DT. A template called LogCal (logbook & calendar) was developed by Menning et al. (Reference Menning, Beyhl, Giese, Weinberg and Nicolai2014) to capture, retrieve and communicate design rationale and support reflection. Jobst et al. (Reference Jobst, Thoring and Badke-Schaub2020) developed and evaluated a tool, the Reflection Canvas, to facilitate reflection activities through sketching and prototyping on the one hand and verbalisation on the other. The tool was found helpful for progressing through nine design process activities. The tool was also supportive of verbalising, prioritising, and documenting information. To foster self-regulated learning and advance design skills through guided reflection, Törlind and Wikberg-Nilsson (Reference Törlind and Wikberg-Nilsson2022) developed a workbook that integrates text, sketches, photos, and renderings.
From the above discussion, the following gaps were identified.
1) Current documentation support tools identified in the literature primarily facilitate the collection, storage, and accessibility of information for different users during DT projects. However, they lack provisions for managing operations and facilitating transformations across various stages within the DT process.
2) Existing LMSs have been designed to offer structured learning environments and serve as universal platforms for online course delivery. However, due to their inherent limitations in adaptability, they fail to capture diverse outcomes and their transitions throughout DT projects effectively.
3) Present commercial digital tools exhibit proficiency in specific tasks such as brainstorming and mind mapping, but their suitability for comprehensive documentation throughout the DT process is limited. They have predefined visual frameworks and do not adapt to the unique circumstances in which DT is employed.
Therefore, the research objective arrived upon for this study are 1. to assist learners in effectively documenting, transforming, and managing natural language-based information throughout the DT process and 2. To aid educators in ensuring traceability and capturing design rationale when assessing the learning. To achieve this, various requirements have been developed to facilitate efficient documentation and assessment activities for the early stages of the DT process.
4. Documentation requirements for various stakeholders
There is a need for comprehensive document support that provides a standardised structure and format across design phases to capture information efficiently. First, the design of the support system should align with the instructions provided in the design activities. The requirements derived from various design activity instructions are given in Table 1. Table 1 shows how various instructions of DT activities call for support that can help document information and its management, such as grouping, tagging, prioritising, synthesising, etc.
Requirements for documentation support
Based on the requirements presented in Table 1, three main system-level requirements are identified. The support should enable: 1. Documenting outcomes (i.e., problems and ideas), along with associated information, 2. Transforming information from one type to another (e.g., converting problem statements into need statements) and 3. Managing information related to documented outcomes (e.g., organizing, tagging, rating, categorizing, and merging). Additionally, the support system should aid the assessment process by ensuring traceability, consistency, and objectivity. In the following section, to address the design requirements obtained for documentation, design features that are implemented through the documentation template are described.
5. Development of instance-driven process documentation templates
Recognising the limitations of activity-based templates in managing and manipulating data, a novel approach was adopted. The inherent challenges of information organisation prompted the creation of problem and solution-focused workbook templates. Drawing inspiration from Unified Modeling Language (UML) and employing a class-object-attributes approach, Instance-driven documentation templates were developed to support documentation of DT outcomes for the early stages of a DT process. The documentation and transformation functions described in the requirements for Stage 1 (Need Identification) were initially mapped to individual classes, each representing a distinct function along with associated attributes. These classes were then integrated into a unified Class Diagram, which served as a comprehensive representation of the system’s structure for Stage 1. The Class Diagram was subsequently transformed into a Problem Card, encapsulating the relevant functions and interactions. The third category of the requirements, managing information effectively, was achieved through the instances of the Problem Card. The same methodology was employed in the development of the Idea Card. The templates are currently developed in the physical form.
5.1. Description of documentation templates for the problem space
The templates cover diverse elements, including problem titles, problem descriptions, user statements, activities, personas, causes, consequences, desired situations/need statements, and requirements (Figure 2A and 2B). Within the framework of problem documentation templates, using string data types serves as a foundation for encapsulating key attributes. The “Problem title” offers a concise glimpse into the core issue, while the “Problem description/User statement” provides an in-depth narrative of the problem's context. “Activities” delineates the processes involved, and “Persona” characterises the user's perspective. In parallel, “Causes” and “Consequences” shed light on contributing factors and potential outcomes. Transitioning towards resolution, the “Desired situation/Need statement” envisions a post-resolution state. The “Requirement” outlines essential conditions for effective problem-solving. This structured approach facilitates comprehensive problem understanding and targeted solutions.
Templates for the Problem space (A. Front side, B. Backside) and the idea space (C. Front side and D. Backside)
Problem documentation templates leverage Boolean and Enumerated data types to capture crucial attributes systematically. “Existence of problems in other habitats” and “Availability of solutions” are binary indicators. “Problem impact” and “Frequency of occurrence” adopt low, medium, or high levels. “Requirement type” distinguishes between scaling, non-scaling, demand, and wished aspects. The category taxonomy—Functional, Usability, Aesthetics, or Other—underpins the “Requirement category.” Priority is stratified as Very High, High, Medium, Low, or Very Low.
5.2. Description of documentation templates for the idea space
The templates feature several key attributes, each catering to distinct facets of the ideation process. “Idea title” provides a concise text field aimed at encapsulating the essence of the generated idea (Figure 2C and 2D). “Idea description” comprehensively describes the idea, its functionality, features, and potential impact. The “Inspiration” field is attributed to the inspirations, underlying motivations or sources that underpin the idea's conception. “Associated activity step” links the idea and the corresponding activity/ process. The field can be used to detail how the idea emerged, evolved, or was influenced during a specific activity. “Associated requirement/ Sub-problem” connects the idea with specific underlying problems or requirements to satisfy which the idea was generated. “Modified idea description” provides space to elaborate on any iterative modifications or enhancements made to the original idea.
This template incorporates key attributes with Boolean or Enumerated data types, facilitating a structured approach to idea documentation. “Idea type,” categorised as Existing, New, or Modified, provides insights into the nature of the generated idea—whether it stems from novel concepts, modifications, or pre-existing solutions. The attribute “Feasibility” is delineated as either Infeasible or Feasible, offering a clear assessment of the practicality and viability of the idea. “Modification based on” involves a range of enumerated factors—economic aspects, user-friendliness, utility, simplicity, safety, environmental considerations, or other pertinent considerations. This attribute underscores the multidimensional nature of idea evolution and the various dimensions on which modifications may be rooted.
Both the problem space and idea space are intended to support all the activities of the first two stages-Identify and ideate-of DT process, since both the stages primarily consist of information whose transformation remains primarily natural language-based.
6. Testing the effectiveness of instance-driven documentation templates
The research objective is formulated to test the effectiveness of instances-driven documentation templates: To assess the effectiveness of instances-driven documentation templates in improving information documentation, transformation, and management as well as to support traceability in the assessment. This study's research objective is translated into the following Research Questions (1 & 2). 1. What is the effectiveness of the instances-driven documentation templates in achieving the desired information documentation, transformation, and management? 2. What is the effectiveness of the instances-driven documentation templates to support traceability in the assessment process? In order to understand the effectiveness of templates and answer the research questions, a DT workshop was carried out with undergraduate students from the engineering, design and architecture domains. Details of the workshops are explained here.
A total of 20 students from the engineering, design, and architecture domains participated in 3 days DT workshop. The students from the engineering domain had specialisation in either mechanical or electronics sub-domains. Similarly, the students from the design domain specialised in product, communication or experience design. Out of the 20, 17 students were in their prefinal or final year of undergraduate study, while three were in their final year of graduate study.
For the experiment, six teams, T1 – T6, were created from the participants, so each team had a minimum of 3 and a maximum of 4 participants. Three teams were given the problem of making the food service system better. This problem was related to the food truck, a medium-sized van with cooking, serving, and selling facilities near the department. The other three teams were given the problem of making the parking system better. This problem was related to the department parking lot where bicycles, two-wheelers, and four-wheelers were parked. All teams were asked to identify the existing problems in the given habitat and solve those problems using the DT process instructions. The mentor was assigned to each team. The mentors were doctoral students pursuing PhDs in various sub-areas of design and having familiarity with DT; all of them have attended a course on DT during their course work. Over three days, students learned DT terms, engaged in activities, followed instructions for each, and documented outcomes using the provided template.
To answer the 1st research question, the workbook templates with documented outcomes were analysed in terms of their capabilities of documenting and managing the outcomes. Further, at the end of the workshop, a questionnaire form was given to each student for feedback. The questions asked to the students were related to the usefulness of the templates in documenting, grouping, sorting, ranking, prioritising, and managing the outcomes. To eliminate the option of ‘neutral’ and encourage more definitive responses, a 4-point scale was used for the feedback questions instead of a 5-point scale. An analysis has been carried out on the data obtained from these feedback forms. With a 5-to-15-minute individual-focused interview, where mentors were asked to provide feedback about the templates’ effectiveness in documenting, transforming, and managing the outcomes and encouraged to comment on any detail they considered relevant. The interview was also meant to confirm the data collected by the questionnaires. The transcribed data and quotes derived from the interview were compared with each other and organized, merged, and split into clusters based on similarities. To answer the 2nd research question, the workbook templates with documented outcomes were analysed in terms of completeness, accuracy, traceability of process and outcomes.
7. Results and discussion
The usefulness of the templates in documenting information:
The feedback gathered from the students’ questionnaires revealed that the students perceived templates as effective in documenting the outcomes. 85% of undergraduate students agreed that the templates’ usefulness was high or very high in documenting outcomes (Q-1, Figure 3).
Perceived usefulness of templates by undergrad students (N=20)
Mentors’ feedback about the usefulness of templates in documenting the outcomes collected via interview shows that students could understand what they want to do, and they were willing to use the templates: “In most cases, students can understand what they want to do and are happy to fill in the templates. They are not reluctant. The templates are capable of capturing what they are doing, and knowledge transfer is happening.” Further, the templates are perceived as useful in guiding the DT process: “The template was pretty useful with respect to the process that one has to follow. The particular framework and type of questions given were okay as far as we understood...”. Mentors also suggested including guidance or prompts on the templates to help students: “You can mention a little bit on the document where they can relate so that they can go back and read it. Inspiration is only required during ideas from nature, so you can mention.”
The usefulness of the templates in transforming information:
The overall flow and order of the templates appear to be well-received. Students find it easy to follow the steps in a particular order, and there are no major issues with the sequence of templates. “The flow was okay, which we follow in DT, starting from task clarification to conceptualisation to embodiment, describing a problem, need and requirement in terms of understandability and how this has been useful to students. The students found it very easy to follow the steps in a particular order. Nothing was missing in the order...” This suggests that the templates provide a structured framework for students to work through the DT process.
The usefulness of the templates in Managing the information:
80% of students agreed that the templates’ usefulness was high in grouping and sorting outcomes (Q-2, Figure 3). Further, 85% of students agreed that the templates’ usefulness was high or very high in ranking and prioritising outcomes (Q-3, Figure 3). Overall, 80% of undergraduate students agreed that the templates’ usefulness was high in managing outcomes (Q-4, Figure 3).
Mentors’ feedback indicates that the templates were useful for providing a structured workflow and order. The use of stickers for grouping in the templates was seen as a valuable feature. It helped students clearly distinguish and group related ideas or solutions.: “For grouping, this is a very good idea, because we are using one page for one problem so that we can see all the problems together, and we are able to make out what the groups we can identify from all the things, so it was very clear in front of us. And it is very easy to club the pages. This goes to this group. If it is not working, then we can change the group. So, it is very easy in that sense.”; “...then they can select any of them to combine into one concept and which match well or not so that it is easily identifiable if we use one page for one idea.”. It can be inferred that using one page for one problem or idea was helpful because it allowed managing problem cards and idea cards.
The usefulness of the templates in supporting assessment:
Documentation templates were pivotal in assessing group performance. Table 2 highlights sample of the assessment criteria and related group performance. In addition to assessing outcomes such as written problems (Completeness), and the transformation of the problem into a need statement (Goodness), the template also proved valuable for process evaluation. Even without being present throughout the entire process, experts could evaluate whether the team had grouped the problems (Completeness), and ensured that the ideas presented during the demonstration were traceable to the documented information along with associated information (Goodness). The structured format provided traceability, enabling cross-referencing of documented content with presentations. This ensured transparency and validated procedural accuracy, enhancing the reliability of outcomes. Overall, templates streamlined the assessment process, promoting consistency, integrity, and objectivity in assessing DT processes and outcomes.
Sample of assessment criteria and associated group performance of DT process and outcomes
Limitations of the templates
Mentors also showed the limitations of the templates. The fixation for the current templates was seen as a limitation. Mentors suggested providing flexibility in terms of what can be documented, allowing for various forms of expression, such as sketches and arrows: “It has to be fluid, not rigid. If they want to sketch, they can sketch; if they want to put an arrow, they should. They want to describe a problem with a sketch that may trigger creativity in that process. So, this is good for documentation and research, but it should not bind students in this format.”
The possibility of digitalizing the templates was raised as a potential improvement. Digital templates could offer advantages in terms of ease of analysis and data processing. They may also be more engaging for students, particularly those in higher grades.: “Can't these templates be digitalised? Pros and cons are there. Your work will be less if it gets digitalised, and analysis will be easier. If students are writing answers in a Word file, then you can use those answers to analyse, and you can use NLP, etc. As a researcher, you will get a lot of benefits...”
The authors also realised that the current version only allows a one-to-one mapping between need statements and requirements. However, ideally, a need can be translated into multiple requirements. To address this, a separate requirement card should be introduced, instead of consolidating the need and requirement into a single problem card.
8. Conclusion and summary
Documentation plays a crucial role in the learning and practice of DT. It serves as a means to capture and organize valuable information throughout the DT process. However, effectively supporting documentation, management, and organization of information during the early phases of DT as well as assessment can be a challenging task.
To address these challenges and enhance the overall learning process, instance-driven documentation templates prove to be a valuable tool. These templates facilitate information documentation, transformation, and management, effectively. By using templates, the process takes a significant step towards achieving the goal of completeness in documentation. These templates enable various operations, such as grouping, tagging, rating, comparing, and combining information, making the documentation process more efficient and comprehensive. Furthermore, these templates support traceability, allowing evaluators to assess design rationale and the quality of outcomes.
Future work could explore how factors such as learners’ experience level and the number of students per group influence the documentation process. Understanding these variables may provide insights into enhancing the quality of documentation for improved outcomes.
Open access
