Concept development: looking to understand user cognition, emotions and behaviour

While cases in both performance categories engaged in user and stakeholder research, reported behaviour in high-performing and midscale sessions differed in the scope of information sought. Teams in high-performance T1 sessions had focused on understanding user cognition and affect in addition to user behaviour and other project factors, laying the foundation for more in-depth perspective taking. This focus was expressed through inquiries about users’ cognition, such as ‘what would they want’, ‘are they interested in this’ and ‘what do they feel’. Further, the focus was shown both when exploring the value of a specific solution conjecture (‘what would [users] prefer to have for visual cues [in our solution conjecture]’), and when more generally aiming to understand users (‘what would [users] want to know if they could anonymously, non-traceably [sic] ask anything [about businesses]’).

In contrast, the goals reported in midscale sessions in T1 were primarily centred on how users behaved around solution conjectures, comprising phrases like ‘would they do it spontaneously’, ‘would they post about this on social media’ and ‘will they buy this’. Cases in the midscale category also recounted research goals that did not support learning directly, such as team building (‘it was fun to make’) and making the deadline (‘at least we have something to show you [the reviewers]’). Explicit techniques to understand users’ and stakeholders’ cognition and emotions were largely absent, suggesting focus on understanding the utility of solutions more than users’ and stakeholders’ perspectives.

These differences in foci were mirrored in the methods chosen for user research. In the midscale category, Team Tennis had only observed users and Team Farming had observed signs that their prototype had been used. These methods resulted in quantitative data, such as how many prototypes had disappeared since the team last saw them. This did not stimulate holistic perspective taking. In the high-performance category, teams had complemented observations with interviews. For example, while Team Water named user behaviour as their primary focus, stating their main goal for prototype testing as ‘[wanting] to know the minimum way of having anyone contribute to water quality monitoring’, they also showed interest in the users’ cognition and affect by interviewing users after having them interact with a prototype.

System design: targeting relevant users and stakeholders

In the system design phase, cases in both performance categories expressed user-centred data gathering patterns akin to the high-performance category in the concept development phase. Cases in both performance categories focused on learning about user cognition, emotions and behaviour alike, making this no longer a differentiating factor across higher and lower design output and process quality.

Despite capturing user cognition and emotions in addition to behaviour, some midscale-performance cases targeted users and stakeholders that were less relevant to understand given the team’s current solution conjecture. For example, Team Finance was working on a business foresight and planning tool in T3, and tested it on established small and medium enterprisess, despite having already learned that ‘many of these small companies don’t see value in long-term planning, because their work is so day-to-day and week-to-week’ and heard that ‘this might be helpful for someone who is new to business or someone who is bigger and has more resources’. However, as the team failed to target their data collection to these promising user groups, their learnings were limited to reinforcing earlier understanding of whose problems the solution conjecture might be fit to address: ‘this appeals more to those who are starting up their business’. In contrast, Team Farming in a high-performing T3 session had been able to find suitable proxies for large-scale farmers to test their prototypes with: agriculture students whose families had large farms, and the caretakers of large farms.

Detailed design: rare and focused user research

In the detailed design phase, cases in the high-performance category discussed users and their perspectives less, increasing instead discussion around feasibility and technical implementation. For example, in T4, Team Farming described an intricate database structure for a digital solution to help farmers learn from each other, and Team Tennis had coded a camera application that could recognise and modify views of tennis courts. As extreme cases, Team Water in T4 and Team Finance in T5 had not done any user research in favour of building a functional and integrated solution. When gathering further data to form perspectives was discussed, the high-performing cases focused on more specific topics than before. For example, Team Water in T5 focused on strengthening their solution conjecture by interviewing managers instead of end-users (‘[our client company] helped us interview the head of a citizen science group’) and by expanding its use case (by inquiring whether citizen scientists would be willing to verify water quality measurements from strangers to supplement the measurements of certified individuals).

The midscale cases, in contrast, still inquired users about more fundamental matters, such as Team Farming in T5 asking how often farmers use agricultural consultants’ help in decision making when their solution was partly focused on connecting farmers to consultants. Similarly, Team Tennis in T5 continued asking users ‘if they would use’ the team’s solution. Overall, while midscale cases were still trying to find overall solutions to deliver value, high-performing cases were assuring that the detailed design of their chosen solution delivered value.

Concept development: sensemaking by moving between generalisations and data

The high-performing cases in T1, enabled by the user data that they had gathered, moved between generalisations and data in their reasoning. They grounded higher-level interpretations in direct and less-ambiguous learnings. For example, Team Water found that laypeople valued seeing water quality being measured in their local area: ‘people really wished for and liked the fact that something is going on locally, that ‘hey, they are actually measuring the water quality’. This generalisation was supported by less-abstract inferences and observations, such as quoting specific users (‘this one lady in particular was very delighted to see that ‘oh, they are measuring it here’, ‘oh, it’s really nice and I really want to help because it’s happening here in my local area’) and describing tangible behaviour (‘even a cyclist stopped to look at it and then moved on’). The team also used a metaphor to further elaborate on the generalisation: ‘it’s not something like, you are reading the newspaper and it’s only at the end, only if you’re bored you get to the part where they say that there was water quality measuring going on. Here, they could see’. The combination of generalised insights and tangible learnings gave user-centred arguments more transparency.

The midscale-performing cases did not move between data and generalisations when discussing users and stakeholders, instead only reporting data. For example, Team Tennis recounted various observations of user behaviour around the prototype, such as ‘they were taking photos’, ‘three people actually played and ‘people just gave the racket to us and then left’, but failed to voice generalisations based on these statements. Team Farming explicitly referenced difficulties in generalising: ‘we don’t know what we learned, we don’t know what it means if no-one bought anything or if everything sold out’. Thus, cases in the midscale-performance category were less effective in building user-centred generalisations.

The midscale cases traced their inability to generalise user-centred knowledge to suboptimal method choices and goals. Team Tennis acknowledged that their test had failed to produce relevant learnings due to various reasons, such as the wrong type of participants (‘we needed people with time, so for example tourists would have been perfect’), the weather (‘the ground was wet, and I think the ball is probably still wet’) and the prototype setup (‘I guess we would’ve needed a more proper setup anyway, as this was kind of sketchy’). Team Farming expressed troubles in planning prototyping and testing (‘we didn’t know which ideas we should make a prototype out of’). While Team Farming wanted to understand users and to design valuable solutions (‘[our focus in the test was too broad] so that we could finally find the key needs so that we can start focusing on something and not go all directions’), they did not know how to do it.

System design: scoping and sensemaking by grouping users

In the system design phase, cases in both performance categories moved between generalisations and data, akin to high-performing cases in the concept development phase.

Also, all cases now grouped users and stakeholders, displaying distinct needs, problems and value propositions for each group. In the high-performance category, Team Tennis at T3 had bracketed users by age and role in the tennis community: young players, middle-aged players, tennis coaches, casual players, competitive players, and tennis club and federation representatives. Thus, the team was able to define distinct types of value that their solution conjecture, a digitally run tennis cup, could provide for each group, such as ‘the value for most casual players is this matchmaking [feature]’, ‘for younger people, the badges, points, statistics, are more motivating’ and ‘[a local club manager] is looking for solutions that would help in engaging the more casual players in the club’. Similarly, Team Water in T2 explored potential value to two user groups, planning to tailor their solution to one or the other: ‘we have to make a decision of whether we’re going to cater [the solution] to citizen scientist groups or if we are still going to try to do something with outdoor enthusiasts’.

While some user groups were portrayed stereotypically (such as younger tennis players liking gamification elements), the grouping brought structure to the design projects, thus supporting further design work. Grouping acknowledges the existence of distinct perspectives and proposes that certain group characteristics create specific cognition, emotions and behaviours in the group. For example, Team Farming in T2 had defined two target groups: ‘farmers that are planning a project, like building a new cowhouse’ and ‘farmers who already have done or are doing a similar project’. They also specified distinct values to both groups, such as the former wanting ‘more knowledge’ and latter always having ‘time for other farmers, since the days go by faster on the tractor if they’re talking to someone’. In this high-performing case, the team further specified interactions between the two groups (‘sometimes they call unknown farmers, sometimes farmers they already know’) and general characteristics beyond the groups (‘they used the old kind of phones’). Detailed user groups also helped identify knowledge gaps and target future user research efforts, with Team Water in T2 describing how they had not ‘talked in-depth with citizen scientists’, a potential user group, but would have ‘at least two video calls with [them] next week’. Overall, grouping users helped high-performing cases scope user research efforts.

However, while cases in the midscale-performance category had narrowed their focus to specific groups, they had not considered the perspectives of some user groups central to the design problem, causing challenges in solution development. For example, Team Finance in T2 had tested a system to market sustainable businesses primarily with business owners, omitting consumer perspectives. While summarising that ‘the idea of free advertising for [small and medium enterprise owners] is fantastic’ along with various aspects that business owners would wish to highlight about their companies, consumers were considered only as recipients of information rather than as a group whose perspectives should be understood: ‘we’d show this info on the window so that all customers could see it’. The team also proposed future work that repeated this stereotypical view of consumers: ‘we thought we might set up a prototype of these ads to see if people passing by look at them’. Ultimately, the team did not know if their solution was valuable or not due to having omitted the consumer perspective: ‘we still don’t know if this will actually make a difference, whether people will go and buy stuff from these stores, or if this will be just another one of those websites or stores that open and then shut down because nobody goes there’. Thus, informed solution design in the system design phase required that teams had identified groups that were central to their solution conjecture.

Detailed design: less moving between generalisations and data, and increased use of nonuser-centred arguments

In the detailed design phase, cases in both performance categories moved between generalisations and data as well as scoped perspectives through user grouping, akin to the high-performance category in the system design phase. However, high-performing cases differed from midscale performers by occasionally omitting generalisation and using nonuser-centred arguments in scoping.

First, instead of moving between generalisations and data, the high-performing cases restated users’ comments and experiences regarding detailed solution features, and immediately moved to using the perspectives in creative processing. For example, Team Water in T5 got requests for added features, which they simply listed as: ‘[a citizen scientist] wished to add some contextual data, for example, what is the weather situation, algae concentration, ….’ When new user information was focused on detailed aspects of the solution, teams may have been able to quickly assess its relevance and put it to use.

Second, instead of scoping understanding through user groups, high-performing cases used project-centred arguments. These included dismissing solution-incompatible comments as minor limitations, deferring them as aspects to work on if the solution was developed further by the client company, and ruling some areas out of scope due to time and other resource limitations. For example, Team Farming in T4 explicitly assigned boundaries to the types of user understanding needed by saying that ‘documentation is not the point of the project’.

In contrast, the midscale-performing cases in this phase adhered to patterns similar to the high-performance category in the system design phase, focusing on generalising user perceptions about the solution. While the cases now specified pertinent user groups, they did not shift their focus onto detailed design. For example, Team Finance in T4 had established the value of their solution conjecture to users by moving between generalisations and data (‘[business owners said that] this [solution] is something they’ve never seen, that it is great to have all this data in one place, and that this helps them find new market areas”), but had not resolved the specifics of delivering this value (“we are struggling with what [types of data] we should add and what [business owners] want’).

Concept development: generalising, improving concepts and articulating value

The use of user and stakeholder perspectives in creative processing could be seen in generalising user-centred knowledge, planning improvements for solution conjectures and defining the value of solution conjectures.

First, high-performing cases in T1 used user perspectives to make user-related generalisations. Specifically, these cases modelled user cognition, namely their thought processes and boundary conditions for decision making. For example, Team Water outlined how to design a sign to encourage people to participate in water quality measurement: ‘first you have to capture people’s attention, then you have to show them that ‘if this happens, then do this’ and then you have to give them instructions, all in the same sign’. Similarly, Team Finance described how price and distance were key influences on how consumers make small purchasing decisions: ‘even though it’s double the price, if [the price of a cinnamon roll] is one euro or two euros it’s so little that if there’s distance [involved], the price doesn’t really affect the decision’. For midscale cases, narrow research foci and suboptimal methods limited user-centred learning, leading to limited capability in applying user perspectives further.

Second, the high-performing cases used user perspectives in improving solution conjectures. Here, the plans presented were actionable and practical, including statements about modifying the solution design (Team Finance: ‘how about we make six choices, like when you go and register your company, you’d have six different [support packages to choose from]’) or focusing on a new aspect of the solution in future work (Team Water: ‘how do we [from the user’s point of view] connect two things that would [physically] be separate’). Further, these teams came to the review sessions having already iterated on their solution conjectures based on user research. For example, Team Water had changed the wording and layout of a sign indicating water quality measurements could be done by passers-by. Midscale cases did not evidence iterations in their review sessions.

Third, high-performing cases used perspective taking to define the potential value of solution conjectures upfront. For example, Team Finance justified solution designs with hypothesised limitations of existing solutions from the perspective of the user (‘it’s not personal nor interesting [to the user]’), users’ unmet needs and frustrations (‘we have heard over and over how hard it is [for business owners] to find time for themselves) and benefits of the new solution (‘we wanted to show [users] that we care about them). In contrast, midscale-performing cases primarily described expected behaviour with their respective prototypes.

Despite their shortcomings in gathering data to form user perspectives, the midscale-performing cases tried to use human-centred reasoning in creative processing. However, rather than referencing specific user insight, these cases often resorted to egotistical or stereotype-based reasoning. For example, Team Farming highlighted a generic farmer comment on pricing, ‘it’s too expensive to buy fertilizer’, and claimed that their prototype was meant to sell fertiliser for a cheaper price in smaller quantities. When questioned by the reviewers about whether smaller containers of fertiliser would be useful for farmers, the team changed their target user on the fly into people with home plants: ‘this is enough for one flower. At least in my home we don’t have that many flowers that I would want to buy the whole package of fertilizer. So that’s why we went with a bit tinier scale’. This indicates that the team had not built a path from what they knew about farmers’ behaviour and cognition to how their solution conjecture would provide value, and were instead reverting to egotistical reasoning about their own needs. Team Tennis, in turn, reverted to stereotypical reasoning when hypothesising that ‘general tennis fans’ would get excited by a pop-up tennis court.

Both midscale-performing cases referenced the time pressure from course deadlines as a reason for limited perspective taking: ‘we didn’t have time [to iterate]’ and ‘we were so last-minute’. Together, suboptimal method choices and pressure to deliver made the midscale cases resort to making, as Team Farming put it, ‘something to show’ because ‘it is better than nothing’. In contrast, despite operating on the same timeline, cases in the high-performance category found ways to extrapolate knowledge from the user perspectives they had formed. For example, Team Finance was able to establish other-oriented value statements concisely, such as ‘we were thinking about what a personal care item would be for a business, and one of us thought it was a massage. You know, business owners are so tired, especially […] when they work on their own, they have no time to take care of themselves’. This shows generalisation of user cognition (no time to care for self) and what the team’s hypothesis for value is (a massage coupon will work as a personal care item).

System design: emphasis on improving solution conjectures

In the system design phase, cases in both performance categories referenced user perspectives in creative processing akin to high-performing cases in the concept development phase. The performance categories were differentiated by their ability to voice plans for improving solution conjectures.

Here, cases in the high-performance category voiced plausible design directions based on what they had inferred about users, such as ideas for modifying the solution design or shifting user research in a different or more specific direction. For example, Team Water in T3 pointed out not only challenges in their solution design, such as users being confused about when to look at a mobile app and when to handle the connected sensor, but also proposed concrete ways to remedy these shortcomings, such as ‘you [could] use the application in a way that you turn it on and set ‘measure now’, then you can put your phone in your pocket or leave it at the shore, then you can go and do the measurement, and then this device will indicate to you with a sound signal and a light that it has now done the measurement, and then you can just pick it up and all the data will automatically be in your phone’. This quote shows that the team had a detailed vision of how to improve their solution conjecture based on users’ perspectives.

In contrast, midscale-performing cases only made user-related generalisations or established the value of their solution conjecture, failing to voice specific plans for improving it based on user insights. This was exemplified by Team Finance in T3: ‘this [solution] would be useful for businesses who are starting up, who don’t have these established systems’. While this finding could lead the team to, for example, tailor their solution towards businesses who are starting up, the team did not describe such plans even when directly prompted, as they had not investigated such businesses. Curiously, in the concept development phase, Team Finance had voiced several promising plans tied to user perspectives. In the system design phase, the team partly resorted to binary (yes/no) statements about the solution’s usefulness rather than voicing underlying needs or structured insights. In T3, the team explained these shortcomings with time pressure (‘we had so little time that we didn’t have a chance to [test with customers] yet’, ‘this was basically created in 24 hours’). Team Tennis in T2 also echoed similar explanations: ‘we aren’t yet finished with this [testing]’. While the midscale-performing cases in this phase did not explicitly admit to doing user tests solely for the sake of course deliverables, the time pressure may have forced them to build and test new versions of the solution conjecture faster than they could decide on what aspects truly needed testing and refinement.

Detailed design: focus on specificity of improvements and value delivery

In the detailed design phase, cases in the midscale-performance category showed similar patterns as high-performing cases in the system design phase. Cases in the high-performance category instead omitted generalising user-centred knowledge, focusing on specific improvements for the solution and convincingly describing how it could deliver value.

First, generalisations were overall less numerous in the detailed design phase than in prior phases and were only made by midscale-performing cases. For example, Team Finance in T4 described how small and medium enterprises carry out social media marketing.

Second, while cases in both performance categories highlighted necessary improvements in their solutions, the high-performing cases focused solely on specific improvements. For example, in their high-performing T5, Team Water described needing to improve the audiovisual signals their device gave: ‘we haven’t exactly figured out what sound [the solution] will make, but it’ll be something less concerning for when it connects, and when you lose the connection it’ll be more annoying’. Team Farming in T5, despite being a midscale-performing case, showed similar attention to detail: ‘it’d be good to have a marker for if a pesticide is dangerous for bees, because farmers don’t want to kill them since it influences yield’. However, the midscale-performing cases admitted that significant portions of their solutions were not yet functional, making it possible that the teams would unearth new needed improvements. For example, when asked whether their prototype was using real data, Team Farming in T5 responded by listing a plethora of aspects yet to be built, including creating ‘simulated data’, ‘farmer profiles with built-in fields’ and the capability to spot pests and see live suggestions from other users. In contrast, Team Finance in T5 had a prototype of a website in which they used real statistics on consumers and small businesses from an official statistics bureau. Thus, the functionality of prototypes influenced how final and polished the proposed improvements were.

Third, while cases in both performance categories established the value of their solutions through user perspectives, the high-performing cases did so more convincingly. For example, Team Water in T5 minimised their solution’s limitations and highlighted positives (‘even though the people we interviewed saw some of these practical problems, they really appreciated the opportunity to expand the usability of the product’) and expressed what users can get from the solution (‘[with this product] they can keep track of and capture data in a more meaningful way’, and ‘[this product] could help the citizen science group reach level two, where their data will be accepted by [country]’). Similarly, Team Finance in T5 had specific taglines (‘it’s a toolkit guiding small businesses to success in a changing environment’) along with positive feedback from the client company. In contrast, Team Tennis in T5 had a bleaker outlook: ‘people said they would use it maybe, well, once or twice. Maybe with extra features they’d use it more often, but it depends on what those features are…’.