1. Introduction
As information technology advances, services are increasingly used within diverse contexts. This trend has elevated the importance of User-Centered Design (UCD) and engineering design, where identifying latent user needs is critical for product innovation (Reference Huang, Qin, Chan and WangHuang et al., 2025). Questionnaires are a common data collection method in UCD, valued for their cost-effectiveness and rapid large-scale administration (Reference Evans and MathurEvans & Mathur, 2018). Typically, questionnaires are designed to capture both quantitative and qualitative data, comprising closed-ended questions for straightforward quantification and open-ended questions to elicit detailed experiences and impressions (Reference ChenChen, 2017).
However, questionnaires cannot probe responses in real time, risking missed insights (Reference Jacobsen, Cox, Griggio and Van BerkelJacobsen et al., 2025). This absence of interaction can also lead to low-quality responses or non-responses. Furthermore, the effort required for open-ended text entry is burdensome, often causing survey fatigue and participant dropout (Reference Porter, Whitcomb and WeitzerPorter et al., 2004). This reflects a common HCI trade-off: in situ methods such as experience sampling and mobile sensing improve temporal fidelity but often increase burden or constrain response depth (Reference Van Berkel, Ferreira and KostakosVan Berkel et al., 2017).
In response to these limitations, the potential for chatbots to “probe” qualitative survey questions has been explored, with numerous studies employing conversational agents that follow a predefined flow to pose questions (Reference Xiao, Zhou, Liao, Mark, Chi, Chen and YangXiao et al., 2020). This line of research suggests that chatbot-based surveys, through their interactive format, can foster greater self-disclosure and gather higher-quality data than static questionnaires (Reference Kim, Lee and GweonKim et al., 2019; Reference Xiao, Zhou, Liao, Mark, Chi, Chen and YangXiao et al., 2020). Recent advancements in Large Language Models (LLMs) further enhance this potential, enabling “in-the-moment” interviews that can elicit deeper user insights and latent needs comparable to human interviewers (Reference Liu, Wang, Cohen, Li and XiongLiu et al., 2025; Reference Korn, Gorsch and VogelsangKorn et al., 2025).
However, a significant constraint has been identified: leaving the dialogue entirely to an uncontrolled LLM often fails to collect the specific data that researchers require (Reference Li and CaiLi & Cai, 2025). Design research currently lacks a systematic framework that codifies when and what to probe (Reference WilsonWilson, 2013), which constitutes the research gap this paper addresses. This stems from the fact that interviewing effectiveness still hinges on the irreproducible tacit knowledge of individual expert interviewers (Reference Hahn and UppaHahn & Uppa, 2013).
Therefore, building on a user interview framework that systematizes designers’ tacit interview strategies (Reference Ding, Taoka and NakataniDing et al., 2026), this study proposes an LLM-driven dialogue method that uses this framework to control probing in ultra-short voice interviews. Our aim is to combine the convenience of questionnaires with the experiential depth of interviews through a reproducible probing mechanism. Specifically, we investigate LLM-based, ultra-short (four-question) voice interviews conducted immediately after user experiences—a context in which users have limited time and traditional chatbot surveys may be impractical. To validate our proposed method (a voice-based dialogue system that takes the framework as input), we address the following Research Questions (RQs):
-
• RQ1: Compared to conventional open-ended questionnaires, can an LLM-based dialogue method collect longer responses and more diverse content?
-
• RQ2: When conducting a voice-based questionnaire using an LLM, what differences emerge in the dialogue based on the presence versus the absence of the proposed framework?
2. Related research
2.1. User information gathering by questionnaire-based survey
Capturing subjective user experience is valuable for hypothesizing system requirements. Methods that investigate this experience after use are crucial techniques within the evaluation and improvement phases of the UCD process. Questionnaires enable researchers and practitioners to capture user concerns using a series of closed-ended or open-ended questions. Open-ended questions are valued for providing deep insights into user experiences and sentiments and are also useful for understanding the rationale behind responses to closed-ended items (Reference ChenChen, 2017).
However, several factors are known to degrade the quality of open-ended responses. Participants can be overwhelmed by the questions, leading to dropout, or they may respond hastily without deep consideration, degrading response quality. The typing burden of open-ended items adds cognitive load (Reference ChenChen, 2017), leading participants to skip or minimize answers (Reference KrosnickKrosnick, 1991).
Research has attempted to mitigate these drawbacks through strategies such as adding simple open-ended probe questions after closed-ended items or incorporating participant’s previous data into survey questions (Reference Velykoivanenko, Salehzadeh Niksirat, Teofanovic, Chapuis, Mazurek and HugueninVelykoivanenko et al., 2024). In contrast, our work adapts human interviewing probes to respond dynamically to user utterances.
2.2. Chatbot-based surveys and in situ elicitation
Research on conversational agents for information elicitation has advanced, with applications reported in survey contexts. In the survey domain, it has been argued that chatbots can present items as personalized, conversational messages, which may enhance response quality. Empirical evidence suggests that chatbots can reduce cognitive effort required to respond, which can reduce “satisficing” (Reference Kim, Lee and GweonKim et al., 2019) and this reduction is associated with higher engagement (Reference Xiao, Zhou, Liao, Mark, Chi, Chen and YangXiao et al., 2020).
Recent studies further highlight that LLM-powered interviews can collect “in-the-moment” qualitative data at scale (Reference Liu, Wang, Cohen, Li and XiongLiu et al., 2025) and foster greater self-disclosure (Reference Papneja and YadavPapneja & Yadav, 2025). However, these studies report that the turn-by-turn interaction of these text-based chatbots increases the time required compared to the efficiency of “quick completion” afforded by questionnaires, without necessarily improving usability or enjoyment (Reference Xiao, Zhou, Liao, Mark, Chi, Chen and YangXiao et al., 2020; Reference Kim, Lee and GweonKim et al., 2019).
This line of work also connects to HCI in situ and longitudinal methods. Experience sampling captures momentary self-reports but can impose burden, whereas mobile sensing scales to long-term behavior yet lacks subjective meaning in context (Reference Van Berkel, Ferreira and KostakosVan Berkel et al., 2017). Diary approaches add this meaning but often face attrition (Reference Li, He, Hu, Jia, Halevy and MaLi et al., 2024). Our work contributes ultra-short voice probing as a lightweight qualitative layer immediately after experiences.
2.3. Interview probing by conversational AI
Modern LLMs exhibit strong capabilities in text understanding and are increasingly applied to interviews and surveys. For example, they can automate requirements elicitation (Reference Korn, Gorsch and VogelsangKorn et al., 2025) and identify novel customer needs from user-generated content (Reference Huang, Qin, Chan and WangHuang et al., 2025). By contrast, in exploratory user-understanding and post-use surveys, unconstrained LLM interviews lacking explicit control of the probing process may fail to obtain desired answers (Reference Li and CaiLi & Cai, 2025). This motivates the introduction of principled probing strategies to improve dialogue quality.
In social-science interviewing, probes play a central role in eliciting rich, nuanced data; they are follow-up questions that surface unarticulated information and deepen understanding. Reference PattonPatton (2014) groups probes into three categories: Detail-oriented (e.g., who/where/when), Elaboration, and Clarification. Extending this work, Reference RobinsonRobinson (2023) proposed the DICE model (Descriptive Detail, Idiographic Memory, Clarifying, and Explanatory), reframing detail-oriented probes as descriptive (including feelings and internal states), redefining elaboration as idiographic memory for one-off events, maintaining clarifying, and adding explanatory probes to uncover reasons behind actions and thoughts.
However, these models specify probe types but provide limited guidance on when to probe, which is essential in dynamic interviews. To address this gap, we previously analyzed interviews with expert service designers to identify conversational cues that warrant probing and to refine probe types for LLM-based implementation. The next section briefly summarizes the resulting framework.
3. The user interview framework
To address the lack of systematic guidance on when and what to probe identified in the Introduction, we employ a user interview framework derived in our prior qualitative study of expert interviewing practices in service design (Reference Ding, Taoka and NakataniDing et al., 2026). In that work, 11.7 hours of interview recordings with 10 professional service designers (Reference Taoka, Tanaka, Nakatani and SaitoTaoka et al., 2024) were coded using inductive qualitative content analysis. We briefly summarize its structure and how we operationalized it for LLM control.
Figure 1 illustrates how the framework was operationalized as a structured prompt that drives a stepwise control sequence for follow-up generation. The LLM initiates the session using the ‘Raise Topic’ category. Upon receiving user response, the model is instructed to detect predefined “Utterance Features” as candidate probing points, select one feature as the current target, choose a corresponding “Direction of In-depth Exploration”, and generate a single follow-up aligned with that direction. If no features are detected in the current utterance, the prompt specifies a fallback to features from earlier turns or to introducing a new topic. “Questioning” and “Listening Techniques” are included only as stylistic guidance and are not modeled as control variables.
The user interview framework
4. Experimental setup and analysis procedure
4.1. Participants and context
To validate the proposed framework-guided LLM interview method, we conducted a field experiment in a large, crowded public exhibition venue where visitors interacted with a short motion-based activity using an exercise-support system (ESS). Immediately after using the ESS, visitors aged 7 to 90 years were invited to participate in an ultra-short interview. All visitors received a brief explanation and provided informed consent. The study protocol was approved by the Institutional Review Board of the authors’ institution (Approval No. 2024182). In total, 359 participants took part (mean age 48.1 years, SD 19.2, median 55; the most frequent age band was 50 to 60 years, 27.9%). Because environmental noise was unavoidable in this public setting, we used an external speaker and microphone. Participants were assigned in near-equal numbers across conditions (A:115, B:114, C:130).
4.2. Voice-based interview system overview
The iOS tablet app used the same UI across conditions: participants answered one fixed opening question followed by three probe turns; each question was shown on screen and spoken via text-to-speech (TTS), and responses were captured via a microphone button and auto-submitted after STT transcription. We used Apple Speech (SFSpeechRecognizer) for speech-to-text (STT) transcription, sent transcripts with a predefined prompt to the LLM API (gemini-2.5-flash-lite-preview-09-2025) with temperature set to 0.2, and synthesized outputs using Google Cloud Text-to-Speech (ja-JP-Neural2-B).
4.3. Procedure and experimental design
We opened with a fixed question “How was your experience with the exercise-support system? Please share your impressions.” followed by three probing turns. The UI, STT, and TTS were identical across conditions; only the question-generation differed. Participants were informed that the system was AI-driven. As shown in Figure 2, follow-ups were produced by one of three pipelines: (A) Framework-guided LLM, which applied a three-step control sequence: feature detection, direction selection and question generation that derived from our framework; (B) Simple LLM, which used a shared role prompt and generated three questions freely; and (C) Fixed Questions, predefined based on Reference Baxter, Courage and CaineBaxter et al. (2015) and shown in Table 1. For (A) and (B), the shared prompt defined the model’s role “You are a service designer eliciting users’ experience impressions; generate questions to investigate their experience”, included a brief description of the ESS, and incorporated the latest user utterance and dialogue history. To reduce time-of-day and venue-traffic confounds, we counterbalanced condition blocks across days by rotating the order of A, B, and C while keeping block durations constant. Sessions concluded with an LLM-generated brief closing remark.
Dialogue flow and question-generation pipelines across conditions (A–C)
Fixed questions used in condition C
4.4. Data analysis
To assess whether the three approaches differed in response volume, we quantified each participant’s utterance length. We counted characters per participant per condition, treating half-width and full-width characters as single characters and excluding punctuation. Because the character-count data did not meet normality, we tested group differences using the Kruskal–Wallis test. Given a significant omnibus result, we performed Dunn’s pairwise comparisons with Bonferroni adjustment.
To examine whether applying the framework yielded qualitative differences, we analyzed all dialogue logs across the three conditions (N = 359). First, we applied deductive coding to follow-up questions in conditions (A) Framework-guided LLM and (B) Simple LLM, using a predefined codebook “Probing-direction” derived from the framework’s “Directions of In-depth Exploration” shown in Table 2. Second, we deductively coded all participant utterances across the three conditions using a predefined “Utterance-features” codebook derived from the framework’s “Utterance Features” shown in Table 3. Emergent recurring themes were incorporated by extending the codebooks according to predefined criteria, and coding decisions were iteratively cross-checked to maintain consistency.
Probing-direction codes for follow-up questions
Utterance-features codes for participant responses
After coding, we aggregated code counts by condition. Probing-direction distributions for Conditions A and B and utterance-feature distributions for Conditions A–C were compared using chi-square tests of independence. We report Cramér’s V as an effect size and, where tests were significant, used standardized residuals to interpret which categories differed by condition. For brevity, we display only codes that account for at least 2% of coded instances in any condition; rarer codes were included in the chi-square analyses but are omitted from Tables 2 and 3 and from Figure 4.
5. Results
5.1. Overview of the collected data
Figure 3 shows utterance length by condition, A (Framework-guided LLM; n=115), B (Simple LLM; n=114), and C (Fixed Questions; n=130). Median lengths were A = 112, B = 110.5, C = 84, indicating that LLM-based methods tended to elicit longer responses than the Fixed Questions. Because lengths were non-normal and sample sizes differed, we used a Kruskal-Wallis test, which indicated a significant difference (H = 7.29, p = .026). Dunn-Bonferroni post hoc tests showed A > C (p = 0.029), with no other significant pairwise differences (Table 4).
Box plots of utterance length by condition (A–C)
Pairwise comparisons of utterance length across conditions
5.2. Distribution of questions and utterances
First, we classified LLM-generated probes using the probing-direction codebook (Table 2). We used eight categories (seven framework-derived plus one emergent category, “Lateral probe”) and coded 408 probes for A and 404 for B (Figure 4-left). A was dominated by “Details: Idiographic” (56.4%, 230), whereas B produced more “Ideas/Ideal situations” (37.1%, 150) and more “Lateral probe”. The distributions differed significantly (χ2 (8) = 165.57, p < .001) with a large effect (Cramér’s V = 0.45). Residual analysis indicated that A generated more “Details: Idiographic”, “Emotions/Sensations”, and “Timing”, while B generated more “Ideas/Ideal situations” and “Lateral probe”; “Concerns” was almost absent in B (0.2%).
Probing directions (A vs. B) (left); utterance features (A–C) (right)
Second, we classified user responses using the utterance-features codebook (Table 3) and added one emergent category, “Refusal”. Coding totals were A = 470, B = 485, and C = 528 (Figure 4-right). The distributions differed significantly (χ² (16) = 165.83, p < .001) with a small to moderate effect (Cramér’s V = 0.236). Residual analysis showed that B elicited more “Ideas/Ideal situations”, A elicited more “Negative emotions/concerns” and “Behavioral/Cognitive changes”, and C elicited more “Positive emotions” and “Refusal”, while yielding fewer “Ideas/Ideal situations” and “Likes or dislikes/Preference”.
Focusing on substantive content, we computed diversity on non-Refusal utterances using Shannon entropy (H) and its effective number of categories exp(H), alongside evenness (J) and the Gini-Simpson index. Diversity ranked A > B > C: A exp(H) = 5.03 (H = 1.62), B = 4.53 (H = 1.51), and C = 3.63 (H = 1.29). A sensitivity check including Refusal preserved the same ordering (A 5.46 > B 4.88 > C 4.52).
5.3. Qualitative results: growth in depth and specificity
In Condition A (Framework-guided LLM), we observed follow-up chains that progressively deepened initially vague user statements (e.g., affect, sensations, ideas) into specific, actionable accounts. Each session comprised eight turns (four questions), labeled T1–T8; AI denotes the system question and USER denotes participant responses, with T1–AI as the shared opening question. In the representative example (Table 5), the vague emotion “It was fun” was concretized through probing down to a specific, actionable behavior.
In Condition B (Simple LLM), as shown in Table 6, similar openings were frequently met with probes that steered the conversation toward future-oriented ideals or feature proposals rather than deepening the experience. In the paired example following “It was fun”, turns from T5 onward diverted toward the participant’s intent to continue and desired features (e.g., “various mountains”)—topics reflecting individual speculation rather than a deeper probe of the experienced emotion itself.
Dialogue example (condition A, participant in their 50s)
Dialogue example (condition B, participant in their 60s)
In contrast, the Fixed Questions (C) condition, which lacks adaptive probing, resulted in simple, single-exchange responses to the “impression”, “good points” and “improvement points” prompts. Consequently, utterances were often vague and concretization was limited. Furthermore, we observed many refusals to answer, such as “I don’t know” (Table 7, Figure 4-right).
Example user responses (condition C, participant in their 70s)
5.4. Qualitative results: recovery after refusals
In Condition A, when participants produced refusals (e.g., “nothing in particular,” “I don’t know”), the system often reconnected to a previously raised utterance feature to resume the dialogue. For example:
T6–USER: “Nothing in particular.”
T7-AI:“Understood, ‘Nothing in particular.’ Could you please tell me a bit more about that feeling you mentioned earlier, about ‘being able to go at your own pace’? […]”
In contrast, when faced with a refusal, the Simple LLM (B) was observed to “jump” to an unrelated topic (often, a generic question about “Ideas/Ideal situations”) rather than probing the original topic, resulting in a shallower investigation. For example:
T6-USER: “I don’t know.”
T7-AI: “Thank you. If there are any points that could be improved, or features you wish it had, to get more people to use this system, could you please tell us?”
And because Condition (C) lacks adaptive probing, refusal responses emerged, leaving the system no choice but to proceed to the next predefined question.
6. Discussion
6.1. What is the effect of introducing an LLM?
We investigated whether a real-time voice-based LLM dialogue method could collect richer data than Fixed Questions (RQ1). While the (A) Framework-guided LLM yielded significantly more utterance volume than (C) Fixed Questions, the (B) Simple LLM showed no significant difference from C. This suggests that simply introducing an LLM does not guarantee more data; rather, the dialogue control appears critical. Condition A’s systematic strategy scaffolded the concretization of vague statements and enabled recovery from refusal, encouraging sustained participant engagement
Beyond volume, we also quantified the diversity of non-refusal content using Shannon entropy and its effective number of categories, diversity ranked A > B > C, with a great concentration on “Positive emotions”. These results indicate that introducing an LLM appears to counteract the skew toward brief positive remarks, thereby broadening the mix of elicited categories.
The finding that the weakly controlled (B) Simple LLM did not statistically outperform (C) Fixed Questions differs from reports such as Reference Xiao, Zhou, Liao, Mark, Chi, Chen and YangXiao et al. (2020). largely due to context and method: their text-based, multi-turn chatbots operated with ample time, whereas our setting enforced short-turn, real-time voice interactions with a different probing implementation. Consistent with this interpretation, our qualitative observations suggest that B favored “Ideas/Ideal situations” and “Lateral” probes, shifting talk to hypotheticals rather than elaborating past experiences and may have constrained volume gains. Nevertheless, both lines of work converge on the benefit that conversational formats can promote self-disclosure (Reference Kim, Lee and GweonKim et al., 2019) relative to static prompts.
Qualitatively, the LLM-based methods (A and B) elicited more multi-faceted content than C. Utterance analysis (Figure 4-right) showed that C yielded more “Positive emotions” and a significantly higher Refusal rate. In contrast, A and B layered listening responses and probes onto the user’s immediate utterance, acquiring deeper information (see examples in the Section 5.3). This aligns with prior findings: in our short-turn voice setting, volume gains require guided control, whereas a conversational format alone still enhances the diversity of information compared to static prompts.
6.2. What is the value of applying the user interview framework to an LLM?
To address RQ2: What differences emerge based on the framework’s presence? While utterance volume was similar between A and B, code distributions revealed clear qualitative differences regarding supported inferences (Reference TavoryTavory, 2020). First, Condition A supported inference about “Open Contexts,” the concrete processes of user experience. Framework-guided (A) generated process-oriented probes (e.g., Details: Idiographic), progressively elaborating vague statements into actionable behaviors (Table 5). This is important for reconstructing user experiences in design research.
Conversely, Simple LLM (B) favored “Ideas/Ideal situations” (Figure 4-right), risking “Closed Contexts” (Reference TavoryTavory, 2020). B’s hypothetical questions often solicited on-the-spot opinions, making responses more likely to be artifacts of the interview situation than reflections of latent needs and thus a weaker basis for design inference. Additionally, B used “Lateral probes” more frequently than A, indicating a tendency to diverge horizontally. In contrast, A more consistently elicited episodic, condition-specific accounts of what happened, when, and why, yielding constraints and failure-related details that are better suited for early requirement reasoning than hypothetical ideals. As one indicator, A surfaced “Negative emotion/concerns” more often than B (A: 21.9% vs. B: 13.2%), signals that can reframe perspectives and stimulate innovation (Reference McDonagh and ThomasMcDonagh & Thomas, 2010).
This distinction is critical for preliminary engineering design. Capturing experience-based knowledge like failure modes minimizes rework costs (Reference Mountney, Gao and WiseallMountney et al., 2007). Unlike Method B’s speculative ideation, Method A provides more grounded evidence for requirement definition, which can help engineering designers mitigate development risks. Practically, our system serves as a practical tool for “reflective design documentation” (Reference Dalsgaard and HalskovDalsgaard & Halskov, 2012). By enforcing probes on idiographic details rather than drifting into shallow ideation, this systematic control supports deeper reflection-on-action, ensuring that data reflects lived experiences rather than on-the-spot speculations.
6.3. Limitations and future work
This study has limitations regarding evaluation, demographics, and context. First, we did not validate the utility of the data with practitioners, interpreting our results instead as proxies based on content specificity. Second, our analysis aggregates a wide age range (7 to 90 years); future work requires a more granular demographic breakdown.
Third, as a field experiment, external factors such as noise and participant haste may have influenced interactions. While randomization mitigated bias, the less adaptive Fixed Questions may be more vulnerable to environmental distractions than the Framework-guided condition. Finally, we did not systematically evaluate accessibility. Although voice and text options were available, further research is needed to assess support for users with speech or hearing impairments.
6.4. Conclusion
We evaluated ultra-short voice interviews in a public venue, comparing framework-guided LLM probing with a simple LLM and Fixed Questions. The simple LLM did not increase response volume, whereas the framework-guided condition elicited significantly more talk than Fixed Questions. It also shifted follow-ups from hypotheticals to concrete lived experience, surfacing negative feedback. While acknowledging the specific constraints of the public field setting, these results indicate that a dialogue control framework capturing designers’ tacit probing logic can make LLM-mediated interviews effective and scalable. These findings suggest a division of labor in which the system handles lightweight, in-the-moment probing to secure a baseline level of qualitative detail, allowing researchers to focus on higher-level work such as sensemaking, creative ideation, and iterative system improvement.
Acknowledgement
This work was supported by JST SPRING (JPMJSP2180), JSPS KAKENHI (JP23K17155, JP24K15615), and the Nippon Foundation HUMAI Program. We are grateful to our lab members for their assistance with the experimental work.
Open access
