publications
2026
- Teacher-centered LLM-based multi-agent systems–towards differentiated educational worksheetsJana Gonnermann-Müller, Jennifer Haase, Konstantin Fackeldey, and Sebastian PokuttaInnovation: The European Journal of Social Science Research, 2026
Classrooms are increasingly heterogeneous with respect to students’ language backgrounds, prior knowledge, and motivational dispositions, making differentiated instruction essential for equitable learning. Yet implementing differentiation remains challenging, as teachers face high workloads, and existing Artificial Intelligence (AI) tools mainly focus on performance while neglecting motivational and emotional factors. This paper presents a teacher-facing, large language model (LLM)-based multi-agent system for generating differentiated mathematics worksheets that account for both cognitive and motivational learner characteristics, supporting teacher-centered, AI-driven personalization in mathematics education. The framework includes three specialized agents: (1) learner agents that simulate diverse profiles incorporating topic proficiency and intrinsic motivation, (2) a teacher agent that adapts instructional content according to didactical principles, and (3) an evaluator agent that provides automated quality assurance. We tested the system using authentic grade 8 mathematics curriculum content and evaluated its feasibility through a) automated agent-based assessment of output quality and b) exploratory feedback from K-12 in-service teachers. Preliminary findings from ten evaluations highlight stability and alignment between generated materials and learner profiles. Teacher feedback particularly emphasized the structure and suitability of tasks. Findings suggest that multi-agent LLM architectures have the potential to enable scalable, context-aware differentiation in heterogeneous classroom settings and outline directions for extending the framework to richer learner profiles and real-world classroom trials.
- Prompting Against Persona Drift: Comparing Intervention Timing and Content in LLM-Simulated ConversationsNicolas Leins, Jennifer Haase, Varvara Geronimus, Jana Gonnermann-Müller, and Sebastian PokuttaPreprint, 2026
Simulating student personas with large language models (LLMs) enables scalable evaluation of educational systems. However, behavioral drift, a progressive decline in persona consistency, can emerge over extended conversations, limiting the validity of such simulations. We evaluate five prompt-level mechanisms using separate monitoring and intervention pipelines. Across 1,200 28-turn conversations spanning four LLMs and two ADHD persona intensities, we varied when to intervene (static vs. adaptive) and what to inject (reinjection vs. reflective reminder), plus a novel adaptive condition in which a monitor generates behavior-specific instructions. Relative to no intervention, reinjection reduced the modeled rate of LLM-rated drift by 35–38%, reflective reminders by 22–27%, and behavior-specific instruction by 87%. None eliminated drift. We found no evidence that adaptive timing outperformed static scheduling. Monitoring therefore appears more useful for deciding what to correct than when to intervene, although behavior-specific instruction requires component-level testing.
- When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task DifficultyNicolas Leins, Nico Pelleriti, Jana Gonnermann-Müller, and Sebastian PokuttaPreprint, 2026
LLM orchestration is often assumed to improve reasoning by allocating additional inference-time computation, yet its gains may not justify its cost. Existing comparisons also frequently overlook differences in optimization effort, making it difficult to isolate the value of orchestration itself. We conduct a controlled evaluation of Self-Refine, Best-of-N, and Debate against task-only and chain-of-thought (CoT) single-call baselines across five LLM backbones and three domains: competitive programming, chess puzzles, and mathematics. For comparability, we optimize each method with GEPA under the same optimization budget and evaluate all methods on the same difficulty-stratified benchmark items. Orchestration yields moderate but benchmark-dependent gains: averaged across backbones within each benchmark, the largest improvement is 4.6 percentage points over optimized CoT inference and 4.5 points over task-only inference, while requiring approximately 2 to 4 times the mean total tokens of task-only inference. Human-derived difficulty is associated with lower absolute accuracy in all three benchmarks, but within-benchmark analyses do not indicate that orchestration effects increase with task difficulty. By contrast, exploratory mixed-effects analyses reveal strong interactions between orchestration method and backbone model across all three benchmarks, showing that orchestration effectiveness depends substantially on the underlying model. Our results suggest that orchestration decisions should be model-specific and account for whether moderate accuracy gains justify the additional inference cost. More broadly, evaluations of LLM orchestrations should control optimization effort and report model-specific accuracy–cost trade-offs rather than treating additional inference-time structure as uniformly beneficial.
- FACET: Multi-Agent AI Supporting Teachers in Scaling Differentiated Learning for Diverse StudentsJana Gonnermann-Müller, Jennifer Haase, Nicolas Leins, Moritz Igel, Konstantin Fackeldey, and Sebastian PokuttaIn Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence (IJCAI 2026), 2026
Classrooms are becoming increasingly heterogeneous, comprising learners with diverse performance and motivation levels, language proficiencies, and learning differences such as dyslexia and ADHD. While teachers recognize the need for differentiated instruction, growing workloads create substantial barriers, making differentiated instruction an ideal that is often unrealized in practice. Current AI educational tools, which promise differentiated materials, are predominantly student-facing and performance-centric, ignoring other aspects that shape learning outcomes. We introduce FACET, a teacher-facing multi-agent framework designed to address these gaps by supporting differentiation that accounts for motivation, performance, and learning differences. Developed with educational stakeholders from the outset, the framework coordinates four specialized agents, including learner simulation, diagnostic assessment, material generation, and evaluation within a teacher-in-the-loop design. School principals (N = 30) shaped system requirements through participatory workshops, while in-service K-12 teachers (N = 70) evaluated material quality. Mixed-methods evaluation demonstrates strong perceived value for inclusive differentiation. Practitioners emphasized both the urgent need arising from classroom heterogeneity and the importance of maintaining pedagogical autonomy as a prerequisite for adoption. We discuss implications for future school deployment and outline partnerships for longitudinal classroom implementation.
- LLM-Based Educational Simulation: Evaluating Temporal Student Persona Stability Across ADHD ProfilesJana Gonnermann-Müller, Jennifer Haase, Nicolas Leins, Thomas Kosch, and Sebastian PokuttaPreprint, 2026
Student simulation with Large language models (LLMs) offers a scalable alternative for educational research and teacher training. Yet, its validity depends on whether models maintain stable personas across extended interactions. We test this prerequisite using a dual-assessment framework measuring self-reported characteristics and observer-rated behavioral expressions. Across two experiments testing four clinically-grounded ADHD persona conditions, five LLMs, and three prompt designs, we quantify between-conversation stability (N=4,968) and within-conversation stability (N=3,952 across 9 turns). Self-reported characteristics remain stable for high intensities, constituting a necessary prerequisite for valid behavioral simulation. Observer-rated behavioral expression reveals selective instability: within-conversation drift occurs in unscripted dialog for high and moderate ADHD personas. Scripted interactions with explicit task prompts eliminate this drift entirely. Stable, persona-aligned simulated learners benefit from a structured interaction design to maintain behavioral coherence, which holds significant implications for teacher training, adaptive tutoring, and any application requiring sustained, path-dependent learner interactions.
- Simulating Eating Disorder Patients with LLMs: Evaluating Psychological Persona Stability in Multi-Turn ConversationsJennifer Haase, Jana Gonnermann-Müller, See Heng Yim, Nicolas Leins, Jan Mendling, and Sebastian PokuttaPreprint, 2026
Large language model (LLM)-based simulations of clinical patients are increasingly used for research and training, yet their validity requires persona stability: coherent maintenance of an assigned psychological profile across and within conversations. We evaluate this prerequisite using eating disorder personas grounded in five published case vignettes, a dual-assessment framework (self-report + independent observer ratings), and validated psychometric instruments (EDE-Q) with known ground-truth scores. Across six LLMs and two experiments (between-conversation stability (Exp. I) and within-conversation stability (Exp. II)), we find that LLMs are paradoxically too stable and too inaccurate: variability is negligible, yet all models systematically overshoot ground-truth severity by 12-30% of the scale range (0.7-1.8 points on a 0-6 scale). The mechanism is selective stereotyping: models differentiate cases on behavioural items (dietary restraint) but maximise cognitive-affective items (body dissatisfaction, weight preoccupation) at ceiling regardless of case severity. Additional conversational context does not improve accuracy; it compounds the overshoot. LLMs can portray severe eating pathology but lack a representation of moderate clinical presentations, a "missing middle".
- Maintaining Stable Personas? Examining Temporal Stability in LLM-Based Human SimulationJana Gonnermann-Müller, Jennifer Haase, Nicolas Leins, Thomas Kosch, and Sebastian PokuttaIn Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems, 2026
Large language models (LLMs) are increasingly employed in Human-Computer Interaction (HCI) research to simulate human behavior for prototype testing and social simulations. The validity of these interactions rests on the assumption that LLMs maintain stable personas. Our work investigates temporal stability in LLM-based human simulation, examining both stability across independent instantiations and within extended interactions. We combined self-reports with observer-ratings of four persona intensity levels (low, moderate, and high ADHD representations, default persona), seven LLMs, and three persona prompts. Results from N = 3,473 conversations and N = 4,054 assessments indicate that LLMs generally reproduce personas across conversations in self-reports and observer ratings, suggesting that LLMs hold promise as tools for simulating human behavior. Within extended 18-turn interactions, observer ratings reveal a decline for moderate and high personas, a discrepancy that warrants further investigation. Our findings indicate methodological considerations for HCI researchers employing LLM-based human simulation and implications for future research.
- It’s Not Just the Prompt: Model Choice Dominates LLM Creative OutputJennifer Haase, Jana Gonnermann-Müller, Paul H. P. Hanel, Nicolas Leins, Thomas Kosch, Jan Mendling, and Sebastian PokuttaIn Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems, 2026
Prompt engineering is often treated as a reliable control mechanism for LLM behavior, yet LLM outputs vary even under similar prompts due to stochasticity. We quantify how much output variance is driven by prompt choice versus model choice and by inherent within-LLM stochasticity by evaluating 12 LLMs on 10 creativity prompts in an open-ended divergent-thinking task (AUT), measuring answer quality (originality) and quantity (number of answers), generating 100 samples per prompt. Then, we partition the variance into model, prompt, within-LLM stochasticity, and model × prompt interaction components. Our findings show that model choice is at least as important as prompt choice in this setting. For originality, the model explains 41% of the variance. Prompts explain 36%, and within-model stochasticity explains 11%. For fluency, prompts explain 4% of the variance. Model choice explains 51%, and within-model stochasticity 34%. Beyond variance decomposition, models exhibit persistent “creative fingerprints” in thematic preferences and formatting habits.
- Beyond Static Instruction: A Multi-agent AI Framework for Adaptive Augmented Reality Robot TrainingNicolas Leins, Jana Gonnermann-Müller, Malte Teichmann, and Sebastian PokuttaIn Companion Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction, 2026
Augmented Reality (AR) offers powerful visualization capabilities for industrial robot training, yet current interfaces remain predominantly static, failing to account for learners’ diverse cognitive profiles. In this paper, we present an AR application for robot training and propose a multi-agent AI framework for future integration that bridges the gap between static visualization and pedagogical intelligence. We report on the evaluation of the baseline AR interface with 36 participants performing a robotic pick-and-place task. While overall usability was high, notable disparities in task duration and learner characteristics highlighted the necessity for dynamic adaptation. To address this, we propose a multi-agent framework that orchestrates multiple components to perform complex preprocessing of multimodal inputs (e.g., voice, physiology, robot data) and adapt the AR application to the learner’s needs. By utilizing autonomous Large Language Model (LLM) agents, the system would dynamically adapt the learning environment based on advanced LLM reasoning in real-time.
- Investigating the Influence of Spatial Ability in Augmented Reality-assisted Robot ProgrammingNicolas Leins, Jana Gonnermann-Müller, Malte Teichmann, and Sebastian PokuttaPreprint, 2026
Augmented Reality (AR) offers promising opportunities to enhance learning, but its mechanisms and effects are not yet fully understood. As learning becomes increasingly personalized, considering individual learner characteristics becomes more important. This study investigates the moderating effect of spatial ability on learning experience with AR in the context of robot programming. A between-subjects experiment (N=71) compared conventional robot programming to an AR-assisted approach using a head-mounted display. Participants’ spatial ability was assessed using the Mental Rotation Test. The learning experience was measured through the System Usability Scale (SUS) and cognitive load. The results indicate that AR support does not significantly improve the learning experience compared to the conventional approach. However, AR appears to have a compensatory effect on the influence of spatial ability. In the control group, spatial ability was significantly positively associated with SUS scores and negatively associated with extraneous cognitive load, indicating that higher spatial ability predicts a better learning experience. In the AR condition, these relationships were not observable, suggesting that AR mitigated the disadvantage typically experienced by learners with lower spatial abilities. These findings suggest that AR can serve a compensatory function by reducing the influence of learner characteristics. Future research should further explore this compensatory role of AR to guide the design of personalized learning environments that address diverse learner needs and reduce barriers for learners with varying cognitive profiles.
- Stable Personas: Dual-Assessment of Temporal Stability in LLM-Based Human SimulationJana Gonnermann-Müller, Jennifer Haase, Nicolas Leins, Thomas Kosch, and Sebastian PokuttaPreprint, 2026
Large Language Models (LLMs) acting as artificial agents offer the potential for scalable behavioral research, yet their validity depends on whether LLMs can maintain stable personas across extended conversations. We address this point using a dual-assessment framework measuring both self-reported characteristics and observer-rated persona expression. Across two experiments testing four persona conditions (default, high, moderate, and low ADHD presentations), seven LLMs, and three semantically equivalent persona prompts, we examine between-conversation stability (3,473 conversations) and within-conversation stability (1,370 conversations and 18 turns). Self-reports remain highly stable both between and within conversations. However, observer ratings reveal a tendency for persona expressions to decline during extended conversations. These findings suggest that persona-instructed LLMs produce stable, persona-aligned self-reports, an important prerequisite for behavioral research, while identifying this regression tendency as a boundary condition for multi-agent social simulation.
2025
- How team organization influences the ability to solve automation failures: an experimental study on human–AI decision-making in teamsMartin Krzywdzinski, Philip Wotschack, Jana Gonnermann-Müller, and Norbert GronauAI & Society, 2025
As production environments become increasingly automated and AI-assisted, managing automation failures is a growing challenge. This study examines how team organization—hierarchical versus self-managed—affects team performance in resolving such failures. Using a laboratory experiment simulating a realistic industrial setting, teams operated automated machinery supported by AI-based assistance. We hypothesize that communication mediates the relationship between team organization and performance outcomes (productivity and quality). The results show that self-managed teams communicate more frequently and with higher quality than hierarchical teams, leading to higher productivity and fewer errors. Structural equation modeling confirms that the effect of team organization on performance is fully mediated by communication. These findings highlight the importance of team communication and suggest that revisiting team organization in AI-driven production—by favoring self-management or enhancing communication in hierarchies—may improve performance. The study contributes to human–AI teaming research by integrating organizational design into experimental analysis.
- FACET: Teacher-Centred LLM-Based Multi-Agent Systems-Towards Personalized Educational WorksheetsJana Gonnermann-Müller, Jennifer Haase, Konstantin Fackeldey, and Sebastian PokuttaPreprint, 2025
The increasing heterogeneity of student populations poses significant challenges for teachers, particularly in mathematics education, where cognitive, motivational, and emotional differences strongly influence learning outcomes. While AI-driven personalization tools have emerged, most remain performance-focused, offering limited support for teachers and neglecting broader pedagogical needs. This paper presents the FACET framework, a teacher-facing, large language model (LLM)-based multi-agent system designed to generate individualized classroom materials that integrate both cognitive and motivational dimensions of learner profiles. The framework comprises three specialized agents: (1) learner agents that simulate diverse profiles incorporating topic proficiency and intrinsic motivation, (2) a teacher agent that adapts instructional content according to didactical principles, and (3) an evaluator agent that provides automated quality assurance. We tested the system using authentic grade 8 mathematics curriculum content and evaluated its feasibility through a) automated agent-based assessment of output quality and b) exploratory feedback from K-12 in-service teachers. Results from ten internal evaluations highlighted high stability and alignment between generated materials and learner profiles, and teacher feedback particularly highlighted structure and suitability of tasks. The findings demonstrate the potential of multi-agent LLM architectures to provide scalable, context-aware personalization in heterogeneous classroom settings, and outline directions for extending the framework to richer learner profiles and real-world classroom trials.
- Specifying ten roles of using ChatGPT in secondary education: a teacher’s perspectivePhilip Schummel, Malte Rolf Teichmann, and Jana Gonnermann-MüllerIn ECIS 2025 Proceedings, 2025
ChatGPT can benefit education; however, challenges arise that demand competent and responsible usage. Competence development needs to start in schools, with the teacher being the gatekeeper for integrating it into the teaching process. Existing research suggests that schoolteachers lack specific guidance on integrating ChatGPT into teaching, as current discussions remain high-level and omit specific use cases throughout the entire teaching process. To guide an informed usage, this paper presents the results of a systematic literature review, analyzing 38 selected papers on investigating ChatGPT in schools. This selection is based on rigorously defined criteria, adhering to the PRISMA guidelines, initially screening 1075 papers. The central value of this paper is to aggregate research into ten roles for secondary teachers on using ChatGPT in schools, presenting corresponding use cases throughout the teaching process. Future research validates the ten roles, conducting interviews with teachers to present an exhaustive framework for integrating ChatGPT in schoolteacher practice.
- Let’s Be Realistic: AI-Recommender Use in a Complex Management SettingJana Gonnermann-Müller, Kristina Sahling, and Jennifer HaaseIn Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 2025
Labels like "AI-powered" or "Human-Expert" activate mental models and shape user decisions. Yet, the transferability of these labels on performance in complex, realistic tasks needs investigation. This study examines how recommender labeling and human factors (mindset, expertise) impact performance in a complex business management scenario. We conducted an online experiment employing a management dashboard, where participants (N = 395) received recommendations labeled as either Artificial Intelligence (AI) or Human-Expert-generated. Unlike previous research, labeling did not significantly influence task performance. Instead, graph literacy and cognitive load were key predictors of performance. Participants with positive attitudes toward AI found recommendations helpful, but their performance did not improve with their use. Expertise seems to be dominant in AI labeling in this context. These findings highlight the interaction between expertise, mindset, and labeling, advocating for further research investigating in which contexts labeling and human factors critically influence performance when using AI recommendations.
- Unlocking Augmented Reality Learning Design Based on Evidence From Empirical Cognitive Load Studies—A Systematic Literature ReviewJana Gonnermann-Müller and Jule M. KrügerJournal of Computer Assisted Learning, 2025
Background: Despite the numerous positive effects of augmented reality (AR) on learning, previous research has shown ambiguous results regarding the cognitive demand on the learner arising from, for example, the overlay of virtual elements or novel interaction techniques. At the same time, the number of evidence-based guidelines on designing AR is limited or focuses on global effects, primarily relying on media comparison studies, whose validity is criticised. Objective: To guide the meaningful design of learning and training settings, this paper systematically reviews empirical research on AR design and synthesises the findings to develop evidence-based recommendations for designing AR systems considering cognitive load. Methods: We conducted a systematic literature review, initially screening 810 distinct papers and ultimately analysing findings from 27 publications, which report on 29 distinct experimental studies. This selection was based on rigorously defined inclusion and exclusion criteria, adhering to the PRISMA guidelines. Results and Conclusion: The central value of this paper is the aggregation of existing evidence from empirical studies, resulting in 15 recommendations for AR design based on six design dimensions: Spatiality-related, Interaction-related, Contextuality-related, Content-related, Guidance-related and Display Selection. Additionally, with three points for future research, this systematic literature review, first, stresses the need for more empirical evidence and value-added studies. Second, learner characteristics that might influence cognitive load in AR-based learning should be examined. Third, it advocates for the inclusion of measurements beyond the NASA-TLX, and including more physiological measurements (e.g., eye-tracking, EEG) to enhance the applicability of the results for learning and training situations.
2024
- Value by Design: Reducing Cognitive Load by Using Visual Guidance in Augmented Reality - An Eye-Tracking StudyJana Gonnermann-Müller, Nicolas Leins, Norbert Gronau, and Thomas KoschIn International Conference on Information Systems Proceedings (ICIS), 2024
Besides the positive effects of Augmented Reality (AR) on work and learning performance, research on its impact on cognitive load has generated ambiguous results. Therefore, this study investigates the effects of visual guidance in AR on cognitive load by highlighting essential virtual input depending on their necessity to the work and learning task. We developed an AR application and ran a training session in a learning factory by experimentally (N=55) investigating two AR applications, with or without visual guidance, compared to paper instructions using questionnaires and eye-tracking. Our data reveal that participants using AR with visual guidance perceived less cognitive load than those without visual guidance or paper instructions. Eye-tracking provides insights into cognitive load changes over three learning rounds, showing that visual guidance is beneficial, especially at the beginning of learning. Based on our findings, we recommend integrating visual guidance into AR work and learning applications, especially for novices.
- Comparing head-mounted and handheld augmented reality for guided assemblyNicolas Leins, Jana Gonnermann-Müller, and Malte TeichmannJournal on Multimodal User Interfaces, 2024
Different Augmented Reality (AR) displays are becoming more commonly used for work since AR promises benefits by offering support, e.g., with additional information or hints. However, most research compares AR with traditional work support, like paper-based or web-based instructions. Since various AR technologies offer device-specific advantages and disadvantages, different AR technologies are more or less suitable to offer support without overwhelming or distracting the worker. Research, therefore, needs to derive empirical results from comparing different AR displays to derive concrete recommendations for action on the use and design of AR for specific contexts. To address this research gap, this experimental study investigates the effect of video-see-through head-mounted AR (Varjo XR-3) vs. handheld AR (Apple iPad) on performance (time and committed failure), motivation, and cognitive load for guided assembly. The study results reveal that both AR displays can successfully guide people in guided assembly tasks. On a descriptive level, the head-mounted AR device reveals slightly better results in terms of time and committed failures. Notably, the impact of technical restrictions on the study results was still evident. Accordingly, further investigation of device-specific differences is of continuing importance.
- Partizipation und Akzeptanz bei organisationalen Veränderungsprozessen: Vorgehensmodell und Praxisbeispiel zur Einführung neuer AnwendungssystemeAndré Ullrich, Jana Gonnermann-Müller, Gergana Vladova, and Norbert GronauIn Handbuch Industrie 4.0 und Digitale Transformation, 2024
2023
- To what extent can individualisation in terms of different types of mode improve learning outcomes and learner satisfaction? A pre-studyJana Gonnermann, Bonny Brandenburger, Gergana Vladova, and Norbert GronauHawaii International Conference on System Sciences (HICSS), 2023
With the latest technological developments and associated new possibilities in teaching, the personalisation of learning is gaining more and more importance. It assumes that individual learning experiences and results could generally be improved when personal learning preferences are considered. To do justice to the complexity of the personalisation possibilities of teaching and learning processes, we illustrate the components of learning and teaching in the digital environment and their interdependencies in an initial model. Furthermore, in a pre-study, we investigate the relationships between the learner’s ability to (digital) self-organise, the learner’s prior-knowledge learning in different variants of mode and learning outcomes as one part of this model. With this pre-study, we are taking the first step towards a holistic model of teaching and learning in digital environments.
- Influence of Pre-Experience on Learning, Usability and Cognitive Load in a Virtual Learning EnvironmentJana Gonnermann-Müller and Malte Rolf TeichmannIn Americas Conference on Information Systems 2023 Proceedings (AMCIS), 2023
Virtual reality can have advantages for education and learning. However, it must be adequately designed so that the learner benefits from the technological possibilities. Understanding the underlying effects of the virtual learning environment and the learner’s prior experience with virtual reality or prior knowledge of the content is necessary to design a proper virtual learning environment. This article presents a pre-study testing the design of a virtual learning environment for engineering vocational training courses. In the pre-study, 12 employees of two companies joined the training course in one of the two degrees of immersion (desktop VR and VR HMD). Quantitative results on learning success, cognitive load, usability, motivation, and qualitative learning process data were presented. The qualitative data assessment shows that overall, the employees were satisfied with the learning environment regardless of the level of immersion and that the participants asked for more guidance and structure accompanying the learning process. Further research is needed to test for solid group differences.