Session A3 - Interactive Systems and Data Visualization

Technology-Enhanced Learning and Heritage

​​Session Host​: Yu’an Chen

Culture Inheritance and Design Innovation: Research Practice at MILAB [Guest Talk]

​​Speaker​​: Haipeng Mi, Tsinghua University

​​Abstract​​: TBD

CoKnowledge: Supporting Assimilation of Time-synced Collective Knowledge in Online Science Videos

​​Speaker​​: Yuanhao Zhang, The Hong Kong University of Science and Technology

​​Abstract​​: Danmaku, a system of scene-aligned, time-synced, floating comments, can augment video content to create `collective knowledge’. However, its chaotic nature often hinders viewers from effectively assimilating the collective knowledge, especially in knowledge-intensive science videos. With a formative study, we examined viewers’ practices for processing collective knowledge and the specific barriers they encountered. Building on these insights, we designed a processing pipeline to filter, classify, and cluster danmaku, leading to the development of CoKnowledge – a tool incorporating a video abstract, knowledge graphs, and supplementary danmaku features to support viewers’ assimilation of collective knowledge in science videos. A within-subject study (N=24) showed that CoKnowledge significantly enhanced participants’ comprehension and recall of collective knowledge compared to a baseline with unprocessed live comments. Based on our analysis of user interaction patterns and feedback on design features, we presented design considerations for developing similar support tools.

Designing LLM-Powered Multimodal Instructions to Support Rich Hands-on Skills Remote Learning: A Case Study with Massage Instructors and Learners

​​Speaker​​: Chutian Jiang, The Hong Kong University of Science and Technology (Guangzhou)

​​Abstract​​: Although remote learning is widely used for delivering and capturing knowledge, it has limitations in teaching hands-on skills that require nuanced instructions and demonstrations of precise actions, such as massage. Furthermore, scheduling conflicts between instructors and learners often limit the availability of real-time feedback, reducing learning efficiency. To address these challenges, we developed a synthesis tool utilizing an LLM-powered Virtual Teaching Assistant (VTA). This tool integrates multimodal instructions that convey precise data, such as stroke patterns and pressure control, while providing real-time feedback for learners and summarizing their performance for instructors. Our case study with instructors and learners demonstrated the effectiveness of these multimodal instructions and the VTA in enhancing massage teaching and learning. We then discuss the tools’ use in other hands-on skills instruction and cognitive process differences in various courses.

Breaking Barriers or Building Dependency? Exploring Team-LLM Collaboration in AI-infused Classroom Debate

​​Speaker​​: Zihan Zhang, Southern University of Science and Technology

​​Abstract​​: Breaking Barriers or Building Dependency? Exploring Team-LLM Collaboration in AI-infused Classroom Debate of debates, the role of AI tools, particularly LLM-based systems, in supporting this dynamic learning environment has been under- explored in HCI. This study addresses this opportunity by investi- gating the integration of LLM-based AI into real-time classroom debates. Over four weeks, 22 students in a Design History course participated in three rounds of debates with support from Chat- GPT. The findings reveal how learners prompted the AI to offer insights, collaboratively processed its outputs, and divided labor in team-AI interactions. The study also surfaces key advantages of AI usage—reducing social anxiety, breaking communication barriers, and providing scaffolding for novices—alongside risks, such as in- formation overload and cognitive dependency, which could limit learners’ autonomy. We thereby discuss a set of nuanced implica- tions for future HCI exploration.

LitLinker: Supporting the Ideation of Interdisciplinary Contexts with Large Language Models for Teaching Literature in Elementary Schools

​​Speaker​​: Haoxiang Fan, Sun Yat-sen University

​​Abstract​​: Teaching literature under interdisciplinary contexts (e.g., science, art) that connect reading materials has become popular in elementary schools. However, constructing such contexts is challenging as it requires teachers to explore substantial amounts of interdisciplinary content and link it to the reading materials. In this paper, we develop LitLinker via an iterative design process involving 13 teachers to facilitate the ideation of interdisciplinary contexts for teaching literature. Powered by a large language model (LLM), LitLinker can recommend interdisciplinary topics and contextualize them with the literary elements (e.g., paragraphs, viewpoints) in the reading materials. A within-subjects study (N=16) shows that compared to an LLM chatbot, LitLinker can improve the integration depth of different subjects and reduce workload in this ideation task. Expert interviews (N=9) also demonstrate LitLinker’s usefulness for supporting the ideation of interdisciplinary contexts for teaching literature. We conclude with concerns and design considerations for supporting interdisciplinary teaching with LLMs.