Human-AI Collaboration in Forecasting Systems
Developing frameworks that enhance expert decision-making through synergistic human-AI interaction in time-series forecasting
Challenge
AI-only forecasting systems struggle under sparse data, unexpected events, and shifting regimes. Expert forecasters routinely adjust model outputs using domain knowledge, yet current systems rarely support structured collaboration between humans and AI. This research addresses the gap by designing collaboration frameworks that calibrate trust, expose model uncertainty, and enable principled human refinement of machine predictions in real-world electricity demand forecasting and related time-series tasks.
Methods
- Built interactive decision support interfaces implementing an AI-first paradigm, where ML models produce initial forecasts that experts refine.
- Collected multimodal behavioral signals (eye-tracking with Tobii Pro, mouse trajectories, screen interaction logs) during forecasting tasks.
- Modeled cognitive state transitions and interaction strategies using sequence models (e.g., HMM/RNN/LSTM) over multimodal features.
- Conducted controlled studies comparing collaboration strategies across participants with varied technical backgrounds and domain familiarity.
Results
- Human-AI collaboration improved forecast accuracy under sparse/volatile conditions compared to AI-only baselines.
- Participants with strong technical backgrounds were especially effective at augmenting AI predictions while managing over-adjustment risks.
- Multimodal signals (gaze, mouse dynamics, interaction timing) revealed distinct decision strategies and informed adaptive interface design.
- Findings consolidated into a human-centered collaboration framework; results disseminated via IJHCS submission and TE2025/INCOSE(CAS) acceptances.
Key Learnings
Effective collaboration requires transparent uncertainty communication, trust calibration, and support for strategy exploration. Multimodal behavioral data enables richer modeling of user cognition and can drive adaptive AI assistance. Designing interfaces that respect expert workflows is essential for achieving reliable, reproducible gains beyond algorithmic performance alone.
Related Publications
Zhang, Y., Hiekata, K., Nakashima, T., Shao, Q. (2025). Towards a Predictive Human Partnership Framework: Designing AI-First Decision Systems with Human Refinement in Time-Series Forecasting International Journal of Human-Computer Studies.
This paper presents a novel framework for Human-AI Collaboration in forecasting systems, proposing an AI-first paradigm where machine learning models generate initial predictions that experts refine through interactive interfaces. We conducted user studies with 25 participants and demonstrate significant improvements in forecast accuracy and user trust.
Zhang, Y., Hiekata, K., Nakashima, T., Shao, Q. (2025). Bridging Human Cognition and AI Systems: A Transdisciplinary Engineering Study of Temporal Interaction Patterns in Collaborative Forecasting Proceedings of the 32nd International Conference on Transdisciplinary Engineering (TE2025). PDF
This study addresses the critical need to understand human-AI collaboration as artificial intelligence becomes increasingly integrated into various aspects of daily life beyond purely technical domains. The authors explore how human adjustments to AI predictions, particularly in judgmental forecasting studies, can sometimes lead to better outcomes than AI-only systems. This highlights the transdisciplinary engineering approach taken to analyze temporal interaction patterns in collaborative forecasting involving both human cognition and AI systems.
Zhang, Y., Shao, Q., Hiekata, K., Nakashima, T. (2025). Multimodal Analysis of Human-AI Collaboration in Adaptive Forecasting in Electricity Demand Prediction INCOSE Complex Adaptive Systems Conference.
As AI technologies advance, human-AI collaboration in adaptive systems becomes crucial for strategic decision-making, particularly in complex tasks like electricity forecasting under uncertainty. This study introduces a transdisciplinary approach to investigate human-AI collaboration, focusing on how multimodal data reveals intricate decision-making patterns. Using Tobii Pro Glasses 3 for eye-tracking, the experiment captured participants' detailed visual attention data along with user interaction logs within a Decision Support System (DSS) designed for real-world electricity forecasting tasks. Previous research has highlighted the importance of human intervention in improving AI predictions, but the underlying mechanisms remain unclear. Multimodal data analysis allows us to reconstruct users' interactive behaviors and decision-making processes when interacting with AI-generated forecasts, highlighting their responses and strategies throughout the experiment. The core contributions of this study include: (1) demonstrating the effectiveness of eye-tracking and interaction log data in capturing individual collaborative behaviors during human-AI interaction; and (2) mapping users' strategic decision-making pathways, illustrating how they adjust their responses based on AI assistance. The insights from this study provide valuable guidance for designing adaptive DSS interfaces that can better facilitate human collaboration with AI in industrial decision-making, driving advancements in strategic planning under dynamic conditions.
