Research case study

Calibrated Intervention in Human–AI Collaborative Forecasting: How Modification Intensity and Direction Interact to Determine Performance

An eye-tracking study of when human corrections improve an AI forecast—and when confident intervention makes it worse.

Yichuan Zhang,Kazuo Hiekata,Takuya NakashimaInternational Journal of Human–Computer Interaction · 2026

Role: First Author
Period: 2026
Status: Completed
Topics: Human–AI Collaboration · Eye Tracking · Forecast Adjustment · Mixed-Effects Models

Conceptual illustration of how the direction and magnitude of human correction can improve or worsen an AI forecast — Conceptual research-question illustration, not experimental data. Actual demand is unknown at decision time: the research asks when human adjustments correct an AI error and when they introduce additional error.

One-line summary

Human oversight adds value only when both the direction and magnitude of correction are calibrated: intervening more can make an AI forecast substantially worse.

At a glance

01Problem: Binary accept-or-reject measures cannot describe continuous oversight, where a user must decide whether, how far and in which direction to change an AI prediction.
02Context: Sixteen participants completed three AI-assisted electricity-demand forecasting tasks; the primary gaze analyses used 38 task observations from 13 participants after quality exclusions.
03My contribution: As first author, I led the conceptualization, methodology, software, formal analysis, investigation, data curation, visualization and original manuscript draft.
04Outcome: A two-dimensional Agency × Effectiveness framework showing that intervention intensity improves forecasts only when the correction direction is well calibrated.

Three-minute walkthrough

Why continuous correction needs a different model of reliance

Most human–AI studies reduce reliance to a binary choice: accept or reject the machine recommendation. Forecasting requires a harder form of oversight. A user can leave the AI prediction unchanged, adjust one or many hourly values, move them by different amounts and move them in either the correct or incorrect direction. The same visible act of intervention can therefore repair an AI error or add new error to an already useful forecast.

A synchronized eye-and-mouse forecasting study

Sixteen university participants worked with an XGBoost electricity-demand model across three within-subject tasks. The interface presented contextual information, historical data, the model forecast, SHAP-based feature attributions and controls for adjusting 24 hourly values. Eye tracking and mouse interaction were synchronized so that information seeking could be aligned with the decision phases in which participants actually modified the forecast. Thirteen participants remained in the primary eye-tracking analyses after pre-defined data-quality exclusions, contributing 38 task-level observations.

Study overview showing the XGBoost forecasting system, five interface areas of interest and the three-task experimental protocol — Figure 1. Study overview. The forecasting model and external context feed a five-region interface: Context Information, Data Analysis and Model Interpretability for reading; User Prediction and Adjustment for acting. Participants completed three AI prediction tasks after briefing and eye-tracker calibration.

Ground truth, AI predictions and participant-adjusted electricity-demand forecasts across three experimental tasks — Figure 2. The three tasks deliberately exposed different AI error structures. Screen 1 contained systematic underestimation, Screen 2 mixed patterns and Screen 3 a comparatively accurate AI forecast. Gray traces show the substantial variation in individual correction strategies.

Separating how much users intervene from whether they intervene correctly

The analysis decomposed each task into three complementary quantities. Agency measures total modification intensity. Effectiveness measures the share of adjusted points moved toward the ground truth. Forecast Value Added (FVA) compares the final user forecast with the original AI baseline. Keeping these quantities separate avoids treating activity itself as evidence of useful oversight.

1 · Agency — total intervention intensity

\mathrm{Agency}=\sum_{h=1}^{24}\left|P^{\mathrm{User}}_h-P^{\mathrm{AI}}_h\right|

Agency is the cumulative absolute change across all 24 hourly predictions. It describes how much the participant intervened, not whether the intervention helped.

2 · Effectiveness — directional calibration

\mathrm{Effectiveness}=\frac{\left|\left\{h:\operatorname{sign}\!\left(P^{\mathrm{User}}_h-P^{\mathrm{AI}}_h\right)=\operatorname{sign}\!\left(A_h-P^{\mathrm{AI}}_h\right)\right\}\right|}{\left|\left\{h:P^{\mathrm{User}}_h\neq P^{\mathrm{AI}}_h\right\}\right|}\times100\%

Effectiveness is the percentage of adjusted points moved in the direction of the actual demand Aₕ. A magnitude-weighted variant was used as a robustness check.

3 · Forecast Value Added — net collaboration outcome

\mathrm{FVA}=\mathrm{MAE}_{\mathrm{AI}}-\mathrm{MAE}_{\mathrm{User}}

Positive FVA means the final human–AI forecast improves on the AI baseline; negative FVA means the intervention degrades it.

Users sensed error magnitude better than correction direction

Across all three tasks, participants adjusted more where AI errors were larger. That magnitude sensitivity did not imply directional calibration. Direction accuracy exceeded chance only on Screen 1, where the AI made a large and systematic error. Screens 2 and 3 remained statistically indistinguishable from random direction guessing even though participants continued to intervene.

Magnitude sensitivity correlations and direction accuracy across the three forecasting tasks — Figure 3. Magnitude and direction separate. Participants adjusted more as AI error increased in every task, but direction accuracy was 72.6% only for the large systematic error and 52.2% / 50.5% for the other two tasks.

Collaboration strategy was a task-contingent state, not a fixed user type

Combining Agency and Effectiveness produced four observable strategy states: Effective Collaboration, Over-intervention, Conservative Correct and Passive Ineffective. The same participant often occupied different states on different tasks. Ten of thirteen participants switched quadrant at least once, cautioning against permanent labels such as an inherently trusting or interventionist user.

Agency by Effectiveness strategy space and participant transitions across the three tasks — Figure 4. The four strategy states produced sharply different outcomes. Mean FVA was +85 MW for Effective Collaboration and −245 MW for Over-intervention. The transition diagram shows that 77% of participants changed state across tasks.

More intervention helps only when the correction is calibrated

The primary mixed-effects model found a significant Agency × Effectiveness interaction (β = 0.016, p < .001, 95% CI [0.011, 0.021]). At high Effectiveness, stronger intervention was associated with positive FVA. At low Effectiveness, the same increase in Agency made outcomes worse. Temporal separation, permutation and bootstrap checks supported the interaction beyond a shared-measurement artifact.

Modelled interaction between Agency and Effectiveness in predicting Forecast Value Added — Figure 5. The FVA surface changes sign with Effectiveness. The zero contour separates regions where greater intervention is harmful from those where it becomes beneficial; the inset shows the fitted Agency slopes at high and low Effectiveness.

Gaze marked willingness to act—not whether the action would be correct

Phase-aligned eye tracking provided process evidence immediately before and during intervention. Greater pre-decision information dwell, broader AOI exploration and shorter mean fixations were associated with higher Agency after controlling for task. None of these aggregate gaze measures showed a reliable association with Effectiveness. The study therefore does not claim that gaze predicts correction quality.

Associations between phase-aligned gaze measures, intervention intensity and intervention effectiveness — Figure 6. Process markers were associated with modification intensity but not accuracy. The pattern remained across preparation-window choices and after false-discovery-rate correction, while the Effectiveness associations remained non-significant.

Making explanations available did not make them part of the workflow

Participants spent about 15% of gaze time on the Model Interpretation panel. The dominant transition was directly from Data to Action: 59% of transitions leaving Data entered the action area. Although 25% moved from Data to Model, the complete Data → Model → Action trigram represented only 1.9% of all transition sequences. Explanation exposure alone was not reliably associated with Effectiveness or FVA.

Distribution of gaze across interface information sources and transitions between data, model explanation and action areas — Figure 7. Attention was concentrated on Data and Action, while Model explanations received less gaze. The transition matrix highlights the dominant Data → Action path and the comparatively rare Data → Model step.

Design implication: support calibration, not intervention for its own sake

The findings motivate interfaces that separate the direction decision from the magnitude decision, surface uncertainty or directional cues at the moment of action and adapt support to the user’s current collaboration state. Explanations should be integrated into the correction workflow rather than placed in a passive panel and assumed to influence behavior. These proposals remain design implications to test, not evaluated product effects from the present study.

What this evidence does—and does not—establish

The fixed order of the three tasks confounds task characteristics with learning or fatigue, even though screen controls and within-screen checks support the primary interaction. The university-student sample and single forecasting domain limit generalization to experts and other high-stakes settings. Associations between gaze and intervention are correlational, and the available sample could not rule out smaller gaze–Effectiveness relationships. The evidence supports calibrated intervention as an analytical framework; it does not show that eye tracking can determine whether a future correction will be right.

Original experimental photograph of a participant wearing eye-tracking glasses during the forecasting task — Figure 8. Original experimental photograph. The study recorded gaze and mouse interaction while participants inspected forecasts and adjusted hourly demand values.

Acknowledgements

The authors thank the SECOM Science and Technology Foundation for facilitating participant recruitment.