In assisted teleoperation for human-robot collaboration, accurate intention prediction is critical for enabling timely and reliable robotic assistance during long-horizon manipulation and assembly tasks. Robot teleoperation demonstrations are costly and hardware-limited, whereas human demonstrations are easier to collect and provide rich temporal structure. To address this challenge, we propose an uncertainty-aware human-to-robot intention prediction framework that combines: (1) hierarchical transfer learning, where MS-TCN++ is pretrained on human hand demonstrations and fine-tuned on limited robot teleoperation data; (2) a conformal prediction module that provides frame-level prediction sets with statistical coverage guarantees for reliable uncertainty quantification; and (3) reliability-gated segment correction, where a reliability analyzer flags uncertain segments, a gap-aware Transformer proposes corrected labels, and a gate accepts a change only when it is confident and transition-consistent.
Experiments on robot assembly demonstrations with 22 action classes show that human-to-robot fine-tuning improves the robot test-set Edit score from 55.23 to 72.51 (mean over 3 seeds) using only 9 robot demonstrations — a gain of 17.28 points over training from scratch. Gated correction further raises Edit to 73.45 without lowering frame accuracy, and conformal prediction sets meet their target coverage. These results show that human demonstrations provide scalable pretraining data for robust, uncertainty-aware robot action segmentation.
Our framework: (1) MS-TCN++ pretrained on human demonstrations is fine-tuned on robot data; (2) conformal prediction quantifies per-frame uncertainty; (3) uncertain segments are re-labelled by a gap-aware Transformer corrector with gated acceptance.
Three components work together to produce reliable, uncertainty-calibrated action segmentation.
MS-TCN++ is pretrained on 83 UMI hand videos encoding action transition priors, then fine-tuned on 18 ALOHA robot videos (9 demonstrations). Both domains share a 22-class vocabulary and 192-dim X3D-M features, enabling complete weight transfer without architectural changes.
Temperature-scaled softmax probabilities are calibrated on a held-out split to produce prediction sets with finite-sample marginal coverage guarantees. Two variants are evaluated: standard (marginal) and class-conditional conformal prediction with per-class thresholds.
A reliability analyzer flags segments likely to be wrong. A gap-aware Transformer, pretrained on human data and fine-tuned on robot data, proposes a new label for each segment. A change is accepted only if the segment was flagged, the proposal confidence is ≥ 0.7, and the new label is transition-compatible with its neighbours.
The 22×22 action transition matrix has 484 entries. The 18 robot training videos contain only 436 observed transitions — insufficient to learn long-range assembly ordering. The 83 human training videos contain 4,025 transitions, and pretraining on them transfers these priors via fine-tuning.
83 training + 20 validation hand videos of toy car assembly captured with an egocentric GoPro camera on a handheld UMI gripper.
Teleoperated demonstrations recorded with dual wrist cameras (480×640). Split: 18 train (9 demos) / 26 val (13 demos) / 34 test (17 demos) / 46 calibration videos.
Evaluated on the robot test set (34 videos, 17 demonstrations) using frame accuracy, Edit score, and F1 at multiple overlap thresholds. Mean ± std over 3 seeds.
Transfer Learning
| Model | Edit ↑ | Acc % ↑ | F1 @10 ↑ | F1 @25 ↑ | F1 @50 ↑ |
|---|---|---|---|---|---|
| Robot-Only (trained from scratch) | 55.23 ± 5.96 | 28.48 ± 1.26 | 32.23 | 24.01 | 10.97 |
| Robot-Only + Gated Corrector | 55.99 ± 4.25 | 28.82 ± 0.94 | 32.99 | 24.69 | 11.25 |
| Human → Robot (ours) | 72.51 ± 2.19 | 34.11 ± 0.55 | 38.52 | 30.81 | 16.86 |
| Human → Robot + Gated Corrector (ours, best) | 73.45 ± 2.81 | 34.19 ± 0.60 | 38.58 | 30.83 | 16.89 |
Conformal Prediction
Standard CP
Class-Conditional CP
| CP Method | Target Level | Robot-Only Coverage | Human→Robot Coverage | + Gated Corrector Coverage | Human→Robot Set Size |
|---|---|---|---|---|---|
| Standard CP (marginal) | 93% | 96.3% | 95.8% | 96.5% | 21.0 / 22 |
| 97% | 99.3% | 97.9% | 98.4% | 21.5 / 22 | |
| Class-Conditional CP | 93% | 94.9% | 95.3% | 95.4% | 17.2 / 22 |
| 97% | 97.4% | 97.1% | 97.3% | 17.9 / 22 |
Segment-level predictions for teleoperated robot video
Human→Robot + gated corrector predictions on test videos v220 / v221
| Start | End | Duration | Ground Truth Action | Predicted Action | IoU | Result |
|---|---|---|---|---|---|---|
| 415 | 876 | 461 | pick up screw | position screw on first wheel | 58.6% | incorrect |
| 876 | 1311 | 435 | position screw on first wheel | position screw on first wheel | 32.1% | correct |
| 1311 | 1430 | 119 | pick up first wheel | pick up first wheel | 36.0% | correct |
| 1430 | 2574 | 1144 | position first wheel | position first wheel | 74.8% | correct |
| 2574 | 3172 | 598 | pick up electric screwdriver | pick up electric screwdriver | 33.4% | correct |
| 3172 | 3996 | 824 | position screwdriver bit | screw first wheel with screwdriver | 44.8% | incorrect |
| 3996 | 4194 | 198 | screw first wheel with screwdriver | pick up screw | 19.7% | incorrect |
| 4194 | 4383 | 189 | put down electric screwdriver | pick up screw | 47.6% | incorrect |
| 4383 | 4604 | 221 | pick up screw | pick up screw | 21.7% | correct |
| 4604 | 5075 | 471 | position screw on second wheel | position screw on second wheel | 44.1% | correct |
| 5075 | 5223 | 148 | pick up second wheel | position screw on second wheel | 13.9% | incorrect |
| 5223 | 6052 | 829 | position second wheel | position second wheel | 26.0% | correct |
| 6052 | 6618 | 566 | pick up electric screwdriver | position second wheel | 56.8% | incorrect |
| 6618 | 6834 | 216 | position screwdriver bit | pick up electric screwdriver | 41.4% | incorrect |
| 6834 | 7680 | 846 | screw second wheel with screwdriver | screw second wheel with screwdriver | 42.8% | correct |
| 7680 | 7972 | 292 | put down electric screwdriver | pick up screw | 64.0% | incorrect |
| 7972 | 8244 | 272 | pick up screw | position screw on side roof | 30.6% | incorrect |
| 8244 | 8982 | 738 | position screw on side roof | position screw on side roof | 54.3% | correct |
| 8982 | 9239 | 257 | pick up electric screwdriver | position screwdriver bit | 45.9% | incorrect |
| 9239 | 9716 | 477 | position screwdriver bit | screw side roof with screwdriver | 24.8% | incorrect |
| 9716 | 10563 | 847 | screw side roof with screwdriver | screw side roof with screwdriver | 71.4% | correct |
| 10563 | 10719 | 156 | put down electric screwdriver | put down electric screwdriver | 61.3% | correct |
11 correct / 22 labelled segments | Human→Robot + gated corrector (seed 42) | Video v220 | Predicted action = best-overlapping predicted segment
