Road use as conversation: a Multimodal Multi-scale Coordinated Action Model (MCAM) for immersive and generative traffic safety applications

Understanding how VRUs—pedestrians, cyclists, and micromobility users—coordinate social action and resolve conflict in mixed-mode urban traffic is not merely a technological but a fundamental human factors problem. The challenge concerns how people perceive, attend to, and act upon the behavioral signals of others under real-world conditions of unpredictability and uncertainty. Intention prediction is a key mechanism through which traffic safety applications attempt to manage this complexity, yet existing frameworks treat coordination as a binary classification task and fail to capture its multi-scale, gradient nature. Current research has major gaps related to data, robustness, and contextual understanding. XR and digital twin systems are deployed for urban traffic safety, yet a critical validity problem persists: these systems are designed and evaluated without ground-truth knowledge of how people look, move, and experience cognitive load in real mixed-mode traffic. Simulator-based studies offer control but sacrifice ecological validity; existing real-world datasets are vehicle-centric, single-modality, and restricted to pedestrian-crossing scenarios. This paper introduces the MCAM, a theoretical framework that reconceptualises urban mobility as structured communicative interaction governed by shared behavioral conventions across three spatiotemporal scales—micro (1–5 s), meso (5–30 s), and macro (longer-term contextual)—operationalising conflict as a five-phase gradient progression from coordinated flow to conflict event . MCAM is operationalised through a multimodal field study using a BIBD at three ecologically distinct urban sites (a signalized intersection, a shared space, and a cycling corridor), observed across peak hours and seasonal conditions. Thirty gender- and age-balanced participants each navigate all three sites as pedestrians, cyclists, and e-scooter riders—a within-subject cross-modal design is often absent from the majority of existing intention detection datasets— would give approximately 60 h of usable recording and 1,500 hours of annotation. Three contributions are presented: (1) a reusable multimodal field methodology for ecologically valid human factors data collection; (2) an MCAM-grounded annotation scheme applicable to both real and simulated environments; and (3) a methodological bridge between real-world human behavior and the design of immersive and generative traffic safety applications.