τ : Learning Touch-Augmented Vision-Language-Action Models from Future Visual SupervisionNing Cheng1, Jinan Xu1, Wanlin Li2, Yangzhi Chen1, Jing Gao1, Yiqun Wang1, Kelan Peng1, Wenjuan Han1∗1Beijing Jiaotong University2Beijing Institute for General Artificial IntelligenceAbstractLearning the informative tactile representation while effectively adapting it to pretrained Vision-Language-Action (VLA) models remains challenging at both the data and modeling levels. At the data level, limited task-specific demonstrations constrain representation quality, whereas large-scale pretraining incurs substantial costs. At the modeling level, existing methods either focus on instantaneous contact states or model temporal interaction dynamics using 6D wrench sequences, leaving high-dimensional tactile signals underexplored. To address these challenges, we present τ , a touchaugmented VLA framework that learns an action-conditioned spatiotemporal tactile representation from future visual supervision inspired by the Joint-Embedding Predictive Architecture (JEPA), and fuses it with vision-language features for action generation under limited data. This supervision operates in latent space and is used only during training, adding no deployment overhead. We also introduce TacAura, a dataset of synchronized vision, proprioception, and vision-based tactile signals across four representative contact-rich manipulation tasks. Experiments show that τ outperforms existing models and generalizes to unseen objects and scenes, delivering improved manipulation performance and robustness.IntroductionRecent advances (Zitkovich et al. 2023; Kim et al. 2025; Black et al. 2024, 2025) in Vision-Language-Action (VLA) models have established a new paradigm for general-purpose robot manipulation by leveraging large-scale multimodal pretraining across diverse tasks and embodiments. Nevertheless, their reliance on vision as the primary modality limits their capability in contact-rich manipulation, where successful execution depends on capturing physical interaction dynamics. Tactile sensing naturally provides this information, such as contact states, force distribution, and deformation patterns, which are dificult to infer from visual observations alone.Despite the promise of tactile sensing, it remains nontrivial to model the physical interaction dynamics captured by tactile signals and efectively integrate the resulting tactile representations with the vision-language features of pretrained VLA models for action generation. Existing methods primarily incorporate touch through specialized fusion architectures (Yu et al. 2025; Huang et al. 2025; Bi et al.2026), synchronous alignment with visual semantics (Cheng et al. 2026) or force measurements (Huang et al. 2026), external tactile modules for action refinement (Zhang et al. 2026c; Liu et al. 2026), or large-scale tactile training (Yuan et al. 2026a; Niu et al. 2026). Despite this progress, these methods mainly focus on instantaneous contact states. Even when temporal tactile information is considered, it is typically modeled as 6D wrench sequences that capture global force–torque dynamics but lack dynamics-aware representation for high-dimensional vision-based tactile signals with fine-grained spatial contact patterns by whether modeling their own temporal evolution or cross-modal predictive supervision (Zhang et al. 2026b; Niu et al. 2026). Therefore, how to model the temporal interaction dynamics for such high-dimensional signals under limited data remains an open problem.To address the problem, we introduce τ, a tactileaugmented VLA framework learns action-conditioned dynamics-aware tactile representations while preserving the capabilities of the pretrained VLA. Specifically, building upon the VLA backbone, τ introduces a tactile encoding and adaptation module that maps vision-based tactile signals into representations compatible with the semantic space of VLA. To encourage these representations to capture temporal interaction dynamics, we further introduce an auxiliary predictive branch for cross-modal self-supervision, inspired by Joint-Embedding Predictive Architecture (JEPA) (LeCun et al. 2022). Conditioned on the current tactile representation and the subsequent actions, a predictor forecasts the resulting change in future visual features. The target feature change is derived from the current and future observations encoded by the VLA vision encoder. A semantic similarity objective aligns the predicted feature change with this detached target, encouraging the tactile module to learn dynamics-aware representations that are predictive of future visual evolution under subsequent actions. Additionally, the predictive branch is used only during training, thereby improving representation quality without increasing inference complexity.We evaluate τ with state-of-the-art visual-only VLA baselines and multimodal policies with tactile perception on multiple representative manipulation tasks requiring fine-grained physical interaction, including plug insertion, USB insertion, stamp press, and whiteboard erasing. The results show that τ consistently outperforms both vision-only VLA baselines and existing tactile-aware multimodal policies. Evaluations on unseen objects and scene configurations further demonstrate the model’s generalizability. In summary, our main contributions are as follows:Vision-Language-Action BackboneFigure 1: τ Framework. Multi-view visual observations, tactile signals, and language instruction are encoded into modalityspecific tokens and fused by the large language model in the vision-language-action model for action chunk prediction. During training, an auxiliary JEPA-style self-supervised objective predicts future visual representations from action-conditioned latent features, enabling predictive multimodal representation learning without pixel-level reconstruction. Notably, the JEPA-style self-supervised learning is applied only during training and is removed during inference.• We propose an action-conditioned cross-modal predictive objective that learns dynamics-aware tactile representations by predicting future visual feature changes from current tactile observations and subsequent actions, without introducing additional inference overhead.• We introduce τ, a tactile-augmented VLA framework that integrates the learned tactile representations into the semantic space of the pretrained VLA model for action generation while preserving its original capabilities.• We conduct extensive real-robot experiments on four contact-rich tasks. The results show that τ surpasses VLA and VTLA baselines and generalizes to unseen objects and scene variations.Related WorkTactile-Aware Robotic ManipulationTactile-aware robotic manipulation aims to leverage tactile information for safe and reliable task execution. Some studies (Xue et al. 2025; Zhu, Huang, and Li 2026; Xu et al. 2026) build on lightweight policy models such as ACT (Zhao et al.and Difusion Policy (Chi et al. 2023), which are eficient to train and deploy but prone to overfitting task-specific datasets. Others (Zhang et al. 2026a) employ VLMs for tactile semantic understanding, cross-modal alignment, and robot action prediction. Still others (Ye et al. 2025; Yuan et al. 2026b; Lou et al. 2026) adopt world models to predict future interaction dynamics, but incur substantial computational overhead. Additionally, several approaches advance tactileaware VLA modeling by either integrating tactile sensing into pretrained VLA backbones—including π0(Black et al. 2024),π0.5\pi _ { 0 . 5 }π0.5​(Black et al. 2025), and OpenVLA (Kim et al. 2025)—with manipulation priors learned from large-scale robotic data (Yu et al. 2025; Bi et al. 2026; Huang et al. 2025; Cheng et al. 2026; Zhang et al. 2026c; Liu et al. 2026), or developing VLA-style tactile foundation policies through large-scale multisensory pretraining (Liu et al. 2025; Yuan et al. 2026a; Niu et al. 2026). In this work, we investigate touch-augmented policy learning within the VLA paradigm under limited data and compute budgets by learning the tactile representation that capture instantaneous contact states and temporal interaction dynamics, and fusing it with visuallanguage features to better ground action generation.Cross-Modal Self-SupervisionCross-modal self-supervision leverages the natural correspondences among diferent sensory modalities to learn effective representations without manual annotations. Early work (Socher et al. 2013; Owens et al. 2016; Lu et al. 2019) mainly focuses on bimodal or trimodal representation learning across vision, language, and audio. Building on these eforts, subsequent studies extend this approach to visualtactile learning by leveraging the synchronization and temporal correlations between visual and tactile signals. Zambelli et al. (Zambelli et al. 2021) explore several cross-modal self-supervised objectives for visual-tactile learning, showing that visual signals can provide efective supervision for tactile representations. Yang et al. (Yang et al. 2022) learn tactile representations for the Gelsight sensor with visuotactile contrastive multiview coding (Tian, Krishnan, and Isola 2020). kerr et al. (Kerr et al. 2022), Yang et al. (Yang et al. 2024), Cheng et al. (Cheng et al. 2025) propose various contrastive pretraining methods. Cao et al. (Cao et al. 2023) apply masked autoencoders to learn tactile representations directly from tactile inputs. Feng et al. (Feng et al. 2025, 2026) exploit masked autoencoders and contrastive learning. Unlike these methods, we introduce JEPA-style self-supervision into touch-augmented VLA policy learning, leveraging correspondences between conditioned actions and future visual changes to learn interaction-aware tactile representations, thereby enabling the policy to more efectively integrate tactile cues with vision-language features and generate contact-aware actions.The τ ModelIn this section, we detail theτ′s\boldsymbol { \tau } ^ { \prime } \mathbf { s }τ′smodel architecture, training strategy, and training dataset.Model ArchitectureAs illustrated in Figure 1, τ consists of three key components: (1) a foundational Vision-Language-Action (VLA) backbone for multimodal perception and action generation, (2) a tactile encoding and adaptation module that bridges tactile signals with the latent space of the pretrained VLA model, and (3) an auxiliary JEPA-style predictive self-supervised branch that facilitates tactile representation learning through future visual latent prediction.Foundational Vision-Language-Action Backbone.τ\tauτis built upon a pretrained VLA modelπ0.5\pi _ { 0 . 5 }π0.5​, which serves as the foundation for multimodal policy learning. At each time step t, the VLA model receives an observationot={ Tt,ℓ~t}o _ { t } = \{ \mathcal { T } _ { t } , \tilde { \ell } _ { t } \}ot​={Tt​,ℓ~t​}whereTt={ Tt1,…,TtN}\mathcal { T } _ { t } = \{ \mathcal { T } _ { t } ^ { 1 } , \ldots , \mathcal { T } _ { t } ^ { N } \}Tt​={Tt1​,…,TtN​}denotes the set of N RGB images, andℓ~t=[ℓt;qt]\tilde { \ell } _ { t } = [ \ell _ { t } ; q _ { t } ]ℓ~t​=[ℓt​;qt​]represents the language instruction formed by concatenating the task descriptionℓt\ell _ { t }ℓt​with the robot’s proprioceptive stateqtq _ { t }qt​. The RGB imagesTt\mathcal { T } _ { t }Tt​are encoded by a vision encoder and projected into the language embedding space as visual tokens, whileℓ~t\tilde { \ell } _ { t }ℓ~t​is tokenized through a word embedding layer. These multimodal tokens are fused by the large language model, and an action expert predicts a future action chunka^t:t+H=[a^t,a^t+1,…,a^t+H−1]\hat { a } _ { t : t + H } = [ \hat { a } _ { t } , \hat { a } _ { t + 1 } , \dots , \hat { a } _ { t + H - 1 } ]a^t:t+H​=[a^t​,a^t+1​,…,a^t+H−1​]from noise via conditional flow matching.Tactile Encoding and Adaptation Module. To endowτ\tauτwith tactile perception capability, we introduce a tactile encoding and adaptation module, which transforms raw tactile signals into latent representations compatible with the multimodal embedding space of the VLA model. Specifically, the observation at time step t is extended fromot={ Tt,ℓ~t}o _ { t } = \{ \mathcal { T } _ { t } , \tilde { \ell } _ { t } \}ot​={Tt​,ℓ~t​}toO~t={ Tt,ℓ~t,Tt}\tilde { \boldsymbol { O } } _ { t } = \{ \mathcal { T } _ { t } , \tilde { \boldsymbol { \ell } } _ { t } , \mathcal { T } _ { t } \}O~t​={Tt​,ℓ~t​,Tt​}, whereTt={ TtL,TtR}\mathcal { T } _ { t } = \{ \mathcal { T } _ { t } ^ { L } , \mathcal { T } _ { t } ^ { R } \}Tt​={