Learning reliable modal weight with transformer for robust RGBT tracking

Highlights：

• A shallow convolutional network is designed to extract and fuse multi-modal information, which significantly simplifies the calculation process. Moreover, an optimal modal weight allocation strategy is proposed to obtain reliable weight for effectively optimizing fused features.

• A classification and regression subnetwork by adding a central branch is adopted to reduce the interference of background, further improving the accuracy of target prediction.

• Sufficient experimental results on four large benchmark datasets, RGBT234 (Li et al. 2019), RGBT210 (Li et al. 2017), GTOT (Li et al. 2016) and LasHeR (Li et al. 2022) indicate that the proposed tracker obtains more outstanding performance compared to the state-of-the-art RGBT trackers.

摘要

•An RGBT tracking framework based on the transformer is designed, which can enhance long-distance feature association and decrease the loss of semantic information. To our knowledge, this is the first time to incorporate the transformer in RGBT tracking.•A shallow convolutional network is designed to extract and fuse multi-modal information, which significantly simplifies the calculation process. Moreover, an optimal modal weight allocation strategy is proposed to obtain reliable weight for effectively optimizing fused features.•A classification and regression subnetwork by adding a central branch is adopted to reduce the interference of background, further improving the accuracy of target prediction.•Sufficient experimental results on four large benchmark datasets, RGBT234 (Li et al. 2019), RGBT210 (Li et al. 2017), GTOT (Li et al. 2016) and LasHeR (Li et al. 2022) indicate that the proposed tracker obtains more outstanding performance compared to the state-of-the-art RGBT trackers.