SFTT: A Spatial-Frequency-Temporal-Based End-to-End Transformer for Heart Rate Estimation

Dey, Rakesh; Palaiahnakote, Shivakumara; Bhattacharya, Saumik; Pal, Umapada; Chanda, Sukalpa

Abstract

Vision-based Heart Rate (HR) estimation in adverse situations, such as changes in skin tone, arbitrary face movements, and complex backgrounds, etc., is challenging. Unlike state-of-the-art models that use color and spatial-temporal information, the present work exploits a Spatial-Frequency-Temporal Transformer (SFTT) for heart rate estimation. For extracting multi-scale contextual features, we propose an end-to-end transformer that encodes contextual information through a pyramidal structure-based approach. Furthermore, to strengthen the features, the proposed model introduces a new attention approach that performs mutual-sharing operations between spatial-temporal and frequency-temporal domains in an end-to-end fashion. Experimental results on four standard datasets, namely UBFC-rPPG, VIPL-HR, OBF, and MMSE-HR, show that the proposed model is generic and invariant to the aforementioned challenges. Further, a comparative study with the state-of-the-art models demonstrates the effectiveness of the proposed method over the existing methods on all four benchmark datasets. Besides, experiments on cross-dataset validation show that the proposed method is reliable and robust.

• Journal Article Type: Article
• Acceptance Date: May 15, 2025
• Online Publication Date: Jul 3, 2025
• Deposit Date: Jul 4, 2025
• Publicly Available Date: Jul 7, 2025
• Journal: IEEE Transactions on Emerging Topics in Computational Intelligence
• Print ISSN: 2168-6750
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• DOI: https://doi.org/10.1109/TETCI.2025.3582841