A New Contrastive Learning-Based Vision Transformer for Sentiment Analysis Using Scene Text Images

Palaiahnakote, Shivakumara; Kapri, Dhruv; Saleem, Muhammad Hammad; Pal, Umapada

Abstract

Sentiment analysis using scene text images is complex and challenging because it has an arbitrary background, and the method should rely on only visual features. Unlike most existing methods that use either text or images or both, this study uses only scene text images for sentiment analysis. The intuition to use only scene text images is that sometimes users express their feelings and emotions or convey their messages by writing text in different shapes with diverse background designs. It is noted that the existing methods ignore such vital cues for sentiment analysis. This work explores a vision transformer to extract visual features that represent contextual information about the appearance of the text image. Further, to strengthen the visual features, the proposed work introduces contrastive learning which maximizes the gap between inter-classes and minimizes the gap between intra-classes of positive, negative, and neutral. To demonstrate the effectiveness of the proposed method, it is tested on our own constructed dataset and benchmark dataset. A comparative study of our method with the existing method shows the proposed method is superior in the classification of positive, negative, and neutral scene text images.

• Journal Article Type: Article
• Acceptance Date: Oct 3, 2024
• Online Publication Date: Dec 23, 2024
• Publication Date: Oct 17, 2024
• Deposit Date: Nov 15, 2024
• Publicly Available Date: Oct 18, 2025
• Journal: International Journal of Pattern Recognition and Artificial Intelligence
• Print ISSN: 0218-0014
• Electronic ISSN: 1793-6381
• Publisher: World Scientific Publishing
• Peer Reviewed: Peer Reviewed
• DOI: https://doi.org/10.1142/s0218001424520293