A new ontology-based multimodal classification system for social media images of personality traits

Biswas, Kunal; Shivakumara, Palaiahnakote; Pal, Umapada; Lu, Tong

Abstract

Number of users of social media is increasing exponentially. People are getting addicted to social media, and because of such addiction, it sometimes causes psychological and mental effects on the users. Understanding user interaction with social media is essential to study personality traits of the users. This paper focuses on classification of personality traits called Big Five factors, namely (i) agreeableness, (ii) conscientiousness, (iii) neuroticism, (iv) extraversion and (v) openness by combining image and textual features through ontology and fully connected neural network (FCNN). The intuition to classify the images of the five classes is that there is a strong correlation between images, profile picture, images of status, text in the images, description of the images uploaded on social media and the person’s mind. To extract such observation, we explore an ontology-based approach, which constructs a weighted undirected graph (WUG) based on labels of the images, profile picture, banner image, text in the images and description of the images in a novel way for feature extraction. The proposed approach uses mini-batch gradient descent to convert WUG to feature vector form for each word. Furthermore, the feature vectors are fed to the FCNN for classification of personality traits-oriented images. The proposed classification method is evaluated by testing on our dataset of five classes containing 5000 images and two benchmark datasets, namely 5-class dataset which provides 33,556 images, and 10-class dataset which provides 2000 images. The results on different datasets show that the proposed approach is superior to the existing methods in terms of average classification rate.

Citation

Biswas, K., Shivakumara, P., Pal, U., & Lu, T. (2023). A new ontology-based multimodal classification system for social media images of personality traits. Signal, Image and Video Processing, 17, 543-551. https://doi.org/10.1007/s11760-022-02259-3

• Journal Article Type: Article
• Acceptance Date: May 4, 2022
• Online Publication Date: Jun 1, 2022
• Publication Date: 2023-03
• Deposit Date: Nov 15, 2024
• Publicly Available Date: Nov 22, 2024
• Journal: Signal, Image and Video Processing
• Print ISSN: 1863-1703
• Publisher: Springer Verlag
• Peer Reviewed: Peer Reviewed
• Volume: 17
• Pages: 543-551
• DOI: https://doi.org/10.1007/s11760-022-02259-3