Primacy of mouth over eyes to perceive audiovisual Mandarin lexical tones

Zeng, Biao; Yu, Guoxing; Hasshim, Nabil; Hong, Shanhu

Abstract

The visual cues of lexical tones are more implicit and much less investigated than consonants and vowels, and it is still unclear what facial areas contribute to facial tones identification. This study investigated Chinese and English speakers’ eye movements when they were asked to identify audiovisual Mandarin lexical tones. The Chinese and English speakers were presented with an audiovisual clip of Mandarin monosyllables (for instance, /ă/, /à/, /ĭ/, /ì/) and were asked to identify whether the syllables were a dipping tone (/ă/, / ĭ/) or a falling tone (/ à/, /ì/). These audiovisual syllables were presented in clear, noisy and silent (absence of audio signal) conditions. An eye-tracker recorded the participants’ eye movements. Results showed that the participants gazed more at the mouth than the eyes. In addition, when acoustic conditions became adverse, both the Chinese and English speakers increased their gaze duration at the mouth rather than at the eyes. The findings suggested that the mouth is the primary area that listeners utilise in their perception of audiovisual lexical tones. The similar eye movements between the Chinese and English speakers imply that the mouth acts as a perceptual cue that provides articulatory information, as opposed to social and pragmatic information.

Citation

Zeng, B., Yu, G., Hasshim, N., & Hong, S. (2023). Primacy of mouth over eyes to perceive audiovisual Mandarin lexical tones. Journal of Eye Movement Research, 16(4), https://doi.org/10.16910/JEMR.16.4.4

• Journal Article Type: Article
• Acceptance Date: Nov 23, 2023
• Publication Date: Nov 29, 2023
• Deposit Date: Feb 6, 2024
• Publicly Available Date: Feb 13, 2024
• Journal: Journal of Eye Movement Research
• Publisher: Bern Open Publishing
• Peer Reviewed: Peer Reviewed
• Volume: 16
• Issue: 4
• DOI: https://doi.org/10.16910/JEMR.16.4.4