An audio-visual system for object-based audio : from recording to listening

Coleman, P; Franck, A; Francombe, J; Liu, Q; de Campos, T; Hughes, RJ; Menzies, D; Galvez, MFS; Tang, Y; Woodcock, JS; Jackson, PJB; Melchior, F; Pike, C; Fazi, FM; Cox, TJ; Hilton, A

Abstract

Object-based audio is an emerging representation for
audio content, where content is represented in a reproduction format-agnostic way and, thus, produced once for consumption on many different kinds of devices. This affords new opportunities for immersive, personalized, and interactive listening experiences. This paper introduces an end-to-end object-based spatial audio pipeline, from sound recording to listening. A high-level system architecture is proposed, which includes novel audiovisual interfaces to support object-based capture and listenertracked rendering, and incorporates a proposed component for objectification, that is, recording content directly into an object-based form. Text-based and extensible metadata enable communication between the system components. An open architecture for object rendering is also proposed. The system’s capabilities are evaluated in two parts. First, listener-tracked reproduction of metadata automatically estimated from two moving talkers is evaluated using an objective binaural localization model. Second, object-based scene capture with audio extracted using blind source separation (to remix between two talkers) and beamforming (to remix a recording of a jazz group) is evaluated

• Journal Article Type: Article
• Online Publication Date: Jan 17, 2018
• Publication Date: Aug 1, 2018
• Deposit Date: Dec 11, 2019
• Publicly Available Date: Dec 11, 2019
• Journal: IEEE Transactions on Multimedia
• Print ISSN: 1520-9210
• Publisher: Institute of Electrical and Electronics Engineers
• Volume: 20
• Issue: 8
• Pages: 1919-1931
• DOI: https://doi.org/10.1109/TMM.2018.2794780
• Publisher URL: https://doi.org/10.1109/TMM.2018.2794780
• Related Public URLs: https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=6046