Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model

Joanna Hong; Se Jin Park; Yong Man Ro

doi:10.18653/v1/2023.findings-emnlp.324

Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model

Joanna Hong
, Se Jin Park
, Yong Man Ro

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Citations (Scopus)

Abstract

We present a novel approach to multilingual audio-visual speech recognition tasks by introducing a single model on a multilingual dataset. Motivated by a human cognitive system where humans can intuitively distinguish different languages without any conscious effort or guidance, we propose a model that can capture which language is given as an input speech by distinguishing the inherent similarities and differences between languages. To do so, we design a prompt fine-tuning technique into the largely pre-trained audio-visual representation model so that the network can recognize the language class as well as the speech with the corresponding language. Our work contributes to developing robust and efficient multilingual audio-visual speech recognition systems, reducing the need for language-specific models.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics
Subtitle of host publication	EMNLP 2023
Publisher	Association for Computational Linguistics (ACL)
Pages	4886-4890
Number of pages	5
ISBN (Electronic)	9798891760615
DOIs	https://doi.org/10.18653/v1/2023.findings-emnlp.324
Publication status	Published - 2023
Event	2023 Findings of the Association for Computational Linguistics: EMNLP 2023 - Hybrid, Singapore Duration: 6 Dec 2023 → 10 Dec 2023

Publication series

Name	Findings of the Association for Computational Linguistics: EMNLP 2023

Conference

Conference	2023 Findings of the Association for Computational Linguistics: EMNLP 2023
Country/Territory	Singapore
City	Hybrid
Period	6/12/23 → 10/12/23

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Access to Document

10.18653/v1/2023.findings-emnlp.324

Cite this

Hong, J., Park, S. J., & Ro, Y. M. (2023). Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 4886-4890). (Findings of the Association for Computational Linguistics: EMNLP 2023). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.findings-emnlp.324

@inproceedings{957066bf75914a5a9b7605785d435154,

title = "Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model",

abstract = "We present a novel approach to multilingual audio-visual speech recognition tasks by introducing a single model on a multilingual dataset. Motivated by a human cognitive system where humans can intuitively distinguish different languages without any conscious effort or guidance, we propose a model that can capture which language is given as an input speech by distinguishing the inherent similarities and differences between languages. To do so, we design a prompt fine-tuning technique into the largely pre-trained audio-visual representation model so that the network can recognize the language class as well as the speech with the corresponding language. Our work contributes to developing robust and efficient multilingual audio-visual speech recognition systems, reducing the need for language-specific models.",

author = "Joanna Hong and Park, \{Se Jin\} and Ro, \{Yong Man\}",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 2023 Findings of the Association for Computational Linguistics: EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

doi = "10.18653/v1/2023.findings-emnlp.324",

language = "English",

series = "Findings of the Association for Computational Linguistics: EMNLP 2023",

publisher = "Association for Computational Linguistics (ACL)",

pages = "4886--4890",

booktitle = "Findings of the Association for Computational Linguistics",

address = "United States",

}

Hong, J, Park, SJ & Ro, YM 2023, Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model. in Findings of the Association for Computational Linguistics: EMNLP 2023. Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics (ACL), pp. 4886-4890, 2023 Findings of the Association for Computational Linguistics: EMNLP 2023, Hybrid, Singapore, 6/12/23. https://doi.org/10.18653/v1/2023.findings-emnlp.324

Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model. / Hong, Joanna; Park, Se Jin; Ro, Yong Man.
Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL), 2023. p. 4886-4890 (Findings of the Association for Computational Linguistics: EMNLP 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model

AU - Hong, Joanna

AU - Park, Se Jin

AU - Ro, Yong Man

PY - 2023

Y1 - 2023

N2 - We present a novel approach to multilingual audio-visual speech recognition tasks by introducing a single model on a multilingual dataset. Motivated by a human cognitive system where humans can intuitively distinguish different languages without any conscious effort or guidance, we propose a model that can capture which language is given as an input speech by distinguishing the inherent similarities and differences between languages. To do so, we design a prompt fine-tuning technique into the largely pre-trained audio-visual representation model so that the network can recognize the language class as well as the speech with the corresponding language. Our work contributes to developing robust and efficient multilingual audio-visual speech recognition systems, reducing the need for language-specific models.

AB - We present a novel approach to multilingual audio-visual speech recognition tasks by introducing a single model on a multilingual dataset. Motivated by a human cognitive system where humans can intuitively distinguish different languages without any conscious effort or guidance, we propose a model that can capture which language is given as an input speech by distinguishing the inherent similarities and differences between languages. To do so, we design a prompt fine-tuning technique into the largely pre-trained audio-visual representation model so that the network can recognize the language class as well as the speech with the corresponding language. Our work contributes to developing robust and efficient multilingual audio-visual speech recognition systems, reducing the need for language-specific models.

UR - https://www.scopus.com/pages/publications/85183309221

U2 - 10.18653/v1/2023.findings-emnlp.324

DO - 10.18653/v1/2023.findings-emnlp.324

M3 - Conference contribution

AN - SCOPUS:85183309221

T3 - Findings of the Association for Computational Linguistics: EMNLP 2023

SP - 4886

EP - 4890

BT - Findings of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

T2 - 2023 Findings of the Association for Computational Linguistics: EMNLP 2023

Y2 - 6 December 2023 through 10 December 2023

ER -

Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this