Predicting Decision-Making in the Future: Human Versus Machine

Hoe Sung Ryu; Uijong Ju; Christian Wallraven

doi:10.1007/978-3-031-02444-3_10

Predicting Decision-Making in the Future: Human Versus Machine

Hoe Sung Ryu, Uijong Ju, Christian Wallraven

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Citation (Scopus)

Abstract

Deep neural networks (DNNs) have become remarkably successful in data prediction, and have even been used to predict future actions based on limited input. This raises the question: do these systems actually “understand” the event similar to humans? Here, we address this issue using videos taken from an accident situation in a driving simulation. In this situation, drivers had to choose between crashing into a suddenly-appeared obstacle or steering their car off a previously indicated cliff. We compared how well humans and a DNN predicted this decision as a function of time before the event. The DNN outperformed humans for early time-points, but had an equal performance for later time-points. Interestingly, spatio-temporal image manipulations and Grad-CAM visualizations uncovered some expected behavior, but also highlighted potential differences in temporal processing for the DNN.

Original language	English
Title of host publication	Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers
Editors	Christian Wallraven, Qingshan Liu, Hajime Nagahara
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	127-141
Number of pages	15
ISBN (Print)	9783031024436
DOIs	https://doi.org/10.1007/978-3-031-02444-3_10
Publication status	Published - 2022
Event	6th Asian Conference on Pattern Recognition, ACPR 2021 - Virtual, Online Duration: 9 Nov 2021 → 12 Nov 2021

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13189 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	6th Asian Conference on Pattern Recognition, ACPR 2021
City	Virtual, Online
Period	9/11/21 → 12/11/21

Bibliographical note

Publisher Copyright:
© 2022, Springer Nature Switzerland AG.

Keywords

Decision-making
Deep learning
Humans versus machines
Video analysis
Video prediction

Access to Document

10.1007/978-3-031-02444-3_10

Cite this

Ryu, H. S., Ju, U., & Wallraven, C. (2022). Predicting Decision-Making in the Future: Human Versus Machine. In C. Wallraven, Q. Liu, & H. Nagahara (Eds.), Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers (pp. 127-141). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13189 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-02444-3_10

Ryu, Hoe Sung ; Ju, Uijong ; Wallraven, Christian. / Predicting Decision-Making in the Future : Human Versus Machine. Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. editor / Christian Wallraven ; Qingshan Liu ; Hajime Nagahara. Springer Science and Business Media Deutschland GmbH, 2022. pp. 127-141 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{db7ad81d75df46fcbf24d56fefc83b7b,

title = "Predicting Decision-Making in the Future: Human Versus Machine",

abstract = "Deep neural networks (DNNs) have become remarkably successful in data prediction, and have even been used to predict future actions based on limited input. This raises the question: do these systems actually “understand” the event similar to humans? Here, we address this issue using videos taken from an accident situation in a driving simulation. In this situation, drivers had to choose between crashing into a suddenly-appeared obstacle or steering their car off a previously indicated cliff. We compared how well humans and a DNN predicted this decision as a function of time before the event. The DNN outperformed humans for early time-points, but had an equal performance for later time-points. Interestingly, spatio-temporal image manipulations and Grad-CAM visualizations uncovered some expected behavior, but also highlighted potential differences in temporal processing for the DNN.",

keywords = "Decision-making, Deep learning, Humans versus machines, Video analysis, Video prediction",

author = "Ryu, {Hoe Sung} and Uijong Ju and Christian Wallraven",

note = "Publisher Copyright: {\textcopyright} 2022, Springer Nature Switzerland AG.; 6th Asian Conference on Pattern Recognition, ACPR 2021 ; Conference date: 09-11-2021 Through 12-11-2021",

year = "2022",

doi = "10.1007/978-3-031-02444-3_10",

language = "English",

isbn = "9783031024436",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "127--141",

editor = "Christian Wallraven and Qingshan Liu and Hajime Nagahara",

booktitle = "Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers",

address = "Germany",

}

Ryu, HS, Ju, U & Wallraven, C 2022, Predicting Decision-Making in the Future: Human Versus Machine. in C Wallraven, Q Liu & H Nagahara (eds), Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13189 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 127-141, 6th Asian Conference on Pattern Recognition, ACPR 2021, Virtual, Online, 9/11/21. https://doi.org/10.1007/978-3-031-02444-3_10

Predicting Decision-Making in the Future: Human Versus Machine. / Ryu, Hoe Sung; Ju, Uijong; Wallraven, Christian.
Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. ed. / Christian Wallraven; Qingshan Liu; Hajime Nagahara. Springer Science and Business Media Deutschland GmbH, 2022. p. 127-141 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13189 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Predicting Decision-Making in the Future

T2 - 6th Asian Conference on Pattern Recognition, ACPR 2021

AU - Ryu, Hoe Sung

AU - Ju, Uijong

AU - Wallraven, Christian

PY - 2022

Y1 - 2022

N2 - Deep neural networks (DNNs) have become remarkably successful in data prediction, and have even been used to predict future actions based on limited input. This raises the question: do these systems actually “understand” the event similar to humans? Here, we address this issue using videos taken from an accident situation in a driving simulation. In this situation, drivers had to choose between crashing into a suddenly-appeared obstacle or steering their car off a previously indicated cliff. We compared how well humans and a DNN predicted this decision as a function of time before the event. The DNN outperformed humans for early time-points, but had an equal performance for later time-points. Interestingly, spatio-temporal image manipulations and Grad-CAM visualizations uncovered some expected behavior, but also highlighted potential differences in temporal processing for the DNN.

AB - Deep neural networks (DNNs) have become remarkably successful in data prediction, and have even been used to predict future actions based on limited input. This raises the question: do these systems actually “understand” the event similar to humans? Here, we address this issue using videos taken from an accident situation in a driving simulation. In this situation, drivers had to choose between crashing into a suddenly-appeared obstacle or steering their car off a previously indicated cliff. We compared how well humans and a DNN predicted this decision as a function of time before the event. The DNN outperformed humans for early time-points, but had an equal performance for later time-points. Interestingly, spatio-temporal image manipulations and Grad-CAM visualizations uncovered some expected behavior, but also highlighted potential differences in temporal processing for the DNN.

KW - Decision-making

KW - Deep learning

KW - Humans versus machines

KW - Video analysis

KW - Video prediction

UR - http://www.scopus.com/inward/record.url?scp=85130252204&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-02444-3_10

DO - 10.1007/978-3-031-02444-3_10

M3 - Conference contribution

AN - SCOPUS:85130252204

SN - 9783031024436

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 127

EP - 141

BT - Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers

A2 - Wallraven, Christian

A2 - Liu, Qingshan

A2 - Nagahara, Hajime

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 9 November 2021 through 12 November 2021

ER -

Ryu HS, Ju U, Wallraven C. Predicting Decision-Making in the Future: Human Versus Machine. In Wallraven C, Liu Q, Nagahara H, editors, Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2022. p. 127-141. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-02444-3_10