Placement scheduling for replication in HDFS based on probabilistic approach

Dinh Mao Bui; Sungyoung Lee

doi:10.1007/978-3-319-39601-9_28

Placement scheduling for replication in HDFS based on probabilistic approach

Dinh Mao Bui
, Sungyoung Lee

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Along with the rapid evolution in Big Data analysis, Apache Hadoop keeps the important role to deliver the high availability on top of computing clusters. Also, to maintain the high throughput access for computation, the Apache Hadoop is equipped with the Hadoop File System (HDFS) for managing the file operations. Besides, HDFS is ensured the reliability and high availability by using a specific replication mechanism. However, because the workload on each computing node is various, keeping the same replication strategy might result in imbalance. Targeting to solve this drawbacks of HDFS architecture, we proposes an approach to adaptively choose the placement for replicas. To do that, the network status and system utilization can be used to create the individual replication placement strategy for each file. Eventually, the proposed approach can provide the suitable destination for replicas to improve the performance. Subsequently, the availability of the system is enhanced while still keeping the reliability of data storage.

Original language	English
Title of host publication	Inclusive Smart Cities and Digital Health - 14th International Conference on Smart Homes and Health Telematics, ICOST 2016, Proceedings
Editors	Carl K. Chang, Hai Jin, Yu Cao, Hamdi Aloulou, Mounir Mokhtari, Lorenzo Chiari
Publisher	Springer Verlag
Pages	314-320
Number of pages	7
ISBN (Print)	9783319396002
DOIs	https://doi.org/10.1007/978-3-319-39601-9_28
Publication status	Published - 2016
Event	14th International Conference on Smart Homes and Health Telematics, ICOST 2016 - Wuhan, China Duration: 25 May 2016 → 27 May 2016

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	9677
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	14th International Conference on Smart Homes and Health Telematics, ICOST 2016
Country/Territory	China
City	Wuhan
Period	25/05/16 → 27/05/16

Bibliographical note

Publisher Copyright:
© Springer International Publishing Switzerland 2016.

Keywords

Execution time
HDFS
Placement scheduling
Probabilistic approach
Throughput

Access to Document

10.1007/978-3-319-39601-9_28

Cite this

Bui, D. M., & Lee, S. (2016). Placement scheduling for replication in HDFS based on probabilistic approach. In C. K. Chang, H. Jin, Y. Cao, H. Aloulou, M. Mokhtari, & L. Chiari (Eds.), Inclusive Smart Cities and Digital Health - 14th International Conference on Smart Homes and Health Telematics, ICOST 2016, Proceedings (pp. 314-320). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9677). Springer Verlag. https://doi.org/10.1007/978-3-319-39601-9_28

Bui, Dinh Mao ; Lee, Sungyoung. / Placement scheduling for replication in HDFS based on probabilistic approach. Inclusive Smart Cities and Digital Health - 14th International Conference on Smart Homes and Health Telematics, ICOST 2016, Proceedings. editor / Carl K. Chang ; Hai Jin ; Yu Cao ; Hamdi Aloulou ; Mounir Mokhtari ; Lorenzo Chiari. Springer Verlag, 2016. pp. 314-320 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{fde6e5ccb3124f8fa68a398914c221e0,

title = "Placement scheduling for replication in HDFS based on probabilistic approach",

abstract = "Along with the rapid evolution in Big Data analysis, Apache Hadoop keeps the important role to deliver the high availability on top of computing clusters. Also, to maintain the high throughput access for computation, the Apache Hadoop is equipped with the Hadoop File System (HDFS) for managing the file operations. Besides, HDFS is ensured the reliability and high availability by using a specific replication mechanism. However, because the workload on each computing node is various, keeping the same replication strategy might result in imbalance. Targeting to solve this drawbacks of HDFS architecture, we proposes an approach to adaptively choose the placement for replicas. To do that, the network status and system utilization can be used to create the individual replication placement strategy for each file. Eventually, the proposed approach can provide the suitable destination for replicas to improve the performance. Subsequently, the availability of the system is enhanced while still keeping the reliability of data storage.",

keywords = "Execution time, HDFS, Placement scheduling, Probabilistic approach, Throughput",

author = "Bui, \{Dinh Mao\} and Sungyoung Lee",

note = "Publisher Copyright: {\textcopyright} Springer International Publishing Switzerland 2016.; 14th International Conference on Smart Homes and Health Telematics, ICOST 2016 ; Conference date: 25-05-2016 Through 27-05-2016",

year = "2016",

doi = "10.1007/978-3-319-39601-9\_28",

language = "English",

isbn = "9783319396002",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "314--320",

editor = "Chang, \{Carl K.\} and Hai Jin and Yu Cao and Hamdi Aloulou and Mounir Mokhtari and Lorenzo Chiari",

booktitle = "Inclusive Smart Cities and Digital Health - 14th International Conference on Smart Homes and Health Telematics, ICOST 2016, Proceedings",

address = "Germany",

}

Bui, DM & Lee, S 2016, Placement scheduling for replication in HDFS based on probabilistic approach. in CK Chang, H Jin, Y Cao, H Aloulou, M Mokhtari & L Chiari (eds), Inclusive Smart Cities and Digital Health - 14th International Conference on Smart Homes and Health Telematics, ICOST 2016, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9677, Springer Verlag, pp. 314-320, 14th International Conference on Smart Homes and Health Telematics, ICOST 2016, Wuhan, China, 25/05/16. https://doi.org/10.1007/978-3-319-39601-9_28

Placement scheduling for replication in HDFS based on probabilistic approach. / Bui, Dinh Mao; Lee, Sungyoung.
Inclusive Smart Cities and Digital Health - 14th International Conference on Smart Homes and Health Telematics, ICOST 2016, Proceedings. ed. / Carl K. Chang; Hai Jin; Yu Cao; Hamdi Aloulou; Mounir Mokhtari; Lorenzo Chiari. Springer Verlag, 2016. p. 314-320 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9677).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Placement scheduling for replication in HDFS based on probabilistic approach

AU - Bui, Dinh Mao

AU - Lee, Sungyoung

N1 - Publisher Copyright: © Springer International Publishing Switzerland 2016.

PY - 2016

Y1 - 2016

N2 - Along with the rapid evolution in Big Data analysis, Apache Hadoop keeps the important role to deliver the high availability on top of computing clusters. Also, to maintain the high throughput access for computation, the Apache Hadoop is equipped with the Hadoop File System (HDFS) for managing the file operations. Besides, HDFS is ensured the reliability and high availability by using a specific replication mechanism. However, because the workload on each computing node is various, keeping the same replication strategy might result in imbalance. Targeting to solve this drawbacks of HDFS architecture, we proposes an approach to adaptively choose the placement for replicas. To do that, the network status and system utilization can be used to create the individual replication placement strategy for each file. Eventually, the proposed approach can provide the suitable destination for replicas to improve the performance. Subsequently, the availability of the system is enhanced while still keeping the reliability of data storage.

AB - Along with the rapid evolution in Big Data analysis, Apache Hadoop keeps the important role to deliver the high availability on top of computing clusters. Also, to maintain the high throughput access for computation, the Apache Hadoop is equipped with the Hadoop File System (HDFS) for managing the file operations. Besides, HDFS is ensured the reliability and high availability by using a specific replication mechanism. However, because the workload on each computing node is various, keeping the same replication strategy might result in imbalance. Targeting to solve this drawbacks of HDFS architecture, we proposes an approach to adaptively choose the placement for replicas. To do that, the network status and system utilization can be used to create the individual replication placement strategy for each file. Eventually, the proposed approach can provide the suitable destination for replicas to improve the performance. Subsequently, the availability of the system is enhanced while still keeping the reliability of data storage.

KW - Execution time

KW - HDFS

KW - Placement scheduling

KW - Probabilistic approach

KW - Throughput

UR - https://www.scopus.com/pages/publications/84976610729

U2 - 10.1007/978-3-319-39601-9_28

DO - 10.1007/978-3-319-39601-9_28

M3 - Conference contribution

AN - SCOPUS:84976610729

SN - 9783319396002

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 314

EP - 320

BT - Inclusive Smart Cities and Digital Health - 14th International Conference on Smart Homes and Health Telematics, ICOST 2016, Proceedings

A2 - Chang, Carl K.

A2 - Jin, Hai

A2 - Cao, Yu

A2 - Aloulou, Hamdi

A2 - Mokhtari, Mounir

A2 - Chiari, Lorenzo

PB - Springer Verlag

T2 - 14th International Conference on Smart Homes and Health Telematics, ICOST 2016

Y2 - 25 May 2016 through 27 May 2016

ER -

Bui DM, Lee S. Placement scheduling for replication in HDFS based on probabilistic approach. In Chang CK, Jin H, Cao Y, Aloulou H, Mokhtari M, Chiari L, editors, Inclusive Smart Cities and Digital Health - 14th International Conference on Smart Homes and Health Telematics, ICOST 2016, Proceedings. Springer Verlag. 2016. p. 314-320. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-39601-9_28

Placement scheduling for replication in HDFS based on probabilistic approach

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this