Detailed Information

Cited 0 time in webofscience Cited 24 time in scopus
Metadata Downloads

Handling data skew in join algorithms using MapReduce

Authors
Myung, JaeseokShim, JunhoYeon, JongheumLee, Sang-goo
Issue Date
Jun-2016
Publisher
PERGAMON-ELSEVIER SCIENCE LTD
Keywords
MapReduce; Join algorithm; Skew handling; Multi-dimensional range partitioning
Citation
EXPERT SYSTEMS WITH APPLICATIONS, v.51, pp 286 - 299
Pages
14
Journal Title
EXPERT SYSTEMS WITH APPLICATIONS
Volume
51
Start Page
286
End Page
299
URI
https://scholarworks.sookmyung.ac.kr/handle/2020.sw.sookmyung/9777
DOI
10.1016/j.eswa.2015.12.024
ISSN
0957-4174
1873-6793
Abstract
One of the major obstacles hindering effective join processing on MapReduce is data skew. Since MapReduce's basic hash-based partitioning method cannot solve the problem properly, two alternatives have been proposed: range-based and randomized methods. However, they still remain some drawbacks: the range-based method does not handle join product skew, and the randomized method performs worse than the basic hash-based partitioning when input relations are not skewed. In this paper, we present a new skew handling method, called multi-dimensional range partitioning (MDRP). The proposed method overcomes the limitations of traditional algorithms in two ways: 1) the number of output records expected at each, machine is considered, which leads to better handling of join product skew, and 2) a small number of input records are sampled before the actual join begins so that an efficient execution plan considering the degree of data skew can be created. As a result, in a scalar skew experiment, the proposed join algorithm is about 6.76 times faster than the range-based algorithm when join product skew exists and about 5.14 times than the randomized algorithm when input relations are not skewed. Moreover, through the worst-case analysis, we show that the input and the output imbalances are less than or equal to 2. The proposed algorithm does not require any modification to the original MapReduce environment and is applicable to complex join operations such as theta joins and multi-way joins. (C) 2016 Elsevier Ltd. All rights reserved.
Files in This Item
Go to Link
Appears in
Collections
공과대학 > 소프트웨어학부 > 1. Journal Articles

qrcode

Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.

Related Researcher

Researcher Shim, Junho photo

Shim, Junho
공과대학 (소프트웨어학부(첨단))
Read more

Altmetrics

Total Views & Downloads

BROWSE