A SLA-based Spark cluster scaling method in cloud environment
- Authors
- Oh Y.; Choi J.; Song E.; Kim M.; Kim Y.
- Issue Date
- Nov-2016
- Publisher
- Institute of Electrical and Electronics Engineers Inc.
- Keywords
- auto-scaling; resource management; SLA; Spark Cluster
- Citation
- 18th Asia-Pacific Network Operations and Management Symposium, APNOMS 2016: Management of Softwarized Infrastructure - Proceedings
- Journal Title
- 18th Asia-Pacific Network Operations and Management Symposium, APNOMS 2016: Management of Softwarized Infrastructure - Proceedings
- URI
- https://scholarworks.sookmyung.ac.kr/handle/2020.sw.sookmyung/3600
- DOI
- 10.1109/APNOMS.2016.7737242
- Abstract
- As the development of Internet and mobile device increases, there is a correspondingly increasing amount of data produced by users of such technology worldwide. It is thus essential to analyze such massive amounts of data reflective of the big data era. Recently, Apache Spark has become popular for analyzing big data, since it can process streaming data and support real-time in-memory computing. Also, it is known to execute applications faster than traditionally used Hadoop. Also cloud technology provides flexible resource utilization environment on-demand. When analyzing big data using Spark in existing environments, it is difficult to provision resources according to the system's changing environment and the influence of other users' executions. Using cloud technology however, it is possible to provision resources more effectively for the execution of jobs through dynamic resource provision methods. In this paper, we propose an auto-scaling framework with corresponding algorithms to manage resources dynamically in virtual environments, in order to meet user-specified SLA (Service Level Agreement) given a set of limited resources. Our experimental results on Spark in OpenStack demonstrate the effectiveness of scaling resources to satisfy user SLAs. © 2016 IEICE.
- Files in This Item
-
Go to Link
- Appears in
Collections - 공과대학 > 소프트웨어학부 > 1. Journal Articles
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.