TIIS (한국인터넷정보학회)
Current Result Document :
| 한글제목(Korean Title) |
An Analytic solution for the Hadoop Configuration Combinatorial Puzzle based on General Factorial Design |
| 영문제목(English Title) |
An Analytic solution for the Hadoop Configuration Combinatorial Puzzle based on General Factorial Design |
| 저자(Author) |
R. Sathia Priya
A. John Prakash
V. Rhymend Uthariaraj
|
| 원문수록처(Citation) |
VOL 16 NO. 11 PP. 3619 ~ 3637 (2022. 11) |
한글내용
(Korean Abstract) |
|
영문내용
(English Abstract) |
Big data analytics offers endless opportunities for operational enhancement by extracting valuable insights from complex voluminous data. Hadoop is a comprehensive technological suite which offers solutions for the large scale storage and computing needs of Big data. The performance of Hadoop is closely tied with its configuration settings which depends on the cluster capacity and the application profile. Since Hadoop has over 190 configuration parameters, tuning them to gain optimal application performance is a daunting challenge. Our approach is to extract a subset of impactful parameters from which the performance enhancing sub-optimal configuration is then narrowed down. This paper presents a statistical model to analyze the significance of the effect of Hadoop parameters on a variety of performance metrics. Our model decomposes the total observed performance variation and ascribes them to the main parameters, their interaction effects and noise factors. The method clearly segregates impactful parameters from the rest. The configuration setting determined by our methodology has reduced the Job completion time by 22%, resource utilization in terms of memory and CPU by 15% and 12% respectively, the number of killed Maps by 50% and Disk spillage by 23%. The proposed technique can be leveraged to ease the configuration tuning task of any Hadoop cluster despite the differences in the underlying infrastructure and the application running on it. |
| 키워드(Keyword) |
ANOVA
Big data
Configuration tuning
General Factorial Design
Hadoop
Impactful parameters
|
| 파일첨부 |
PDF 다운로드
|
