High performance hardware accelerator for optimizing Hadoop

摘要：

Today, data analysts in Hadoop world are facing performance
issues from processing large volumes of data. The amount of
data is rising so does the requirement of server performance.
The scientists working with Hadoop also have been focusing
on software solutions for performance improvement and have
delivered quite a lot of solutions. We are trying a
different angle of hardware machine architectural perspective
to improve performance of the big data processing.
Within Hadoop architecture, data I/O could often be the main
bottleneck during data analysis processing. We develop 2
MapReduce-based hardware-accelerators: one is to speed up a
simple data processing case by reducing I/O rate. The other
is a computational case of Monte Carlo method for the
approximation of Pi. We also build a compression/
decompression hardware accelerator to offload CPU in Hadoop
compression/decompression scenario.
Data scientists often use Apache Hive to explore data and
make queries. Hive accelerator working with Tez/LLAP and
even Spark is also our main development target.
This presentation will describe how hardware accelerated
Hadoop works, how it’s developed to improve Hadoop
performance and how much it gains for data analysts.

講者簡介：

張智淳 Peter Chang
研發/銷售/創業家, 前IBM技術專案經理

2015 商周奇點創新大賽決賽
2011 IBM Eminence and Excellence Awards (Team Awards)
2010 The Best of IBM (Outstanding Performance in ServerProven Planning)
2009 台灣國際專案管理師協會 認證講師 (專案管理實務)

• ← 利用 Jupyter + Ansible 快速佈署 Spark
• From Understanding to Problem Solving： Applying Deep Learning to Defend Against the Deceptive Advertising and Phone Scams →