Dfs-Container: Achieving Containerized Block I/O For Distributed File Systems

Keywords

CGroups; HDFS; Proportional I/O Allocation

Abstract

Today BigData systems commonly use resource management systems such as TORQUE, Mesos, and Google Borg to share the physical resources among users or applications. Enabled by virtualization, users can run their applications on the same node with low mutual interference. Container-based virtualizations (e.g., Docker and Linux Containers) offer a lightweight virtualization layer, which promises a near-native performance and is adopted by some Big-Data resource sharing platforms such as Mesos. Nevertheless, using containers to consolidate the I/O resources of shared storage systems is still at an early stage, especially in a distributed file system (DFS) such as Hadoop File System (HDFS). To overcome this issue, we propose a distributed middleware system, DFS-Container, by further containerizing DFS.We also evaluate and analyze the unfairness of using containers to proportionally allocate the I/O resource of DFS. Based on these analyses and evaluations, we propose and implement a new mechanism, IOPS-Regulator, which improve the fairness of proportional allocation by 74.4% on average.

Publication Date

9-24-2017

Publication Title

SoCC 2017 - Proceedings of the 2017 Symposium on Cloud Computing

Number of Pages

660-

Document Type

Article; Proceedings Paper

Personal Identifier

scopus

DOI Link

https://doi.org/10.1145/3127479.3132568

Socpus ID

85032439257 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/85032439257

This document is currently not available here.

Share

COinS