Skip to main navigation Skip to search Skip to main content

Towards enabling dynamic resource estimation and correction for improving utilization in an apache mesos cloud environment

  • State University of New York Binghamton University
  • Northeastern University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Academic cloud infrastructures require users to specify an estimate of their resource requirements. The resource usage for applications often depends on the input file sizes, parameters, optimization flags, and attributes, specified for each run. Incorrect estimation can result in low resource utilization of the entire infrastructure and long wait times for jobs in the queue. We have designed a Resource Utilization based Migration (RUMIG) system to address the resource estimation problem. We present the overall architecture of the two-stage elastic cluster design, the Apache Mesos-specific container migration system, and analyze the performance for several scientific workloads on three different cloud/cluster environments. In this paper we (b) present a design and implementation for container migration in a Mesos environment, (c) evaluate the effect of right-sizing and cluster elasticity on overall performance, (d) analyze different profiling intervals to determine the best fit, (e) determine the overhead of our profiling mechanism. Compared to the default use of Apache Mesos, in the best cases, RUMIG provides a gain of 65% in runtime (local cluster), 51% in CPU utilization in the Chameleon cloud, and 27% in memory utilization in the Jetstream cloud.

Original languageEnglish
Title of host publicationProceedings - 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages188-197
Number of pages10
ISBN (Electronic)9781728109121
DOIs
StatePublished - May 2019
Event19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2019 - Larnaca, Cyprus
Duration: May 14 2019 → May 17 2019

Publication series

NameProceedings - 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2019

Conference

Conference19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2019
Country/TerritoryCyprus
CityLarnaca
Period05/14/19 → 05/17/19

Keywords

  • Apache mesos
  • Chameleon cloud
  • Cloud optimization
  • Container migration
  • Docker
  • Estimation
  • Jetstream
  • Resource utilization
  • Right sizing

Fingerprint

Dive into the research topics of 'Towards enabling dynamic resource estimation and correction for improving utilization in an apache mesos cloud environment'. Together they form a unique fingerprint.

Cite this