A cloud reservation system for big data applications
Loading...
Files
Accepted Version
Date
2017-03
Authors
Marinescu, Dan C.
Paya, Ashkan
Morrison, John P.
Journal Title
Journal ISSN
Volume Title
Publisher
IEEE
Published Version
Abstract
Emerging Big Data applications increasingly require resources beyond those available from a single server and may be expressed as a complex workflow of many components and dependency relationships-each component potentially requiring its own specific, and perhaps specialized, resources for its execution. Efficiently supporting this type of Big Data application is a challenging resource management problem for existing cloud environments. In response, we propose a two-stage protocol for solving this resource management problem. We exploit spatial locality in the first stage by dynamically forming rack-level coalitions of servers to execute a workflow component. These coalitions only exist for the duration of the execution of their assigned component and are subsequently disbanded, allowing their resources to take part in future coalitions. The second stage creates a package of these coalitions, designed to support all the components in the complete workflow. To minimize the communication and housekeeping overhead needed to form this package of coalitions, the technique of combinatorial auctions is adapted from market-based resource allocation. This technique has a considerably lower overhead for resource aggregation than the traditional hierarchically organized models. We analyze two strategies for coalition formation: the first, history-based uses information from past auctions to pre-form coalitions in anticipation of predicted demand; the second one is a just-in-time-that builds coalitions only when support for specific workflow components is requested.
Description
Keywords
Big Data , Cloud computing , Combinatorial mathematics , Just-in-time , Network servers , Protocols , Resource allocation , Cloud reservation system , Big Data applications , Component-dependency relationships , Resource management , Cloud environments , Two-stage protocol , Dynamically forming rack-level server coalitions , Workflow component execution , Combinatorial auctions , Market-based resource allocation , Servers , Organizations , Computers , Google , Cloud resource management , Hierarchical organization
Citation
Marinescu, D. C., Paya, A. and Morrison, J. P. (2017) 'A Cloud Reservation System for Big Data Applications', IEEE Transactions on Parallel and Distributed Systems, 28(3), pp. 606-618. DOI: 10.1109/TPDS.2016.2594783
Collections
Copyright
© 2017, IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.