A cloud reservation system for big data applications

Simple item page
dc.contributor.authorMarinescu, Dan C.
dc.contributor.authorPaya, Ashkan
dc.contributor.authorMorrison, John P.
dc.contributor.funderNational Science Foundationen
dc.contributor.funderEuropean Commissionen
dc.contributor.funderIDA Irelanden
dc.contributor.funderEnterprise Irelanden
dc.date.accessioned2019-08-29T14:19:09Z
dc.date.available2019-08-29T14:19:09Z
dc.date.issued2017-03
dc.date.updated2019-08-29T14:08:37Z
dc.description.abstractEmerging Big Data applications increasingly require resources beyond those available from a single server and may be expressed as a complex workflow of many components and dependency relationships-each component potentially requiring its own specific, and perhaps specialized, resources for its execution. Efficiently supporting this type of Big Data application is a challenging resource management problem for existing cloud environments. In response, we propose a two-stage protocol for solving this resource management problem. We exploit spatial locality in the first stage by dynamically forming rack-level coalitions of servers to execute a workflow component. These coalitions only exist for the duration of the execution of their assigned component and are subsequently disbanded, allowing their resources to take part in future coalitions. The second stage creates a package of these coalitions, designed to support all the components in the complete workflow. To minimize the communication and housekeeping overhead needed to form this package of coalitions, the technique of combinatorial auctions is adapted from market-based resource allocation. This technique has a considerably lower overhead for resource aggregation than the traditional hierarchically organized models. We analyze two strategies for coalition formation: the first, history-based uses information from past auctions to pre-form coalitions in anticipation of predicted demand; the second one is a just-in-time-that builds coalitions only when support for specific workflow components is requested.en
dc.description.sponsorshipEnterprise Ireland (Irish Centre for Cloud Computing and Commerce); IDA Ireland (Irish Centre for Cloud Computing and Commerce)en
dc.description.statusPeer revieweden
dc.description.versionAccepted Versionen
dc.format.mimetypeapplication/pdfen
dc.identifier.citationMarinescu, D. C., Paya, A. and Morrison, J. P. (2017) 'A Cloud Reservation System for Big Data Applications', IEEE Transactions on Parallel and Distributed Systems, 28(3), pp. 606-618. DOI: 10.1109/TPDS.2016.2594783en
dc.identifier.doi10.1109/TPDS.2016.2594783en
dc.identifier.endpage618en
dc.identifier.issn1045-9219
dc.identifier.issued3en
dc.identifier.journaltitleIEEE Transactions On Parallel and Distributed Systemsen
dc.identifier.startpage606en
dc.identifier.urihttps://hdl.handle.net/10468/8414
dc.identifier.volume28en
dc.language.isoenen
dc.publisherIEEEen
dc.relation.projectinfo:eu-repo/grantAgreement/NSF//1525943/US/AF: Small: Is the Simulation of Quantum Many-Body Systems Feasible on the Cloud?/en
dc.relation.projectinfo:eu-repo/grantAgreement/EC/H2020::RIA/643946/EU/Self-Organising, Self-Managing Heterogeneous Cloud/CloudLightningen
dc.relation.urihttps://ieeexplore.ieee.org/document/7523396
dc.rights© 2017, IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.en
dc.subjectBig Dataen
dc.subjectCloud computingen
dc.subjectCombinatorial mathematicsen
dc.subjectJust-in-timeen
dc.subjectNetwork serversen
dc.subjectProtocolsen
dc.subjectResource allocationen
dc.subjectCloud reservation systemen
dc.subjectBig Data applicationsen
dc.subjectComponent-dependency relationshipsen
dc.subjectResource managementen
dc.subjectCloud environmentsen
dc.subjectTwo-stage protocolen
dc.subjectDynamically forming rack-level server coalitionsen
dc.subjectWorkflow component executionen
dc.subjectCombinatorial auctionsen
dc.subjectMarket-based resource allocationen
dc.subjectServersen
dc.subjectOrganizationsen
dc.subjectComputersen
dc.subjectGoogleen
dc.subjectCloud resource managementen
dc.subjectHierarchical organizationen
dc.titleA cloud reservation system for big data applicationsen
dc.typeArticle (peer-reviewed)en
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
1432.pdf
Size:
442.99 KB
Format:
Adobe Portable Document Format
Description:
Accepted Version
Download
License bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
license.txt
Size:
2.71 KB
Format:
Item-specific license agreed upon to submission
Description:
Download
Collections
Computer Science - Journal Articles