| Title | Generalizing MapReduce as a Unified Cloud and HPC Runtime |
| Publication Type | Conference Proceedings |
| Year of Publication | 2011 |
| Date Published | 11/14/2011 |
| Authors | Qiu, J. |
| Refereed Designation | Unknown |
| Conference Name | Petascale Data Analytics: Challenges, and Opportunities Workshop(PDAC-11) at The International Conference for High Performance Computing, Networking, Storage and Analysis (SC11) |
| Series Title | Proceedings of the Petascale Data Analytics: Challenges, and Opportunities |
| Conference Location | Seattle, Washington, USA |
| Publication Language | eng |
| Keywords | Cloud, data analysis, fault tolerance, HPC, Interoperability, Iterative MapReduce, Map Collective, scaling performance |
| Abstract | Computational simulation and analysis were one of the keys to the future in data-intensive science as a “fourth paradigm” of scientific discovery but facing a major challenge as handling the incredible increases in dataset sizes. This requires attractive powerful programming models that address issues of portability with scaling performance and fault tolerance. Further, one must meet these challenges for both computation and storage. We build on the success of our research on Iterative MapReduce with successful prototypes Twister (on HPC) and Twister4Azure (on clouds). We have designed a novel Map Collective runtime which generalizes previous work in both HPC and MapReduce communities, which we hypothesize can be used as the runtime for data analysis (mining) interoperably between clouds and clusters. |
| URL | Follow Link |
