Abstract

Big data workflows often require the assembly and exchange of complex, multi-element datasets. For example, in biomedical applications, the input to an analytic pipeline can be a dataset consisting thousands of images and genome sequences assembled from diverse repositories, requiring a description of the contents of the dataset in a concise and unambiguous form. Typical approaches to creating datasets for big data workflows assume that all data reside in a single location, requiring costly data marshaling and permitting errors of omission and commission because dataset members are not explicitly specified. We address these issues by proposing simple methods and tools for assembling, sharing, and analyzing large and complex datasets that scientists can easily integrate into their daily workflows. These tools combine a simple and robust method for describing data collections (BDBags), data descriptions …

Metadata

publication

2016 Ieee international conference on big data (big data), 319-328, 2016

year

2016

publication date

2016/12/5

authors

Kyle Chard, Mike D'Arcy, Ben Heavner, Ian Foster, Carl Kesselman, Ravi Madduri, Alexis Rodriguez, Stian Soiland-Reyes, Carole Goble, Kristi Clark, Eric W Deutsch, Ivo Dinov, Nathan Price, Arthur Toga

link

https://ieeexplore.ieee.org/abstract/document/7840618/

resource_link

https://research.manchester.ac.uk/files/45989205/bagminid.pdf

conference

2016 Ieee international conference on big data (big data)

pages

319-328

publisher

IEEE