Chiminey: A user-friendly platform that helps scientists process vast quantities of data
As information technology advances at pace, scientists have more opportunities to study new phenomena and discover new natural laws. To fully seize such opportunities, however, they must grasp what these new technologies can do and how research can exploit them. This is a demanding and time-consuming process for anyone without computer science expertise.
Now, help is at hand, with the user-friendly Chiminey platform, which provides a sophisticated computing and data management service. Ian Thomas and his colleagues at RMIT University in Melbourne, Australia, developed Chiminey. They present it in the Elsevier journal Big Data Research, along with a hands-on tutorial on how to use it.
Chiminey targets scientists performing experiments that involve big data – computations and datasets too large to execute on a regular PC or laptop. These scientists often employ cloud computing – shared, internet-based processing resources and data – and high-performance computing (HPC), using extremely fast supercomputers.
These days, experiments with such high data demands are common in many scientific domains, including quantum mechanics, weather forecasting, climate research, oil and gas exploration, and molecular modelling. High-profile examples include the Large Hadron Collider, the world’s largest and most powerful particle collider, based at CERN, and the Sloan Digital Sky Survey, which processes astronomical data.
Thomas notes that scientists really want to engage with cloud and high-performance computing but are “often stymied by the initial learning curve associated with the new tools and techniques.” When using cloud computing platforms, for example, a scientist needs to learn how to set up an ‘execution environment’. That is, to create and set up virtual machines, collect the experiment’s results, and then destroy the virtual machines.
The Chiminey platform works by providing a set of drop-in components called ‘smart connectors’. Each one supports a certain computation type and provides a particular computing infrastructure. Smart connectors can interact with a service in the cloud and coordinate a cloud-based infrastructure for the user.
“The platform uses smart connectors that allow a researcher’s computational tasks to be packaged up and deployed on a number of different cloud computing and HPC platforms,” Thomas explains. As a result, the scientists can process their data without needing to know the execution environment’s exact set-up.
As well as being user-friendly, smart connectors are adaptable for different tasks. “Smart connectors can be flexibly adapted and extended for addressing researchers’ specific problems,” Thomas adds.
Free online until 31st December 2017.
Thomas, I.E., et al.: “Chiminey: Connecting scientists to HPC, cloud and big data,” Big Data Research (2017)