SKA and Amazon Web Services have teamed up to offer an AstroCompute in the Cloud grant.

Square Kilometre Array (SKA) Organisation is teaming up with Amazon Web Services (AWS) to use innovative cloud computing solutions to explore ever-increasing amounts of astronomy data in ways that were previously unimaginable. Located in Australia and Africa, the SKA will be the world’s largest radio telescope and is often considered to be the largest public science Big Data project, set to ultimately produce massive amounts of data –several times the global Internet traffic.
SKA Organisation Headquarters, Tuesday 28 April - Today, SKA Organisation and AWS are launching the AstroCompute in the Cloud grant program to accelerate the development of innovative tools and techniques for processing, storing and analysing the global astronomy community’s vast amounts of astronomic data in the cloud. Grant recipients will have access to credits for AWS cloud services over a two-year period and up to one petabyte (PB) of storage for data contributed by SKA partners, which AWS will make available as a public dataset. Anyone associated with or using radio astronomical telescopes or radio astronomical data resources around the world is welcome to apply.
"With the SKA, we will be generating more data than the entire Internet traffic at any single time," said Tim Cornwell, the SKA Organisation Architect and administrator of the grant. "So we're looking into innovative cloud solutions to help us cope with never-seen-before volumes of data, using techniques that are yet to be invented."
"This is an exciting opportunity, not only for our partner institutions, but for all companies and research facilities around the world dealing with astronomy data,” said Professor Philip Diamond, SKA Organisation Director-General. “The call is to help us explore how cloud computing can help process the data deluge we are expecting in astronomy in the 21st century – and in particular with the SKA."
In its first phase of construction, SKA will include two game-changing telescopes, one consisting of more than one hundred thousand so-called ‘dipole’ antennas, and one with about two hundred large dishes. Supercomputers will translate the enormous volume of raw data coming from the telescopes into a useable form for astronomers. With observations expected to run full-time, data will flow continuously and supercomputers will process it on the fly, transmitting useful data to an archive and deleting contaminated or otherwise unnecessary data in real time. To handle the data, and develop the know-how to process it, new smart algorithms and software will be required.
“Through our Scientific Computing program, our grants and our public datasets, we’ve found that when researchers have access to the tools and data they need, they find innovative ways of solving big data challenges,” said Jamie Kinney, senior manager for scientific computing, Amazon Web Services, Inc. “The SKA is an ambitious project which presents an unprecedented opportunity to leverage a tremendous amount of data to explore the universe.”
Beyond the field of astronomy the development of cloud processing and data analysis and visualization tools is certain to have major applications in everyday life. Supercomputing is increasingly used by pharmaceutical companies to design better drugs, by weather forecasting to refine predictions up to a week in advance, and by engineers to design smarter infrastructure.
"There's an increasingly strong link between fundamental research and computing, with all the potential spinoffs benefitting society that come with it,” said Tim Cornwell. “CERN, the European Organization for Nuclear Research, realised very early they would face a challenge to distribute the amount of data from their experiments to physicists around the world. To solve it, they created the World Wide Web. SKA is the next step."
Statements from SKA partners
Peter Quinn, Executive Director of the International Centre for Radio Astronomy Research (ICRAR), Perth, Australia: “We're pleased to see Amazon Web Services support the Square Kilometre Array project. ICRAR has been actively using AWS for several years to prototype data and processing systems for the SKA and to demonstrate the benefits of cloud technologies for radio astronomy. This is a great example of how the SKA and industry are working together to innovate in areas that will not only help science but also generate down-to-earth benefits for the global community."
Lewis Ball, Director of CSIRO Astronomy and Space Science, Marsfield, Australia: "The management of big data is now a challenge faced by researchers worldwide. For example, once fully operational, our ASKAP telescope (one of the SKA precursor telescopes) will generate about five petabytes of data per year. Big computing resources will support this vital scientific research, expand capabilities and enable exciting new discoveries - not only for astronomy but also other data-intensive investigations right across the scientific spectrum."
Justin Jonas, Associate Director at SKA South Africa: “The computing needs of the SKA and its pathfinder and precursor facilities present some problems that are unique to radio astronomy, but others are common with other Big Data and High Performance Computing applications. One of our current challenges is to identify the most appropriate compute platforms for these two classes of applications. Cloud computing is an attractive option that is already being used to good effect by MeerKAT (one of the SKA precursor telescopes) scientists, engineers and software developers.The AWS grant will allow us to fully explore the capabilities of cloud computing in the context of MeerKAT and SKA data processing and delivery. We foresee that students in our Human Capital Development Programme will also benefit from this grant, giving them first-hand experience in this cutting edge computing environment.”
Mike Garrett, ASTRON Director, the Netherlands: "It makes sense for a globally distributed project like the SKA to be an early adopter of cloud technology. The cloud will impact every possible aspect of the project, from telescope maintenance and operations, to collaborative data sharing and the nature and process of scientific discovery itself.”
Brian Glendenning, Head of the Data Management and Software Department at the NRAO: "The National Radio Astronomy Observatory (NRAO) is pleased that grants are available to all radio astronomy users, including users with NRAO data. NRAO has recently started a pilot project to configure and implement a supported instance of its radio interferometric data reduction software package (CASA) on AWS. NRAO will be able to assist (e.g., via providing supported CASA virtual machines with tuned parallelization parameters) the radio community as it makes the transition to this era of on demand computing."