Page 18 - FCW, June 2017

Commentary|JED SUNDWALL
JED SUNDWALL leads the global open-data practice at Amazon Web Services.
The cloud takes open data to new heights
The volume of government data is growing exponentially, and agencies must make it easier for innovators to access and use it
The volume of data being pro- duced every day is growing at a rate of 2.5 exabytes — equivalent to 250,000 Libraries of Congress — every single day, according to IDC.
Much of that data is open, or usable by anyone for any purpose without licensing fees. It has been a boon for entrepreneurs, scien- tists and public servants who use it to create new products, acceler- ate scientific discovery and pro- vide better services to the public.
Unfortunately, the infrastructure typically used to serve that data
is not keeping up with growth. Users often have to download and store their own copies of the data. That’s fine with a few gigabytes, but as volumes increase, that approach doesn’t work.
For example, the National Oceanic and Atmospheric Admin- istration’s new weather satellite, GOES-16, produces an estimated 1 terabyte of data per day, over 100 times that of its predecessor. Very few people have the hard disks (or patience) to download a terabyte of data.
Using the old model of distribu- tion reduces the data’s value to end users and ultimately to the taxpayer.
When open data is made avail- able in the cloud, however, users can take advantage of on-demand computing resources to query as much or as little of the data as desired.
When their analysis is done, they can save the results and turn
off the virtual servers, never hav- ing to worry about owning a copy of the original data.
For example, the U.S. Geo- logical Survey has made Landsat images of the Earth’s land surface available at no cost since 2010. However, users have been lim- ited by their ability to download
Using the old model of distribution reduces the data’s value to end users and ultimately to the taxpayer.
and store significant amounts of data. End users told us they had big ideas about what they could do with Landsat data, but they couldn’t access it fast enough or couldn’t afford to store their own copies.
Amazon S3 started hosting imagery from the Landsat 8 satel- lite in 2015 and logged over 1 bil- lion requests from 147 countries in the first year. Businesses such as Esri, Mapbox and MathWorks immediately created tools to take advantage of the new, easy-to- access Landsat archive.
In addition, NOAA officials recognized early on that the cloud would be essential to fulfilling
their mission. In 2015, they made several hundred terabytes of high- resolution Nexrad data available in the cloud and immediately saw data use spike by 130 percent while simultaneously seeing a 50 percent decrease in the use of the agency’s servers.
That open-data initiative also made the full Nexrad archive avail- able on demand, thereby creating new analysis and discovery possi- bilities. For example, Eli Bridge at the University of Oklahoma used the public dataset to estimate the size and better understand purple martin roosts, which appear as ring-shaped patterns on radar images.
Once open data is uploaded to a public cloud, the potential uses are virtually endless, and agencies are just getting started.
Governments around the world are investing billions in new sen- sors. They range from internet of things devices in parking meters to Earth-observing satellites, which are producing huge volumes of data.
The best way to get a return on such investments is to make it easy for innovators to access the data and put it to work. With a modernized data distribution method and some imagination, open data can be unleashed to become a tremendous force for the public good. n
12 June 2017 FCW.COM