Page 16 - GCN, Jan/Feb 2016

EMERGING TECH
BY PATRICK MARSHALL
NASA and Esri speed delivery of cloud-based imagery data
IN DECEMBER 2015, NASA and Esri announced that they are publicly sharing a jointly developed raster file format and a unique compression program designed to handle large amounts of mission- critical imagery data.
As a result, that imag- ery can be moved to and from the cloud much, much faster, said Peter Becker, Esri’s imagery product manager. “In comparison to things like JPEG 2000, it’s something like 10 to 15 times faster,” he added.
“We’ve always had a concern over how we access large volumes of imagery, especially from cloud storage,” Becker
said, noting that storage- area networks and network-attached storage don’t scale well. “We real- ized we had to make use
of the elasticity and avail- ability of cloud storage,
but we couldn’t connect
our servers directly to cloud storage because our servers are expecting file shares. So we were trying to work out how to resolve that.”
Fortunately, a program- mer who had worked on the Meta Raster Format (MRF) at NASA recently took a po- sition at Esri and suggested that the format’s architec- ture might be just what the team was looking for.
“As we looked more at the
MRF format, we realized that what they had invented there was actually really simple,” Becker said.
MRF’s special power is that it breaks files into three parts that can be cached separately, making it espe-
The other challenge the Esri team faced was com- ing up with a compression strategy appropriate for critical geolocation data. Most “lossy” compression al- gorithms for imagery throw out data that the human eye
compression. LERC allows analysts to define a toler- ance for file compression. One might, for example, in- struct LERC to compress an elevation file as much as it can while ensuring accuracy within 10 centimeters.
As a bonus, Becker said the team soon real- ized that LERC, which was patented last year, not only worked well
on elevation files, it also worked very well in com- pressing high-bit-depth satellite imagery.
By using LERC within MRF, users not only get faster performance but they also lower their storage requirements. Becker estimated that using the technology with cloud storage costs ap- proximately one-third of what traditional file-based enterprise storage costs.
“This can result in very significant cost reductions for organizations with
large volumes of imagery,” he said.
In addition to releas-
ing the technologies to the public, Becker said Esri is continuing to work on im- proving them by developing better application program- ming interfaces. The team is also looking to incorporate new features into MRF, including versioning of imagery and time-tracking stamps. •
12 GCN JANUARY/FEBRUARY 2016 • GCN.COM
Image-compression technology and a raster format that breaks files into three parts promise to improve performance and reduce storage requirements for data-hungry visualizations.
cially well suited to cloud storage.
“Since the files are broken up, we can bring [down] the metadata files and store those locally,” he said. Users can examine data on file contents and download
the data-heavy portions only when needed. “We’re minimizing the number of requests that are going back and forth between the serv- ers and storage.”
won’t miss — hence, the “lossy” tag — but compress- ing elevation data can cause trouble.
“The problem with lossy compression is that you end up not knowing the accu- racy of your results,” Becker said. “If you’re flying a plane in the mountains and your elevation data is off by 20 meters, that is critical.”
The answer Esri devel- oped is limited error raster
EARTHDATA.NASA.GOV