Page 50 - MSDN Magazine, August 2017

AZURE
Batch Processing Using a
Serverless Architecture
Joseph Fultz
For Azure enterprise customers, a key challenge in managing their cloud footprint is the ability to control what they spend and to charge those costs back to the consumers. Fortu- nately, there are several vendors that provide tools, such as Cloud Cruiser, Cloudyn, and Cloudability, to help with collecting usage data and generating a rich set of reports. Additionally, you can find many good examples of how to pull data programmatically, such as the post from a former co-worker of mine, Ed Mondek, in which he shows how to pull data into Excel and view it (bit.ly/2rzDOPI). However, if you want to pull that data regularly and enable his- torical, present trend and predictive views, you need to store a lot more data. For a large enterprise with thousands of resources per subscription, that amount of data can be daunting and is certainly not what you’d want to fetch and keep on a local machine.
Luckily, there’s another way. In this article I’m going to walk you through the serverless Extract-Transform-Load (ETL) process I set up to extract such data, provide a little enrichment and store the data to a place where further work (analytics, map-reduce and so forth)
can be done. I’ll touch on the overall design, key decision points and important things to consider in taking a serverless approach.
Determining Consumption
The first decision is choosing between the Enterprise Agreement (EA) Billing API and the Azure Billing API, which centers its requests around specific subscriptions. My prototype is targeted at enterprise customers with multiple enrollments in an EA. In the scenario with which I’m working, subscriptions are being used as part of the management boundaries for both specific product groups and for separating production from non-production resources. This could result in a fairly high number of subscriptions in flux due to the volatile proof-of-concept (PoC) type of work being cre- ated as new groups and new product lines start up in Azure. Thus, I chose to work with the EA API because it reduced the scope of work in that I don’t have to create a discovery mechanism for subscriptions. This leaves me with the noted challenge of not having any data for subscriptions created outside of the enrollments for the enterprise. While this is an important area to tackle, it comes with a number of other process and management challenges that have to be solved organizationally and is outside the scope of the work I want to accomplish.
Requirements and Logical Flow
In any architecture, it’s the intersections between systems that require the most scrutiny in design and testing. A serverless architecture doesn’t change the need to consider the volume of data that moves
This article discusses:
• Design considerations in a serverless architecture
• Integrating multiple Azure Platform-as-a-Service capabilities • Billing data retrieval
Technologies discussed:
Azure CosmosDB, Azure Functions, Azure Blob Storage
44 msdn magazine