For Those Building New Apps in the Cloud

Azure Cloud on Ulitzer

Subscribe to Azure Cloud on Ulitzer: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get Azure Cloud on Ulitzer: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Azure Authors: Elizabeth White, Liz McMillan, Yeshim Deniz, Greg Schulz, Todd Matters

Related Topics: Cloud Computing, Apache Web Server Journal, Azure Cloud on Ulitzer, Green Storage Journal, Microsoft Developer, CIO/CTO Update, Java in the Cloud, Big Data on Ulitzer

Blog Feed Post

Windows Azure Recipe: Big Data

We know that there are patterns hidden inside this data that might provide competitive insight into market trends

As the name implies, what we’re talking about here is the explosion of electronic data that comes from huge volumes of transactions, devices, and sensors being captured by businesses today. This data often comes in unstructured formats and/or too fast for us to effectively process in real time. Collectively, we call these the 4 big data V’s: Volume, Velocity, Variety, and Variability. These qualities make this type of data best managed by NoSQL systems like Hadoop, rather than by conventional Relational Database Management System (RDBMS).

We know that there are patterns hidden inside this data that might provide competitive insight into market trends.  The key is knowing when and how to leverage these “No SQL” tools combined with traditional business such as SQL-based relational databases and warehouses and other business intelligence tools.

Drivers

  • Petabyte scale data collection and storage
  • Business intelligence and insight

Solution

The sketch below shows one of many big data solutions using Hadoop’s unique highly scalable storage and parallel processing capabilities combined with Microsoft Office’s Business Intelligence Components to access the data in the cluster.

image

Ingredients

  • Hadoop – this big data industry heavyweight provides both large scale data storage infrastructure and a highly parallelized map-reduce processing engine to crunch through the data efficiently. Here’s are the key pieces of the environment:
    • Pig - a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs.
    • Mahout - a machine learning library with algorithms for clustering, classification and batch based collaborative filtering that are implemented on top of Apache Hadoop using the map/reduce paradigm.
    • Hive - data warehouse software built on top of Apache Hadoop that facilitates querying and managing large datasets residing in distributed storage. Directly accessible to Microsoft Office and other consumers via add-ins and the Hive ODBC data driver.
    • Pegasus - a Peta-scale graph mining system that runs in parallel, distributed manner on top of Hadoop and that provides algorithms for important graph mining tasks such as Degree, PageRank, Random Walk with Restart (RWR), Radius, and Connected Components.
    • Sqoop - a tool designed for efficiently transferring bulk data between Apache Hadoop and structured data stores such as relational databases.
    • Flume - a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large log data amounts to HDFS.
  • Database – directly accessible to Hadoop via the Sqoop based Microsoft SQL Server Connector for Apache Hadoop, data can be efficiently transferred to traditional relational data stores for replication, reporting, or other needs.
  • Reporting – provides easily consumable reporting when combined with a database being fed from the Hadoop environment.

Training

These links point to online Windows Azure training labs where you can learn more about the individual ingredients described above.

image Hadoop Learning Resources (20+ tutorials and labs)

Huge collection of resources for learning about all aspects of Apache Hadoop-based development on Windows Azure and the Hadoop and Windows Azure Ecosystems

SQL Azure (7 labs)

Microsoft SQL Azure delivers on the Microsoft Data Platform vision of extending the SQL Server capabilities to the cloud as web-based services, enabling you to store structured, semi-structured, and unstructured data.

See my Windows Azure Resource Guide for more guidance on how to get started, including links web portals, training kits, samples, and blogs related to Windows Azure.

More Stories By Clint Edmonson

Clint Edmonson is an Architect Evangelist for Microsoft focused on Windows Azure and Cloud Computing. He has been developing software with Microsoft technologies for the last 20 years, working on everything from custom shrink-wrapped software at small startups to enterprise architecture and methodology adoption at fortune 500 companies. He spends his days helping companies understand Microsoft’s latest developer cloud offerings and developing strategic roadmaps for their adoption. He lives in St. Louis, Missouri and is the host of www.notsotrivial.net. You can follow Clint on twitter at @clinted.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.