In our previous episodes of the AI Show, we’ve learned all about the Azure Anomaly detector, how to bring the service on premises, and some awesome tips and tricks for getting the service to work well for you.

In this episode of the AI Show, Qun Ying shows us how to build an end-to-end solution using the Anomaly Detector and Azure Databricks. This step by step demo detects numerical anomalies from streaming data coming through Azure Event Hubs.

Anomaly Detection on Streaming Data Using Azure Databricks Related Links

David Giard recently posted a how-to article on creating an Azure DataBricks service. Check it out!

Azure Databricks is a web-based platform built on top of Apache Spark and deployed to Microsoft’s Azure cloud platform. Databricks provides a web-based interface that makes it simple for users to create and scale clusters of Spark servers and deploy jobs and Notebooks to those clusters. Spark provides a […]

CloudAcademy has an intro piece Apache Spark on Azure DataBricks.

Apache Spark is an open-source framework for doing big data processing. It was developed as a replacement for Apache Hadoop’s MapReduce framework. Both Spark and MapReduce process data on compute clusters, but one of Spark’s big advantages is that it does in-memory processing, which can be orders of magnitude faster than the disk-based processing that MapReduce uses. There are plenty of other differences between the two systems, as well, but we don’t need to go into the details here.

Databricks first introduced MLflow in last June. Immediately, startups and larger enterprises started using it to manage their machine learning lifecycles. Since then, more than 80 contributors from some 40 companies have worked on the open source machine learning tool, and it regularly sees more than 500,000 downloads per month.

And check out this recent news:

Unveiled at the Spark + AI Summit 2019, sponsored by Databricks, the new Databricks and Microsoft collaboration is a sign of the companies’ deepening ties, but it is also too new to say how effectively the partnership will advance MLflow for developers, said Mike Gualtieri, a Forrester analyst.

In this talk, Andrei Varanoch demonstrates the blueprint for such a Lambda Architecture implementation in Microsoft Azure, with Azure Databricks — a PaaS Spark offering – as a key component.  The term “Lambda Architecture” stands for a generic, scalable and fault-tolerant data processing architecture. As the hyper-scale now offers a various PaaS services for data ingestion, storage and processing, the need for a revised, cloud-native implementation of the lambda architecture is arising.