Advancing Analytics answers the question that has come up a few times in my day to day work: it looks great, but how much is this going to cost?

There has been a whole lot of excitement around Azure Purview, the new governance and classification tool in Azure, but also a few worried noises about the cost.

In today’s video, Simon looks through the available cost information for Purview, talks about what that means for the product’s target audience and what we can see about the future state.

The Purview Pricing page can be found here – https://azure.microsoft.com/en-us/pricing/details/azure-purview/

Advancing Analytics takes a closer look at the two new runtimes available for Databricks.

We have not just one but two new Databricks Runtimes currently in preview – 7.6 brings several new features focussing on making Autoloader more flexible, improving performance of Optimize and Structured Streaming.

Runtime 8.0 is a much wider change, seeing the shift to Spark 3.1, introducing new language versions for Python, Scala and R.

This shift brings a large swathe of functionality, performance and feature changes, so take some time to look through the docs.

Simon walks through the high level notes, pulling out some interesting features and improvements.

Simon from Advancing Analytics explores the Atlas API that’s exposed under the covers of the new Azure Purview data governance offering.

There are a couple of different libraries available currently, so don’t be surprised if we see a lot of shifts & changes as the preview matures!

In this video, Simon takes a look at how you can get started with the API in a Databricks Notebook to register a custom lineage between two entities

For more info on the pyapacheatlas library used, see: https://pypi.org/project/pyapacheatlas/