One of the new features of Synapse Analytics is Synapse Link – the ability to query a live analytics store within CosmosDB with only tiny amounts of setup. We’ve recently seen it rolled out for the SQL On-Demand endpoint, meaning we can write both Spark and SQL directly over this analytics store!

In today’s video, Simon demonstrates how we can use Synapse Link to build up a Lambda Architecture, which enables near real-time querying with relatively little fuss!

More information on Synapse Link can be found here: https://azure.microsoft.com/en-us/updates/azure-synapse-link-for-azure-cosmos-db-sql-serverless-runtime-support-in-preview/

For the OG Lambda Architecture, check out Nathan Marz’s book “Big Data” here – https://www.manning.com/books/big-data

In this talk, Andrei Varanoch demonstrates the blueprint for such a Lambda Architecture implementation in Microsoft Azure, with Azure Databricks — a PaaS Spark offering – as a key component.  The term “Lambda Architecture” stands for a generic, scalable and fault-tolerant data processing architecture. As the hyper-scale now offers a various PaaS services for data ingestion, storage and processing, the need for a revised, cloud-native implementation of the lambda architecture is arising.