Chris Seferlis introduce us to the newly added Apache Spark Pools in Azure Synapse Analytics for Big Data, Machine Learning, and Data Processing needs.

From the description:

I give an overview of what Spark is, and where it came from; why the Synapse Team added it to the suite of offering, and some sample workloads why you might use it.In this video I introduce the newly added Apache Spark Pools in Azure Synapse Analytics for Big Data, Machine Learning, and Data Processing needs. I give an overview of what Spark is, and where it came from; why the Synapse Team added it to the suite of offering, and some sample workloads why you might use it.

Delta Lake is an open-source storage management system (storage layer) that brings ACID transactions and time travel to Apache Spark and big data workloads.

The latest and greatest of Delta Lake 0.7.0 requires Apache Spark 3 and among the features is a full coverage of SQL DDL and DML commands.

One of the new features of Synapse Analytics is Synapse Link – the ability to query a live analytics store within CosmosDB with only tiny amounts of setup. We’ve recently seen it rolled out for the SQL On-Demand endpoint, meaning we can write both Spark and SQL directly over this analytics store!

In today’s video, Simon demonstrates how we can use Synapse Link to build up a Lambda Architecture, which enables near real-time querying with relatively little fuss!

More information on Synapse Link can be found here: https://azure.microsoft.com/en-us/updates/azure-synapse-link-for-azure-cosmos-db-sql-serverless-runtime-support-in-preview/

For the OG Lambda Architecture, check out Nathan Marz’s book “Big Data” here – https://www.manning.com/books/big-data

Learn about seven different database paradigms and what they do best.

Contents:

  • 00:00 Intro
  • 00:45 Key-value
  • 01:48 Wide Column
  • 02:47 Document
  • 04:05 Relational
  • 06:21 Graph
  • 07:22 Search Engine
  • 08:27 Multi-model

The Microsoft Azure channel explains how KPMG Japan uses Azure Arc to build out a seamless data solution.

KPMG Ignition Tokyo, the centerpiece of KPMG Japan’s digital strategy, delivers specialty software solutions to its global clients. With a multi-cloud and hybrid approach, the firm is rolling out its next-generation, AI-based audit software built on Azure, and implementing Azure Arc to deliver seamless solutions for clients across multiple hybrid data estates.

Adam Marczak explains Azure Data Factory Mapping Data Flow in this video.

With Azure Data Factory Mapping Data Flow, you can create fast and scalable on-demand transformations by using visual user interface. In just minutes you can leverage power of Spark with not a single line of code written.

In this episode I give you introduction to what Mapping Data Flow for Data Factory is and how can it solve your day to day ETL challenges. In a short demo I will consume data from blob storage, transform movie data, aggregate it and save multiple outputs back to blob storage.

Sample code and data: https://github.com/MarczakIO/azure4everyone-samples/tree/master/azure-data-factory-mapping-data-flows 

Learn how to extract value from your data to bring the impact of your low-code solutions to a whole new level.

PowerApps already enable creation of useful business applications with minimal effort.

In this session, you will learn about how and why to connect your applications to Azure services responsible for Big Data.

You will see an example of an application that keeps track of NYC taxi logs and provides logistical information for greater business insights. You will leave this session with confident understanding of what Big Data connection options PowerApps provide, how to connect your application to Big Data, as well as how to reference and visualize it.

Additional Resources: Power Apps Devs