ETL – Frank's World of Data Science & AI

Azure Data Factory
Closed
18 sec read

How to Get Metadata Activity in Azure Data Factory

Azure Data Factory is an amazingly powerful tool that can accomplish just about every ETL/ELT task.

In this video, Frank Liu shows how to use the Get Metadata activity to get a list of file names.

Azure Data
Closed
31 sec read

The Modern Data Warehouse in Azure – Data Processing

In this video, Chris Seferlis continues discussing the Modern Data Platform in Azure with Part 3: Data Processing.

Tools Discusssed:

Azure Data Factory Data Flows – https://docs.microsoft.com/en-us/azure/data-factory/concepts-data-flow-overview
Azure Databricks – https://azure.microsoft.com/en-us/services/databricks/
Azure HDInsight – https://azure.microsoft.com/en-us/services/hdinsight/
Azure Marketplace – https://azuremarketplace.microsoft.com/en-us/marketplace/

Azure Big Data Databricks
Closed
24 sec read

How to Build a Cloud Data Platform with Databricks Part 2 – ETL Processing

Learn how to use Apache Spark and Delta Lake on Databricks to perform ETL processing, manage late arriving data, and repair corrupted data.

Companies look to support both business analytics and machine learning initiatives within their organization, but often face challenges with complex operations, proprietary technologies, and unreliable data.

Azure Big Data
Closed
26 sec read

Introduction to the Modern Data Warehouse in Azure

Chris Seferlis will be publishing on the modern data warehouse in Azure.

Here he starts with an overview of the stages of a data warehouse and the implications of ELT vs ETL as we move from sources, ingestion, storage, transformation, staging and presentation.

Learn more:

https://azure.microsoft.com/en-in/solutions/architecture/modern-data-warehouse/

Azure SQL Server
Closed
40 sec read

Azure Synapse Analytics – Next-gen Azure SQL Data Warehouse

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs. All of this leverages our limitless Azure Data Lake Storage service for any type of data.

Microsoft Mechanics explains.

Azure Data Data Science
Closed
15 sec read

Kappa vs Lambda Architecture

Chris Seferlis describes some key differences between the Kappa and Lambda Architectures, advantages and disadvantages of each, and why you might choose one over the other on the Azure platform.

Big Data Data Data Driven
Closed
29 sec read

Trey Johnson on ETL, Data, and Fixing Up Old Cars

In this episode of Data Driven, Andy and I talk to Trey Johnson about ETL, Data, and fixing up old cars. Also, I apologize for the delay in production for this episode

Press the play button below to listen here or visit the show page at DataDriven.tv

Big Data Spark
Closed
57 sec read

Azure Databricks for Data Engineers and Data Developers

Data engineering is about 70% of any data pipeline today, and without having the experience to implement a data engineering pipeline well, there is no value to be accumulated from your data.

In this session from Microsoft Ignite we discuss the best practices and demonstrate how a data engineer can develop and orchestrate the big data pipeline, including: data ingestion and orchestration using Azure Data Factory; data curation, cleansing and transformation using Azure Databricks; data loading into Azure SQL Data Warehouse for serving your BI tools.

Watch and learn how to effectively do the ETL/ELT process combined with advanced capabilities such as monitoring the jobs, getting alerts, jobs retrial, set permissions, and much more.

Data
Closed
1 min read

The New Import Flat File Wizard in SSMS 17.3

The latest episode of Data Exposed welcomes Alan Yu, a PM in the SQL Server engineering group focusing on SQL Tools. He shows off the latest SQL Server Management Studio 17.3 update.

Alan starts off by highlighting two new features in the latest release: Import Flat File Wizard and XEvent Profiler. Import Flat File Wizard is a new tool that helps streamline the import experience for a user who wants to import .csv or .txt files without specialized domain knowledge. The XEvent profiler is a live viewer window of extended events that can be quickly launched. For the purpose of this demo, Alan chose to focus on showcasing the Import Flat File Wizard.

To learn more about SSMS 17.3, check out this blog post: https://blogs.technet.microsoft.com/dataplatforminsider/2017/10/10/whats-new-in-sql-server-management-studio-17-3/

Download the latest version of SSMS: https://docs.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms

To learn more about PROSE, check out: https://microsoft.github.io/prose/

Tag: ETL