Here’s an interesting talk Albert Franziu Cros on a CI/CD setup composed by a Spark Streaming job in K8s consuming from Kafka.

Over the last year, we have been moving from a batch processing jobs setup with Airflow using EC2s to a powerful & scalable setup using Airflow & Spark in K8s.

The increasing need of moving forward with all the technology changes, the new community advances, and multidisciplinary teams, forced us to design a solution where we were able to run multiple Spark versions at the same time by avoiding duplicating infrastructure and simplifying its deployment, maintenance, and development.

In our talk, we will be covering our journey about how we ended up with a CI/CD setup composed by a Spark Streaming job in K8s consuming from Kafka, using the Spark Operator and deploying with ArgoCD.

Guy Hizkiau, VP R&D and Product at SpecterX, a rising startup in External Data Management Solution, Joins Adi Polak to discuss how Data management evolves and how organizations should reconsider the way they share secure data during external organization collaboration. 

Guy shares how they leverage Office 365, Azure Information Protection, and Microsoft Cloud App Security to build their solution and support various secure data management operations organizations need.  

dotnet-monitor is a command line tool that makes it easier to get access to diagnostics information in a dotnet process.

In the episode, Rich is joined by Sourabh who explains to us the importance of gathering application diagnostics and also gives us a demo of how to run dotnet-monitor in Kubernetes

Useful Links