Hosted by Origin Workspace and Bud, sponsored by Solar Battery Scheduler

Location: 40 Berkeley Square Bristol, BS8 1HP

AGENDA

18.00 – 18:30 Meet & Greet Grab a soft drink, chat and meet people. Feel free to bring your own alcoholic drink.

--------------

18:30 - 19:15 Delta Lake Tables 101 by Kamil Nowinski

There are more and more file formats nowadays: Parquet format is not the best shiny star any longer. Now, the Delta Lake takes the prim. Why people do confuse it with Parquet and always talk about files in this case?

In this session, we'll take a look at the evolution of ETL into ELT and its storage aspect, which explain why it is "a must" for modern data warehouse solutions and how is it related Delta Lake technology in cloud environments like Databricks or Synapse Analytics. Finally, we'll check what's Delta-Parquet creature presented in Microsoft Fabric OneLake recently. We will see also what data layers (stages) are commonly set up and why they make sense.

--------------

19:15 - 19:45 Pizza and Networking

--------------

19:45 - 20:30 Better ETL with Managed Airflow in ADF by Niall Langley

Building complex data workflows using Azure Data Factory can get a little clunky - as you orchestration needs get more complex you hit limitations like not being able to nest loops or conditionals, running simple Python, bash or PowerShell scripts is difficult, and costs can grow quickly as you are charged per task execution. Recently another option become available, Managed Airflow in ADF.

Apace Airflow is a code-centric open-source platform for developing, scheduling and monitoring batch-based data workflows, built using the python language Data Engineers know and love. But until Managed Airflow, getting it working in Azure was a complex task for customers more used to PaaS services such as ADF, Databricks and Fabric. It is also an important ETL orchestrator on AWS and GCP, so cross cloud compatibility becomes simpler to achieve.

In this session we’ll look at what Airflow is, how it’s different from ADF, and what advantages Managed Airflow in ADF gives us. We talk about the idea of a DAG for building the workflow, and then work through some demos to show just how easy it is to use Python to write an Airflow DAG’s and import them into the Managed Airflow Environment as pipelines. We then dive into the excellent monitoring UI and find out just how easy is it to trigger a pipeline, view it to see the dependencies between tasks, and monitor runs.

By the end of the session attendees will have a good understanding of what Airflow is, when to use it, and how it fits into the Azure Data Platform.

--------------

20:30 - Pub

--------------

About Origin Workspace - https://originworkspace.co.uk/

Origin Workspace is designed to meet the changing needs of today's entrepreneurs, small and growing businesses, ambitious consultants, freelancers and remote workers. We know that your workplace needs to be more than a desk with internet connection. You need networking opportunities as well as IT support, a place to hold meetings. And all in a supportive environment that inspires you to connect and grow, and in a convenient location next to Bristol's Clifton Triangle.

About Bud - https://bud.co.uk/

Bud is a training management platform, designed to streamline the processes involved in delivering apprenticeships and skills training. Built around workflows that require key items of data to be captured only once, Bud’s customers benefit from reduced administrative time and improved data accuracy.

About Solar Battery Scheduler - https://www.solarbatteryscheduler.com/

Make the most of your Solar Power Generation with Eco Intelligent Charging of your Home Battery.

---- Photos We ask that you do NOT take photos at this meetup.

We will invite people to be included in a group photo/s during the event. Speakers will let you know if it's okay to photograph their presentation (excluding other attendees).

You may see organisers taking photos during the talks. These will be of speakers, if they have agreed to this, and will not include faces of attendees.