Data engineer 101

Data engineering is a critical field in data science that involves preparing the "big data" infrastructure to be analyzed by data scientists. In this episode we are discussing the differences and how important each is with our guests.

Guests

• Mahmoud Fettal

• Salim Jannah

• Omaima Khalil

Notes

0:00:00 - Introduction and welcoming

0:02:50 - What is data engineering?

0:08: 43 - What are the key skills required for a data engineer?

0:16:40 - How does data engineering differ from data science?

0:20:00 - Data analyst vs data engineer vs data scientist

0:22:41 - What are the common tools used in data engineering?

0:28:57 - What are data pipelines?

0:34:54 - What challenges do data engineers face?

0:42:12 - Q&A

0:53:42 - How important is real -time data processing in data engineering?

1:02:35 - What is a data lake, and how does it differ from a data warehouse?

1:12:52 - How do data engineers use machine learning?

1:18:01 - Types of projects really involved with Data engineering

1:32:17 - What future trends should data engineers be aware of?

1:41:00 - Geeksblabla Picks

2:18:30 - Conclusion and Goodbye

Links

• Apache Airflow vs Mage.ai

• Lakehouse paper

• Open Source Agent for Data Analysis

• Simplifying Data Engineering and Analytics with Delta

Prepared and Presented by

• Meriem Zaid