Astronomer

👻 Sneak peak from the webinar this Thursday: Debugging Apache Airflow's scariest issues! It is spooky season and Kenten Danas and I selected some of the spookiest errors we've encountered to tackle together with our agentic friends! You'll learn how to tackle the kind of issues where the error trace does not help (or does not exist) as well as how to give your agents the best shot at being helpful assistants in debugging Airflow. Join us live on Thursday Oct 29th at 11am ET / 4pm CET! Or register and watch the recording afterwards (but this one is much more fun live 😄 ) Link in the comments!

Your agent calls many different tools, BUT some are dangerous! Now you can review a given tool call before any disaster 👇 Airflow just introduced tool-call approvals 🤗 Works on @task.agent and AgentOperator available in Airflow 3.3+. How does it work? Simple: ✅ toolset.approval_required(func) on any toolset, SQLToolset and MCPToolset included ✅ Tool(refund_order, requires_approval=True) inside a FunctionToolset for a single function Let's say we have a refund_order tool. The function gets the tool name and the arguments, and requires human review for refunds over $50 in one line. What happens at the call: 1️⃣ The task pauses in awaiting_input 💡 No worker slot held, like a deferred task 2️⃣ The request shows up on Required Actions 💡 Tool name and arguments, secrets masked Then you: 3️⃣ Approve 💡 The call runs, the agent carries on 4️⃣ Reject 💡 The task doesnt fail. The agent is told your reason and carries on without the call You can specify tool_approval_timeout to define the wait. On timeout, you get fail or deny, and you can even set tool_approval_assigned_users to who can decide 😎 Now you can finally better control your agent! Try it, you gonna love it Enjoy ❤️ P.S: Like and share to help your teammates #airflow #apacheairflow #dataengineer #dataengineering

You’re deep in legacy data, broken pipelines, and surprise schema changes. Now there’s Jev, new open source models, and Airflow Dags with loops??? If you’re feeling overwhelmed, you’re not alone. Join data engineers Tamara and Volker on November 18 for an honest conversation about what’s worth your attention and what you can safely ignore. Here’s what to expect: → Two data engineers trying to decipher the current trends → Where your role is headed (it’s not all bad!) → Resources and code to take home

You’re deep in legacy data, broken pipelines, and surprise schema changes. Now there’s Jev, new open source models, and Airflow Dags with loops??? If you’re feeling overwhelmed, you’re not alone. Join data engineers Tamara and Volker on November 18 for an honest conversation about what’s worth your attention and what you can safely ignore. Here’s what to expect: → Two data engineers trying to decipher the current trends → Where your role is headed (it’s not all bad!) → Resources and code to take home

AI-generated code is shipping fast but the data foundation isn’t ready and you’re on the hook for data quality. That's a tricky spot to be in. So, how do you maintain data quality without becoming the bottleneck for shipping AI? (Hint: it comes before the semantic layer) Steven Hillion will share how our enterprise customers are putting orchestration first to improve data quality at the source to build the data foundation required for AI.

You don't want to miss 👇