{"id":940,"date":"2023-06-11T00:03:02","date_gmt":"2023-06-11T00:03:02","guid":{"rendered":"https:\/\/bandittracker.com\/?p=940"},"modified":"2023-06-15T09:22:56","modified_gmt":"2023-06-15T09:22:56","slug":"apache-airflow-for-managers","status":"publish","type":"post","link":"https:\/\/bandittracker.com\/apache-airflow-for-managers\/","title":{"rendered":"Quick Intro To Apache Airflow For Managers"},"content":{"rendered":"\n<p>Apache Airflow is an open-source platform for creating, scheduling, and monitoring data pipelines.<\/p>\n\n\n\n<p>Because Airflow is written in Python, your developers can use Python to define the tasks and dependencies of the steps in a pipeline. They can also use version control, code reuse, and automated tests when developing workflows.<\/p>\n\n\n\n<p>Airflow\u2019s scheduler provides an interactive web-based user interface that helps with visualizing pipelines running in production, monitoring progress, and troubleshooting issues.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_65 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-66285cb9d7775\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-66285cb9d7775\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/bandittracker.com\/apache-airflow-for-managers\/#Origins_Of_Airflow\" title=\"Origins Of Airflow\">Origins Of Airflow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/bandittracker.com\/apache-airflow-for-managers\/#Essential_Terms_To_Understand\" title=\"Essential Terms To Understand\">Essential Terms To Understand<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/bandittracker.com\/apache-airflow-for-managers\/#Workflows_As_Code\" title=\"Workflows As Code\">Workflows As Code<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/bandittracker.com\/apache-airflow-for-managers\/#Scheduling_With_Airflow\" title=\"Scheduling With Airflow\">Scheduling With Airflow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/bandittracker.com\/apache-airflow-for-managers\/#Five_Typical_Use_Cases_For_Airflow\" title=\"Five Typical Use Cases For Airflow\">Five Typical Use Cases For Airflow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/bandittracker.com\/apache-airflow-for-managers\/#What_Do_You_Need_To_Install_Apache_Airflow\" title=\"What Do You Need To Install Apache Airflow?\">What Do You Need To Install Apache Airflow?<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Origins_Of_Airflow\"><\/span>Origins Of Airflow<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Apache Airflow was originally developed by Airbnb in 2014 as a solution to manage the company&#8217;s increasingly complex tasks and dependencies that made up their data pipelines.<\/p>\n\n\n\n<p>Airbnb open-sourced Airflow in 2015, making it available to a broader community. The following year, Apache Software Foundation took Airflow under its umbrella. They made it a top-level Apache project in 2019, acknowledging its wide adoption and active community.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Essential_Terms_To_Understand\"><\/span>Essential Terms To Understand<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>You need to get familiar with these concepts (I&#8217;ll explain each in turn):<\/p>\n\n\n\n<ul>\n<li>Tasks<\/li>\n\n\n\n<li>DAG<\/li>\n\n\n\n<li>Operators<\/li>\n\n\n\n<li>Executors<\/li>\n\n\n\n<li>Workers<\/li>\n\n\n\n<li>Hooks<\/li>\n<\/ul>\n\n\n\n<p>A task in Airflow is a single unit of work. This could be running a Python script, a Shell command, or a Spark job.<\/p>\n\n\n\n<p>A DAG is a collection of tasks that operate on your data. The term stands for \u201cDirected Acyclic Graph\u201d. \u201cDirected\u201d means that it runs in a specific order, and \u201cAcyclic\u201d means that a task can\u2019t loop back to a previous task.<\/p>\n\n\n\n<p>Operators determine what gets done by a task. Airflow comes with many built-in operators for common tasks, such as PythonOperator for running Python code, BashOperator for bash commands, and so on.<\/p>\n\n\n\n<p>Executors are what actually runs the tasks within a DAG. Airflow comes with several executors with different characteristics, such as SequentialExecutor, LocalExecutor,<\/p>\n\n\n\n<p>Workers are the machines within a distributed system that the tasks run on.<\/p>\n\n\n\n<p>Hooks are interfaces to external platforms and databases like <a href=\"https:\/\/bandittracker.com\/python-read-write-snowflake-table\/\" data-type=\"post\" data-id=\"899\">Snowflake<\/a>, S3, MySQL, Postgres, etc. They allow Airflow to interact with external systems.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/snowflake-deltalake-s3-azure.jpg\" alt=\"\" class=\"wp-image-921\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/snowflake-deltalake-s3-azure.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/snowflake-deltalake-s3-azure-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Workflows_As_Code\"><\/span>Workflows As Code<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>One of the key features of Apache Airflow is that workflows are written as code, typically in Python.<\/p>\n\n\n\n<p>Let\u2019s contrast that with GUI-based workflow tools. Implementing code means that teams can use loops and conditional logic to generate tasks dynamically.<\/p>\n\n\n\n<p>This means that tasks can be created based on certain conditions or variables that may change at runtime, rather than hard-coding each task separately.<\/p>\n\n\n\n<p>Let&#8217;s say you want to create a backup task for each database in your database management system (e.g. PostgreSQL). <\/p>\n\n\n\n<p>Instead of creating a task manually for each database, you can dynamically generate these tasks with a loop that uses the metadata (list of databases) of the DBMS. When a new database is created, the new backup task will also appear.<\/p>\n\n\n\n<p>This of course is achieved with a Python script that can be version controlled. It can also be peer-reviewed and run through automated unit tests.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Scheduling_With_Airflow\"><\/span>Scheduling With Airflow<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Scheduling and monitoring are essential for any data pipeline, ensuring tasks are running at the correct time and providing visibility into their progress and status.<\/p>\n\n\n\n<p>In Airflow, each DAG has a schedule defined by a start date and a schedule interval.<\/p>\n\n\n\n<p>Airflow&#8217;s scheduler monitors all tasks and all DAGs and triggers the task instances whose dependencies have been met. It&#8217;s designed to run continuously and keep a constant state of what tasks are due to run.<\/p>\n\n\n\n<p>Airflow provides a rich web-based user interface for monitoring and managing your DAGs and tasks. It lets you:<\/p>\n\n\n\n<ul>\n<li>View the list of DAGs and their status.<\/li>\n\n\n\n<li>Drill down into a DAG to see tasks, their dependencies, and status.<\/li>\n\n\n\n<li>View the logs for each task instance.<\/li>\n\n\n\n<li>Trigger DAG runs manually.<\/li>\n\n\n\n<li>Mark task instances as successful or failed.<\/li>\n\n\n\n<li>Clear the state of task instances. dependencies.<\/li>\n<\/ul>\n\n\n\n<p>In addition, Airflow can send alerts through emails or services like Slack or PagerDuty.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Five_Typical_Use_Cases_For_Airflow\"><\/span>Five Typical Use Cases For Airflow<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Here are five examples of how organizations use Airflow in real-world scenarios:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">ETL (Extract, Transform, Load) Pipelines<\/h3>\n\n\n\n<p>This is one of the most common use cases for Airflow. ETL involves extracting data from multiple sources, transforming it into a suitable format, and loading it into a data warehouse for analysis.<\/p>\n\n\n\n<p>Each of these stages can be modeled as tasks in a DAG, with dependencies between them ensuring they run in the correct order.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Machine Learning Workflows<\/h3>\n\n\n\n<p>Complex machine learning workflows often involve many steps, such as data extraction, data cleaning, feature extraction, model training, model evaluation, and model deployment.<\/p>\n\n\n\n<p>Airflow can orchestrate these tasks, ensuring that each step is carried out at the right time and in the right order.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Data Quality Checks<\/h3>\n\n\n\n<p>Airflow can be used to run tasks that check the quality of your data.<\/p>\n\n\n\n<p>For example, you might have a task that verifies the integrity of your data, checks for duplicates, or validates that the data conforms to certain rules or constraints.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Automated Report Generation<\/h3>\n\n\n\n<p>Reports that need to be generated and distributed regularly can be automated using Airflow.<\/p>\n\n\n\n<p>The pipeline might involve extracting the latest data, running it through some analysis code, generating a report (such as a PDF or a spreadsheet), and then emailing it to the relevant people.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Data Lake Ingestion<\/h3>\n\n\n\n<p>Ingesting data into a data lake at regular intervals from various sources is another common use case.<\/p>\n\n\n\n<p>Airflow can manage the orchestration of these complex ingestion processes, handling failures and retries as necessary.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_Do_You_Need_To_Install_Apache_Airflow\"><\/span>What Do You Need To Install Apache Airflow?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Because it\u2019s a Python package, installing Airflow is relatively straightforward.<\/p>\n\n\n\n<p>A support team can install it on a linux system using the pip installer like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip install 'apache-airflow'<\/code><\/pre>\n\n\n\n<p>But before you tell them to do so, have a think about which database system will be used for its configuration. The default is SQLite, but many organizations choose PostgreSQL instead. <\/p>\n\n\n\n<p>If that\u2019s what you want, the pip command looks like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip install 'apache-airflow&#91;postgres]'<\/code><\/pre>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Apache Airflow is an open-source platform for creating, scheduling, and monitoring data pipelines. Because Airflow is written in Python, your developers can use Python to define the tasks and dependencies of the steps in a pipeline. They can also use version control, code reuse, and automated tests when developing workflows. Airflow\u2019s scheduler provides an interactive &#8230; <a title=\"Quick Intro To Apache Airflow For Managers\" class=\"read-more\" href=\"https:\/\/bandittracker.com\/apache-airflow-for-managers\/\" aria-label=\"More on Quick Intro To Apache Airflow For Managers\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[29],"tags":[],"_links":{"self":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/940"}],"collection":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/comments?post=940"}],"version-history":[{"count":1,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/940\/revisions"}],"predecessor-version":[{"id":941,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/940\/revisions\/941"}],"wp:attachment":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/media?parent=940"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/categories?post=940"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/tags?post=940"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}