{"id":956,"date":"2023-06-19T16:12:04","date_gmt":"2023-06-19T16:12:04","guid":{"rendered":"https:\/\/bandittracker.com\/?p=956"},"modified":"2023-06-19T16:24:52","modified_gmt":"2023-06-19T16:24:52","slug":"calling-one-databricks-notebook-from-another","status":"publish","type":"post","link":"https:\/\/bandittracker.com\/calling-one-databricks-notebook-from-another\/","title":{"rendered":"How To Call One Databricks Notebook From Another"},"content":{"rendered":"\n<p>You will often want to reuse code from one Databricks notebook in another. This step-by-step beginner guide shows you how to:<\/p>\n\n\n\n<ul>\n<li>Import a function from one notebook to another.<\/li>\n\n\n\n<li>Run one notebook from another.<\/li>\n\n\n\n<li>Return a value from a called notebook.<\/li>\n\n\n\n<li>Pass parameters to a notebook.<\/li>\n\n\n\n<li>Return large datasets from a called notebook.<\/li>\n<\/ul>\n\n\n\n<p>If you\u2019re new to this technology, don\u2019t worry. I assume that you know <a href=\"https:\/\/bandittracker.com\/what-is-a-databricks-notebook\/\" data-type=\"post\" data-id=\"926\">the basics of notebooks in Databricks<\/a>. But that\u2019s all you need to follow along.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_64 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-661502ef374d5\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-661502ef374d5\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/bandittracker.com\/calling-one-databricks-notebook-from-another\/#Typical_Use_Case\" title=\"Typical Use Case\">Typical Use Case<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/bandittracker.com\/calling-one-databricks-notebook-from-another\/#How_To_Import_Functions_With_The_run_Magic_Command\" title=\"How To Import Functions With The %run Magic Command\">How To Import Functions With The %run Magic Command<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/bandittracker.com\/calling-one-databricks-notebook-from-another\/#Step-By-Step_Guide_To_Importing_Functions_From_Other_Notebooks\" title=\"Step-By-Step Guide To Importing Functions From Other Notebooks\">Step-By-Step Guide To Importing Functions From Other Notebooks<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/bandittracker.com\/calling-one-databricks-notebook-from-another\/#Advantages_And_Disadvantages_Of_The_run_Command\" title=\"Advantages And Disadvantages Of The %run Command\">Advantages And Disadvantages Of The %run Command<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/bandittracker.com\/calling-one-databricks-notebook-from-another\/#How_To_Run_One_Notebook_From_Another_With_dbutilsnotebookrun\" title=\"How To Run One Notebook From Another With  dbutils.notebook.run\">How To Run One Notebook From Another With  dbutils.notebook.run<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/bandittracker.com\/calling-one-databricks-notebook-from-another\/#Step-By-Step_Guide_To_Using_dbutilsnotebookrun\" title=\"Step-By-Step Guide To Using dbutils.notebook.run\">Step-By-Step Guide To Using dbutils.notebook.run<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/bandittracker.com\/calling-one-databricks-notebook-from-another\/#Advantages_And_Disadvantages_Of_dbutilsnotebookrun\" title=\"Advantages And Disadvantages Of  dbutils.notebook.run\">Advantages And Disadvantages Of  dbutils.notebook.run<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/bandittracker.com\/calling-one-databricks-notebook-from-another\/#How_To_Pass_Parameters_From_One_Notebook_To_Another\" title=\"How To Pass Parameters From One Notebook To Another\">How To Pass Parameters From One Notebook To Another<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/bandittracker.com\/calling-one-databricks-notebook-from-another\/#How_To_Return_Large_Datasets_From_One_Notebook_To_Another\" title=\"How To Return Large Datasets From One Notebook To Another\">How To Return Large Datasets From One Notebook To Another<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Typical_Use_Case\"><\/span>Typical Use Case<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Here is a typical scenario. You are working on a project that requires analyzing different datasets coming from multiple sources. Each dataset should go through the same set of preprocessing steps such as handling missing values.<\/p>\n\n\n\n<p>You&#8217;ve written a function, preprocess_data(), that takes raw data as input, goes through the pre-processing steps, and returns the cleaned data.<\/p>\n\n\n\n<p>You could copy and paste your preprocess_data() function into each of them, but that\u2019s a bad idea. Not only is it time-consuming, but it also creates room for potential errors or inconsistencies. <\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/frustrated-coder.jpg\" alt=\"\" class=\"wp-image-957\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/frustrated-coder.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/frustrated-coder-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<p>Another drawback is that If you ever need to update the function, you would need to do so in each notebook.<\/p>\n\n\n\n<p>This is a clear case where code reuse can significantly streamline your workflow. By writing your preprocess_data() function in one notebook and importing it into others, you save time, reduce the risk of errors, and make your code easier to maintain.<\/p>\n\n\n\n<p>Now, let\u2019s dive into practical ways to deal with this scenario.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_To_Import_Functions_With_The_run_Magic_Command\"><\/span>How To Import Functions With The %run Magic Command<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Magic commands in Databricks are special commands in a Python notebook cell that are not part of the Python language itself. They are preceded by the percent symbol: %.<\/p>\n\n\n\n<p>The <strong>%run<\/strong> command in Databricks enables the execution of an entire notebook within another notebook.<\/p>\n\n\n\n<p>The command acts as an importer and executor. It imports the entire code from the specified notebook and runs it in the context of the current notebook.<\/p>\n\n\n\n<p>This means that any functions, classes, or variables are imported and available for use in the notebook where the %run command is used.<\/p>\n\n\n\n<p>Let\u2019s say that your required functionality is in a notebook called \u201cProcessing\u201d in a subfolder called \u201ctasks\u201d. The call would look like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>%run \u201c.\/tasks\/Processing\u201d<\/code><\/pre>\n\n\n\n<p>Once you have run this command in a cell, later cells have access to the function in the other notebook.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Drawback<\/h3>\n\n\n\n<p>Because the %run command runs the entire notebook, any executable statements (like function calls or print statements) will also execute when you use %run.<\/p>\n\n\n\n<p>So, you probably don\u2019t want to use %run with standard notebooks with lots of print statements or reporting displays.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Best Tip<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/tips-and-tricks.jpg\" alt=\"\" class=\"wp-image-958\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/tips-and-tricks.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/tips-and-tricks-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<p>Here\u2019s the setup that I advise: create a \u201clibrary\u201d notebook that only contains functions and classes. This notebook should have no runnable code.<\/p>\n\n\n\n<p>This means that when you use the run command to import all the functions into another notebook, you don\u2019t get an avalanche of hard-to-follow messages.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step-By-Step_Guide_To_Importing_Functions_From_Other_Notebooks\"><\/span>Step-By-Step Guide To Importing Functions From Other Notebooks<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>If you\u2019re already familiar with working with notebooks in Databricks, then the previous section should get you going with importing functions. But if this is new to you, here&#8217;s a step-by-step guide that assumes you are a complete beginner.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Create a function in a new notebook<\/h3>\n\n\n\n<p>Create a new notebook in Databricks and add this simple function to it. All it does is add two numbers together.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def add_numbers(a, b):\n     \n     return a + b<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Save your notebook<\/h3>\n\n\n\n<p>Rename the notebook to something like \u201cUtils\u201d to show that it\u2019s to hold a collection of utility functions.<\/p>\n\n\n\n<p>Make sure to take note of the notebook&#8217;s path in your Databricks workspace.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Create a second notebook and use the %run command<\/h3>\n\n\n\n<p>Create a new notebook and use the %run command in a cell like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>%run \"\/path\/to\/Utils\"<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Call the imported function<\/h3>\n\n\n\n<p>After running the %run command, the add_numbers function from the &#8220;Utils&#8221; notebook is available in your current notebook. You can now call the function as you normally would:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>result = add_numbers(5, 7)\n\nprint(result)&nbsp; # Outputs: 12<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Advantages_And_Disadvantages_Of_The_run_Command\"><\/span>Advantages And Disadvantages Of The %run Command<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The %run command is a quick and straightforward solution to importing functions from one notebook to another. But it comes with advantages and disadvantages.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Advantages<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-up-1.jpg\" alt=\"\" class=\"wp-image-960\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-up-1.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-up-1-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<h4 class=\"wp-block-heading\">Simplicity<\/h4>\n\n\n\n<p>The %run command is easy to use and just needs one line of code.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Code Reusability and Modularity<\/h4>\n\n\n\n<p>The command helps you avoid duplicating code across multiple notebooks. This promotes better code organization and modularity.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Shared State<\/h4>\n\n\n\n<p>When a notebook is run using the %run command, it shares the same SparkContext with the calling notebook.<\/p>\n\n\n\n<p>This means that any Spark jobs that are kicked off by the called notebook are part of the same Spark application as the calling notebook.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Disadvantages<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-down.jpg\" alt=\"\" class=\"wp-image-961\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-down.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-down-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<h4 class=\"wp-block-heading\">Entire Notebook Execution<\/h4>\n\n\n\n<p>The %run command runs the entire notebook, not just specific functions or variables. This can be a disadvantage if you only need to use a specific function from the imported notebook.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">No Isolation<\/h4>\n\n\n\n<p>The functions and variables from the imported notebook become available in the global scope of the calling notebook. This can lead to naming conflicts if the same name is used in both notebooks.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_To_Run_One_Notebook_From_Another_With_dbutilsnotebookrun\"><\/span>How To Run One Notebook From Another With  dbutils.notebook.run<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The dbutils.notebook.run also lets you run a notebook from within another notebook. The function lets you return a string result from the called notebook.<\/p>\n\n\n\n<p>This is the syntax:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>output = dbutils.notebook.run(\"\/path\/to\/notebook\", timeout_seconds)<\/code><\/pre>\n\n\n\n<p>There are three main differences with the %run command:<\/p>\n\n\n\n<ul>\n<li>Context<\/li>\n\n\n\n<li>Timeout (optional)<\/li>\n\n\n\n<li>Functions are not imported<\/li>\n<\/ul>\n\n\n\n<p>Let&#8217;s look at these aspects in turn:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Context<\/h3>\n\n\n\n<p>A major difference is that this method executes the called notebook in a separate context. This is essentially treating the second notebook as a separate job.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Optional Timeout<\/h3>\n\n\n\n<p>The optional timeout parameter lets you specify the maximum time in seconds that the notebook is allowed to run. If it doesn\u2019t complete, the execution will be interrupted.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Functions are not imported<\/h3>\n\n\n\n<p>Unlike %run, dbutils.notebook.run does not import functions or variables from the called notebook into the calling notebook. Instead, it returns the last expression in the called notebook as a string.<\/p>\n\n\n\n<p>This is useful when you want to call a notebook that produces a specific output, but you don&#8217;t need to use any of its functions or variables in your current notebook.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step-By-Step_Guide_To_Using_dbutilsnotebookrun\"><\/span>Step-By-Step Guide To Using dbutils.notebook.run<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>While dbutils.notebook.run can&#8217;t directly import functions like %run, it can be used to run a notebook that returns the output of a function. Here&#8217;s how you can use it:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Create a function in a new notebook<\/h3>\n\n\n\n<p>Create a new notebook in Databricks and add this simple function to it. All it does is add two numbers together.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def add_numbers(a, b):\n\n&nbsp;&nbsp;&nbsp; return a + b<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Return the result of the function<\/h3>\n\n\n\n<p>At the end of the same notebook, call the function with the desired parameters and return its output:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>result = add_numbers(5, 7)\n\nreturn result<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Save your notebook<\/h3>\n\n\n\n<p>Rename the notebook to something like \u201cCalc\u201d to indicate the usage.<\/p>\n\n\n\n<p>Make sure to take note of the notebook&#8217;s path in your Databricks workspace.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Create a second notebook and use dbutils<\/h3>\n\n\n\n<p>Create a new notebook and assign the output of the first notebook to a variable like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>output = dbutils.notebook.run(\"\/path\/to\/Calc\", 600)<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 5: Use the output<\/h3>\n\n\n\n<p>The dbutils.notebook.run command returns the last expression of the called notebook as a string.<\/p>\n\n\n\n<p>In our example, output would be &#8217;12&#8217;. If you need to use the output as an integer, you&#8217;ll need to convert it:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>output = int(output)<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Advantages_And_Disadvantages_Of_dbutilsnotebookrun\"><\/span>Advantages And Disadvantages Of  dbutils.notebook.run<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Here are some of the benefits and limitations of this second method.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Advantages<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-up-1.jpg\" alt=\"\" class=\"wp-image-960\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-up-1.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-up-1-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<h4 class=\"wp-block-heading\">Isolation<\/h4>\n\n\n\n<p>dbutils.notebook.run executes the called notebook in a separate context. This means that the called notebook doesn&#8217;t affect the state of the calling notebook.<\/p>\n\n\n\n<p>This helps prevent naming conflicts and other unexpected issues.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Controlled Output<\/h4>\n\n\n\n<p>Unlike %run, dbutils.notebook.run returns the last expression from the called notebook as a string.<\/p>\n\n\n\n<p>This means that you can specifically control what data you want to pass back to the calling notebook.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Timeout Control<\/h4>\n\n\n\n<p>You can optionally set a timeout for the execution of the called notebook.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Disadvantages<\/h3>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-down.jpg\" alt=\"\" class=\"wp-image-961\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-down.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/thumbs-down-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<h4 class=\"wp-block-heading\">String Output<\/h4>\n\n\n\n<p>The fact that the output is always a string can be a pain in the neck. You will often need to convert it.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">No Function or Variable Import<\/h4>\n\n\n\n<p>dbutils.notebook.run does not import functions or variables from the called notebook into the calling notebook.<\/p>\n\n\n\n<p>If you need to use a function from another notebook, you&#8217;ll need to structure your notebooks differently or use the %run command instead.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Resource Consumption<\/h4>\n\n\n\n<p>Since dbutils.notebook.run executes the called notebook as a separate job, it consumes separate resources.<\/p>\n\n\n\n<p>This can lead to increased resource usage if you are <a href=\"https:\/\/bandittracker.com\/run-databricks-notebooks-in-parallel\/\" data-type=\"post\" data-id=\"951\">running several notebooks concurrently<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_To_Pass_Parameters_From_One_Notebook_To_Another\"><\/span>How To Pass Parameters From One Notebook To Another<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>You can pass parameters when using dbutils.notebook.run by using the widgets function that is part of the dbutils library.<\/p>\n\n\n\n<p>Widgets in Databricks notebooks are user-interactive graphical elements that let your notebooks accept input from users. There are lots of fancy choices like buttons and dropdowns. But when passing parameters, you will use the simple \u201ctext\u201d widget.<\/p>\n\n\n\n<p>Here\u2019s a step-by-step guide.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Define the Notebook to be Called<\/h3>\n\n\n\n<p>Let\u2019s call the target notebook \u201cCalledNotebook\u201d. You can define a widget to accept a parameter like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.widgets.text(\"input\", \"defaultValue\")<\/code><\/pre>\n\n\n\n<p>This code will create a widget named input with a default value of &#8220;defaultValue&#8221;. You can get the value of the widget (and thus the passed parameter) like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>input_value = dbutils.widgets.get(\"input\")<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Call the Notebook with Parameters<\/h3>\n\n\n\n<p>Let\u2019s name the notebook that calls the target as \u201cCallingNotebook\u201d. In this notebook, you use the dbutils.notebook.run function to call CalledNotebook and pass parameters. Here&#8217;s an example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.notebook.run(\"CalledNotebook\", 600, {\"input\": \"Hello, world!\"})<\/code><\/pre>\n\n\n\n<p>This code will run CalledNotebook and pass the string &#8220;Hello, world!&#8221; as the input parameter. The 600 is a timeout value in seconds, you can adjust this based on your needs.<\/p>\n\n\n\n<p>When CalledNotebook is run, the input widget will have a value of &#8220;Hello, world!&#8221;. The input_value variable will also be set to &#8220;Hello, world!&#8221;.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_To_Return_Large_Datasets_From_One_Notebook_To_Another\"><\/span>How To Return Large Datasets From One Notebook To Another<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>We&#8217;ve already looked at how the dbutils notebook.run function will return a string. But what if you want to return something more complex &#8211; like a DataFrame?<\/p>\n\n\n\n<p>The problem here is that a DataFrame is usually too large to be converted to a string representation. This would not be an efficient method, and mightn&#8217;t even work.<\/p>\n\n\n\n<p>The solution is to have the called notebook write the DataFrame to a temporary view or to a file. The calling notebook can then read and access the data.<\/p>\n\n\n\n<p>This approach has the advantage of being efficient, because Spark&#8217;s file reading and writing operations are designed to handle large datasets. <\/p>\n\n\n\n<p>It also allows you to control the format of your data. For example, you can choose to write either to a CSV file or a Parquet file, depending on your use case.<\/p>\n\n\n\n<p>Bear in mind that if you write to a temporary view, the data will only be accessible while your Spark session is active. If you need more permanent storage, writing to a file is a better option.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>You will often want to reuse code from one Databricks notebook in another. This step-by-step beginner guide shows you how to: If you\u2019re new to this technology, don\u2019t worry. I assume that you know the basics of notebooks in Databricks. But that\u2019s all you need to follow along. Typical Use Case Here is a typical &#8230; <a title=\"How To Call One Databricks Notebook From Another\" class=\"read-more\" href=\"https:\/\/bandittracker.com\/calling-one-databricks-notebook-from-another\/\" aria-label=\"More on How To Call One Databricks Notebook From Another\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[29],"tags":[],"_links":{"self":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/956"}],"collection":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/comments?post=956"}],"version-history":[{"count":4,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/956\/revisions"}],"predecessor-version":[{"id":965,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/956\/revisions\/965"}],"wp:attachment":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/media?parent=956"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/categories?post=956"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/tags?post=956"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}