{"id":1036,"date":"2023-07-07T14:26:23","date_gmt":"2023-07-07T14:26:23","guid":{"rendered":"https:\/\/bandittracker.com\/?p=1036"},"modified":"2023-07-07T14:26:24","modified_gmt":"2023-07-07T14:26:24","slug":"run-a-job-from-a-databricks-notebook","status":"publish","type":"post","link":"https:\/\/bandittracker.com\/run-a-job-from-a-databricks-notebook\/","title":{"rendered":"How To Run A Job From A Databricks Notebook"},"content":{"rendered":"\n<p>A common Databricks task is to schedule a job that calls a notebook. However, sometimes you will want to do this in reverse. In other words, you want to run a notebook that launches a job.<\/p>\n\n\n\n<p>This is most likely when you have set up a job that you want to test without creating a new task. It&#8217;s also handy when you are testing how a job performs repeatedly i.e. you want to run it multiple times in a loop.<\/p>\n\n\n\n<p>There are two ways to do this. Both methods use the REST API that Databricks provides to interact with jobs. This article walks through each method with examples.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_65 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-66285c213beb9\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-66285c213beb9\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/bandittracker.com\/run-a-job-from-a-databricks-notebook\/#What_You_Need_In_Advance\" title=\"What You Need In Advance\">What You Need In Advance<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/bandittracker.com\/run-a-job-from-a-databricks-notebook\/#Method_1_Call_A_Job_Via_Python_Code_In_The_Notebook\" title=\"Method 1: Call A Job Via Python Code In The Notebook\">Method 1: Call A Job Via Python Code In The Notebook<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/bandittracker.com\/run-a-job-from-a-databricks-notebook\/#Method_2_Use_A_Databricks_Magic_Command_To_Call_A_Job_From_A_Notebook\" title=\"Method 2: Use A Databricks Magic Command To Call A Job From A Notebook\">Method 2: Use A Databricks Magic Command To Call A Job From A Notebook<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/bandittracker.com\/run-a-job-from-a-databricks-notebook\/#Tips_For_Running_Jobs_From_Notebooks\" title=\"Tips For Running Jobs From Notebooks\">Tips For Running Jobs From Notebooks<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_You_Need_In_Advance\"><\/span>What You Need In Advance<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Regardless of the method that you choose, you will need:<\/p>\n\n\n\n<ul>\n<li>the job ID<\/li>\n\n\n\n<li>your Databricks instance name<\/li>\n\n\n\n<li>an access token<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">How To Find The Job ID<\/h3>\n\n\n\n<p>You will need the unique Job ID that is assigned when a job is created within Databricks. To get the ID, follow these steps in the web interface:<\/p>\n\n\n\n<ol>\n<li>Click on Workflows in the sidebar.<\/li>\n\n\n\n<li>Choose the job that you want to call from your notebook.<\/li>\n\n\n\n<li>Find the ID at the end of the URL in the browser.<\/li>\n<\/ol>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/07\/databricks-workflows-sidebar.jpg\" alt=\"\" class=\"wp-image-1037\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/07\/databricks-workflows-sidebar.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/07\/databricks-workflows-sidebar-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<p>Let&#8217;s say that the browser shows: <\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><em>https:\/\/acme.cloud.databricks.com\/<strong>#<\/strong>job\/987654<\/em><\/p>\n<\/blockquote>\n\n\n\n<p>Your job ID is 987654 i.e. the digits at the end of the URL.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How To Find Your Databricks Instance Name<\/h3>\n\n\n\n<p>You can find the instance name in the same way that you got the job ID. The database instance is part of the URL displayed in the job details page.<\/p>\n\n\n\n<p>In the example I presented for the job ID above, &#8220;<em>acme.cloud.databricks.com<\/em>&#8221; is the Databricks instance.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How To Set Up An Access Token<\/h3>\n\n\n\n<p>Because your notebook will be accessing the Databricks REST API, the code will need to authenticate that it has the appropriate access.<\/p>\n\n\n\n<p>You can provide this by creating a personal access token to use in the code. <\/p>\n\n\n\n<p><em>One alternative to the method I present here is to hard-code a username and password. But this is not best practice.<\/em><\/p>\n\n\n\n<p>To set up a personal access token, follow the steps below. <em>Note that you can&#8217;t do this in the Community Edition<\/em>.<\/p>\n\n\n\n<ol>\n<li>Click on the settings gear cog at the bottom left of the web interface.<\/li>\n\n\n\n<li>Choose &#8220;Admin Settings&#8221;.<\/li>\n\n\n\n<li>Choose &#8220;User Settings&#8221; from the drop-down menu to open that settings page.<\/li>\n\n\n\n<li>Click the &#8220;generate token&#8221; button.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Method_1_Call_A_Job_Via_Python_Code_In_The_Notebook\"><\/span>Method 1: Call A Job Via Python Code In The Notebook<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>This section provides you with sample code. As well as placing your job ID into the relevant position, you will also need to know your Databricks instance. I shows you how to get both in the previous section.<\/p>\n\n\n\n<p>The following code does the following:<\/p>\n\n\n\n<ol>\n<li>imports the requests library to access the Databricks REST API.<\/li>\n\n\n\n<li>creates a JSON structure with the job ID that you identified earlier.<\/li>\n\n\n\n<li>creates a JSON structure with the access token you created earlier.<\/li>\n\n\n\n<li>uses the requests library to make the API call to run the job.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>import requests.json\n\njob_json = {\"job_id\": &lt;place-your-job-id-here>}\n\nauth_json = {\"Authorization\": \"Bearer: &lt;place-your-access-token-here>\"}\n\nresponse = requests.post(\"https:\/\/&lt;your-databricks-instance>\/api\/2.0\/jobs\/runnow\", json=job_json, headers=auth_json).json()\n<\/code><\/pre>\n\n\n\n<p>If the API call is a success, the response variable (last line) will contain the run ID of the triggered job.<\/p>\n\n\n\n<p>If there is an error, it will be returned as a text message in the response variable.<\/p>\n\n\n\n<p>Note that an error from calling the job does not raise an exception in your Python code. This makes it a little tricky to <a href=\"https:\/\/bandittracker.com\/debug-databricks-notebooks\/\" data-type=\"post\" data-id=\"1021\">debug the notebook<\/a>. You will need to check the message explicitly.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Method_2_Use_A_Databricks_Magic_Command_To_Call_A_Job_From_A_Notebook\"><\/span>Method 2: Use A Databricks Magic Command To Call A Job From A Notebook<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>With this method, you shell out to the command line from within the notebook to make the Databricks API call.<\/p>\n\n\n\n<p>Use %sh as the magic command at the top of the notebook cell that is calling the job. (Check out our <a href=\"https:\/\/bandittracker.com\/what-is-a-databricks-notebook\/\" data-type=\"post\" data-id=\"926\">introduction to Databricks notebooks<\/a> if this is unfamiliar).<\/p>\n\n\n\n<p>Before I give you the statement to use, make sure you have these detail (I show you how to get them in an earlier section):<\/p>\n\n\n\n<ul>\n<li>access token<\/li>\n\n\n\n<li>Databricks instance<\/li>\n\n\n\n<li>job ID<\/li>\n<\/ul>\n\n\n\n<p>This is the statement to enter into the notebook cell (with the %sh command):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl --netrc --request POST --header \"Authorization: Bearer &lt;your-access-token>\" https:\/\/&lt;your-databricks-instance-name>\/api\/2.0\/jobs\/run-now --data '{\"job_id\": &lt;your-job-id>}'<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Tips_For_Running_Jobs_From_Notebooks\"><\/span>Tips For Running Jobs From Notebooks<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Your workspace is limited to one thousand concurrent runs. If you are using techniques to <a href=\"https:\/\/bandittracker.com\/run-databricks-notebooks-in-parallel\/\" data-type=\"post\" data-id=\"951\">run Databricks notebooks in parallel<\/a> and they are each launching jobs, be careful that you don&#8217;t exceed this limit.<\/p>\n\n\n\n<p>Regardless of what you are using a notebook for, each individual cell can output a maximum of 8 MB. There is another upper limit whereby all the cells in the notebook have a total limit of 20 MB.<\/p>\n\n\n\n<p>If these limits are hit, then the running job will be cancelled and marked as failed. To avoid problems with these limitations, you should attach your notebook to an all-purpose cluster.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A common Databricks task is to schedule a job that calls a notebook. However, sometimes you will want to do this in reverse. In other words, you want to run a notebook that launches a job. This is most likely when you have set up a job that you want to test without creating a &#8230; <a title=\"How To Run A Job From A Databricks Notebook\" class=\"read-more\" href=\"https:\/\/bandittracker.com\/run-a-job-from-a-databricks-notebook\/\" aria-label=\"More on How To Run A Job From A Databricks Notebook\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[29],"tags":[],"_links":{"self":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/1036"}],"collection":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/comments?post=1036"}],"version-history":[{"count":1,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/1036\/revisions"}],"predecessor-version":[{"id":1038,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/1036\/revisions\/1038"}],"wp:attachment":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/media?parent=1036"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/categories?post=1036"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/tags?post=1036"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}