{"id":1021,"date":"2023-06-30T15:07:07","date_gmt":"2023-06-30T15:07:07","guid":{"rendered":"https:\/\/bandittracker.com\/?p=1021"},"modified":"2023-06-30T15:14:02","modified_gmt":"2023-06-30T15:14:02","slug":"debug-databricks-notebooks","status":"publish","type":"post","link":"https:\/\/bandittracker.com\/debug-databricks-notebooks\/","title":{"rendered":"How To Debug Databricks Notebooks"},"content":{"rendered":"\n<p>Debugging is an essential part of the coding process. Even the most experienced programmers make mistakes. It&#8217;s through debugging that these mistakes are found and fixed.<\/p>\n\n\n\n<p>Each cell in a Databricks notebook has its own output, and any error messages are also displayed in the cell output. This is usually the starting point for your investigation.<\/p>\n\n\n\n<p>But Databricks provides a number of additional ways to help you debug your notebooks. This article shows you how to use:<\/p>\n\n\n\n<ul>\n<li>traditional print and logging methods<\/li>\n\n\n\n<li>the interactive Python debugger<\/li>\n\n\n\n<li>the new Databricks Variable Explorer<\/li>\n\n\n\n<li>the Spark Cluster UI and the Cluster Event Logs<\/li>\n<\/ul>\n\n\n\n<p>Before we dive into these methods, let&#8217;s start with common errors you&#8217;ll be dealing with.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_63 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-660a635e0c400\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-660a635e0c400\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/bandittracker.com\/debug-databricks-notebooks\/#Common_Errors_And_Exceptions_In_Databricks_Code\" title=\"Common Errors And Exceptions In Databricks Code\">Common Errors And Exceptions In Databricks Code<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/bandittracker.com\/debug-databricks-notebooks\/#Basic_Debugging_With_Print_Statements\" title=\"Basic Debugging With Print Statements\">Basic Debugging With Print Statements<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/bandittracker.com\/debug-databricks-notebooks\/#Error_Logging\" title=\"Error Logging\">Error Logging<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/bandittracker.com\/debug-databricks-notebooks\/#How_To_Use_The_Interactive_Python_Debugger_In_A_Notebook\" title=\"How To Use The Interactive Python Debugger In A Notebook\">How To Use The Interactive Python Debugger In A Notebook<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/bandittracker.com\/debug-databricks-notebooks\/#How_To_Set_A_Breakpoint_With_The_Debugger\" title=\"How To Set A Breakpoint With The Debugger\">How To Set A Breakpoint With The Debugger<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/bandittracker.com\/debug-databricks-notebooks\/#The_Databricks_Variable_Explorer\" title=\"The Databricks Variable Explorer\">The Databricks Variable Explorer<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/bandittracker.com\/debug-databricks-notebooks\/#Other_Databricks_Tools_For_Debugging\" title=\"Other Databricks Tools For Debugging\">Other Databricks Tools For Debugging<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/bandittracker.com\/debug-databricks-notebooks\/#Alternatives_To_Using_Databricks_For_Debugging\" title=\"Alternatives To Using Databricks For Debugging\">Alternatives To Using Databricks For Debugging<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Common_Errors_And_Exceptions_In_Databricks_Code\"><\/span>Common Errors And Exceptions In Databricks Code<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n<div class=\"wp-block-image is-style-rounded\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/bug.jpg\" alt=\"\" class=\"wp-image-1025\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/bug.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/bug-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<p>The errors and exceptions you encounter when running code in Databricks fall into five categories:<\/p>\n\n\n\n<ul>\n<li>Syntax errors<\/li>\n\n\n\n<li>Dependency errors<\/li>\n\n\n\n<li>Runtime errors<\/li>\n\n\n\n<li>Spark errors<\/li>\n\n\n\n<li>Cluster errors<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Syntax Errors<\/h3>\n\n\n\n<p>These are probably the most straightforward to debug.<\/p>\n\n\n\n<p>Review your code with an eye out for typographical errors. You may also fall foul of indentation issues, which tends to come with inexperience.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Dependency Errors<\/h3>\n\n\n\n<p>You\u2019ll get dependency errors when the code in a cell depends on a package or library that hasn\u2019t been installed on the cluster.<\/p>\n\n\n\n<p>If you\u2019re sure that you installed the appropriate packages, check whether the cells are running in a different order than what you expected (a few print statements can help).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Runtime Errors<\/h3>\n\n\n\n<p>These are what to watch out for:<\/p>\n\n\n\n<ul>\n<li>Type errors e.g. concatenating a string with an integer<\/li>\n\n\n\n<li>Index errors e.g. accessing an element in a list that doesn\u2019t exist<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Spark Errors<\/h3>\n\n\n\n<p>Spark exceptions are most likely to occur when you\u2019re executing a Spark job that hits an error. Watch out for jobs that:<\/p>\n\n\n\n<ul>\n<li>Run out of memory<\/li>\n\n\n\n<li>Have serialization errors<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Cluster Errors<\/h3>\n\n\n\n<p>Your notebook depends on the cluster it is attached to, so problems underneath the hood can bubble up with cryptic messages.<\/p>\n\n\n\n<p>Watch out for the cluster running out of resources or having connectivity issues. <em>Check out our article on <a href=\"https:\/\/bandittracker.com\/what-are-databricks-clusters\/\" data-type=\"post\" data-id=\"918\">databricks clusters<\/a> if you need a quick intro to what they do.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Basic_Debugging_With_Print_Statements\"><\/span>Basic Debugging With Print Statements<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>If you\u2019re used to the sophisticated debugging features of modern coding IDEs like Visual Studio, then you may feel a little underpowered with notebooks.<\/p>\n\n\n\n<p>But you can go a long way with the old-fashioned (and effective) technique of using print statements to debug your code.<\/p>\n\n\n\n<p>In Python, you can use the print() function to display the value of variables or expressions. Similarly, in Scala, you can use println(). You will usually use print statements to track:<\/p>\n\n\n\n<ul>\n<li>Variable values<\/li>\n\n\n\n<li>Execution flow<\/li>\n\n\n\n<li>DataFrame content<\/li>\n<\/ul>\n\n\n\n<p>Bugs are often due to variables not having the values you expect. By printing out the value of variables at different points in your program, you can narrow down the section of code where something goes awry.<\/p>\n\n\n\n<p>Print statements can be used to trace the execution flow of the program. By adding print statements in conditional blocks or loops, you can check the order of operations and how many times certain loops are being executed.<\/p>\n\n\n\n<p>Here\u2019s an example of checking that a DataFrame in a variable named &#8220;df&#8221; has the expected volume of data (a zero means you probably have to investigate):<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>print(\"The count of rows in the dataframe is\", df.count())<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Error_Logging\"><\/span>Error Logging<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>While print statements are great for debugging during development, they are not recommended for production code. They lack the flexibility and configurability of proper logging. <\/p>\n\n\n\n<p>As you get ready to promote your code into production, you should switch to using a logging library. This allows you to set different logging levels, output logs to different destinations, and more.<\/p>\n\n\n\n<p>To do so, you&#8217;ll need to import the logging module and set up basic configuration. This only needs to be done once, typically at the beginning of your notebook. Your code will look like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import logging\n\nlogging.basicConfig(level=logging.INFO)<\/code><\/pre>\n\n\n\n<p>The basicConfig function sets up the default handler so that debug messages are written to the console. This can be changed to writing to a physical log. <em>Our tutorial on <a href=\"https:\/\/bandittracker.com\/reading-and-writing-to-s3-with-databricks\/\" data-type=\"post\" data-id=\"974\">writing from Databricks to S3<\/a> will give you some pointers.<\/em><\/p>\n\n\n\n<p>The level argument sets the root logger level to the specified severity level. Setting it to INFO as in our example will produce a lot of logging. You can filter it to WARNING or ERROR to be more selective.<\/p>\n\n\n\n<p>You can replace the print statement from the previous section with a logging statement as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>logging.info(\"The count of rows in the dataframe is %s\", df.count())<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_To_Use_The_Interactive_Python_Debugger_In_A_Notebook\"><\/span>How To Use The Interactive Python Debugger In A Notebook<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Python cells give you access to what\u2019s known as the <strong>pdb<\/strong> \u2013 the Python debugger. It\u2019s clunkier than debuggers you may be used to in desktop IDEs, but it gets the job done.<\/p>\n\n\n\n<p>When your code throws an error, the notebook will stop and highlight the error. At this point, you can enter debug mode by running this command in the next cell:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>%debug<\/code><\/pre>\n\n\n\n<p>The output will show the current state of the variables set within the problem cell. You will also get the ipdb (interactive python debugger) interface embedded within the cell.<\/p>\n\n\n\n<p>This picture shows the code stopped by a division by zero exception. I have entered the %debug command in the cell below.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"450\" height=\"482\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/division-by-zero.png\" alt=\"\" class=\"wp-image-1022\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/division-by-zero.png 450w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/division-by-zero-280x300.png 280w\" sizes=\"(max-width: 450px) 100vw, 450px\" \/><\/figure><\/div>\n\n\n<p>At the bottom of the picture, the interactive Python debugger box (ipdb) has opened. Here, you can type in the name of other variables to display their state.<\/p>\n\n\n\n<p>When you\u2019re finished with the ipdb, you can type:<\/p>\n\n\n\n<ul>\n<li>C (or continue) to run the next cell<\/li>\n\n\n\n<li>Exit<\/li>\n\n\n\n<li>Cancel<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_To_Set_A_Breakpoint_With_The_Debugger\"><\/span>How To Set A Breakpoint With The Debugger<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>You can use the Python debugger to set breakpoints in specific cells.<\/p>\n\n\n\n<p>To do so, you should import the library at the top of your notebook or before you set the first breakpoint:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Import pdb<\/code><\/pre>\n\n\n\n<p>Then you can add breakpoints to specific areas with this line:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pdb.set_trace()<\/code><\/pre>\n\n\n\n<p>In the picture below, the last line will have a division by zero error. But before that line is executed, I have added a breakpoint with the set_trace method.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"419\" height=\"397\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/pdb-set-breakpoint.png\" alt=\"\" class=\"wp-image-1023\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/pdb-set-breakpoint.png 419w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/pdb-set-breakpoint-300x284.png 300w\" sizes=\"(max-width: 419px) 100vw, 419px\" \/><\/figure><\/div>\n\n\n<p>The notebook stops executing at the trace line and opens the interactive Python debugger command line. You can execute the next line by entering C or continue.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"The_Databricks_Variable_Explorer\"><\/span>The Databricks Variable Explorer<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Databricks rolled out the new Variable Explorer in May 2023 to assist in debugging.<\/p>\n\n\n\n<p>Instead of typing in variable names in the Python debugger, you now have a visual tool to view all the variables. It displays on the right hand side of the screen.<\/p>\n\n\n\n<p>It\u2019s particularly useful for looking at DataFrames. The Visual Explorer doesn\u2019t just give you a preview of the data, it lets you quickly explore the schema. This is very handy when dealing with a wide set of columns.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Other_Databricks_Tools_For_Debugging\"><\/span>Other Databricks Tools For Debugging<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>You should get familiar with these tools within the Databricks environment. &nbsp;They will be invaluable in your debugging efforts.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Spark Cluster UI<\/h3>\n\n\n\n<p>The easiest way to access the Spark UI when you\u2019re working in your notebook, follow these steps:<\/p>\n\n\n\n<ol type=\"1\">\n<li>Expand the cluster dropdown in the top right of the page.<\/li>\n\n\n\n<li>Click on the attached cluster.<\/li>\n\n\n\n<li>Click on \u201cSpark UI\u201d in the menu.<\/li>\n<\/ol>\n\n\n\n<p>This opens the Spark UI page in a separate tab. Here, you can view the details of your Spark job, including tasks, stages, and storage. You can also see if there are any failed tasks and inspect the error messages and logs for those tasks.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Cluster Event Logs<\/h3>\n\n\n\n<p>I showed in the previous section how to access the cluster detail page.<\/p>\n\n\n\n<p>The cluster detail page has several tabs, including the Event Log. If you are experiencing cluster exceptions (e.g. loss of connection), you may get more detail here.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Alternatives_To_Using_Databricks_For_Debugging\"><\/span>Alternatives To Using Databricks For Debugging<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>There are plenty of critics who consider the Databricks environment to be unsuitable for debugging a complex code base. They prefer to use the more extensive features and tools provided by IDEs like Visual Studio or PyCharm.<\/p>\n\n\n\n<p>Personally, I think that most tasks required for moving and transforming data around the enterprise can be debugged within a notebook using the techniques described in this article.<\/p>\n\n\n\n<p>However, complex requirements with lots of conditional logic and execution paths are probably not suitable for notebooks. They\u2019ll work fine when running perfectly. It just takes too much labor and time to figure out what\u2019s happening when they go wrong.<\/p>\n\n\n\n<p>I suggest that this complex code is only prototyped in notebooks. Then it should be moved to versioned scripts that can be stepped through within IDEs with more sophisticated debugging features.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Debugging is an essential part of the coding process. Even the most experienced programmers make mistakes. It&#8217;s through debugging that these mistakes are found and fixed. Each cell in a Databricks notebook has its own output, and any error messages are also displayed in the cell output. This is usually the starting point for your &#8230; <a title=\"How To Debug Databricks Notebooks\" class=\"read-more\" href=\"https:\/\/bandittracker.com\/debug-databricks-notebooks\/\" aria-label=\"More on How To Debug Databricks Notebooks\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[29],"tags":[],"_links":{"self":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/1021"}],"collection":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/comments?post=1021"}],"version-history":[{"count":5,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/1021\/revisions"}],"predecessor-version":[{"id":1032,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/1021\/revisions\/1032"}],"wp:attachment":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/media?parent=1021"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/categories?post=1021"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/tags?post=1021"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}