{"id":966,"date":"2023-06-20T11:03:23","date_gmt":"2023-06-20T11:03:23","guid":{"rendered":"https:\/\/bandittracker.com\/?p=966"},"modified":"2023-06-20T11:03:25","modified_gmt":"2023-06-20T11:03:25","slug":"databricks-dbfs-beginner-guide","status":"publish","type":"post","link":"https:\/\/bandittracker.com\/databricks-dbfs-beginner-guide\/","title":{"rendered":"Working With Databricks DBFS (For Beginners)"},"content":{"rendered":"\n<p>The local file system in Databricks is known as the DBFS. This article explains the underlying concepts of DBFS for Databricks beginners and people who are new to cloud storage.<\/p>\n\n\n\n<p>Once you\u2019ve grasped the concepts, the article shows you how to:<\/p>\n\n\n\n<ul>\n<li>Access and browse the DBFS with the web interface.<\/li>\n\n\n\n<li>List DBFS files from a Notebook.<\/li>\n\n\n\n<li>List DBFS files using the REST API.<\/li>\n\n\n\n<li>List DBFS files using the CLI.<\/li>\n<\/ul>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_65 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-66285cc3d6e05\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-66285cc3d6e05\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/bandittracker.com\/databricks-dbfs-beginner-guide\/#What_Is_Databricks_DBFS\" title=\"What Is Databricks DBFS?\">What Is Databricks DBFS?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/bandittracker.com\/databricks-dbfs-beginner-guide\/#How_DBFS_Relates_To_Other_Files_Systems\" title=\"How DBFS Relates To Other Files Systems\">How DBFS Relates To Other Files Systems<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/bandittracker.com\/databricks-dbfs-beginner-guide\/#How_To_Access_And_Browse_DBFS_In_The_Web_Inteferace\" title=\"How To Access And Browse DBFS In The Web Inteferace\">How To Access And Browse DBFS In The Web Inteferace<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/bandittracker.com\/databricks-dbfs-beginner-guide\/#How_To_Get_A_List_Of_Files_In_DBFS_In_A_Notebook\" title=\"How To Get A List Of Files In DBFS In A Notebook\">How To Get A List Of Files In DBFS In A Notebook<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/bandittracker.com\/databricks-dbfs-beginner-guide\/#How_To_List_DBFS_Files_Using_The_Rest_API\" title=\"How To List DBFS Files Using The Rest API\">How To List DBFS Files Using The Rest API<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/bandittracker.com\/databricks-dbfs-beginner-guide\/#How_To_List_DBFS_Files_Using_The_Databricks_CLI\" title=\"How To List DBFS Files Using The Databricks CLI\">How To List DBFS Files Using The Databricks CLI<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_Is_Databricks_DBFS\"><\/span>What Is Databricks DBFS?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Even if you\u2019re new to the Cloud, you will be familiar with the local file system on your personal laptop or desktop computer. The visual layout organizes your files into folders and sub-folders.<\/p>\n\n\n\n<p>The underlying storage system in your computer hardware may be structured differently (e.g. pointers and addresses). But a file system gives you a simplified way of navigating and accessing what you need.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/file-system.jpg\" alt=\"\" class=\"wp-image-968\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/file-system.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/file-system-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure>\n\n\n\n<p>&#8220;DBFS&#8221; is an acronym for Databricks File System. It is a simplified layer that provides a single layout to any form of cloud storage that Databricks supports.<\/p>\n\n\n\n<p>For example, if you\u2019re not used to working with AWS S3 buckets, then you\u2019ve been spared the weird (distributed) way that the AWS file system represents a hierarchy of data objects.<\/p>\n\n\n\n<p>With DBFS between you and your chosen type of cloud storage, you don\u2019t have to worry about the complexities of distributed systems. DBFS looks like the kind of local file system you are used to.<\/p>\n\n\n\n<p>That\u2019s called \u201cunified access to an abstraction layer\u201d.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_DBFS_Relates_To_Other_Files_Systems\"><\/span>How DBFS Relates To Other Files Systems<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>There are two primary types of file systems: distributed file systems and local file systems.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Distributed File Systems<\/h3>\n\n\n\n<p>These are file systems that allow access to files from multiple hosts sharing via a computer network. This is the realm of systems like:<\/p>\n\n\n\n<ul>\n<li>Hadoop Distributed File System (HDFS)<\/li>\n\n\n\n<li>Amazon S3<\/li>\n\n\n\n<li>Azure Blob Storage<\/li>\n\n\n\n<li>Google Cloud Storage<\/li>\n<\/ul>\n\n\n\n<p>DBFS acts as an interface to these types of systems, providing a unified, simple access layer.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/snowflake-deltalake-s3-azure.jpg\" alt=\"\" class=\"wp-image-921\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/snowflake-deltalake-s3-azure.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/snowflake-deltalake-s3-azure-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Local File Systems<\/h3>\n\n\n\n<p>A local file system refers to the file system of a local machine or a Databricks cluster in this context.<\/p>\n\n\n\n<p>Files in a local file system of a <a href=\"https:\/\/bandittracker.com\/what-are-databricks-clusters\/\" data-type=\"post\" data-id=\"918\">Databricks cluster<\/a> are typically only accessible from that cluster and are lost when the cluster is terminated.<\/p>\n\n\n\n<p>DBFS provides persistence beyond the lifecycle of a cluster, as data stored in DBFS remains intact even after a cluster is terminated.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_To_Access_And_Browse_DBFS_In_The_Web_Inteferace\"><\/span>How To Access And Browse DBFS In The Web Inteferace<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Databricks provides a web-based user interface where you can access and browse DBFS. This feature can be enabled or disabled by the workspace admin user.<\/p>\n\n\n\n<p>If you\u2019re using the Community Edition, it\u2019s disabled by default. To enable it, follow these steps:<\/p>\n\n\n\n<ol type=\"1\">\n<li>Click the settings gearbox in the left pane (down at the bottom)<\/li>\n\n\n\n<li>Go to the Workspace Settings tab.<\/li>\n\n\n\n<li>Scroll down to the Advanced section.<\/li>\n\n\n\n<li>Enable \u201cDBFS File Browser\u201d.<\/li>\n\n\n\n<li>Refresh the page.<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/databricks-enable-dbfs-file-browser.jpg\" alt=\"\" class=\"wp-image-967\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/databricks-enable-dbfs-file-browser.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/databricks-enable-dbfs-file-browser-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure>\n\n\n\n<p>When the browser is enabled, you can access it with these steps:<\/p>\n\n\n\n<ol type=\"1\">\n<li>Chose the Data tab on the sidebar of your Databricks workspace<\/li>\n\n\n\n<li>Click on the &#8220;DBFS&#8221; button at the top of the page.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_To_Get_A_List_Of_Files_In_DBFS_In_A_Notebook\"><\/span>How To Get A List Of Files In DBFS In A Notebook<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>If you&#8217;re working within a Databricks notebook, you can use the dbutils.fs module to list the contents of a directory in DBFS.<\/p>\n\n\n\n<p>Here&#8217;s an example that lists the contents of the root directory:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>display(dbutils.fs.ls(\"\/\"))<\/code><\/pre>\n\n\n\n<p><em>If you&#8217;re just getting started, check out our beginner <a href=\"https:\/\/bandittracker.com\/what-is-a-databricks-notebook\/\" data-type=\"post\" data-id=\"926\">tutorial on using Databricks notebooks<\/a>.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_To_List_DBFS_Files_Using_The_Rest_API\"><\/span>How To List DBFS Files Using The Rest API<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The DBFS API offers a programmatic interface to DBFS. The API consists of a set of operations that can be used to manage files and directories in DBFS.<\/p>\n\n\n\n<p>The DBFS API is a RESTful API that uses standard HTTP methods like GET, PUT, and DELETE to interact with files and directories in DBFS.<\/p>\n\n\n\n<p>This example lists files in the root directory using a curl command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl https:\/\/&lt;databricks-url>\/api\/2.0\/dbfs\/list -d '{\n\n\"path\": \"\/\"\n\n}'<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_To_List_DBFS_Files_Using_The_Databricks_CLI\"><\/span>How To List DBFS Files Using The Databricks CLI<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>When you install the Databricks CLI on your local machine, you can use it to list files in a directory in DBFS.<\/p>\n\n\n\n<p>Here\u2019s an example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>databricks fs ls dbfs:\/<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>The local file system in Databricks is known as the DBFS. This article explains the underlying concepts of DBFS for Databricks beginners and people who are new to cloud storage. Once you\u2019ve grasped the concepts, the article shows you how to: What Is Databricks DBFS? Even if you\u2019re new to the Cloud, you will be &#8230; <a title=\"Working With Databricks DBFS (For Beginners)\" class=\"read-more\" href=\"https:\/\/bandittracker.com\/databricks-dbfs-beginner-guide\/\" aria-label=\"More on Working With Databricks DBFS (For Beginners)\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[29],"tags":[],"_links":{"self":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/966"}],"collection":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/comments?post=966"}],"version-history":[{"count":1,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/966\/revisions"}],"predecessor-version":[{"id":969,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/966\/revisions\/969"}],"wp:attachment":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/media?parent=966"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/categories?post=966"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/tags?post=966"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}