{"id":970,"date":"2023-06-20T11:52:51","date_gmt":"2023-06-20T11:52:51","guid":{"rendered":"https:\/\/bandittracker.com\/?p=970"},"modified":"2023-06-20T11:52:53","modified_gmt":"2023-06-20T11:52:53","slug":"configure-permissions-and-roles-for-databricks-and-s3","status":"publish","type":"post","link":"https:\/\/bandittracker.com\/configure-permissions-and-roles-for-databricks-and-s3\/","title":{"rendered":"Configure Permissions And Roles For Databricks And S3"},"content":{"rendered":"\n<p>Permissions in AWS are defined by IAM roles (Identity and Access Management).<\/p>\n\n\n\n<p>To work with files in S3, your Databricks deployment will need an IAM role that has read\/write permissions to an S3 bucket. You have two configuration options:<\/p>\n\n\n\n<ol type=\"1\">\n<li>Use the Unity Catalog in Databricks.<\/li>\n\n\n\n<li>Set up the roles and permissions in AWS.<\/li>\n<\/ol>\n\n\n\n<p>The Unity Catalog is a new feature from Databricks this aims to simplify what you otherwise must set up in AWS. However, you may not have access to it.<\/p>\n\n\n\n<p>This article walks you through setting up the roles and permissions directly.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_63 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-660e6464409ea\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-660e6464409ea\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/bandittracker.com\/configure-permissions-and-roles-for-databricks-and-s3\/#Should_You_Mount_S3_on_DBFS\" title=\"Should You Mount S3 on DBFS?\">Should You Mount S3 on DBFS?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/bandittracker.com\/configure-permissions-and-roles-for-databricks-and-s3\/#Prerequisites_For_This_Tutorial\" title=\"Prerequisites For This Tutorial\">Prerequisites For This Tutorial<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/bandittracker.com\/configure-permissions-and-roles-for-databricks-and-s3\/#Four-Stage_Process\" title=\"Four-Stage Process\">Four-Stage Process<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/bandittracker.com\/configure-permissions-and-roles-for-databricks-and-s3\/#Stage_1_Create_The_Instance_Profile_IAM_Role\" title=\"Stage 1: Create The Instance Profile (IAM Role)\">Stage 1: Create The Instance Profile (IAM Role)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/bandittracker.com\/configure-permissions-and-roles-for-databricks-and-s3\/#Stage_2_Link_Back_From_the_Bucket_To_the_Instance_Profile\" title=\"Stage 2: Link Back From the Bucket To the Instance Profile\">Stage 2: Link Back From the Bucket To the Instance Profile<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/bandittracker.com\/configure-permissions-and-roles-for-databricks-and-s3\/#Stage_3_Add_The_Intance_Profile_To_The_EC2_Instance_Policy\" title=\"Stage 3: Add The Intance Profile To The EC2 Instance Policy\">Stage 3: Add The Intance Profile To The EC2 Instance Policy<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/bandittracker.com\/configure-permissions-and-roles-for-databricks-and-s3\/#Stage_4_Hook_Up_The_Instance_Profile_To_Your_Workspace\" title=\"Stage 4: Hook Up The Instance Profile To Your Workspace\">Stage 4: Hook Up The Instance Profile To Your Workspace<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Should_You_Mount_S3_on_DBFS\"><\/span>Should You Mount S3 on DBFS?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>There used to be a third option of mounting S3 buckets on DBFS. We used to do this by using the dbutils.fs.mount command.<\/p>\n\n\n\n<p>However, this method is deprecated i.e. Databricks doesn\u2019t want us to do this anymore.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/stop.jpg\" alt=\"\" class=\"wp-image-971\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/stop.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/stop-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure>\n\n\n\n<p>Instead, use the Unity Catalog or directly configure the IAM roles as shown in the rest of this article.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Prerequisites_For_This_Tutorial\"><\/span>Prerequisites For This Tutorial<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>I assume that you have a Databricks account and <a href=\"https:\/\/bandittracker.com\/what-is-a-databricks-workspace\/\" data-type=\"post\" data-id=\"944\">workspace<\/a>. You will also need an AWS S3 bucket for Databricks to work with.<\/p>\n\n\n\n<p>Follow these steps:<\/p>\n\n\n\n<ol type=\"1\">\n<li>Login into AWS and click on the \u201cCreate Bucket\u201d button.<\/li>\n\n\n\n<li>Set the name and region and save your choices.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Four-Stage_Process\"><\/span>Four-Stage Process<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The Databricks\/S3 permissions configuration is a four-stage process:<\/p>\n\n\n\n<ul>\n<li>Stage 1: create an IAM role that grants access from EC2 to an S3 bucket.<\/li>\n\n\n\n<li>Stage 2: create a bucket policy that links back to the stage 1 role.<\/li>\n\n\n\n<li>Stage 3: add the stage 1 role to the EC2 role you used when deploying Databricks.<\/li>\n\n\n\n<li>Stage 4: hook up the stage 1 role to your Databricks workspace.<\/li>\n<\/ul>\n\n\n\n<p>Confused? Thought that the whole point of Databricks was to simplify all this? <\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"400\" height=\"500\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/confused.jpg\" alt=\"\" class=\"wp-image-907\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/confused.jpg 400w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/confused-240x300.jpg 240w\" sizes=\"(max-width: 400px) 100vw, 400px\" \/><\/figure>\n\n\n\n<p>Well, that\u2019s what the Unity Calalog is for.<\/p>\n\n\n\n<p>But don\u2019t worry. Just follow the steps in the next sections.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Stage_1_Create_The_Instance_Profile_IAM_Role\"><\/span>Stage 1: Create The Instance Profile (IAM Role)<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Databricks calls this IAM role for EC2 access an \u201cInstance Profile\u201d. This is how to set it up:<\/p>\n\n\n\n<ol type=\"1\">\n<li>Log into AWS.<\/li>\n\n\n\n<li>Click on \u201cRoles\u201d in the left pane and choose \u201cCreate Role\u201d.<\/li>\n\n\n\n<li>Choose \u201cAWS Service\u201d as the trusted entity and \u201cEC2\u201d as the use case.<\/li>\n\n\n\n<li>Click \u201cNext\u201d twice.<\/li>\n\n\n\n<li>Give the role a name and click \u201cCreate Role\u201d.<\/li>\n<\/ol>\n\n\n\n<p>Now that you have your Instance Profile, you need to give it access to the S3 bucket you just created.<\/p>\n\n\n\n<ol type=\"1\">\n<li>Find the role in Role List and click on the name.<\/li>\n\n\n\n<li>Go to the Permissions tab.<\/li>\n\n\n\n<li>Click \u201cAdd Permissions -&gt; Inline Policy\u201d.<\/li>\n\n\n\n<li>Go to the JSON tab.<\/li>\n<\/ol>\n\n\n\n<p>Here is sample JSON. You just need to replace the resource details with your bucket name.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": &#91;\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": &#91;\n        \"s3:ListBucket\"\n      ],\n     \"Resource\": &#91;\n        \"arn:aws:s3:::&lt;s3-bucket-name>\"\n      ]\n    },\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": &#91;\n        \"s3:PutObject\",\n        \"s3:GetObject\",\n        \"s3:DeleteObject\",\n        \"s3:PutObjectAcl\"\n      ],\n      \"Resource\": &#91;\n         \"arn:aws:s3:::&lt;s3-bucket-name>\/*\"\n      ]\n    }\n  ]\n}<\/code><\/pre>\n\n\n\n<p>Once you\u2019ve pasted in your policy and reviewed it, give it a name and click \u201cCreate Policy\u201d.<\/p>\n\n\n\n<p>The next display shows a summary of what has been created. <strong>At this point, you should take a copy of the Role ARN (you need this for stage 4).<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Stage_2_Link_Back_From_the_Bucket_To_the_Instance_Profile\"><\/span>Stage 2: Link Back From the Bucket To the Instance Profile<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Now you tell AWS that the bucket connects back to the instance profile.<\/p>\n\n\n\n<ol type=\"1\">\n<li>Open the IAM console.<\/li>\n\n\n\n<li>Click on \u201cPolicies\u201d in the left pane and choose \u201cCreate Policy\u201d.<\/li>\n\n\n\n<li>Use the template below to add a policy with read\/write\/list permissions.<\/li>\n\n\n\n<li>Name the policy and create it.<\/li>\n<\/ol>\n\n\n\n<p>Here is a sample policy where you should replace:<\/p>\n\n\n\n<ul>\n<li>The AWS Account ID<\/li>\n\n\n\n<li>The Instance Profile role<\/li>\n\n\n\n<li>The bucket name<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": &#91;\n    {\n      \"Sid\": \"Example permissions\",\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"AWS\": \"arn:aws:iam::&lt;aws-account-id-databricks>:role\/&lt;iam-role-for-s3-access>\"\n      },\n      \"Action\": &#91;\n        \"s3:GetBucketLocation\",\n        \"s3:ListBucket\"\n      ],\n      \"Resource\": \"arn:aws:s3:::&lt;s3-bucket-name>\"\n    },\n    {\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"AWS\": \"arn:aws:iam::&lt;aws-account-id-databricks>:role\/&lt;iam-role-for-s3-access>\"\n      },\n      \"Action\": &#91;\n        \"s3:PutObject\",\n        \"s3:GetObject\",\n        \"s3:DeleteObject\",\n        \"s3:PutObjectAcl\"\n      ],\n      \"Resource\": \"arn:aws:s3:::&lt;s3-bucket-name>\/*\"\n    }\n  ]\n}\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Stage_3_Add_The_Intance_Profile_To_The_EC2_Instance_Policy\"><\/span>Stage 3: Add The Intance Profile To The EC2 Instance Policy<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The Instance Profile is what you created in Stage 1. You also need the details of the IAM role that you used when you first created your Databricks account. To get this:<\/p>\n\n\n\n<ol type=\"1\">\n<li>Go to Workspaces.<\/li>\n\n\n\n<li>Click your workspace name.<\/li>\n\n\n\n<li>Scroll down to the credentials box.<\/li>\n\n\n\n<li>Grab the role from the long \u201cARN\u201d string.<\/li>\n<\/ol>\n\n\n\n<p>Suppose the Role ARN is this: <strong><em>rn:aws:iam:999988887777:role\/data-tasks<\/em><\/strong><\/p>\n\n\n\n<p>The role name is \u201cdata-tasks\u201d in this example.<\/p>\n\n\n\n<p>Now you know the role name, follow these steps:<\/p>\n\n\n\n<ol type=\"1\">\n<li>Open the IAM console.<\/li>\n\n\n\n<li>Click on \u201cRoles\u201d in the left pane and find the Databricks deployment role.<\/li>\n\n\n\n<li>Go to the Permissions tab.<\/li>\n\n\n\n<li>Edit the poilcy.<\/li>\n\n\n\n<li>Add the section below to the JSON.<\/li>\n<\/ol>\n\n\n\n<p>Replace \u201ciam-role-instance-profile\u201d with the role from Stage 1.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>    {\n      \"Effect\": \"Allow\",\n      \"Action\": \"iam:PassRole\",\n      \"Resource\": \"arn:aws:iam::&lt;aws-account-id-databricks>:role\/&lt;iam-role-instance-profile>\"\n    }<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Stage_4_Hook_Up_The_Instance_Profile_To_Your_Workspace\"><\/span>Stage 4: Hook Up The Instance Profile To Your Workspace<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Whew, nearly finished! Follow these steps in Databricks:<\/p>\n\n\n\n<ol type=\"1\">\n<li>Open the admin settings.<\/li>\n\n\n\n<li>Go to the Instance Profiles tab.<\/li>\n<\/ol>\n\n\n\n<p>Add the Instance Profile you created in Stage 1.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Permissions in AWS are defined by IAM roles (Identity and Access Management). To work with files in S3, your Databricks deployment will need an IAM role that has read\/write permissions to an S3 bucket. You have two configuration options: The Unity Catalog is a new feature from Databricks this aims to simplify what you otherwise &#8230; <a title=\"Configure Permissions And Roles For Databricks And S3\" class=\"read-more\" href=\"https:\/\/bandittracker.com\/configure-permissions-and-roles-for-databricks-and-s3\/\" aria-label=\"More on Configure Permissions And Roles For Databricks And S3\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[29],"tags":[],"_links":{"self":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/970"}],"collection":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/comments?post=970"}],"version-history":[{"count":2,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/970\/revisions"}],"predecessor-version":[{"id":973,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/970\/revisions\/973"}],"wp:attachment":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/media?parent=970"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/categories?post=970"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/tags?post=970"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}