{"id":1009,"date":"2023-06-29T09:53:39","date_gmt":"2023-06-29T09:53:39","guid":{"rendered":"https:\/\/bandittracker.com\/?p=1009"},"modified":"2023-06-29T09:53:41","modified_gmt":"2023-06-29T09:53:41","slug":"what-languages-does-databricks-use","status":"publish","type":"post","link":"https:\/\/bandittracker.com\/what-languages-does-databricks-use\/","title":{"rendered":"What Languages Does Databricks Use? (Explained)"},"content":{"rendered":"\n<p>In Databricks, notebooks provide the ability to develop real-time machine learning, data enginering, and data analytics workflows. Notebooks support four programming languages:<\/p>\n\n\n\n<ul>\n<li>Python<\/li>\n\n\n\n<li>R<\/li>\n\n\n\n<li>Scala<\/li>\n\n\n\n<li>SQL<\/li>\n<\/ul>\n\n\n\n<p>You can use one or all languages within a single notebook. If you&#8217;re starting out, you may be wondering if one of these languages is better for your purposes.<\/p>\n\n\n\n<p>Does the company favor a specific language? Which runs faster on the platform? Which scales better? Read on to find out.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_63 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6606570e2a82a\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6606570e2a82a\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/bandittracker.com\/what-languages-does-databricks-use\/#What_Language_Is_Databricks_Written_In\" title=\"What Language Is Databricks Written In?\">What Language Is Databricks Written In?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/bandittracker.com\/what-languages-does-databricks-use\/#Which_Languages_Are_Used_Inside_The_Company\" title=\"Which Languages Are Used Inside The Company?\">Which Languages Are Used Inside The Company?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/bandittracker.com\/what-languages-does-databricks-use\/#Python_As_The_Most_Commonly_Used_Databricks_Language\" title=\"Python As The Most Commonly Used Databricks Language\">Python As The Most Commonly Used Databricks Language<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/bandittracker.com\/what-languages-does-databricks-use\/#Should_You_Use_Python_Or_PySpark_In_Databricks\" title=\"Should You Use Python Or PySpark In Databricks?\">Should You Use Python Or PySpark In Databricks?<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_Language_Is_Databricks_Written_In\"><\/span>What Language Is Databricks Written In?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Databricks as a platform is written in Scala, one of the four languages it supports. <\/p>\n\n\n\n<p>This is because the platform runs on top of Apache Spark, which is written in Scala. Databricks provides an abstract layer to simplify operations on the distributed Spark clusters beneath it.<\/p>\n\n\n\n<p>Because Spark is Scala-based, the language is a natural choice for the internal workings of APIs that run on top of it. <\/p>\n\n\n\n<p>However, when Databricks was first conceived as a commercial venture, it would have been technically possible for the founders to choose a different programming language. For example, they could have written their layer in Java. After all, Scala itself runs on the JVM (Java virtual machine).<\/p>\n\n\n\n<p>So, why did the original Databricks developers choose Scala? That&#8217;s easy to understand when you know the background of the <a href=\"https:\/\/bandittracker.com\/who-created-databricks\/\" data-type=\"post\" data-id=\"988\">seven co-founders of Databricks<\/a>. They were all involved in the early development of Spark (written in Scala).<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/databricks.jpg\" alt=\"\" class=\"wp-image-991\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/databricks.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/databricks-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Which_Languages_Are_Used_Inside_The_Company\"><\/span>Which Languages Are Used Inside The Company?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Scala is also widely used within the company when implementing all the surrounding tools and glue that facilitates <a href=\"https:\/\/bandittracker.com\/what-is-a-databricks-notebook\/\" data-type=\"post\" data-id=\"926\">notebooks<\/a>, <a href=\"https:\/\/bandittracker.com\/delta-tables-in-databricks\/\" data-type=\"post\" data-id=\"1002\">delta tables<\/a>, jobs, and workflows. A company blog post in 2021 described Scala as:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>a kind of lingua franca<\/p>\n<cite><a href=\"https:\/\/www.databricks.com\/blog\/2021\/12\/03\/scala-at-scale-at-databricks.html\" target=\"_blank\" rel=\"noreferrer noopener\">Company blog post<\/a><\/cite><\/blockquote>\n\n\n\n<p>However, it&#8217;s not the only language in use &#8211; as you&#8217;d expect from a medium-sized developer workshop. Reynold Xin, one of the co-founders, answered a question on Quora in 2021 as to what languages are used in Databricks. This is what he said:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>The top languages for our internal development are: Scala, Typescript, JavaScript, Python, Jsonnet, Java, and some Go. <\/p>\n<cite><a rel=\"noreferrer noopener\" href=\"https:\/\/www.quora.com\/What-does-Databricks-do\" target=\"_blank\">Reynold Xin<\/a><a rel=\"noreferrer noopener\" href=\"https:\/\/www.quora.com\/What-does-Databricks-do\" target=\"_blank\">,<\/a><a rel=\"noreferrer noopener\" href=\"https:\/\/www.quora.com\/What-does-Databricks-do\" target=\"_blank\"> <\/a><a rel=\"noreferrer noopener\" href=\"https:\/\/www.quora.com\/What-does-Databricks-do\" target=\"_blank\">2<\/a><a rel=\"noreferrer noopener\" href=\"https:\/\/www.quora.com\/What-does-Databricks-do\" target=\"_blank\">0<\/a><a rel=\"noreferrer noopener\" href=\"https:\/\/www.quora.com\/What-does-Databricks-do\" target=\"_blank\">2<\/a><a href=\"https:\/\/www.quora.com\/What-does-Databricks-do\" target=\"_blank\" rel=\"noreferrer noopener\">1<\/a><\/cite><\/blockquote>\n\n\n\n<p>So, there you have it from the horse&#8217;s mouth. Now that you know the importance of Scala, you may be thinking that it must be the preferred language when using the platform. But that&#8217;s not actually true.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Python_As_The_Most_Commonly_Used_Databricks_Language\"><\/span>Python As The Most Commonly Used Databricks Language<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>In some ways, it doesn&#8217;t really matter what is used within the company. The choice of the customer base is more important.<\/p>\n\n\n\n<p>Xin expanded on his 2021 Quora comment with this:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote\">\n<p>Our customers use primarily Python, SQL, and some R and Scala.<\/p>\n<cite><a href=\"https:\/\/www.quora.com\/What-does-Databricks-do\" target=\"_blank\" rel=\"noreferrer noopener\">Reynold Xin, 2021<\/a><\/cite><\/blockquote>\n\n\n\n<p>It doesn&#8217;t surprise me that Python is the number one. All the surveys amongst data engineers rank it as the most popular language in the industry. This is shown in the <a href=\"https:\/\/www.tiobe.com\/tiobe-index\/\" data-type=\"URL\" data-id=\"https:\/\/www.tiobe.com\/tiobe-index\/\" target=\"_blank\" rel=\"noreferrer noopener\">TIOBE index<\/a> where Python was number 1 in 2022 and 2023.<\/p>\n\n\n\n<p>These are the rankings for the four languages in 2023:<\/p>\n\n\n\n<ul>\n<li>Python: 1<\/li>\n\n\n\n<li>SQL: 9<\/li>\n\n\n\n<li>R: 17<\/li>\n\n\n\n<li>Scala: 23<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Should_You_Use_Python_Or_PySpark_In_Databricks\"><\/span>Should You Use Python Or PySpark In Databricks?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Some data scientists who are new to Databricks may be mostly familiar with using Python and Pandas to work with their data.<\/p>\n\n\n\n<p>It could be tempting to continue to work exclusively within this framework. But that would be a mistake if you are working with large datasets. I advise that you become familiar with PySpark.<\/p>\n\n\n\n<p>Here&#8217;s the reason why. The Databricks platform is designed to distribute your large data workloads across nodes working in parallel on a cluster. But the language that you use must be able to tell the platform to do so.<\/p>\n\n\n\n<p>Plain old Python runs in a single process unless you do clever things to make it launch multiple processes in parallel. (<em>You can <a href=\"https:\/\/bandittracker.com\/run-databricks-notebooks-in-parallel\/\" data-type=\"post\" data-id=\"951\">run Databricks notebooks in parallel<\/a> but it takes a bit of extra coding<\/em>).<\/p>\n\n\n\n<p>Similarly to Python, the Pandas library is designed to run on a single machine.<\/p>\n\n\n\n<p>In contrast, PySpark is an API that distributes your Python load statements, queries, and transformations to run in parallel across multiple nodes.<\/p>\n\n\n\n<p>To help with Pandas, Databricks launched an API called Koalas that implemented the single-process Pandas DataFrames as multi-node Spark RDD data structures. <\/p>\n\n\n\n<p>But wait &#8211; you don&#8217;t have to get familiar with the marsupial-named library. Koalas was conveniently merged into PySpark in 2021.<\/p>\n\n\n\n<p>To get the benefits of using Pandas with parallel DataFrames (really being RDDs under the hood), all you need to do is add a single line of code at the top of your notebooks. It will look like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from pyspark.pandas import read_csv<\/code><\/pre>\n\n\n\n<p>From there, you can simply perform the usual Pandas data import and transformations that you&#8217;re used to. The PySpark library implements them as distributed Spark RDDs.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In Databricks, notebooks provide the ability to develop real-time machine learning, data enginering, and data analytics workflows. Notebooks support four programming languages: You can use one or all languages within a single notebook. If you&#8217;re starting out, you may be wondering if one of these languages is better for your purposes. Does the company favor &#8230; <a title=\"What Languages Does Databricks Use? (Explained)\" class=\"read-more\" href=\"https:\/\/bandittracker.com\/what-languages-does-databricks-use\/\" aria-label=\"More on What Languages Does Databricks Use? (Explained)\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[29],"tags":[],"_links":{"self":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/1009"}],"collection":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/comments?post=1009"}],"version-history":[{"count":2,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/1009\/revisions"}],"predecessor-version":[{"id":1011,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/1009\/revisions\/1011"}],"wp:attachment":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/media?parent=1009"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/categories?post=1009"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/tags?post=1009"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}