John Summer In today’s fast-evolving technological landscape, data has been coined as the new oil. Databricks is a powerful data analytics platform at the heart of this data revolution. You may be wondering if this new platform has already been adopted by large well-known companies. The answer is an indisputable yes! This article looks at its far-reaching … Read more
A common Databricks task is to schedule a job that calls a notebook. However, sometimes you will want to do this in reverse. In other words, you want to run a notebook that launches a job. This is most likely when you have set up a job that you want to test without creating a … Read more
Debugging is an essential part of the coding process. Even the most experienced programmers make mistakes. It’s through debugging that these mistakes are found and fixed. Each cell in a Databricks notebook has its own output, and any error messages are also displayed in the cell output. This is usually the starting point for your … Read more
In Databricks, notebooks provide the ability to develop real-time machine learning, data enginering, and data analytics workflows. Notebooks support four programming languages: You can use one or all languages within a single notebook. If you’re starting out, you may be wondering if one of these languages is better for your purposes. Does the company favor … Read more
Delta tables are a core feature of Databricks. This article assumes you are completely new to the platform but have some familiarity with SQL or Spark basics. Read on to get an understanding of what they are, how they differ from other storage formats, and how to create and use them. What Is A Delta … Read more
The default SQL standard used by Databricks has changed since the platform first launched. Databricks originally used Spark SQL as the default SQL dialect, but changed the standard in late 2021. The default dialect in Databricks is currently ANSI Standard SQL. Because there are differences between the two dialects, developers and analysts should be aware … Read more
If you want the story of how a group of seven PhD students and professors at Berkeley came to co-found a fast-growing data and AI company, then this article is for you. If you just want the list of co-founders of Databricks, then here you are (in alphabetical order): Read on to learn more about … Read more
Did you know that we can trace the origins of Apache Spark back to Google’s published research on distributed processing? But Google didn’t create Spark. This article looks at what came before Spark, how and where the innovative technology was created, and the key individuals involved in its inception. Let’s start at the beginning and … Read more
Before you start exchanging data between Databricks and S3, you need to have the necessary permissions in place. We have a separate article that takes you through configuring S3 permissions for Databricks access. The rest of this article provides code examples for common use cases when reading and writing data with Databricks and S3. Create … Read more
Permissions in AWS are defined by IAM roles (Identity and Access Management). To work with files in S3, your Databricks deployment will need an IAM role that has read/write permissions to an S3 bucket. You have two configuration options: The Unity Catalog is a new feature from Databricks this aims to simplify what you otherwise … Read more
