{"id":978,"date":"2023-06-22T15:15:59","date_gmt":"2023-06-22T15:15:59","guid":{"rendered":"https:\/\/bandittracker.com\/?p=978"},"modified":"2023-06-27T13:36:58","modified_gmt":"2023-06-27T13:36:58","slug":"who-created-apache-spark","status":"publish","type":"post","link":"https:\/\/bandittracker.com\/who-created-apache-spark\/","title":{"rendered":"Who Created Apache Spark? (Origins Explained)"},"content":{"rendered":"\n<p>Did you know that we can trace the origins of Apache Spark back to Google\u2019s published research on distributed processing? But Google didn\u2019t create Spark.<\/p>\n\n\n\n<p>This article looks at what came before Spark, how and where the innovative technology was created, and the key individuals involved in its inception.<\/p>\n\n\n\n<p>Let&#8217;s start at the beginning and trace the <em>spark<\/em> (pun intended) that ignited the big data revolution.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_63 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6601c210d28ea\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6601c210d28ea\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/bandittracker.com\/who-created-apache-spark\/#Googles_MapReduce_Where_It_All_Began\" title=\"Google\u2019s MapReduce: Where It All Began\">Google\u2019s MapReduce: Where It All Began<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/bandittracker.com\/who-created-apache-spark\/#Origins_Of_Hadoop_MapReduce\" title=\"Origins Of Hadoop MapReduce\">Origins Of Hadoop MapReduce<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/bandittracker.com\/who-created-apache-spark\/#Origins_Of_Apache_Spark\" title=\"Origins Of Apache Spark\">Origins Of Apache Spark<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/bandittracker.com\/who-created-apache-spark\/#Early_Key_Contributors_To_Spark\" title=\"Early Key Contributors To Spark\">Early Key Contributors To Spark<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Googles_MapReduce_Where_It_All_Began\"><\/span>Google\u2019s MapReduce: Where It All Began<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>In the early 2000s, Google\u2019s search engine was struggling to deal with indexing an exponentially growing amount of data on the web.<\/p>\n\n\n\n<p>The company needed a more efficient way to process the vast amount of data. Google had several research teams working on a problem that threatened its future.<\/p>\n\n\n\n<p>In 2004, Google engineers Jeffrey Dean and Sanjay Ghemawat released a <a href=\"https:\/\/static.googleusercontent.com\/media\/research.google.com\/en\/\/archive\/mapreduce-osdi04.pdf\" target=\"_blank\" rel=\"noreferrer noopener\">seminal paper<\/a> titled &#8220;MapReduce: Simplified Data Processing on Large Clusters&#8221;. This described a new framework for indexing.<\/p>\n\n\n\n<p>The framework introduced a programming model with two stages: Map and Reduce.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/Google-MapReduce.jpg\" alt=\"\" class=\"wp-image-985\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/Google-MapReduce.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/Google-MapReduce-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Map and Reduce<\/h3>\n\n\n\n<p>The Map stage divided the problem into numerous small sub-problems (the Map stage). These sub-problems could be processed independently across many different machines in a distributed system.<\/p>\n\n\n\n<p>The next stage would take the results from the many different machines. The data would be aggregated, sorted, and further processed. This is the Reduce stage.<\/p>\n\n\n\n<p>Distributed computing wasn\u2019t a new concept here. But the goal of MapReduce was to manage two significant challenges:<\/p>\n\n\n\n<ol type=\"1\">\n<li>High complexity<\/li>\n\n\n\n<li>Multiple machines and components for failure<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Simplifying Complexity<\/h3>\n\n\n\n<p>The programming model for processing large amounts of data across many machines is necessarily complex. But the MapReduce model hid that complexity from developers who could focus on the &#8220;map&#8221; and &#8220;reduce&#8221; functions.<\/p>\n\n\n\n<p>The underlying framework would handle the complexities of data distribution, redundancy, and fault tolerance.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Improving Reliability<\/h3>\n\n\n\n<p>A large distributed system is based on clusters of commodity hardware. There are many different parts and components that can fail and potentially bring down the entire system.<\/p>\n\n\n\n<p>The MapReduce framework included many built-in features to manage hardware or network failures.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Two Consequences Of MapReduce<\/h3>\n\n\n\n<p>It\u2019s fair to say that for Google, the main consequence of MapReduce was to make the search engine the behemoth it is today.<\/p>\n\n\n\n<p>The new approach allowed Google to process enormous amounts of data more efficiently. It was a major contributor to their ability to index the rapidly expanding internet.<\/p>\n\n\n\n<p>The second consequence was to the wider tech community. Google\u2019s MapReduce inspired other open-source distributed computing projects. That includes Apache Hadoop, that brought similar capabilities to the broader technology world.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Origins_Of_Hadoop_MapReduce\"><\/span>Origins Of Hadoop MapReduce<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The story of Hadoop starts with Doug Cutting and Mike Cafarella in the early 2000s.<\/p>\n\n\n\n<p>Cutting was working at Xerox\u2019s PARC (Palo Alto Research Center) in 2002, while Cafarell was a graduate student at the University of Washington.<\/p>\n\n\n\n<p>The pair collaborated on an independent project to develop an open-source web-search engine that could compete with commercial versions (i.e. Google, Yahoo, and others of that time).<\/p>\n\n\n\n<p>This was known as the \u201cNutch\u201d project. Their main stumbling block was handling large-scale data processing.<\/p>\n\n\n\n<p>Google published a paper on distributed file systems (Google File System or GFS) in 2003. A year later, they published their MapReduce paper.<\/p>\n\n\n\n<p>Doug Cutting realized that these technologies could solve their challenges with Nutch. He started working on an open-source implementation that included:<\/p>\n\n\n\n<ul>\n<li>a distributed file system (this became the Hadoop Distributed File System, or HDFS)<\/li>\n\n\n\n<li>a programming model that became MapReduce.<\/li>\n<\/ul>\n\n\n\n<p>Cutting named this new implementation \u201cHadoop\u201d.<\/p>\n\n\n\n<p>Why? This was the name of a toy elephant that belonged to his son. <em>Cute, eh?<\/em> <em>And here&#8217;s the logo to go with the whimsy&#8230;<\/em><\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"300\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/Apache-Hadoop-1.jpg\" alt=\"\" class=\"wp-image-983\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/Apache-Hadoop-1.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/Apache-Hadoop-1-300x150.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Enter Yahoo<\/h3>\n\n\n\n<p>Yahoo hired Doug Cutting in 2006. At the time, the company was a giant in the field of internet search. They allowed Cutting to devote his time to developing Hadoop.<\/p>\n\n\n\n<p>It\u2019s fair to say that Yahoo was a significant player in pushing Hadoop toward where it became. The commercial company gave Cutting and a small development team the space to let the open-source framework mature.<\/p>\n\n\n\n<p><em>Ironically, younger readers may be wondering\u2026Yahoo?..Who?<\/em><\/p>\n\n\n\n<p>The first usable version of Hadoop was released in 2008.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Consequences<\/h3>\n\n\n\n<p>Over time, Hadoop became the foundation for the big data ecosystem. Other systems, such as Apache Hive and Apache HBase, were built on top of the framework.<\/p>\n\n\n\n<p>Its ability to process large volumes of data across clusters of computers fundamentally changed the way companies and organizations could work with big data.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Origins_Of_Apache_Spark\"><\/span>Origins Of Apache Spark<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>A year after the Hadoop release of 2008, a research student named Matei Zaharia was starting his Ph.D. project at Berkeley\u2019s AMPLab at the University of California. AMPLab stands for the Algorithms, Machines, and People Lab.<\/p>\n\n\n\n<p>Zaharia wanted to fix two major issues with Hadoop. The first was the speed limitations with massive data due to Hadoop being disk-based (i.e. frequently writing to physical hardware).<\/p>\n\n\n\n<p>The second was that the framework was focused on batch processing and Zaharia wanted to extend the capabilities for interactive queries and streaming.<\/p>\n\n\n\n<p>He started collaborating with other distributed computing PhD students on a class project that they called the Mesos platform. Zaharia wrote a machine learning engine to run on top of the platform. He called it Spark. <\/p>\n\n\n\n<p>Meanwhile, a machine learning PhD student called Lester Mackey hit Zaharia up for some help. Mackey was a member of a student team trying to win a programming competition sponsored by Netflix. <\/p>\n\n\n\n<p>The goal was to build a recommendation engine. But Mackey was concerned that his team&#8217;s models were running too slow on Hadoop. <\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"400\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/slow-computer-frustration.jpg\" alt=\"\" class=\"wp-image-919\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/slow-computer-frustration.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/slow-computer-frustration-300x200.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<p>Zaharia helped him get set up on Spark which drove their competition entry to second place.<\/p>\n\n\n\n<p>The following year, Zaharia open-sourced the early software under a BSD license. This is the Berkeley Software Distribution that allows other developers to freely use, modify, and distribute the software.<\/p>\n\n\n\n<p>An increasing number of Berkeley students and professors were excited by the capabilities of this new project and contributed to the development of Spark. <\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Enter Apache<\/h3>\n\n\n\n<p>The Apache Software Foundation (ASF) is a non-profit corporation established in 1999. Its mission is to provide software for the public good.<\/p>\n\n\n\n<p>Each Apache project is managed by a self-selected team of technical experts who are active contributors to the project.<\/p>\n\n\n\n<p>The Berkeley team switched the licensing to the Apache Software Foundation in 2013. It was so popular with the developer community that it became a <a rel=\"noreferrer noopener\" href=\"https:\/\/spark.apache.org\/news\/spark-becomes-tlp.html\" target=\"_blank\">Top-Level project<\/a> the following year.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"300\" src=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/Apache-Spark.jpg\" alt=\"\" class=\"wp-image-984\" srcset=\"https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/Apache-Spark.jpg 600w, https:\/\/bandittracker.com\/wp-content\/uploads\/2023\/06\/Apache-Spark-300x150.jpg 300w\" sizes=\"(max-width: 600px) 100vw, 600px\" \/><\/figure><\/div>\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Early_Key_Contributors_To_Spark\"><\/span>Early Key Contributors To Spark<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>There were many more contributors to the early Spark versions than I can mention here, but I\u2019ll give a few.<\/p>\n\n\n\n<p>Michael Franklin wasn\u2019t a developer per se, but as the Director of the AMPLab, he played a crucial leadership role in steering the project.<\/p>\n\n\n\n<p>Mosharaf Chowdhury was a Ph.D. student at the time who contributed several features to the code base. He has continued in the academic world and became professor of computer science at the University of Michigan.<\/p>\n\n\n\n<p>Tathagata Das also contributed to early streaming aspects for real-time data processing.<\/p>\n\n\n\n<p>I\u2019ve already mentioned Ion Stoica. As a professor and Matei Zaharia\u2019s Ph.D. supervisor, he also co-authored academic papers underlying the framework.<\/p>\n\n\n\n<p>Of course, Matei Zaharia was the catalyst for the Spark project. He went on to co-found Databricks with some other key contributors.<\/p>\n\n\n\n<p>Reynold Xin was a lead developer and one of the earliest committers of code on the project. Andy Konwinski and Patrick Wendell were other early contributors. <\/p>\n\n\n\n<p>Several of these individuals became <a href=\"https:\/\/bandittracker.com\/who-created-databricks\/\" data-type=\"post\" data-id=\"988\">co-founders of Databricks<\/a> (another interesting story!).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Did you know that we can trace the origins of Apache Spark back to Google\u2019s published research on distributed processing? But Google didn\u2019t create Spark. This article looks at what came before Spark, how and where the innovative technology was created, and the key individuals involved in its inception. Let&#8217;s start at the beginning and &#8230; <a title=\"Who Created Apache Spark? (Origins Explained)\" class=\"read-more\" href=\"https:\/\/bandittracker.com\/who-created-apache-spark\/\" aria-label=\"More on Who Created Apache Spark? (Origins Explained)\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[29],"tags":[],"_links":{"self":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/978"}],"collection":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/comments?post=978"}],"version-history":[{"count":6,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/978\/revisions"}],"predecessor-version":[{"id":1008,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/posts\/978\/revisions\/1008"}],"wp:attachment":[{"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/media?parent=978"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/categories?post=978"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bandittracker.com\/wp-json\/wp\/v2\/tags?post=978"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}