The following video by Cloudera CEO Mike Olson elaborates on the significance of Apache Spark in the Hadoop landscape, with a particular focus on its differentiation from MapReduce. The video prefigures Cloudera’s One Platform Initiative aimed at rendering Spark a viable alternative to MapReduce.
Cloudera recently announced a One Platform Initiative that aspires to make Apache Spark the default framework for processing analytics in Hadoop, ahead of MapReduce. Cloudera’s One Platform Initiative will focus on bolstering the security of Apache Spark, rendering Spark more scalable, enhancing management functionality and augmenting Spark Streaming, the Spark component that focuses on ingesting massive volumes of streaming data for use cases such as the internet of things. Cloudera’s efforts to improve the security of Apache Spark will focus on ensuring the encryption of data at rest as well as over the wire. Meanwhile, the initiative to improve the scalability of Apache Spark aims to render it scalable to as many as 10,000 nodes including enhanced ability to handle computational workloads by means of an integration with Intel’s Math Kernel Library. With respect to management, Cloudera plans to deepen Spark’s integration with YARN by creating metrics that provide insight into resource utilization as well as improvements to multi-tenant performance. Regarding Spark Streaming, Cloudera plans to render Spark Streaming more broadly available to business users via the addition of SQL semantics and the ability to support 80% of common streaming workloads.
Cloudera’s larger goal is to enhance the enterprise-readiness of Apache Spark with a view to promoting it as a viable alternative to MapReduce. All of Cloudera’s enhancements to Spark will be contributed to the Apache Spark open source project. That said, Cloudera’s leadership in stewarding the acceleration of the enterprise-readiness of Apache Spark as a MapReduce alternative promises to position it strongly as the undisputed market share and thought leader in the Hadoop distribution space, particularly given the range of its intended contributions to Spark and the depth of its vision for subsequent Spark enhancements in forthcoming months.
Cloudera and Trillium Software recently announced a collaboration whereby the Trillium Big Data solution is certified for Cloudera’s Hadoop distribution. As a result of the partnership, Cloudera customers can take advantage of Trillium’s data quality solutions to profile, cleanse, de-duplicate and enrich Hadoop-based data. Trillium responds to a problem in the Big Data industry wherein the customer focus on deployment and management of Hadoop-based data repositories eclipses concerns about data quality. In the case of Hadoop-based data, data quality solutions predictably face challenges associated with the sheer volume of data that requires cleansing or quality improvements. Trillium’s Big Data Solution for data quality cleanses data natively within Hadoop because identifying data with data quality issues and then transporting it to another infrastructure becomes costly and complex. The collaboration between Trillium Software and Cloudera illustrates the relevance of data quality solutions for Hadoop despite the increased attention currently devoted to Big Data analytics and data visualization solutions. As such, Trillium fills a critical niche within the Big Data processing space and its alliance with Cloudera positions it strongly to consolidate its early traction within the space of solutions dedicated to data quality in the Big Data space.
MapR has declined the invitation to participate in the Open Data Platform (ODP) after careful consideration, as noted in a recent blog post by John Schroeder, the company’s CEO and co-founder. Schroeder claims that the Open Data Platform is redundant with the governance provided by the Apache Software Foundation, that it purports to “solve” Hadoop-related problems that do not require solving and that it fails to accurately define the core of the Open Data Platform as it relates to Hadoop. With respect to software governance, Schroeder notes that the Apache Software Foundation has done well to steward the development of Apache Hadoop as elaborated below:
The Apache Software Foundation has done a wonderful job governing Hadoop, resulting in the Hadoop standard in which applications are interoperable among Hadoop distributions. Apache governance is based on a meritocracy that doesn’t require payment to participate or for voting rights. The Apache community is vibrant and has resulted in Hadoop becoming ubiquitous in the market in only a few short years.
Here, Schroeder credits the Apache Software Foundation with creating a Hadoop ecosystem in which Hadoop-based applications interoperate with one another and wherein the governance structure is based on a meritocracy that does not mandate monetary contributions in order to garner voting rights. In addition, the blog post observes that whereas the Open Data Platform defines the core of Apache Hadoop as MapReduce, YARN, Ambari and HDFS, other frameworks such as “Spark and Mesos, are gaining market share” and stand to complicate ODP’s definition of the core of Hadoop. Meanwhile, Cloudera’s Chief Strategy Officer Mike Olson explained why Cloudera also declined to join the Open Data Platform by noting that Hadoop “won because it’s open source” and that the partnership between Pivotal and Hortonworks was “antithetical to the open source model and the Apache way.” Given that 75% of Hadoop implementations use either MapR or Cloudera, ODP looks set to face some serious challenges despite support from IBM, Pivotal and Hortonworks, although the precise impact of the schism over the Open Data Platform on the Hadoop community remains to be seen.
Cloudera And Cask Partner To Align Cask’s Application Development Platform With Cloudera’s Hadoop Product Portfolio
Cloudera and Cask recently announced a strategic collaboration marked by a commitment to integrate the product roadmaps of both companies into a unified vision based around the goal of empowering developers to more easily build and deploy applications using Hadoop. As part of the collaboration, Cloudera made an equity investment in Cask, the company formerly known as Continuity. Cask’s flagship product consists of the Cask Data Application Platform (CDAP), an application platform used to streamline and simplify Hadoop-based application development in addition to delivering operational tools for integrating application components and performing runtime services. The integration of Cask’s open source Cask Data Application Platform with Cloudera’s open source Hadoop distribution represents a huge coup for Cask insofar as its technology stands to become tightly integrated with one of the most popular Hadoop distributions in the industry and correspondingly vie for potential acquisition by Cloudera as its product develops further. Cloudera, on the other hand, stands to gain from Cask’s progress in building a platform for facilitating Big Data application development that runs natively within a Hadoop infrastructure. By aligning its product roadmap with Cask, Cloudera adds yet another feather to its cap vis-à-vis tools and platforms within its ecosystem that enhance and accelerate the experience of Hadoop adoption. Overall, the partnership strengthens Cloudera’s case for going public by illustrating the astuteness and breadth of its vision when it comes to strategic partners and collaborators such as Cask, not to mention the business and technological benefits of the partnership. Expect Cloudera to continue aggressively building out its partner ecosystem as it hurtles toward an IPO that it may well be already preparing, at least as reported in Venturebeat.
Teradata and Cloudera Partner To Optimize Integration Of Big Data Platforms In Teradata Unified Architecture
Teradata today announced a partnership with enterprise Hadoop vendor Cloudera marked by the optimization of the integration between Teradata’s integrated data warehouse and Cloudera’s enterprise data hub. The collaboration between Teradata and Cloudera streamlines access to multiple data sources by means of the Teradata Unified Data Architecture (UDA). As a result of the integration, the Teradata Unified Data Architecture can access data from the Cloudera enterprise data hub by way of a unified Big Data infrastructure that has the capacity to perform data operations and analytics on massive, heterogeneous datasets featuring structured and unstructured data. As part of today’s announcement, Teradata also revealed details of Cloudera-certified connectors that can integrate with Apache Hadoop. Other components of the UDA that interface with Cloudera’s enterprise data hub include the Teradata QueryGrid, which allows users to pose analytical questions of data in both Teradata’s integrated data warehouse and the Cloudera enterprise data hub, in addition to the Teradata Loom, which enables tracking, exploration, cleansing and transformation of Hadoop files. Today’s announcement of the integration between Teradata’s integrated data warehouse and Cloudera’s enterprise data hub signals an important development in the Big Data space insofar because the alignment of the product roadmaps of the two vendors promises to position Teradata strongly via-a-via the development of Big data analytics and processing functionality. On Cloudera’s side, the partnership renders its enterprise data hub even more compatible with one of the industry’s most respected Big Data analytic platforms and prefigures the inking of even more partnerships between Hadoop and Big Data management vendors as a means of continuing to foster deeper hardware and software integration in the Hadoop management space.
Cloudera recently acquired big data security vendor Gazzang to strengthen its security offerings for its Hadoop distribution and related offerings. Cloudera’s acquisition of Gazzang will provide “enterprise-grade data encryption and key management.” In addition, the Gazzang team will constitute the foundation of the Cloudera Center for Security Excellence dedicated to the development of comprehensive Hadoop security solutions. Cloudera’s acquisition of Gazzang comes weeks after the announcement of the acquisition of XA Secure by Hortonworks to obtain access to a comprehensive security solution for Hadoop that addresses issues such as user authentication, authorization and audit and control. That Cloudera and Hortonworks acquired dedicated Hadoop security companies in the space of a month illustrates the intensity of the need in the Big Data space to package proven Hadoop security technologies in conjunction with Hadoop deployments and third party tools for optimizing Hadoop analytics and data management. Cloudera, for example, actively contributes to the open source initiative Project Rhino that seeks to augment the data protection functionality of Hadoop and contribute the resulting code back to the Apache Software Foundation. The bottom line is that Hadoop security has suddenly emerged as an urgent vertical within the Big Data space that testifies to the increasing prevalence and scale of the deployment of Hadoop distributions in the enterprise.