Pivotal Open Sources Its Big Data Suite And Announces Partnership With Hortonworks

Pivotal recently announced the open sourcing of key components of its Pivotal Big Data Suite. Parts of the Pivotal Big Data Suite that will be outsourced include the MPP Pivotal Greenplum Database, Pivotal HAWQ and Pivotal GemFire, the NoSQL in-memory database. Pivotal’s decision to open source the core of its Big Data suite builds upon its success monetizing the Cloud Foundry platform and intends to accelerate the development of analytics applications that leverage big data and real-time streaming big data sets. The open sourcing of Greenplum, Pivotal’s SQL on Hadoop platform HAWQ and GemFire renders Pivotal’s principal analytics and database platforms more readily accessible to the developer community and encourages enterprises to experiment with Pivotal’s solutions. Sundeep Madra, VP of the Data Product Group, Pivotal, remarked on Pivotal’s decision to open source its Big Data suite as follows:

Pivotal Big Data Suite is a major milestone in the path to making big data truly accessible to the enterprise. By sharing Pivotal HD, HAWQ, Greenplum Database and GemFire capabilities with the open source community, we are contributing to the market as a whole the necessary components to build solutions that make up a next generation data infrastructure. Releasing these technologies as open source projects will only help accelerate adoption and innovation for our customers.

Pivotal’s announcement of the open sourcing of its Big Data suite comes in tandem with a strategic alliance aimed at synergistically maximizing the competencies of both companies to deliver best-in-class Hadoop capabilities for the enterprise. The partnership with Hortonworks includes product roadmap alignment, integration and the implementation of a unified vision with respect to leveraging the power of Apache Hadoop to facilitate the capability to derive actionable business intelligence on a scale rarely performed within the contemporary enterprise. In conjunction with the collaboration with Hortonworks, Pivotal revealed its participation in the Open Data Platform, an organization dedicated toward promoting the use of Big Data technologies centered around Apache Hadoop whose Platinum members include GE, Hortonworks, IBM, Infosys, Pivotal and SAS. The Open Data Platform intends to ensure components of the Hadoop ecosystem such as Apache Storm, Apache Spark and Hadoop-analytics applications integrate with and optimally support one another.

All told, Pivotal’s decision to open source its Big Data suite represents a huge coup for the Big Data analytics community at large insofar as organizations now have access to some of the most sophisticated Hadoop-analytics tools in the industry at no charge. More striking, however, is the significance of Pivotal’s alignment with Hortonworks, which stands to tilt the balance of the struggle for Hadoop market share toward Hortonworks and away from competitors Cloudera and MapR, at least for the time being. Thus far, Cloudera has enjoyed notable traction in the financial services sector and within the enterprise more generally, but the enriched analytics available to the Hortonworks Data Platform by means of the partnership with Pivotal promise to render Hortonworks a more attractive solution, particularly for analytics-intensive use cases and scenarios. Regardless, Pivotal’s strategic evolution as represented by its open source move, its collaboration with Hortonworks and leadership position in the Open Data Platform constitute a seismic moment in Big Data history wherein the big data world shakes as the world’s most sophisticated big data analytics firm qua Pivotal unites with Hortonworks, the company responsible for the first publicly traded Hadoop distribution. The obvious question now is how Cloudera and MapR will respond to the Open Data Platform and the extent to which Pivotal’s partnership with Hadoop distributions remains exclusive to, or focused around Hortonworks in the near future.

OpenStack Innovator Joshua McKenty Leaves Piston Cloud Computing For Pivotal

In a stunning announcement, Joshua McKenty, co-creator of OpenStack and co-founder and CTO of Piston Cloud Computing (Piston), revealed this week that he has accepted a position at Pivotal as field CTO. McKenty transitions to Pivotal after a three year stint at commercial OpenStack vendor Piston, during which time he was central to raising approximately $20M in capital, growing the team by a factor of 15 and increasing sales by a multiplier of 1000. McKenty helped build Piston from a “fledgling startup” into “a real business with an incredible group of customers, an established leadership team, and a mature product” as noted in a blog post reflecting on his tenure at Piston. More importantly, however, McKenty served as an OpenStack evangelist that demonstrated the commercial viability of OpenStack as a prominent alternative to proprietary Infrastructure as a Service technologies such as Amazon Web Services and Microsoft Azure. As a former NASA luminary, McKenty’s position as an OpenStack board member and his outspoken elaboration of OpenStack’s potential gifted OpenStack with technical credibility that ultimately led to its adoption by companies such as IBM, Red Hat, HP, Ericsson and a slew of technology behemoths that now collectively contribute financial and engineering-related resources to the OpenStack project.

OpenStack’s success aside and notwithstanding, however, McKenty noted that his burning technical interest involves simplifying application development and deployment as follows:

This was the central issue that I was recruited into NASA to address in 2008—how to improve both security and efficiency by unifying NASA’s application development into a common platform. Originally called NASA.net, this project quickly ran into a then-common roadblock—the lack of agile and programmatic infrastructure to support this platform.

The past 5 years have been a detour to address this lack of agile infrastructure—a detour that could now be easily termed the “OpenStack” years.

Here, McKenty remarks on how his recruitment to NASA to work on unifying application development was derailed by the lack of adequate technical infrastructure, which ultimately led to the development and refinement of OpenStack over a five year period. McKenty commented further on the complexity of contemporary software development by noting:

Fast forward 30 years, and I now have two daughters, both of whom are trying to “learn to code”. And while the intervening decades have made computers vastly more capable, they have also made them more complex. What was once possible with a single machine (the one sitting in your living room or kitchen, no less), now requires the use of “the cloud”, and an arcane set of tasks with a poorly defined mental model called…deployment.

At Pivotal, McKenty will have precisely this opportunity to work on simplifying application development in conjunction with the Cloud Foundry project. Previously, he had worked with Pivotal on the integration of Cloud Foundry with OpenStack and confessed that much of his interest in OpenStack dissipated subsequent to the success of the integration of the two platforms. McKenty’s transition from Piston to Pivotal signals the end of an era in the history of OpenStack. On one hand, the OpenStack collaboration has already reached a tipping point such that it will continue with the momentum and innovation commensurate to its backing by the world’s most successful technology companies. Nevertheless, the OpenStack world will miss McKenty’s technical acumen, intellectual passion for open source technologies and unique ability to contextualize the place of one technology within the larger technology landscape. McKenty’s move to Pivotal aptly illustrates the story of a man following his passion and intellectual interests as enabled by a truly unique opportunity to join one of the most innovative and powerful technology companies in the world. While McKenty will be missed by the OpenStack community at large, his presence at Cloud Foundry promises to usher in a new era for application development marked by increased simplicity and streamlined processes that render it easier for everyone to write, or at least understand how to write code.

Pivotal And Hortonworks Collaborate To Advance Apache Ambari For Hadoop Management

Pivotal and Hortonworks will collaborate to accelerate development of Apache Ambari, the open source framework for provisioning, managing and monitoring Hadoop clusters. Pivotal will dedicate engineers toward advancing the “installation, configuration and management capabilities” of Apache Ambari as part of the larger project of contributing to software that promotes adoption of Apache Hadoop. In a blog post, Pivotal’s Jamie Buckley elaborated on the value of Apache Ambari to the Hadoop ecosystem as follows:

Apache Hadoop projects are central to our efforts to drive the most value for the enterprise. An open source, extensible and vendor neutral application to manage services in a standardized way benefits the entire ecosystem. It increases customer agility and reduces operational costs and can ultimately help drive Hadoop adoption.

Here, Buckley remarks on the way in which Ambari enhances the process of deploying and managing Hadoop by reducing costs and increasing the flexibility of customer choices regarding the operationalization of Hadoop. Meanwhile, Shaun Connolly, VP Strategy at Hortonworks, commented on the significance of Pivotal’s contribution to the Apache Ambari project as follows:

Pivotal has a strong record of contribution to open source and has proven their commitment with projects such as Cloud Foundry, Spring, Redis and more. Collaborating with Hortonworks and others in the Apache Hadoop ecosystem to further invest in Apache Ambari as the standard management tool for Hadoop will be quite powerful. Pivotal’s track record in open source overall and the breadth of skills they bring will go a long way towards helping enterprises be successful, faster, with Hadoop.

Connolly highlights Pivotal’s historical commitment to open source projects such as Cloud Foundry and its track record of success helping enterprises effectively utilize Apache Hadoop. Hortonworks stands to gain from Pivotal’s extraordinary engineering talent and reputation for swiftly releasing production-grade code for Big Data management and analytics applications. Meanwhile, Pivotal benefits from enriching an open source project that both vendors refer to in the context of a “standard” management tool for the Apache Hadoop ecosystem. The real winner, however, is Hortonworks, who now can claims the backing of Pivotal for the open source project Ambari incubated by some of its engineers, but also reaps the benefits of dedicated engineering staff from Pivotal that will almost certainly accelerate the rate of development of Ambari. The only qualification, here, is that Pivotal’s collaboration with Hortonworks is likely to ensure the optimization of Ambari for both the Pivotal HD and Hortonworks distribution, with the ancillary consequence that Ambari may be less suited for other Hadoop distributions such as Cloudera and MapR. Regardless, the collaboration between Hortonworks and Pivotal promises to serve as a huge coup for the Big Data industry at large both with respect to expediting development of Apache Ambari, and constituting a model for collaboration between competitors in the Big Data space that will ultimately enhance Hadoop adoption and effective utilization.

Pivotal Releases Pivotal Big Data Suite With Pricing Per Core And Annual Subscription

Not to be outdone by the slew of product and price announcements from Google, Amazon Web Services and Microsoft over the past week, EMC-VMware spinoff Pivotal announced a new product offering branded the Pivotal Big Data Suite on Wednesday. The platform delivers Pivotal Greenplum Database, Pivotal GemFire, Pivotal SQLFire, Pivotal GemFire XD and Pivotal HAWQ, in addition to unlimited use of Pivotal’s Hadoop distribution Pivotal HD. Because the Pivotal Big Data Suite is priced on the basis of an annual subscription for all software and services, in addition to per core pricing for computing resources, customers need not fear additional fees related to software licensing or customer support over and beyond the subscription price. Moreover, customers essentially have access to a commercial-grade Hadoop distribution for free as part of the subscription price. Pivotal compares the Big Data Suite to a “swiss army knife for Big Data” that enables customers to “use whatever tool is right for your problem, for the same price.” Customers have access to products such as Greenplum’s massively parallel processing (MPP) architecture-based data warehouse, GemFire XD’s in-memory distributed Big data store for real-time analytics with a low latency SQL interface and HAWQ’s SQL-querying ability for Hadoop. Taken together, the Pivotal Big Data Suite edges towards the realization of Pivotal One, an integrated solution that performs Big Data management and analytics for ecosystems of applications, real-time data feeds and devices that can serve the data needs of the internet of things, amongst other use cases. More importantly, the Pivotal Big Data Suite represents the most systematic attempt to productize Big Data solutions in the industry at large, even if it is composed of an assemblage of heterogeneous products under one roof. The combination of access to a commercial grade Hadoop distribution (Pivotal HD), a data warehouse designed to store petabytes of data (Pivotal Greenplum) and closed loop real-time analytics solutions (Pivotal GemFire XD) within a unified product offering available via an annual subscription and per core pricing constitutes an offer not easy to refuse for anyone seriously interested in exploring the capabilities of Big Data. The bottom line is that Pivotal continues to push the envelope with respect to Big Data technologies although it now stands to face the challenge posed by cash flush Cloudera, which recently finalized $900M in funding and a strategic and financial partnership with Intel.

Pivotal HD 2.0 Features Support For Apache Hadoop 2.2 And General Availability Of GemFire XD

This week, EMC and VMware spinoff Pivotal announced the availability of Pivotal HD 2.0, a commercial distribution of Apache Hadoop that now features support for Apache Hadoop 2.2. Moreover, Pivotal also revealed the general availability of Pivotal GemFire XD, a SQL compliant, in-memory database designed for real-time analytics for Big Data processing. In its initial release, Pivotal GemFire XD represents an in-memory distributed data store that “provides a low-latency SQL interface to in-memory table data, while seamlessly integrating data that is persisted in HDFS.” Because GemFire brings the power of real-time analytics to Hadoop, it empowers mobile providers to run complex algorithms on incoming calls to route the call appropriately, or geospatial navigation systems to alter suggested routes based on incoming data about traffic and weather conditions. Like Apache Spark, a parallel data processing framework that facilitates real-time analytics on Hadoop, GemFire enables real-time Big Data analytics but is explicitly designed for data environments with high demands for scalability and availability. Michael Cucchi, Pivotal’s senior director of product marketing, commented on Pivotal’s interest in Spark and GemFire XD in an interview with InformationWeek as follows:

We’re excited about Spark and will support it, but it’s generally used for [data] ingest or caching,” GemFire XD is an ANSI-compliant SQL database with high-availability features, and it can run over wide-area networks, so you can have an instance in Europe and another in North America with replication.

Built on the vFabric SQLFire product that belongs to the category of NewSQL databases noted for high performance and scalability, GemFire XD is adds features such as HDFS-persistence and off-heap memory storage for table data. In addition to GemFire XD, Pivotal 2.0 also features an integration with GraphLab for graphing analytics as well as enhancements to HAWQ such as support for MADlib, R, Python, Java, and Parquet. Overall, Pivotal 2.0 represents a notable advancement over Pivotal 1.1 that brings the power of YARN, real-time analytics via GemFire XD and graphing technology to Hadoop and Big Data processing and analytics. With Pivotal HD 2.0 released less than 6 months after the November 1, 2013 release of Pivotal HD 1.1, Pivotal promises to innovate in the Big Data space at the same dizzying rate with which Amazon Web Services innovates with regard to cloud computing technologies and platforms. Expect to hear more about the conjunction of real-time analytics and graphing technologies on Hadoop via Pivotal 2.0 as customer use cases proliferate and circulate throughout the Big Data space.

Pivotal Releases Pivotal One, An Enterprise PaaS Marked By The Convergence Of Cloud And Big Data

On Tuesday, Pivotal announced details of Pivotal One, an integrated platform for application development that aims to bring the scalability and performance enjoyed by tech giants such as Facebook, Google and Twitter to enterprise computing. Pivotal One delivers a suite of application and database services around Pivotal CF, an enterprise grade distribution of Cloud Foundry that customers can deploy within on premise environments, VMware or OpenStack-based cloud infrastructures or on Amazon Web Services. Pivotal CF features the Pivotal CF Elastic Runtime Service, which ensures a scalable runtime environment and the Pivotal CF Operations Manager, which provides a comprehensive suite of services for managing and upgrading the Pivotal CF platform in addition to integrating it with IaaS environments.

Pivotal CF is surrounded by an ecosystem of Pivotal One products such as Pivotal™ HD, Pivotal AX, Pivotal RabbitMQ™, and MySQL. Pivotal HD represents Pivotal’s distribution of Apache Hadoop, Pivotal AX delivers analytic services for querying, analyzing, discovering and visualizing data, Pivotal RabbitMQ provides a messaging service for applications and MySQL gives customers the option of a familiar relational database with which to build their applications. The conjunction of Pivotal CF with Pivotal One’s wraparound products marks an offering that integrates the versatility of Cloud Foundry’s tools for agile application development with Pivotal’s tools for big data deployments and data analytics. The offering strives to increase the degree of respect accorded to Platform as a Service in contemporary cloud computing by delivering an enterprise PaaS that surpasses the agility, performance and scalability of application development options available by means of today’s IaaS platforms.