The following guest blog post was authored by Ashwin Viswanath, Director of Cloud Product Marketing, Talend.
While moving to the cloud is becoming a common business decision, many organizations are still trying to figure out the best path to meet their business needs and IT infrastructure goals. The subsequent discussions often result in a hybrid approach where organizations tap the value of both public and private clouds for different services.
Hybrid integration, as it has evolved over the years, plays a very positive role in the creation of a variety of complex value-chain ecosystems supporting numerous digital business initiatives.
The definition of hybrid integration, reflecting the changing nature of these initiatives, has evolved as well. It ranges from: supporting SaaS apps to B2B data sources from partners and suppliers presented as a mashup in a portal; to integrating big data and backend systems to support mobile applications. As these and other initiatives continue to proliferate, the definition of hybrid integration will continue to morph as well.
Time for an Upgrade
The rapid evolution of existing initiatives, and an accelerated pace in the creation of new use cases is a clear indication that it’s time to upgrade your hybrid integration capabilities.
For example, imagine that you work in a line-of-business (LOB) that is highly data-driven such as a Customer Success Department. Access to up-to-the minute information about your customers is critical to your operation. You no longer can wait for the traditional 24-hour IT cycle to obtain operational reports that tell you if your customers need help, details their latest purchases, and tells you which customers are critical to contact.
These days if customers are having bad experiences with your product or service, they will take advantage of other options and churn almost immediately. This is particularly true in today’s subscription-based online environments, which allow a customer to turn off the service and go elsewhere with a few keystrokes. You need to be able to predict if and when such an event may occur so that you can take corrective action before the customer says “I’ve had enough.”
As a member of our hypothetical Customer Success Department, you also need tools that can analyze a number of different data points gathered from customer support tickets that indicate how often they responded to marketing offers. You even want to read the tweets they have posted about you and your company or be notified immediately if a steady, big ticket customer drops off your radar.
It’s readily apparent that collecting data from all these different data sources, aggregating that data, and deploying the correct filtered business rules and triggers to alert the Customer Success management team represents a lot of work for IT. And the Customer Success Department is only one use case – there are many others such as marketing, finance, and the supply chain – all can benefit from being notified when pre-determined business parameters are not being met. This alert has to be issued before a critical threshold has been reached so the LOB can take corrective action in a timely and decisive manner.
To deploy these hybrid integration services in a world that has become increasingly data-driven and characterized by a 24/7 competitive environment will probably require an upgrade of your data integration service, including all layers of the cloud stack. This kind of intensive rework extends beyond the infrastructure and platform layers, or a few tweaks to your software-as-a-Service (SaaS) applications.
To achieve the kind of insights described in the Customer Success Department applications described above requires a solid cloud-based infrastructure built to support your most demanding Big Data needs. This includes the ability to handle great quantities of streaming data.
The data technologies you select – whether they are NoSQL databases, or some other major real-time big data protocol, must be specifically tailored to work with the large volumes of real time streaming data that are required in today’s competitive environment. This includes a secure cloud-based integration platform that allows your users to connect their applications with one another and transfer data between them. Browser-based graphic development tools are complemented by ready-made integration actions, flow templates, components and connectors that make data integration easy.
Five Phases of Hybrid Integration
In this two-part blog series (part I, part II), I outlined the five phases of hybrid integration, which allow you to easily assess for yourself, which phase of hybrid integration you’re in and how you can move to a more advanced phase. The phase descriptions follow a trajectory – from older, more mature phases to more recent (and potentially disruptive) scenarios. In brief, the five phases are:
- Phase 1: Replicating SaaS apps to on-premise databases – Companies in this initial stage either need business critical information from their SaaS apps; or they are sending SaaS data to a staging database so that it can be picked up by other on-premises apps
- Phase 2: Integrating SaaS Apps directly with on-premises apps – Each LOB has their preferred SaaS app. Sales departments use Salesforce, Marketing has Marketo, HR prefers Workday and Finance has NetSuite. These SaaS apps need to connect to a back-office ERP on-premises system such as SAP R/3 or Oracle EBS. Implementing a pure-play cloud ERP platform is still very much a work in progress, as the blog explains.
- Phase 3: Hybrid Data Warehousing with the Cloud – As the volume and variety of data grows, you need a strategy to move your data from an on-premises data warehouse to newer, more advanced Big Data resources in the cloud. Choosing the right Big Data protocols and getting underway by at least creating a data lake in the cloud with cloud-based services such as AWS S3 or Azure Blobs, is part of this phase.
- Phase 4: Real-time analytics on streaming data – It is during this phase that you implement systems that can provide you with the tools to work with real-time streaming data. To realize the full benefit of real-time analytics you need the support of a hybrid integration infrastructure. This infrastructure will vary depending on your use case – for example, supporting software that analyzes weblogs, clickstream data, sensor data, database logs, or social media sentiment.
- Phase 5: Machine learning for optimized app experience – In the not too distant future, every experience will be delivered as an app through mobile device – that includes enterprise as well as consumer mobile apps. The hybrid integration infrastructure will be architected to provide the ability to discover patterns buried deep within the data using machine learning so the applications can be more responsive to user needs. Advanced algorithms will allow value to be extracted from immense and disparate data sources that go beyond the capabilities of human analysis. For developers, machine learning will apply business-critical analytics to applications to do everything from improving customer experience or providing product recommendations to presenting hyper-personalized content.
Eventually, if current trends continue, everything will move into the cloud, supporting increasing opportunities for converging applications and data integration and redefining yet again what is meant by hybrid integration.
The importance of this transformation was underscored by a recent IDC report. DC expects that cloud IT infrastructure spending will grow at a compound annual growth rate (CAGR) of 15.1 percent and will reach $53.1 billion by 2019 accounting for 46% of the total spending on enterprise IT infrastructure. At the same time, spending on non-cloud IT infrastructure will decline at -1.7 percent CAGR. Spending on public cloud IT infrastructure will grow at a higher rate than spending on private cloud IT infrastructure – at 16.3 percent vs. 13.2 percent CAGR. In 2019, IDC expects service providers will spend $33.6 billion on IT infrastructure for delivering public cloud services, while spending on private cloud IT infrastructure will reach $19.4 billion[1].
Benefits of Hybrid Integration
With advanced hybrid integration in place and cloud integration tools at hand, companies are able to carry out “do it yourself” data integration projects. There is no need for users to constantly go back to the IT department to modify business rules and parameters. They have a turnkey solution that allows them to quickly and easily deal with changing business requirements. And the end user is far more familiar with those business requirements and knows which technical parameters need to be changed to provide the answers they are looking for.
In addition, these new integration workflows developed by the user community can be leveraged by other business units as well. This creates an exponential expansion of contextual information between LOBs. IT is freed up to build platforms for future innovation by researching Big Data algorithms and leveraging the cloud infrastructure to maximize these new algorithms, rather than constantly putting patches on old information workflows.
In short, hybrid integration in the cloud offers enormous advantages for companies seeking the best of private and public clouds for scalability, price, control and flexibility.
[1] “Worldwide Quarterly Cloud IT Infrastructure Tracker,” IDC, October 2015.
Cloud Computing Today 