Technology Insight for Modern Business Intelligence and Analytics Platforms

23 October 2015 | ID:G00291721

Analyst(s):

Josh Parenteau, Rita L. Sallam, Cindi Howson, Joao Tapadinhas, Thomas W. Oestreich, Kurt Schlegel

Summary

The BI and analytics platform market's multiyear shift of focus from IT-led reporting to business-led self-service analytics has reached a tipping point. Modern BI platforms support organizational needs for greater accessibility, agility and analytical insight from a diverse range of data sources.

Overview

Key Findings

• Most business intelligence (BI) programs have adjusted their scope and shifted their focus away from delivering reporting with some guided analysis and toward enabling free-form analysis with some reporting.

• Traditional reporting-based BI platforms and supporting processes are not designed to support the current pace and dynamic nature of business change or the exponential growth in terms of the sources, volume and complexity of data.

• Modern BI platforms must be agile, user-friendly and trusted to support the expanded — increasingly mission-critical, time-urgent and dynamic — role of analytics in driving competitiveness and creating business value.

• Traditional BI platforms are being augmented with more agile solutions, often purchased by individual business units — a development that challenges IT departments to support a broader portfolio.

Recommendations

BI leaders should:

• Use modern BI platforms to expand user adoption and deliver business value from analytics investments. They should start with new line-of-business-driven analytics projects.

• Plan changes to IT roles and responsibilities to enable wider access to data, and support collaborative content development that includes how users are trained and supported.

• Determine how their existing investments in traditional BI technologies can be used in the context of a modern BI platform to promote analytic governance and deliver actionable content to information consumers.

• Realize that simply adding modern capabilities to a platform that requires traditional data access and/or data ingestion and preparation does not make that platform modern.

Strategic Planning Assumptions

By 2018, most business users and analysts in organizations will have access to self-service tools to prepare data for analysis as part of the shift to deploying modern BI platforms.

By 2018, most stand-alone self-service data preparation offerings will either have expanded into end-to-end analytical platforms or been integrated as features of existing analytics platforms.

By 2018, smart, governed, Hadoop-based, search-based and visual-based data discovery will converge in a single form of next-generation data discovery that will include self-service data preparation and natural-language generation.

Analysis

Analytics is increasingly strategic to businesses and is now central to most business roles. Every business is an analytics business, every business process is an analytics process, and every business user is an analytics user. It is no longer possible for chief marketing officers to be experts solely in branding and the placement of advertisements — they must also be experts in customer analytics. The same is true of HR, supply chain and finance roles in most industries.

To meet the time-to-insight demands of today's competitive business environment, many organizations aim to democratize analytics with self-service capabilities. The associated insertion of analytics deeper into lines of business has significantly changed the traditional BI model and end-user requirements. The modern BI platform is characterized by agility, flexibility and ease of use, all of which are critical in every phase of the analytic workflow, from data source acquisition to execution of actionable insights.

Definition

A modern BI platform supports IT-enabled analytic content development. It is defined by a self-contained architecture that enables nontechnical users to autonomously execute full-spectrum analytic workflows from data access, ingestion and preparation to interactive analysis and the collaborative sharing of insights. By contrast, traditional BI platforms are designed to support modular development of IT-produced analytic content, and specialized tools and skills and significant upfront data modeling, coupled with a predefined metadata layer, are required to access their analytic capabilities.

Description

There are five key components of an analytic workflow that define a BI platform: data source, data ingestion and preparation, content authoring, analysis and insight delivery. In this regard, differences between the approaches of modern and traditional BI platforms are evident. They are outlined in Table 1.

Source: Gartner (October 2015)

These five components are reflected in the modern BI and analytics platform architecture shown in Figure 1. Each is described in more detail below to emphasize the differences between traditional and modern BI platforms.

BI = business intelligence; OLAP = online analytical processing

Source: Gartner (October 2015)

Data Source: Upfront Modeling — From Required to Optional

The most significant difference between modern and traditional BI platforms in the area of data acquisition lies in the amount of upfront modeling required to prepare data for analysis.

A traditional BI platform usually requires a lengthy, IT-led enterprise data warehouse development effort as a prerequisite for accessing the platform's reporting and analytic capabilities through a semantic layer. In addition, some traditional BI platforms need further downstream processing in the form of online analytical processing (OLAP) cubes or datamarts, which require significant IT involvement to design, build and maintain, as well as data-modeling expertise generally confined to IT departments. When the focus of BI was to promote a "single source of truth," in order to deliver consistent and trusted descriptive analytic content sourced primarily from structured internal transactional data, this traditional approach to BI supported the IT supplier/business user consumer model of BI delivery enabled by traditional BI platforms.

However, the democratization of analytics, coupled with exponential growth in the volume, variety and complexity of data, has changed the paradigm of BI delivery. It has also changed the long-standing requirement for upfront modeling by the IT department. The traditional top-down, requirement-driven approach to data curation and provisioning does not meet organizations' requirements for agility as analytic sophistication extends beyond descriptive reports and dashboards and into data discovery and predictive modeling enhanced by new multistructured data sources.

For a BI platform to be considered modern, access to data from within the platform cannot be restricted to formally modeled data structures that have to be designed and built before users can access the platform's analytic capabilities.

Examples of unmodeled data to which a modern platform should allow end users autonomous access include:

• Unmodeled data residing in a relational database management system, in-memory database, Hadoop Distributed File System (HDFS) or NoSQL system, whether on-premises or in the cloud, which has been created by IT staff or power users from:

  • Replicated transactional system data/operational data stores.

  • Syndicated data.

  • Openly available, free data sources.

  • Sensor/machine-generated data.

  • Integrated/enriched datasets resulting from the combination of disparate data sources.

  • Organizational "dark data" traditionally hidden from, or beyond the scope of, BI technologies.

• Direct access to cloud application data sources.

• Direct access to social media streams.

• Direct access to JSON/XML output.

• Personal productivity files, such as Microsoft Excel and comma-separated values (CSV) files. (However, access to these file types alone does not make an otherwise traditional platform modern.)

It is important to note that, in addition to these examples, modern BI platforms may optionally source from traditional, IT-modeled data structures to promote governance and reusability across the organization. In fact, in many organizations, extending IT-modeled structures in an agile manner and combining them with any of the sources listed above is a core requirement.

With a modern BI platform, users can prototype data models and test new data sources in the context of existing data structures without first needing IT staff to update the existing data model and load data into it for users to access.

The key differentiator of a modern BI platform is that users have the option to use centrally modeled and curated environments but are not required to do so in order to access the platform's analytic capabilities.

Data Ingestion and Preparation: From IT-Produced to IT-Enabled

Given their focus on promoting consistency through centrally defined metrics and a "single source of truth," most traditional BI platform architectures are defined by an IT-built semantic (metadata) layer that serves as the end user's single point of access to underlying modeled data structures for all BI platform modules. In traditional BI platforms, data ingestion is generally not supported as the semantic layer functions as a business-focused abstraction tier and virtual representation of the more complex underlying data structures. Minimal data preparation is supported in traditional BI platforms, as much of the work to extract, transform and load data occurs outside the BI platform and within IT-built data structures outlined in the previous section. An IT-built semantic layer as a prerequisite for accessing the analytic capabilities is a characteristic of a traditional BI platform.

A modern BI platform does not require exclusive access through a predefined semantic layer. Rather, it allows users to autonomously and iteratively ingest and prepare data enabled by columnar/in-memory data storage to support analytical speed and agility. It accomplishes this through the self-service data preparation capabilities listed below (aspects of "smart data preparation" that use machine learning may automate some of the work).

Agile modeling:

These are capabilities to prepare data for analysis that are accessed by IT staff or power users/data analysts. They are manual but support an iterative and agile workflow because of the self-contained nature of the BI platform. Capabilities include:

• Data transformation

• Data cleansing/data quality

• Data enrichment

• Custom grouping and calculations

Auto-inference:

These capabilities introduce automation into the self-service data preparation workflow, thereby enabling self-service data preparation for a broader range of nontechnical users. Capabilities include:

• Inference of relationships between dataset entities (schema-on-read)

• Automated classification of dimensions/measures

• Built-in geocoding

• Built-in date conversion and time series functions

• Data profiling with data quality improvement recommendations

• Machine learning to predict analysts' content usage

Data blending:

This is the ability to join datasets before loading them into a columnar or in-memory BI platform datastore, or for the purpose of creating a federated/logical view of data for direct access using the following capabilities:

• Manual definition of table joins and relationships

• Inference of table joins and relationships

Direct query:

In some cases, it is desirable to draw on the performance capabilities and/or security infrastructure and associated rules of the underlying environment hosting the data, but to perform some self-service data preparation capabilities in the BI platform layer without storing a result set in the platform's columnar/in-memory datastore.

While a semantic layer is generally considered a required component of a traditional BI platform, it can also serve as an effective option for governing a modern BI platform through the promotion of user-built data models to the system-of-record semantic layer where there is a broader organizational need to share and standardize analytic content. An existing IT-built semantic layer can also serve as a data source in a modern BI platform, provided that end users can integrate objects defined in the semantic layer with other data sources using the self-service data preparation capabilities of a modern BI platform outlined above.

Content Authoring: From IT-Created to Business-User-Created

The type of analytic content typically supported by traditional BI platforms is generally descriptive and answers the question "What happened?" (see "Extend Your Portfolio of Analytics Capabilities" ). For this, information must be collected, modeled and stored, which typically requires significant IT involvement following the traditional data management practices outlined in the data sections above. For use cases where system-of-record reporting is a requirement or where management dashboards are used to display key performance indicators (KPIs) central to the business, IT-centric reporting with predefined navigation, drill paths and interactivity sourced from modeled environments designed for these requirements is appropriate. Traditional BI platforms are also well-suited to analytic application development where custom development is required, as with customer-facing analytic applications.

However, the BI market has evolved significantly, and matured. This has resulted in the emergence of new capabilities that offer an alternative to traditional analytic content authoring, and that have become central to modern BI platforms.

For a BI platform to be considered modern, it must offer self-service capabilities that enable nontechnical users to do the following without assistance from their IT department:

• Develop free-form and intuitive interactive analytic content

• Design and create analytic dashboard applications

• Create storyboards/narratives

In contrast to traditional BI platforms that primarily address the descriptive end of the analytic spectrum, modern BI platforms enable nontechnical content authors to find important insights using a broad range of more advanced analytical capabilities. They do so by means of intuitive drag-and-drop authoring environments for interactive visualization and embedded advanced analytics.

Modern BI platforms support an analytic workflow that enables content authors to begin with interactive exploration and rapid prototyping, the result being deployment of production content for distribution and consumption. This is in stark contrast to the traditional approach to BI whereby all requirements had to be identified in the initial phases of a project — the traditional BI platform being designed for a top-down approach, in contrast to the modern bottom-up approach. This represents a major change for many organizations, but it is a key factor contributing to the tipping point that we have reached as the scope and focus of BI programs shift from delivering reporting with some guided analysis to enabling free-form analysis with some reporting.

A critical point to note is that simply adding modern capabilities, such as visualization or collaboration, to a BI platform that requires traditional data access and/or data ingestion/preparation does not make that platform modern. All aspects of the analytic workflow must be modern for a BI platform to be categorized as such.

Analysis: From Predefined Reporting to Free-Form Visual Exploration

As noted in the previous section, much of the analytic content that will be created for consumption by a growing population of business users in the modern BI world will originate as interactive exploration accessed by nontechnical users through modern BI platforms. This need will continue to evolve and expand as organizations seek to increase the degree to which analytics drives organizational decision making and as access to data and analytic capabilities is further democratized. The types of analysis that a modern BI platform will be required to support include:

• Interation with, and free-form interaction iterative modeling of, data through visual exploration

• Exploration and navigation of analytic dashboards and embedded analytic content

• Consumption of storyboards and infographics

• Consumption of dynamic analytic narratives describing key insights

The evolution and expansion of analytic interaction required by business users and analysts is blurring the lines between producers, enablers and consumers, and has changed requirements for BI platforms. Traditional BI platforms were designed to support a supplier-consumer model of BI whereby all reporting and analytic needs were defined upfront, so that IT staff could translate data into production-grade reports and dashboards for end users to consume. From an end-user perspective, traditional BI platforms suppot the following types of interaction:

• Interaction with IT-built reports and dashboards

• Interaction with a semantic/metadata layer for ad hoc analysis

• Interaction with a multidimensional OLAP cube for ad hoc analysis

It is further expected that a modern BI platform will support a diverse range of interaction options that enable end users to access analytic capabilities from any physical location via a Web browser or any form factor of mobile device. Traditional BI delivery through client/server-based platforms that require desktop software for every content user or additional IT-led development efforts to enable mobile device access do not support the modern need for speed and agility.

Business users and analysts are also receptive to new ways of interacting with data through automated recommendations, new visualizations types, smart data discovery (see "Smart Data Discovery Will Enable a New Class of Citizen Data Scientist" ) and natural-language Q&A. All of these will continue to evolve and mature to the point where they become assumed features of modern BI platforms.

Insight Delivery: From Distribution to Collaborative Storytelling

The ability to schedule and deliver IT-built static or parameterized management-style reports to large numbers of users, who often print them, is a key feature of traditional BI platforms. IT-authored, KPI-centric dashboards with scheduled refreshes and notifications based on changes are also important features that enable users to monitor what has happened in terms of the performance measures that IT and business executives have identified as the most important drivers of the business.

Key traditional BI insight delivery and sharing capabilities include:

• Scheduled refresh and delivery to consumers of reports that are often printed

• Scheduled refresh of dashboards, with notification of updates of new data or new business events

However, without a modern BI platform, interactivity is limited and often restricted by the content author, so users frequently export report data to other formats (such as Microsoft Excel) to continue to analyze it and to combine it with other sources. The way insights are shared and socialized by modern BI platforms is one of their key attractions.

In modern BI platforms, sharing, collaborating and socializing content through a portal and/or social and collaboration capabilities is crucial for creating a concensus about the interpretation of results and for deciding the best resulting actions. This activity is enhanced by the ability to string together insights as interactive and shared storyboards that "walk" collaborators through a series of insights that lead to conclusions and actions. Modern BI platforms also enable users to define triggers and thresholds that produce automatic alerts and notifications when certain user-specified conditions are met or threshholds exceeded.

Natural-language generation (NLG) is an emerging capability that automatically generates a context-sensitive narrative that focuses the user/consumer of shared content on what is statistically significant in the data. The creation of storyboards will increasingly be facilitatated, and potentially automated, by NLG capabilities, which will increase the scalability and reach of analytics within organizations. Visualizations contextualized and enriched by easy-to-understand narratives, either analyst- or NLG-created, accelerate the time to insight. They also improve the accuracy of insights and conclusions made from analytic content delivered to, and shared with, an expanded set of users who may have limited analytics skills.

Key modern BI platform capabilities for sharing insights include:

• Content sharing and collaboration of modern analytic content through a portal

• Social interaction about analytic content using live chats, annotations, conversation timelines, likes/dislikes, content rating, ability to follow, recommendations and so on

• Alerts and notifications based on manually defined or automated triggers/thresholds

• Open APIs to allow external access to user-built data models and analytic content developed within the modern BI platform's self-contained architecture

Benefits and Uses

The benefits of modern BI platforms are the enabling of fast time-to-insight, the ability to change and evolve at the speed of business, and the ability to use information as a strategic competitive advantage.

The innovation that comes with an agile and iterative modern BI platform delivers the necessary building blocks to increase analytic maturity and derive value from data. This allows organizations to gain new, previously undiscovered insights with which to improve decision making. It also creates opportunities for differentiation and new revenue streams as new insights are packaged and extended to customers, partners and suppliers.

Adoption Rate

We are past the point where demand for modern BI platforms, driven by the requirement to make analytics pervasive without causing chaos, has become mainstream across all industries, business domains and company sizes.

Risks

BI leaders that fail to adopt modern platforms will be unable to respond to the faster pace of business and to users' voracious appetite for fast insights from more data sources than ever before. A traditional centralized and rigidly IT-centric approach cannot meet these new, time-sensitive requirements.

But despite the obvious benefits of BI platform modernization, organizations must assess the potential drawbacks of a shift to a modern BI platform. These include:

• The possible need for significant change management and additional resources to help the IT organization support the new BI operating model.

• The potential complexity of integrating existing BI infrastructure with a new modern BI platform — especially when the task involves multiple locations and business units, and different information latency and data quality requirements.

• The risk of information chaos as a result of inadequate information management discipline (with regard to data consistency, data quality and so on), insufficient expertise and the low maturity levels of modern BI platform users — that is, business users.

Evaluation Factors

In early 2016, Gartner will publish a detailed Critical Capabilities report as a companion to a new edition of the "Magic Quadrant for Business Intelligence and Analytics Platforms." It will provide an in-depth comparison of products and vendors that meet the definition of a modern BI platform. Additionally, we shall publish an RFP template covering the detailed functionality that a modern BI platform should have.

Recommendations

BI leaders should use modern BI platforms' capabilities to introduce agility and iterative development of analytic content into their organizations. Rather than focus on upfront modeling and development of data structures that will prove difficult to adapt to changing business conditions, the IT organization must adopt an enablement role and empower business users and analysts to derive insights autonomously.

Existing investments in traditional BI platforms, supporting processes and data structures should still be used, where possible, to promote consistency; but BI leaders must balance the need for consistency and scalability with the ever-increasing need for speed and agility. They must determine how to support both ends of the spectrum and provide an appropriate mix of skills, training and support for each user and use case.

Finally, BI leaders must take a holistic approach to BI platform modernization. They need to realize that, without appropriate change management and an updated, comprehensive BI and analytics strategy, adding modern capabilities to a traditional, IT-led program will not yield the full benefits of a modern BI platform.