Go real-time with your data

Sharing data across an organisation brings the most benefits when the data is as 'fresh' as possible. But pulling that off is not as simple as it sounds. Quocirca's Louella Fernandes explains the ins and outs of using real-time intelligence.

Responding to changing business needs through timely access to accurate information is a critical driver of business success. This has given rise to the notion of the 'real-time enterprise' and vendors of time business intelligence (BI) software are increasingly talking about how their BI tools can enable faster decision-making at the right time.

But what exactly is involved in implementing a real business intelligence platform - and does your organisation really need one?

First, what does 'real-time' mean? Real-time is ultimately defined by the business environment an organisation is operating in. For some organisations, real-time response may be defined as 'within a minute', while for other organisations working in situations where instantaneous response is required it may be as little as a few milliseconds.

The sweet spot for real-time business intelligence is operations in industries such as financial services, manufacturing and retail. These are areas where users can access and act upon timely and accurate information, in a way that helps them to manage day-to-day business activities which impact operational efficiency such as inventory control.

Achieving real-time business intelligence presents many challenges but one of the main hurdles to overcome is slow query performance due to limitations of the current BI infrastructure.

Today many organisations have a data warehouse that stores a version of their transactional information and makes it available for queries and analysis. Data warehouses are updated with periodic snapshots of operational data which are 'cleaned' before being loaded into the warehouse. This process is commonly referred to as extraction, transformation and load (ETL), and can take anywhere from a few hours to several days to complete. As a result, the data in the warehouse can be at least a day or two out of date - or even a week or a month behind the source systems.

Whilst this latency is generally acceptable for trend analysis and forecasting, traditional data warehouses simply can't keep pace with today's BI requirements for fast and accurate data. They were not designed to deliver complex analytics on terabytes of data quickly and, as the volume of data used in organisations grows exponentially, extracting information becomes more time-consuming and complex.

Increasingly database vendors are building time-sensitive data warehouse features into their database products. In real-time data warehousing there is no latency between the source system and the data warehouse, so changes in the source system are reflected in the data warehouse instantaneously or at least in near real-time. This enables easy and efficient processing for query and analysis at any time.

With terabyte-scale databases readily available, the new generation of data warehouse appliances offer real benefits to organisations analysing large volumes of transactions - such as financial services and telecoms.

Investing in real-time BI can be costly, and generally those applications that require the most real-time capabilities also need high performance systems. Financial organisations dependent on time-sensitive applications - such as credit card fraud detection - are demanding and willing to pay for very low latency access.

Retail is also an area where real-time sales and inventory analytics can give a critical competitive edge. Virgin Megastore, for example, has implemented a loss prevention solution from Microsoft using BizTalk server for real time reporting, trend analysis and fraud detection.

There are a host of vendors offering or adapting products for real-time data integration. Business Objects and Informatica have incorporated real-time capabilities to the ETL tools they provide for moving data into data warehouses. Sunopsis, recently acquired by Oracle, uses the E-LT (extract - load and transform) approach, which does not require an ETL server. Datamirror's Transformation server loads data warehouses in real-time, without impacting the performance of an organisation's production systems. And GoldenGate Software offers transactional data management (TDM) technology for capturing and moving transactional data in real-time across databases.

Clearly the key to effective reporting is fast query performance into data warehouses or live operational data. Today the traditional reporting approach of using Olap (on line analytical processing) solutions is being superseded by in-memory data management which promises to take BI to the next level. These tools avoid the classic problems associated with Olap cube technology (cubes arrange data along predefined business dimensions for easy access and analysis, resulting in long pre-calculation times and problems with managing ever expanding 'cube farms').

A number of BI vendors are enhancing their real-time reporting capabilities to use in-memory analysis. Cognos, for example, recently acquired Celequest, a dashboarding solution offered as an appliance or via a software as a service model, which allows for fast and low-cost deployment. Information Builder's WebFocus has for some time supported real-time data access and reporting through the use of iWay Software's native data and application adapters.

The area of in-memory analysis is one of the most innovative areas of BI today, with many smaller niche companies offering solutions that can be complementary to the major BI platforms.

Vendor QlikTech's QlikView offers powerful in-memory analysis reporting aimed at enabling all users in an organisation with fast analysis, without the complexity of using traditional Olap cubes. Advizor, Spotfire and Tableau also offer in-memory data management and advanced data visualisation capabilities aimed at providing fast, easy to use and (relatively) affordable analysis. And SeeWhy uses event stream processing based on SOA to analyse transactions as they occur, and so build BI into an organisation's operational business processes.

Indeed, if operational users can access timely and accurate information in this way, they can respond to events as they happen - the ultimate goal of a real-time enterprise.

However, many companies today are not equipped and ready to take advantage of real-time business intelligence. Organisations are still facing the challenge of collecting and analysing information from scattered and often incompatible data sources, meaning that real-time BI enterprise is still a few years away.

Along with addressing data quality problems, companies must put in place the policies and procedures to enable workers to act on real-time alerts and data analysis.

Ultimately, the bottleneck in the real-time enterprise is almost always human but with the right tools, real-time BI will be the key to improving business performance by enabling business insight to all employees, leading to better, faster, more relevant decisions.