Smart Bridge Trading Solutions Neeraj Nathani Trade Promotion Forecasting

Trade Promotion Forecasting

Trade Promotion Forecasting (TPF) is the process that attempts to discover multiple correlations between trade promotion characteristics and historic demand in order to provide accurate demand forecasting for future campaigns. The ability to distinguish the uplift or demand due to the impact of the trade promotion as opposed to baseline demand is fundamental to model promotion behavior. Model determination enables what-if analysis to evaluate different campaign scenarios with the goal of improving promotion effectiveness and ROI at the product-channel level by selecting the best scenario.

Trade promotion forecasting challenges

Trade Promotion spending is one of the consumer goods industry’s largest expenses with costs for major manufacturers ranging from 10 percent to 20 percent of gross sales. Understandably, 67 percent of respondents to a recent survey said they were concerned about the return on investment (ROI) gained from such spending. Quantifying ROI depends heavily on the ability to accurately identify the “baseline” demand (the demand that would exist without the impact of the trade promotion) and the uplift.^[1]

In fact, forecast accuracy plays a critical role in the success of consumer goods companies. Aberdeen Group research found that best-in-class companies (with an average forecast accuracy of 72 percent) have an average promotion gross margin uplift of 28 percent, while laggard companies (with an average forecasting accuracy of only 42 percent) have a gross margin uplift of less than 7 percent.^[2]

A bottom-up sales forecast at the SKU-account/POS level requires taking into account product attributes, historical sales levels and store specifics. A complicating factor is that the large number of different variables which describe the product, the store and the promotion attributes, both quantitative and qualitative, could potentially have many different values. Selecting the most important variables and incorporating them into a prediction model is a challenging task.^[3]

Despite these challenges, two-thirds of companies in the consumer supply chain consider forecast accuracy a high business priority. 74 percent said it would be helpful to develop a bottom-up forecast based on stock-keeping unit (SkU) by key customer.^[4]

Traditional trade promotion forecasting methods

Many companies forecast the impact of trade promotions primarily through a human expert approach. Human experts are unable to take into account all the variables involved and also cannot provide an analytic prediction of campaign behavior and trends. A recent survey by Aberdeen Group showed that 78 percent of companies used Microsoft Excel spreadsheets as their primary trade promotion technology tool. The limitations of relying upon spreadsheets for trade promotion planning and forecasting include lack of visibility, ineffectiveness and difficulty in tracking deductions.^[5]

Specialized applications have been developed and become more common. 35 percent of companies now use legacy systems, 30 percent use Sales and Operations Planning (S&OP) applications, 26 percent use integrated Enterprise Resource Planning (ERP) modules and 17percent use home grown trade promotion solutions. These applications support the planning process, while still primarily relying on human knowledge and intuition for forecasting. One problem with this approach is that humans tend to make optimistic assumptions when forecasting and planning. The result is that forecasts most commonly err on the optimistic side and that human forecasters also tend to underestimate the amount of uncertainty in their forecasts.^[6]

A further issue is that the majority of manufacturers use legacy trade promotion systems that contribute to internal fragmentation of trade marketing data. Many of these companies are currently using assumption-based forecasts with limited accuracy.^[7]

Analytic approaches to trade promotion forecasting

TPF is complicated by the fact that campaigns are described by both quantitative (such as price and discount) and qualitative (such as display space and support by sales representatives) variables. New approaches are being developed to address this and other challenges. Most of these approaches attempt to incorporate large amounts of heterogeneous data in the forecasting process. One researcher validated the ability of multivariate regression models to forecast the impact on sales of a product of many variables including price, discount, visual merchandizing, etc.^[8]

The term Big Data describes the increasing volume and velocity of heterogeneous data that is coming into the enterprise. The challenge is to combine this data across all of the silos within the organization for a single view. The data can be used to improve trade promotion forecast accuracy because it usually contains real connections and causation that can help to better understand what customers are buying, where they are buying it, why they are buying and how they are buying.^[9]

Traditional methods are insufficient to assimilate and process such a large volume of data. Therefore more sophisticated modeling and algorithms have been developed to address the problem. Some companies have begun using machine learning methods to utilize the massive volumes of unstructured and structured data they already hold to better understand these connections and causality.^[10]

Machine learning can make it possible to recognize the shared characteristics of promotional events and identify their effect on normal sales. Learning machines use universal approximations of nonlinear functions to model complex nonlinear phenomena. Learning machines process sets of input and output data and develop a model of their relationship. Based on this model, learning machines forecast outputs associated with new sets of input data.^[10]

Intelligible Machine Learning (IML) is an implementation of Switching Neural Networks that has been applied to TPF. Starting from a collection of promotional characteristics, IML is able to identify and present in intelligible form existing correlations between relevant attributes and uplift. This approach is designed to automatically select the most suitable uplift model in order to describe the future impact of a planned promotion. In addition, new promotions are automatically classified using the previously trained model, thus providing a simple way of studying different what-if scenarios.^[11]

TPF systems should be capable of correlating and analyzing vast amounts of raw data in different formats such as corporate sales histories and online data from social media. The analysis should be able to be performed very quickly so planners can respond quickly to demand signals.^[12]

Groupe Danone used machine learning technology for trade promotion forecasting of a range of fresh products characterized by dynamic demand and short shelf life. The project increased forecast accuracy to 92 percent resulting in an improvement in service levels to 98.6 percent, a 30 percent reduction in lost sales and a 30 percent reduction in product obselecense.^[13

Source: Wikipedia.

Smart Bridge Trading Solutions Neeraj Nathani Trade Promotion Management (TPM)

Trade Promotion Management (TPM)

Trade Promotion Management typically refers to one or more software applications that assist companies in managing their complex trade promotion activity. Trade Promotion Management is a challenge faced by most CPG/FMCG companies around the globe. Consumer goods companies spend substantial amounts of time and money—14 percent of revenue, according to an AMR Research study—on promotions with retailers designed to boost revenue or increase/protect market share (or both).^[1] Key Account Managers and Category/Brand Managers are responsible for developing and implementing a promotional program that delivers the maximum return on investment and manages the trade off between incremental volume and maintenance of a reasonable level of trade spend.^[2]

Gartner

Gartner believes that technologies related to managing trade promotions have never been more relevant, as the average revenue expended by manufacturers for promotions now exceeds 20%. More and more companies are leaving spreadsheets for automated technologies, while others are adding promotion optimization capabilities

Key functions

• sales forecasting
• Promotion planning and budgeting
• Predictive modeling/optimization
• Promotion execution and monitoring
• Settlement
• Post event analysis

Business problems addressed

Historically, there have been many solutions to trade promotion management. Commonly, companies use their accounting systems or spreadsheets, but as the complexity of trade increases software solutions have been developed and implemented to fill the needs of companies in various industries including consumer goods, food manufacturing, food service and others.

 Lack of accurate and timely information to support trade promotion decision-making

Trade promotion decisions are often rushed and based on sub-par data. While sales and marketing managers are surrounded by promotion information, questions on retail commitment and product forecast accuracy can hinder the process. Multiple data sources and conflicting needs from various departments further complicate the issue.

Inability to plan promotions based on analytics

Historical trade promotion data should be analyzed in order to continually improve trade promotions. If a company does not utilize processes and systems that measure trade promotion performance, future trade promotion executions could be less effective than if they’d been planned using past analytical information.

Ineffective organization and partner integration

Lack of integration both internally and with external partners can hinder trade promotion success. Key elements of organizational integration include standardized metrics, regular information sharing, cross-functional department collaboration, and collaborative processes4. Integration with retail partners is important to executing promotions successfully, as well as maintain strong relationships with retailers over time.

Lack of appropriate Key Performance Indicators (KPI)

KPIs tell manufacturers and retailers how trade promotions performed relative to their pre-determined objectives. A lack of understanding on what trade promotion data to measure and how to measure performance can hinder the overall process. Manufacturers and retailers will not know what made a promotion effective or ineffective unless they have predetermined data points to measure and analyze.

Lack of control and the ability to accurately monitor the effectiveness and efficiency of Trade Promotions is the key area of concern. Many companies still utilise vast arrays of fragmented spreadsheet style reports to manage their TPM activities.

The complexities of the typical Trade Promotion Management process can be appreciated in this diagram which is based on just one individual retailer account.

Not all promotions are created equal and there are many ways of looking at a promotion and evaluating the effective and efficient use of trade funds. The characteristics of your product portfolio eg beer, toilet paper, toothpaste and yoghurt will impact on the performance of your trade promotion even if the same promotional strategy is used.

For example: with the same size and display location, toilet tissue wipes up the competition with an 82% display driven lift, with yoghurt a distant second at 28%, beer at 15% and toothbrushes at 14%.

(Source: Nielsenwire “Six Trade Promotion Tips: Why Less Can be More”)

Why Promote?

Promotions are undertaken with the intention of driving an increase in consumer sales. This may be instigated as the Brand Manager has a target/KPI to achieve and the promotion is designed to close the gap between the baseline sales and the target. Generally the manufacturer will sell their products to retailers at a discount to help fund the promotion. This discounting is of no benefit if there are no additional sales as a result of the promotion.

Promax delivers key information that helps Brand Managers answer questions, such as:

• What to promote?
• How often to promote?
• Timing – when to promote?
• Duration – how long to promote?
• Promotional level – how much should I offer?
• Product selection – which bundle of products to select?
• Are the trade deals aligned with the Brand positioning?
• What is the level of sales lift during the promotional period?
• Has the promotion created a long-term impact on base-line consumer sales?
• What happens if I change the frequency of my promotions?
• Do my promotions clash amongst various retailers?
• Are specific promotions cannibalising sales of other products in my portfolio and/or competitor products?
• What is the optimum scenario in terms of volume and profitability?
• What impact do these promotions have on customer profitability?
• How do I communicate and implement these promotions via the salesforce?

There are a number of good reasons why one might choose a simple price promotion:

• Offensive sales gain – to offset competitive threats. Typically this may mean a situation where the supplier is prepared to forego short term profits.
• Recover market share – sustaining a nominated loss of contribution to gain a position over competitors
• Stimulate Sales – without a loss of contribution and hopefully an increase in overall profitability. This is probably the most common rationale for promotions. Implicit in this is that the company will increase sales revenue and thus the total profit from the promotion will be greater than if there were no promotion at all.
• Reward – for existing brand loyal users. This may be a short-term loss of profitability and may even make money if a lot of brand loyal consumers stock up.
• Switch – However, if there are a large number of promotions targeted at “deal loyal” users then they will switch brand frequently. When promoting to this group the best course of action is to make sure that the promotion is profitable as there may be no long-term future gains to be had.
• Gets new users to Trial – for new products every new trial is valuable; this incentive is premised on the assumption that the long-term value of a new user may be worth many times the cost of the initial purchase, so the supplier is prepared to invest heavily to obtain a new consumer. This may justify the high cost of a promotion. Trade promotions of new items often need to be supported by other vehicles such as demonstrations and media advertising.
• Reward to Trade. This often happens to get a buyer off your back. If however, it is viewed as being good for both parties, there should be a profit opportunity.
• Because the trade threatens deletion – in many instances buyers have co-op budgets to achieve. There is pressure on suppliers to maintain ranging and promotion of your products in the category ensures support for your brand.

Each of these scenarios has different parameters of cost, price point, mechanic, vehicle and incremental volume gain. In reality the sales team gets a certain amount of money to spend on promotions, some say that these funds are a “cost of doing business”. The more money that is provided the more we will sell. If only this were true! We really need to understand how we can effectively apply the limited trade funds to maximise their efficiency.

Source: Wikipedia.

Data warehouse Neeraj Nathani

Data warehouse

In computing, a data warehouse or enterprise data warehouse (DW, DWH, or EDW) is a database used for reporting and data analysis. It is a central repository of data which is created by integrating data from one or more disparate sources. Data warehouses store current as well as historical data and are used for creating trending reports for senior management reporting such as annual and quarterly comparisons.

The data stored in the warehouse are uploaded from the operational systems (such as marketing, sales etc., shown in the figure to the right). The data may pass through an operational data store for additional operations before they are used in the DW for reporting.

The typical ETL-based data warehouse uses staging, data integration, and access layers to house its key functions. The staging layer or staging database stores raw data extracted from each of the disparate source data systems. The integration layer integrates the disparate data sets by transforming the data from the staging layer often storing this transformed data in an operational data store (ODS) database. The integrated data are then moved to yet another database, often called the data warehouse database, where the data is arranged into hierarchical groups often called dimensions and into facts and aggregate facts. The combination of facts and dimensions is sometimes called a star schema. The access layer helps users retrieve data.^[1]

A data warehouse constructed from an integrated data source systems does not require ETL, staging databases, or operational data store databases. The integrated data source systems may be considered to be a part of a distributed operational data store layer. Data federation methods or data virtualization methods may be used to access the distributed integrated source data systems to consolidate and aggregate data directly into the data warehouse database tables. Unlike the ETL-based data warehouse, the integrated source data systems and the data warehouse are all integrated since there is no transformation of dimensional or reference data. This integrated data warehouse architecture supports the drill down from the aggregate data of the data warehouse to the transactional data of the integrated source data systems.

Data warehouses can be subdivided into data marts. Data marts store subsets of data from a warehouse.

This definition of the data warehouse focuses on data storage. The main source of the data is cleaned, transformed, cataloged and made available for use by managers and other business professionals for data mining, online analytical processing, market research and decision support (Marakas & O'Brien 2009). However, the means to retrieve and analyze data, to extract, transform and load data, and to manage the data dictionary are also considered essential components of a data warehousing system. Many references to data warehousing use this broader context. Thus, an expanded definition for data warehousing includes business intelligence tools, tools to extract, transform and load data into the repository, and tools to manage and retrieve metadata.

Benefits of a data warehouse

A data warehouse maintains a copy of information from the source transaction systems. This architectural complexity provides the opportunity to:

• Congregates data from multiple sources into a single database so a single query engine can be used to present data.
• Mitigates the problem of database isolation level lock contention in transaction processing systems caused by attempts to run large, long running, analysis queries in transaction processing databases.
• Maintain data history, even if the source transaction systems do not.
• Integrate data from multiple source systems, enabling a central view across the enterprise. This benefit is always valuable, but particularly so when the organization has grown by merger.
• Improve data quality, by providing consistent codes and descriptions, flagging or even fixing bad data.
• Present the organization's information consistently.
• Provide a single common data model for all data of interest regardless of the data's source.
• Restructure the data so that it makes sense to the business users.
• Restructure the data so that it delivers excellent query performance, even for complex analytic queries, without impacting the operational systems.
• Add value to operational business applications, notably customer relationship management (CRM) systems.

Generic data warehouse environment

The environment for data warehouses and marts includes the following:

• Source systems that provide data to the warehouse or mart;
• Data integration technology and processes that are needed to prepare the data for use;
• Different architectures for storing data in an organization's data warehouse or data marts;
• Different tools and applications for the variety of users;
• Metadata, data quality, and governance processes must be in place to ensure that the warehouse or mart meets its purposes.

In regards to source systems listed above, Rainer states, “A common source for the data in data warehouses is the company’s operational databases, which can be relational databases” (130).
Regarding data integration, Rainer states, “It is necessary to extract data from source systems, transform them, and load them into a data mart or warehouse” (131).
Rainer discusses storing data in an organization’s data warehouse or data marts. “There are a variety of possible architectures to store decision-support data” (131).
Metadata are data about data. “IT personnel need information about data sources; database, table, and column names; refresh schedules; and data usage measures (133).
Today, the most successful companies are those that can respond quickly and flexibly to market changes and opportunities. A key to this response is the effective and efficient use of data and information by analysts and managers (Rainer, 127). A “data warehouse” is a repository of historical data that are organized by subject to support decision makers in the organization (128). Once data are stored in a data mart or warehouse, they can be accessed.

Facts

A fact is a value or measurement, which represents a fact about the managed entity or system.
Facts as reported by the reporting entity are said to be at raw level.
E.g. if a BTS received 1,000 requests for traffic channel allocation, it allocates for 820 and rejects the remaining then it would report 3 facts or measurements to a management system:

• tch_req_total = 1000
• tch_req_success = 820
• tch_req_fail = 180

Facts at raw level are further aggregated to higher levels in various dimensions to extract more service or business-relevant information out of it. These are called aggregates or summaries or aggregated facts.
E.g. if there are 3 BTSs in a city, then facts above can be aggregated from BTS to city level in network dimension. E.g.

• 
• 

[edit] Dimensional vs. normalized approach for storage of data

There are two leading approaches to storing data in a data warehouse — the dimensional approach and the normalized approach.
The dimensional approach, whose supporters are referred to as “Kimballites”, believe in Ralph Kimball’s approach in which it is stated that the data warehouse should be modeled using a Dimensional Model/star schema. The normalized approach, also called the 3NF model, whose supporters are referred to as “Inmonites”, believe in Bill Inmon's approach in which it is stated that the data warehouse should be modeled using an E-R model/normalized model.
In a dimensional approach, transaction data are partitioned into "facts", which are generally numeric transaction data, and "dimensions", which are the reference information that gives context to the facts. For example, a sales transaction can be broken up into facts such as the number of products ordered and the price paid for the products, and into dimensions such as order date, customer name, product number, order ship-to and bill-to locations, and salesperson responsible for receiving the order.
A key advantage of a dimensional approach is that the data warehouse is easier for the user to understand and to use. Also, the retrieval of data from the data warehouse tends to operate very quickly. Dimensional structures are easy to understand for business users, because the structure is divided into measurements/facts and context/dimensions. Facts are related to the organization’s business processes and operational system whereas the dimensions surrounding them contain context about the measurement (Kimball, Ralph 2008).
The main disadvantages of the dimensional approach are:

1. In order to maintain the integrity of facts and dimensions, loading the data warehouse with data from different operational systems is complicated, and
2. It is difficult to modify the data warehouse structure if the organization adopting the dimensional approach changes the way in which it does business.

In the normalized approach, the data in the data warehouse are stored following, to a degree, database normalization rules. Tables are grouped together by subject areas that reflect general data categories (e.g., data on customers, products, finance, etc.). The normalized structure divides data into entities, which creates several tables in a relational database. When applied in large enterprises the result is dozens of tables that are linked together by a web of joins. Furthermore, each of the created entities is converted into separate physical tables when the database is implemented (Kimball, Ralph 2008). The main advantage of this approach is that it is straightforward to add information into the database. A disadvantage of this approach is that, because of the number of tables involved, it can be difficult for users both to:

1. join data from different sources into meaningful information and then
2. access the information without a precise understanding of the sources of data and of the data structure of the data warehouse.

It should be noted that both normalized and dimensional models can be represented in entity-relationship diagrams as both contain joined relational tables. The difference between the two models is the degree of normalization.
These approaches are not mutually exclusive, and there are other approaches. Dimensional approaches can involve normalizing data to a degree (Kimball, Ralph 2008).
In Information-Driven Business (Wiley 2010),^[6] Robert Hillard proposes an approach to comparing the two approaches based on the information needs of the business problem. The technique shows that normalized models hold far more information than their dimensional equivalents (even when the same fields are used in both models) but this extra information comes at the cost of usability. The technique measures information quantity in terms of Information Entropy and usability in terms of the Small Worlds data transformation measure.^[7]

Source: Wikipedia.

Neeraj Nathani The Big Value In Big Data: Seeing Customer Buying Patterns

The Big Value In Big Data: Seeing Customer Buying Patterns

Is Big Data simply a popular catchphrase or the launch of a new era? Overuse doesn’t automatically transform a buzzword into a best business practice, and several factors, such as measurable results, focus and sustainability, determine whether an idea is much more than just that. The core question is this: does big data actually solve real-world business problems? The short answer is yes – and here’s a real world example of how leveraging Big Data can solve the complexity around product proliferation by helping companies align product offering and supply chain based on customer-buying patterns.

Big Data: More Than Just A Trend

According to Gartner, unstructured and structured data held by enterprises continues to grow at explosive rates. However, volume and velocity of data – what the business world is beginning to understand as the “Big Data Problem” – are becoming less of an issue than the variety of data. Each silo within the enterprise – operations, supply management, sales, marketing – faces its own data variety challenges, where bits exist in a multitude of formats and types.

Neeraj Nathani:

Due to the variability of data across silos, systems can’t “speak” to one another, and gaining an accurate, enterprise-wide view of demand and performance seems impossible. In fact, most business and IT managers accept the lack of intersystem collaboration as a given, an inevitable limit that must be worked around. As a result, what we know is being increasingly outpaced by the things we don’t know. Performance within individual silos is clear, but this view does little to inform effective strategic direction for the organization.

There is a better way to tackle this challenge in variety and capture the opportunity posed by Big Data.

Patterns And Connections
Traditionally business silos have individually sorted, stored and managed data that is relevant to the needs within the silo. This has resulted in enterprise-related data being disrupted into disconnected pieces across the silos. The Big Data challenge requires aggregating the data across these silos for a single view of the organization. The purpose of the aggregation is to reveal the intelligence and causality across the business functions for strategic insight. This Big Data challenge requires solutions that can harness the intelligence from the data and deliver actionable intelligence to the business user. Conventional business intelligence and data warehouse tools aren’t designed to analyze, identify and surface critical data linkages and causality. As a result, insights, contexts and market opportunities remain hidden from view because users don’t know what they don’t know.
These critical connections and causalities are the key to managing big data and allowing companies to see a more comprehensive picture of their product and product variants based on actual customer-buying patterns. The connection patterns reveal valuable information quickly, accurately and allow for faster, more relevant decision-making.
Freeing the data to reveal connections and causation through pattern-based analytics solutions will paint a bigger picture – one that can better manage product variants and streamline sales by shedding light on what customers are buying, when, where and how. Currently companies pour their non-standard data into spreadsheets that then require teams of data analysts to interpret and derive meaning from it. This is not scalable and often misses the mark. Big data demands applications that can interpret and deliver immediate actionable intelligence to business users

Leveraging Data To Address Product Proliferation

One of the major business problems companies are facing today is the complexity caused by product proliferation – one of the biggest drivers of material cost and inventory levels. This is a complex issue that isn’t easily resolved with traditional approaches, but can be addressed with a systematic and enterprise-wide pattern-based analytics approach to leveraging the data across the silos within the organization.

For high value manufacturing, product variety is the single biggest driver of cost as it dictates the material requirements and inventory, which is typically over 80 percent of the total cost. Chasing diverse customer demands with an explosion of product variants dramatically increases total cost and causes volatility across the supply chain. But what are customers really buying? Are customers really buying all that companies are offering? Is there a way to satisfy them with fewer variations? Or alternate variations that improve supply chain efficiency? This is crucial knowledge for not only satisfying the demand, but also reducing supply chain costs and increasing margins.

As product proliferation has increased, so have operational complexity and cost structure; “what-if” scenarios with alternate product portfolios that meet the majority of customer demand have been an underutilized lever and represent the next frontier of business process improvement. A poor product mix will drive complexity throughout the value chain – impacting supply chain, marketing and sales, and service and support. In the same vein, a good product mix can dramatically improve delivery costs and increase profits.

The product portfolio also impacts the sales efficiency and top line revenue. To understand the opportunity cost with sales, consider the typical OEM salesperson: Let’s say he or she has a $4 million quota and spends 10.5% of his or her time defining a customer solution, configuring and pricing it, and then tracking delivery. Reducing that time by just 1 percent for a 100-person sales force represents an opportunity cost of $42 million.

Pattern-based analytics solutions are already being leveraged by some companies, giving them a much-needed competitive advantage in a high-risk environment by providing insights into customer-buying patterns to guide the product offering, supply chain planning and execution. For example, when NCR needed to optimize product configurations across their ATM line, they implemented pattern-based analytics solutions to analyze customer-buying patterns. With insight into what’s selling where, to whom, when and how often, NCR optimized the product line, defined customer segments and then seamlessly pushed this to the sales team to help reduce lead times.

At NCR, this resulted in a dramatic improvement in sales efficiency and supply chain performance. Pattern-based analytics can reveal deep insight into what customers are buying, and leverages that to offer and guide customers to the best choices based on availability and product margin. This approach uses customer-buying patterns to create the best product offering, and simultaneously guides customers to the best choices based on what is on hand. This is a win-win as the supply chain builds what is being bought and the sales reps sell what is in stock.

Source: Wikipedia.

Neeraj_Nathani Smart Bridge Big Data Technology Evaluation Checklist

Big Data Technology Evaluation Checklist

Anyone who’s been following the rapid-fire technology developments in the world that is becoming known as “big data” sees a new capability, product, or company founded literally every week. The ambition of all of these players, established and newcomer, is tremendous, because the potential value to business is enormous. Each new arrival is aimed at addressing the pain that enterprises are experiencing around unrelenting growth in the velocity, volume, and variety of the data their operations generate.

What’s being lost, however, in some of this frothy marketing activity, is that it’s still early for big data technologies. There are vexing problems slowing the growth and the practical implementation of big data technologies. For the technologies to succeed at scale, there are several fundamental capabilities they should contain, including stream processing, parallelization, indexing, data evaluation environments and visualization.

When evaluating big data technology, it can be valuable to ask companies about their ability to deliver some of these fundamental capabilities. If you get an unsophisticated answer, you may find that the company is not as serious or capable as you might have expected. (For my research on this topic please see: Designing a Scalable and Agile Big Data Platform.)

In this article, we examine some of the big data requirements that are partially defined or in early stages of maturity. Any big data vendor worth considering should be able to address these requirements now or in the near future, or confidently explain their position. We sat down with Mike Driscoll, CTO of Metamarkets, a big data company that delivers predictive analytics solutions for digital media, to develop a checklist for evaluating new solutions and their fit criteria against the challenges of big data:

Some general questions to begin the evaluation process:

• Does the solution allow for stream processing, and incremental calculation of statistics?
• Does the solution parallelize processing and take advantage of distributed computing?
• Does the solution perform summary indexing to accelerate queries of huge datasets?
• What are the solution’s data exploration and evaluation environments that enable a quick understanding of the value of new datasets?
• How does a solution directly provide or easily integrate with visualization tools?
• What is the strategy for verticalization of the technology?
• What is the ecosystem strategy? How does the solution provider fill the gaps in its capabilities through partnerships?

Getting these questions answered will put most vendors to the test and help improve your understanding of the technology you are evaluating.

Stream processing

As the pace of business has increased, and the number of instrumented business processes has expanded, increasingly our attention is focused not on “data sets,” but on “data streams.”

“Decision-makers are interested putting their finger on the pulse of their organization, but to get answers in real time; they require architectures that can process streams of data as they happen,” Driscoll says. “Current database technologies are not well suited to do this kind of stream processing.”

For example, calculating an average over a group of data can be done in a traditional batch process, but far more efficient algorithms exist for calculating a moving average of data as it arrives, incrementally, unit by unit. If you want to take a repository of data and perform almost any statistical analysis, that can be accomplished with open source products like R or commercial products like SAS. But if you want to create a set of streaming statistics, to which you incrementally add or remove a chunk of data as a moving average, the libraries either don’t exist or are immature.

“The entire ecosystem around streaming data is underdeveloped,” says Driscoll.

In other words, if you’re talking to a vendor about a big data project, you have to determine whether this kind of stream processing is important to your project, and if it is, whether they have a capability to provide it. This axiom extends all the way down, to not just the analytical algorithms that run over the streams, but also to the way in which those streams are queued, ingested, managed and ultimately processed Many architectures exist for ingesting and queuing data streams, some of which are proprietary. TIBCO, Esper and ZeroMQ all offer solutions. But those solutions are only about moving packets of data around. Actually doing analysis of the streams requires practitioners to build on a lower level, for which there is another rapidly evolving toolset.
Parallelization
There are many definitions of big data. Here’s a useful one: “Small data” is data that fits in-memory on a single desktop or machine with a capacity of between 1 GB and 10 GB of disk space. “Medium data” fits on a single hard drive of 100 GB to 1 TB in size. “Large data” is distributed over many machines, comprising 1 TB to multiple petabytes.
“If you want to work with distributed data, and you expect to have any hope of processing that data in a reasonable amount of time, that requires distributed processing,” Driscoll says.
Parallel processing comes to the fore in distributed data. Hadoop is one of the better-known examples of distributed or parallelized processing. Hadoop can do more than distributed processing. It can also conduct distributed queries, which have been a subject of interest recently for designers of massively parallel processing (MPP) databases, whose object is to take a query and parallelize it across a set of nodes. Each of those nodes does partial work for the query in parallel, and then combines those partial answers into a single unified answer, Driscoll explains. Parallelizing queries is not a simple affair. When something is being analyzed in a parallel stream, each new unit of data must be combined with an existing unit of data in order to produce an answer.
Therefore, if a vendor is trying to sell you a solution for addressing big data at scale, their salespeople should be able to articulate their special secret sauce and strategy for parallelization.
“One of the most important features that current data warehouse vendors must offer is the ability to do parallel copy from Hadoop into their warehouse,” Driscoll says. “So, whether it’s EMC Greenplum offering the ability to do distributed parallel copy from Hadoop to Greenplum, Netezza or Oracle, the ability to parallelize data transfer is a critical feature.”
Hadoop has a massively distributed file system and can support distributed queries on top of it. But it does not inherently support parallelization—running a parallel process on Hadoop without an algorithm for optimizing queries can significantly slow down the process, taking minutes to return an answer. This is acceptable for some queries, but it won’t support real-time analytics in a big data world. The power and speed of that algorithm will be a determining factor of the robustness and cost of the solution, and that should be appropriately scaled to your needs, says Driscoll.
Summary indexing
Summary indexing is the process of creating a pre-calculated summary of data to speed up running queries. The problem with summary indexing is that it requires you to plan in advance what kind of queries you are going to run, so it is limiting.
The most common form of summary indexing is the star schema used to support speedy searches in data warehouses. A star schema prioritizes one master dimension (such as location or product) in advance of running a multi-dimensional data cube, organizing subordinate dimensions in relation to the master. The technique works well, but has one huge problem. If you want to ask a new question, it takes a long time to reconfigure the schema and associated data cubes and recompute them. When we begin to ask questions of all our data, not just the structured data, it is practically impossible to create a star schema to answer every possible question. The problem is not pre-processing; it’s the difficulty of reconfiguring the pre-processing as needed to ask new questions and get a speedy response.
The ideal solution would easily adjust the summaries being created as new questions arose. With a quickly created, summarized form of the data, it would then be possible to use data-analysis tools such as QlikView, Tableau, or TIBCO Spotfire for exploration and analysis. But there is currently a gap in available tools to make this summary creation easier, as many available tools don’t reach down to machine-level data, says Driscoll. The result is that the IT department becomes involved in building a custom query.

Some help is on the way for this problem. Vendors such as Splunk have emerged with a solution based on their search language that makes creating summary indexes far faster than other approaches, like star schemas. The designers of technology like SAP HANA, 1010 Data, and Metamarkets recommend an in-memory approach that completely abandons summarizing, by keeping vast amounts of data in in-memory systems.

But data volumes are growing fast and the need for summarizing will never go away completely. For the short and medium term, vendors must have a strategy for agile creation of summary indexes.
Data evaluation environments
How does your vendor’s solution understand, or allow you to understand the meaning of new datasets and incorporate them into your analysis?

For example, if your business has a retail store, and you are studying transactions, you could pick up the movements of people around the web site, or even around brick-and-mortar stores through opt-in GPS signals and cell-phone tracking. Once you acquire the data:

• How do you incorporate and understand what that data can tell you?
• How do you develop a model of the store that will help you analyze customer behavior?
• How do you understand when those movements become events?
• How fast can you figure that out—before the customer leaves the store?

To answer these questions, your solution needs to be able to join disparate datasets. Very few vendors have a distinguished capability to join datasets. And, just as the most critical areas of a building are at the joints, the same is true of data architecture. Where data interfaces, tremendous value can be unlocked, Driscoll says.

“The holy grail for anyone in the online retail space is to understand the connection between online impression events that lead to actions, such as clicks, or some level of engagement that eventually lead to purchasing behaviors, which eventually lead to long-term customer adoption,” Driscoll says. “Right now, all of these datasets live in different places. American Express knows where you’ve purchased your Starbucks coffee and Foursquare knows where you checked in at Starbucks, and Yahoo! knows when you clicked on an ad discount for a Starbucks latte on a hot summer day. And yet, people are struggling to thread these various data streams together.”

As enterprises begin to draw in disparate data feeds—particularly mobile data feeds—it’s critical that a vendor has a solution for rapidly joining disparate datasets, because the information they contain is critical for enterprises.

Visualization

There are two broad categories of visualization tools, according to Driscoll.

Exploratory visualization describes tools that allow a decision-maker and an analyst to explore different axes of the data for relationships, which usually involves some kind of visual “mining for insights.” Tools such as Tableau and TIBCO Spotfire, and to a lesser extent QlikView, fit into this category, Driscoll says.

Narrative visualizations are designed to examine a particular axis of the data in a particular way. For instance, if you say want to look at a time series visualization of sales broken up by geography for an enterprise, a format for that visualization can be pre-created. The data can be played back month-by-month for every geography, and is sorted into a pre-cast formula. Vendors such as Perceptive Pixel fit into this category.

In narrative visualizations, “Certain knobs are free to explore the data, but it’s not completely open to ask any question,” Driscoll says. “These visualizations are designed to tell a certain story about the data, just as certain pre-computations or reports are designed to tell a certain story. And some tools are better for the first, ad-hoc exploratory model, and others are better for the second, constrained narrative.”

Mind the Verticals

There are nearly as many types of decision-making needs in different verticals as there are ways to collect, process and analyze data. Vendors should be wary that different decisions makers within an organization and between verticals have different kinds of visualizations that they are accustomed to seeing.

“Any vendor that wants to serve the needs of those decisions-makers ought to be well aware of what those expected narratives are, because that will speed the adoption of that visualization tool,” Driscoll says, citing the preference for candlestick plots in the financial services industry.
Ecosystem Strategy
The largest most successful companies all spend tens of millions creating ecosystems around their products. The ecosystems are supported by product features and business models that allow the product to do its job but also work with other technologies or partners who extend the product or craft it to special uses. If a product doesn’t have an ecosystem strategy, you may find that it is hard to adapt to your needs and that finding expertise to help with implementation and configuration may be hard to come by.
This list of requirements for big data technology is not exhaustive, but it is a good start. Using these topics when evaluating big data technology will only lead to deeper understanding.

Source: Wikipedia.