PERFORMANCE BOOSTERS FOR ETL PROGRAMS: Neeraj Nathani-Smart Bridge

PERFORMANCE BOOSTERS FOR ETL PROGRAMS:

Challenges

ETL processes can involve considerable complexity, and significant operational problems can occur with improperly designed ETL systems.

The range of data values or data quality in an operational system may exceed the expectations of designers at the time validation and transformation rules are specified. Data profiling of a source during data analysis can identify the data conditions that will need to be managed by transform rules specifications. This will lead to an amendment of validation rules explicitly and implicitly implemented in the ETL process.

Data warehouses are typically assembled from a variety of data sources with different formats and purposes. As such, ETL is a key process to bring all the data together in a standard, homogeneous environment.

Design analysts should establish the scalability of an ETL system across the lifetime of its usage. This includes understanding the volumes of data that will have to be processed within service level agreements. The time available to extract from source systems may change, which may mean the same amount of data may have to be processed in less time. Some ETL systems have to scale to process terabytes of data to update data warehouses with tens of terabytes of data. Increasing volumes of data may require designs that can scale from daily batch to multiple-day microbatch to integration with message queues or real-time change-data capture for continuous transformation and update

Performance

ETL vendors benchmark their record-systems at multiple TB (terabytes) per hour (or ~1 GB per second) using powerful servers with multiple CPUs, multiple hard drives, multiple gigabit-network connections, and lots of memory. The fastest ETL record is currently held by Syncsort,^[1] Vertica and HP at 5.4TB in under an hour which is more than twice as fast as the earlier record held by Microsoft and Unisys.

In real life, the slowest part of an ETL process usually occurs in the database load phase. Databases may perform slowly because they have to take care of concurrency, integrity maintenance, and indices. Thus, for better performance, it may make sense to employ:

• Direct Path Extract method or bulk unload whenever is possible (instead of querying the database) to reduce the load on source system while getting high speed extract
• most of the transformation processing outside of the database
• bulk load operations whenever possible.

Still, even using bulk operations, database access is usually the bottleneck in the ETL process. Some common methods used to increase performance are:

• Partition tables (and indices). Try to keep partitions similar in size (watch for null values which can skew the partitioning).
• Do all validation in the ETL layer before the load. Disable integrity checking (disable constraint ...) in the target database tables during the load.
• Disable triggers (disable trigger ...) in the target database tables during the load. Simulate their effect as a separate step.
• Generate IDs in the ETL layer (not in the database).
• Drop the indices (on a table or partition) before the load - and recreate them after the load (SQL: drop index ...; create index ...).
• Use parallel bulk load when possible — works well when the table is partitioned or there are no indices. Note: attempt to do parallel loads into the same table (partition) usually causes locks — if not on the data rows, then on indices.
• If a requirement exists to do insertions, updates, or deletions, find out which rows should be processed in which way in the ETL layer, and then process these three operations in the database separately. You often can do bulk load for inserts, but updates and deletes commonly go through an API (using SQL).

Whether to do certain operations in the database or outside may involve a trade-off. For example, removing duplicates using distinct may be slow in the database; thus, it makes sense to do it outside. On the other side, if using distinct will significantly (x100) decrease the number of rows to be extracted, then it makes sense to remove duplications as early as possible in the database before unloading data.

A common source of problems in ETL is a big number of dependencies among ETL jobs. For example, job "B" cannot start while job "A" is not finished. You can usually achieve better performance by visualizing all processes on a graph, and trying to reduce the graph making maximum use of parallelism, and making "chains" of consecutive processing as short as possible. Again, partitioning of big tables and of their indices can really help.

Another common issue occurs when the data is spread between several databases, and processing is done in those databases sequentially. Sometimes database replication may be involved as a method of copying data between databases - and this can significantly slow down the whole process. The common solution is to reduce the processing graph to only three layers:

• Sources
• Central ETL layer
• Targets

This allows processing to take maximum advantage of parallel processing. For example, if you need to load data into two databases, you can run the loads in parallel (instead of loading into 1st - and then replicating into the 2nd).

Of course, sometimes processing must take place sequentially. For example, you usually need to get dimensional (reference) data before you can get and validate the rows for main "fact" tables.

Parallel processing

A recent^[update] development in ETL software is the implementation of parallel processing. This has enabled a number of methods to improve overall performance of ETL processes when dealing with large volumes of data.

ETL applications implement three main types of parallelism:

• Data: By splitting a single sequential file into smaller data files to provide parallel access.
• Pipeline: Allowing the simultaneous running of several components on the same data stream. For example: looking up a value on record 1 at the same time as adding two fields on record 2.
• Component: The simultaneous running of multiple processes on different data streams in the same job, for example, sorting one input file while removing duplicates on another file.

All three types of parallelism usually operate combined in a single job.

An additional difficulty comes with making sure that the data being uploaded is relatively consistent. Because multiple source databases may have different update cycles (some may be updated every few minutes, while others may take days or weeks), an ETL system may be required to hold back certain data until all sources are synchronized. Likewise, where a warehouse may have to be reconciled to the contents in a source system or with the general ledger, establishing synchronization and reconciliation points becomes necessary.

Source: Wikipedia.

Smart Bridge Trading Solutions Neeraj Nathani The Big Value In Big Data: Seeing Customer Buying Patterns

The Big Value In Big Data: Seeing Customer Buying Patterns

Is Big Data simply a popular catchphrase or the launch of a new era? Overuse doesn’t automatically transform a buzzword into a best business practice, and several factors, such as measurable results, focus and sustainability, determine whether an idea is much more than just that. The core question is this: does big data actually solve real-world business problems? The short answer is yes – and here’s a real world example of how leveraging Big Data can solve the complexity around product proliferation by helping companies align product offering and supply chain based on customer-buying patterns.

Big Data: More Than Just A Trend

According to Gartner, unstructured and structured data held by enterprises continues to grow at explosive rates. However, volume and velocity of data – what the business world is beginning to understand as the “Big Data Problem” – are becoming less of an issue than the variety of data. Each silo within the enterprise – operations, supply management, sales, marketing – faces its own data variety challenges, where bits exist in a multitude of formats and types.

Neeraj Nathani:

Due to the variability of data across silos, systems can’t “speak” to one another, and gaining an accurate, enterprise-wide view of demand and performance seems impossible. In fact, most business and IT managers accept the lack of intersystem collaboration as a given, an inevitable limit that must be worked around. As a result, what we know is being increasingly outpaced by the things we don’t know. Performance within individual silos is clear, but this view does little to inform effective strategic direction for the organization.

There is a better way to tackle this challenge in variety and capture the opportunity posed by Big Data.

Patterns And Connections
Traditionally business silos have individually sorted, stored and managed data that is relevant to the needs within the silo. This has resulted in enterprise-related data being disrupted into disconnected pieces across the silos. The Big Data challenge requires aggregating the data across these silos for a single view of the organization. The purpose of the aggregation is to reveal the intelligence and causality across the business functions for strategic insight. This Big Data challenge requires solutions that can harness the intelligence from the data and deliver actionable intelligence to the business user. Conventional business intelligence and data warehouse tools aren’t designed to analyze, identify and surface critical data linkages and causality. As a result, insights, contexts and market opportunities remain hidden from view because users don’t know what they don’t know.
These critical connections and causalities are the key to managing big data and allowing companies to see a more comprehensive picture of their product and product variants based on actual customer-buying patterns. The connection patterns reveal valuable information quickly, accurately and allow for faster, more relevant decision-making.
Freeing the data to reveal connections and causation through pattern-based analytics solutions will paint a bigger picture – one that can better manage product variants and streamline sales by shedding light on what customers are buying, when, where and how. Currently companies pour their non-standard data into spreadsheets that then require teams of data analysts to interpret and derive meaning from it. This is not scalable and often misses the mark. Big data demands applications that can interpret and deliver immediate actionable intelligence to business users

Leveraging Data To Address Product Proliferation

One of the major business problems companies are facing today is the complexity caused by product proliferation – one of the biggest drivers of material cost and inventory levels. This is a complex issue that isn’t easily resolved with traditional approaches, but can be addressed with a systematic and enterprise-wide pattern-based analytics approach to leveraging the data across the silos within the organization.

For high value manufacturing, product variety is the single biggest driver of cost as it dictates the material requirements and inventory, which is typically over 80 percent of the total cost. Chasing diverse customer demands with an explosion of product variants dramatically increases total cost and causes volatility across the supply chain. But what are customers really buying? Are customers really buying all that companies are offering? Is there a way to satisfy them with fewer variations? Or alternate variations that improve supply chain efficiency? This is crucial knowledge for not only satisfying the demand, but also reducing supply chain costs and increasing margins.

As product proliferation has increased, so have operational complexity and cost structure; “what-if” scenarios with alternate product portfolios that meet the majority of customer demand have been an underutilized lever and represent the next frontier of business process improvement. A poor product mix will drive complexity throughout the value chain – impacting supply chain, marketing and sales, and service and support. In the same vein, a good product mix can dramatically improve delivery costs and increase profits.

The product portfolio also impacts the sales efficiency and top line revenue. To understand the opportunity cost with sales, consider the typical OEM salesperson: Let’s say he or she has a $4 million quota and spends 10.5% of his or her time defining a customer solution, configuring and pricing it, and then tracking delivery. Reducing that time by just 1 percent for a 100-person sales force represents an opportunity cost of $42 million.

Pattern-based analytics solutions are already being leveraged by some companies, giving them a much-needed competitive advantage in a high-risk environment by providing insights into customer-buying patterns to guide the product offering, supply chain planning and execution. For example, when NCR needed to optimize product configurations across their ATM line, they implemented pattern-based analytics solutions to analyze customer-buying patterns. With insight into what’s selling where, to whom, when and how often, NCR optimized the product line, defined customer segments and then seamlessly pushed this to the sales team to help reduce lead times.

At NCR, this resulted in a dramatic improvement in sales efficiency and supply chain performance. Pattern-based analytics can reveal deep insight into what customers are buying, and leverages that to offer and guide customers to the best choices based on availability and product margin. This approach uses customer-buying patterns to create the best product offering, and simultaneously guides customers to the best choices based on what is on hand. This is a win-win as the supply chain builds what is being bought and the sales reps sell what is in stock.

Source: Wikipedia.

Smart Bridge INDEXES IN SQL Neeraj Nathani Smart Bridge Trading Solutions

INDEXES IN SQL

Indexes are special lookup tables that the database search engine can use to speed up data retrieval. Simply put, an index is a pointer to data in a table. An index in a database is very similar to an index in the back of a book.

For example, if you want to reference all pages in a book that discuss a certain topic, you first refer to the index, which lists all topics alphabetically, and are then referred to one or more specific page numbers.

An index helps speed up SELECT queries and WHERE clauses, but it slows down data input, with UPDATE and INSERT statements. Indexes can be created or dropped with no effect on the data.

Creating an index involves the CREATE INDEX statement, which allows you to name the index, to specify the table and which column or columns to index, and to indicate whether the index is in ascending or descending order.

Indexes can also be unique, similar to the UNIQUE constraint, in that the index prevents duplicate entries in the column or combination of columns on which there's an index.

The CREATE INDEX Command:

The basic syntax of CREATE INDEX is as follows:

CREATE INDEX index_name ON table_name;

Single-Column Indexes:

A single-column index is one that is created based on only one table column. The basic syntax is as follows:

CREATE INDEX index_name

ON table_name (column_name);

Unique Indexes:

Unique indexes are used not only for performance, but also for data integrity. A unique index does not allow any duplicate values to be inserted into the table. The basic syntax is as follows:

CREATE INDEX index_name

on table_name (column_name);

Composite Indexes:

A composite index is an index on two or more columns of a table. The basic syntax is as follows:

CREATE INDEX index_name

on table_name (column1, column2);

Whether to create a single-column index or a composite index, take into consideration the column(s) that you may use very frequently in a query's WHERE clause as filter conditions.

Should there be only one column used, a single-column index should be the choice. Should there be two or more columns that are frequently used in the WHERE clause as filters, the composite index would be the best choice.

Implicit Indexes:

Implicit indexes are indexes that are automatically created by the database server when an object is created. Indexes are automatically created for primary key constraints and unique constraints.

The DROP INDEX Command:

An index can be dropped using SQL DROP command. Care should be taken when dropping an index because performance may be slowed or improved.

The basic syntax is as follows:

DROP INDEX index_name;

You can check INDEX Constraint chapter to see actual examples on Indexes.

When should indexes be avoided?

Although indexes are intended to enhance a database's performance, there are times when they should be avoided. The following guidelines indicate when the use of an index should be reconsidered:

• Indexes should not be used on small tables.
• Tables that have frequent, large batch update or insert operations.
• Indexes should not be used on columns that contain a high number of NULL values.
• Columns that are frequently manipulated should not be indexed.

---

How are Indexes used?

Indexes have three main uses:

• To quickly find specific rows by avoiding a Full Table Scan

We've already seen above how a Unique Scan works. Using the phone book metaphor, it's not hard to understand how a Range Scan works in much the same way to find all people named "Gallileo", or all of the names alphabetically between "Smith" and "Smythe". Range Scans can occur when we use >, <, LIKE, or BETWEEN in a WHERE clause. A range scan will find the first row in the range using the same technique as the Unique Scan, but will then keep reading the index up to the end of the range. It is OK if the range covers many blocks.

• To avoid a table access altogether

If all we wanted to do when looking up Gallileo in the phone book was to find his address or phone number, the job would be done. However if we wanted to know his date of birth, we'd have to phone and ask. This takes time. If it was something that we needed all the time, like an email address, we could save time by adding it to the phone book.

Oracle does the same thing. If the information is in the index, then it doesn't bother to read the table. It is a reasonably common technique to add columns to an index, not because they will be used as part of the index scan, but because they save a table access. In fact, Oracle may even perform a Fast Full Scan of an index that it cannot use in a Range or Unique scan just to avoid a table access.

• To avoid a sort

This one is not so well known, largely because it is so poorly documented (and in many cases, unpredicatably implemented by the Optimizer as well). Oracle performs a sort for many reasons: ORDER BY, GROUP BY, DISTINCT, Set operations (eg. UNION), Sort-Merge Joins, uncorrelated IN-subqueries, Analytic Functions). If a sort operation requires rows in the same order as the index, then Oracle may read the table rows via the index. A sort operation is not necessary since the rows are returned in sorted order.

Partitioned Tables And Indexes

Maintenance of large tables and indexes can become very time and resource consuming. At the same time, data access performance can reduce drastically for these objects. Partitioning of tables and indexes can benefit the performance and maintenance in several ways.

• Partition independance means backup and recovery operations can be performed on individual partitions, whilst leaving the other partitons available.
• Query performance can be improved as access can be limited to relevant partitons only.
• There is a greater ability for parallelism with more partitions.

All the examples shown here use the users tablespace for all partitions. In a real situation it is likely that these partitions would be assigned to different tablespaces to reduce device contention.

• Range Partitioning Tables
• Hash Partitioning Tables
• Composite Partitioning Tables
• Partitioning Indexes
• Local Prefixed Indexes
• Local Non-Prefixed Indexes
• Global Prefixed Indexes
• Global Non-Prefixed Indexes
• Partitioning Existing Tables

Related articles.

• Partitioning Enhancements In Oracle9i
• Hash Partitioned Global Indexes in Oracle 10g
• Partitioning Enhancements in Oracle Database 11g Release 1
• Partitioning an Existing Table using DBMS_REDEFINITION
• Partitioning an Existing Table using EXCHANGE PARTITION

Source: Wikipedia.

Top-down versus bottom-up design methodologies

Big Data Technology Evaluation Checklist Neeraj nathani

Big Data Technology Evaluation Checklist

Anyone who’s been following the rapid-fire technology developments in the world that is becoming known as “big data” sees a new capability, product, or company founded literally every week. The ambition of all of these players, established and newcomer, is tremendous, because the potential value to business is enormous. Each new arrival is aimed at addressing the pain that enterprises are experiencing around unrelenting growth in the velocity, volume, and variety of the data their operations generate.

What’s being lost, however, in some of this frothy marketing activity, is that it’s still early for big data technologies. There are vexing problems slowing the growth and the practical implementation of big data technologies. For the technologies to succeed at scale, there are several fundamental capabilities they should contain, including stream processing, parallelization, indexing, data evaluation environments and visualization.

When evaluating big data technology, it can be valuable to ask companies about their ability to deliver some of these fundamental capabilities. If you get an unsophisticated answer, you may find that the company is not as serious or capable as you might have expected. (For my research on this topic please see: Designing a Scalable and Agile Big Data Platform.)

In this article, we examine some of the big data requirements that are partially defined or in early stages of maturity. Any big data vendor worth considering should be able to address these requirements now or in the near future, or confidently explain their position. We sat down with Mike Driscoll, CTO of Metamarkets, a big data company that delivers predictive analytics solutions for digital media, to develop a checklist for evaluating new solutions and their fit criteria against the challenges of big data:

Some general questions to begin the evaluation process:

• Does the solution allow for stream processing, and incremental calculation of statistics?
• Does the solution parallelize processing and take advantage of distributed computing?
• Does the solution perform summary indexing to accelerate queries of huge datasets?
• What are the solution’s data exploration and evaluation environments that enable a quick understanding of the value of new datasets?
• How does a solution directly provide or easily integrate with visualization tools?
• What is the strategy for verticalization of the technology?
• What is the ecosystem strategy? How does the solution provider fill the gaps in its capabilities through partnerships?

Getting these questions answered will put most vendors to the test and help improve your understanding of the technology you are evaluating.

Stream processing

As the pace of business has increased, and the number of instrumented business processes has expanded, increasingly our attention is focused not on “data sets,” but on “data streams.”

“Decision-makers are interested putting their finger on the pulse of their organization, but to get answers in real time; they require architectures that can process streams of data as they happen,” Driscoll says. “Current database technologies are not well suited to do this kind of stream processing.”

For example, calculating an average over a group of data can be done in a traditional batch process, but far more efficient algorithms exist for calculating a moving average of data as it arrives, incrementally, unit by unit. If you want to take a repository of data and perform almost any statistical analysis, that can be accomplished with open source products like R or commercial products like SAS. But if you want to create a set of streaming statistics, to which you incrementally add or remove a chunk of data as a moving average, the libraries either don’t exist or are immature.

“The entire ecosystem around streaming data is underdeveloped,” says Driscoll.

In other words, if you’re talking to a vendor about a big data project, you have to determine whether this kind of stream processing is important to your project, and if it is, whether they have a capability to provide it. This axiom extends all the way down, to not just the analytical algorithms that run over the streams, but also to the way in which those streams are queued, ingested, managed and ultimately processed Many architectures exist for ingesting and queuing data streams, some of which are proprietary. TIBCO, Esper and ZeroMQ all offer solutions. But those solutions are only about moving packets of data around. Actually doing analysis of the streams requires practitioners to build on a lower level, for which there is another rapidly evolving toolset.
Parallelization
There are many definitions of big data. Here’s a useful one: “Small data” is data that fits in-memory on a single desktop or machine with a capacity of between 1 GB and 10 GB of disk space. “Medium data” fits on a single hard drive of 100 GB to 1 TB in size. “Large data” is distributed over many machines, comprising 1 TB to multiple petabytes.
“If you want to work with distributed data, and you expect to have any hope of processing that data in a reasonable amount of time, that requires distributed processing,” Driscoll says.
Parallel processing comes to the fore in distributed data. Hadoop is one of the better-known examples of distributed or parallelized processing. Hadoop can do more than distributed processing. It can also conduct distributed queries, which have been a subject of interest recently for designers of massively parallel processing (MPP) databases, whose object is to take a query and parallelize it across a set of nodes. Each of those nodes does partial work for the query in parallel, and then combines those partial answers into a single unified answer, Driscoll explains. Parallelizing queries is not a simple affair. When something is being analyzed in a parallel stream, each new unit of data must be combined with an existing unit of data in order to produce an answer.
Therefore, if a vendor is trying to sell you a solution for addressing big data at scale, their salespeople should be able to articulate their special secret sauce and strategy for parallelization.
“One of the most important features that current data warehouse vendors must offer is the ability to do parallel copy from Hadoop into their warehouse,” Driscoll says. “So, whether it’s EMC Greenplum offering the ability to do distributed parallel copy from Hadoop to Greenplum, Netezza or Oracle, the ability to parallelize data transfer is a critical feature.”
Hadoop has a massively distributed file system and can support distributed queries on top of it. But it does not inherently support parallelization—running a parallel process on Hadoop without an algorithm for optimizing queries can significantly slow down the process, taking minutes to return an answer. This is acceptable for some queries, but it won’t support real-time analytics in a big data world. The power and speed of that algorithm will be a determining factor of the robustness and cost of the solution, and that should be appropriately scaled to your needs, says Driscoll.
Summary indexing
Summary indexing is the process of creating a pre-calculated summary of data to speed up running queries. The problem with summary indexing is that it requires you to plan in advance what kind of queries you are going to run, so it is limiting.
The most common form of summary indexing is the star schema used to support speedy searches in data warehouses. A star schema prioritizes one master dimension (such as location or product) in advance of running a multi-dimensional data cube, organizing subordinate dimensions in relation to the master. The technique works well, but has one huge problem. If you want to ask a new question, it takes a long time to reconfigure the schema and associated data cubes and recompute them. When we begin to ask questions of all our data, not just the structured data, it is practically impossible to create a star schema to answer every possible question. The problem is not pre-processing; it’s the difficulty of reconfiguring the pre-processing as needed to ask new questions and get a speedy response.
The ideal solution would easily adjust the summaries being created as new questions arose. With a quickly created, summarized form of the data, it would then be possible to use data-analysis tools such as QlikView, Tableau, or TIBCO Spotfire for exploration and analysis. But there is currently a gap in available tools to make this summary creation easier, as many available tools don’t reach down to machine-level data, says Driscoll. The result is that the IT department becomes involved in building a custom query.

Some help is on the way for this problem. Vendors such as Splunk have emerged with a solution based on their search language that makes creating summary indexes far faster than other approaches, like star schemas. The designers of technology like SAP HANA, 1010 Data, and Metamarkets recommend an in-memory approach that completely abandons summarizing, by keeping vast amounts of data in in-memory systems.

But data volumes are growing fast and the need for summarizing will never go away completely. For the short and medium term, vendors must have a strategy for agile creation of summary indexes.
Data evaluation environments
How does your vendor’s solution understand, or allow you to understand the meaning of new datasets and incorporate them into your analysis?

For example, if your business has a retail store, and you are studying transactions, you could pick up the movements of people around the web site, or even around brick-and-mortar stores through opt-in GPS signals and cell-phone tracking. Once you acquire the data:

• How do you incorporate and understand what that data can tell you?
• How do you develop a model of the store that will help you analyze customer behavior?
• How do you understand when those movements become events?
• How fast can you figure that out—before the customer leaves the store?

To answer these questions, your solution needs to be able to join disparate datasets. Very few vendors have a distinguished capability to join datasets. And, just as the most critical areas of a building are at the joints, the same is true of data architecture. Where data interfaces, tremendous value can be unlocked, Driscoll says.

“The holy grail for anyone in the online retail space is to understand the connection between online impression events that lead to actions, such as clicks, or some level of engagement that eventually lead to purchasing behaviors, which eventually lead to long-term customer adoption,” Driscoll says. “Right now, all of these datasets live in different places. American Express knows where you’ve purchased your Starbucks coffee and Foursquare knows where you checked in at Starbucks, and Yahoo! knows when you clicked on an ad discount for a Starbucks latte on a hot summer day. And yet, people are struggling to thread these various data streams together.”

As enterprises begin to draw in disparate data feeds—particularly mobile data feeds—it’s critical that a vendor has a solution for rapidly joining disparate datasets, because the information they contain is critical for enterprises.

Visualization

There are two broad categories of visualization tools, according to Driscoll.

Exploratory visualization describes tools that allow a decision-maker and an analyst to explore different axes of the data for relationships, which usually involves some kind of visual “mining for insights.” Tools such as Tableau and TIBCO Spotfire, and to a lesser extent QlikView, fit into this category, Driscoll says.

Narrative visualizations are designed to examine a particular axis of the data in a particular way. For instance, if you say want to look at a time series visualization of sales broken up by geography for an enterprise, a format for that visualization can be pre-created. The data can be played back month-by-month for every geography, and is sorted into a pre-cast formula. Vendors such as Perceptive Pixel fit into this category.

In narrative visualizations, “Certain knobs are free to explore the data, but it’s not completely open to ask any question,” Driscoll says. “These visualizations are designed to tell a certain story about the data, just as certain pre-computations or reports are designed to tell a certain story. And some tools are better for the first, ad-hoc exploratory model, and others are better for the second, constrained narrative.”

Mind the Verticals

There are nearly as many types of decision-making needs in different verticals as there are ways to collect, process and analyze data. Vendors should be wary that different decisions makers within an organization and between verticals have different kinds of visualizations that they are accustomed to seeing.

“Any vendor that wants to serve the needs of those decisions-makers ought to be well aware of what those expected narratives are, because that will speed the adoption of that visualization tool,” Driscoll says, citing the preference for candlestick plots in the financial services industry.
Ecosystem Strategy
The largest most successful companies all spend tens of millions creating ecosystems around their products. The ecosystems are supported by product features and business models that allow the product to do its job but also work with other technologies or partners who extend the product or craft it to special uses. If a product doesn’t have an ecosystem strategy, you may find that it is hard to adapt to your needs and that finding expertise to help with implementation and configuration may be hard to come by.
This list of requirements for big data technology is not exhaustive, but it is a good start. Using these topics when evaluating big data technology will only lead to deeper understanding.

Source: Wikipedia.