Neeraj Nathani

Monday, 11 February 2008

Picking Winners In Big Data Neeraj Nathani Smart bridge

Picking Winners In Big Data

Big data solutions are picking up speed in the IT industry. There’s a Cambrian explosion of interesting start-ups, and all the database and business intelligence incumbents have moved to create big data offerings, or rebrand into the new universe.

The key to seeing the value of big data is understanding that it’s a business problem, not a matter of picking the right tools and waiting for the magic to happen. That said, technology choices still need to be made in an increasingly crowded and confused market.

The biggest question for anybody wanting to invest or adopt technology in this area is how to pick the winner? Glancing through marketing materials will do little to help you: everybody claims relevance to big data.

As I am often asked my opinion about big data companies, I thought I’d share some of the principles I use to help me think about the industry.

Where’s the value?

We need to understand where the actual value is in the data world. For the most part, this value doesn’t lie purely in the software. Over the past decade we have seen a rising tide of commoditization of the software stack: from operating system, to relational databases, to Hadoop itself. In fact, without this, we wouldn’t have the big data revolution as we know it.

As Hadoop has become a de facto standard, so has the notion of building on top of it with open source. There is some advantage in software innovation, but it is momentary. Once something is known to be possible, there are enough smart programmers out there that reproducing it becomes straightforward. (Because of this, I gloomily predict no shortage of patent battles in the not-too-distant future.)

Despite the flux in the software world, two things about big data remain constant: the need for compute and storage, and data itself. It’s ultimately to ownership of one or both of these factors that IT industry value will gravitate.

Compute and storage

The ever growing need for computing power and storage bodes well for companies providing the basics. These fall into two categories: hardware manufacturers, and cloud infrastructure providers. Not that these two markets are immune to their own fluctuations, thanks to standardization and commoditization, but fundamentally, getting paid for use of metal is the name of the game.

In this respect, it’s not too much of a mystery why storage company EMC has plowed so early and so deep into the big data world. Neither is it hard to see the reasoning behind Intel creating its own optimized Hadoop distribution.

Data

Value resting in data is the more subtle of the two axes of big data success.

Big data is ultimately about the smart use of data to drive a business. There are two kinds of data: data about your business, and data external to your business that you can create value from.

It’s easy to see who might get success from the latter, external data. We can expect that massive data owners such as Google, Facebook, Thomson Reuters, Bloomberg will experience ongoing success for as long as they are able to create product from their data.

Who owns your data, though? The obvious answer, you, isn’t the only answer. In fact, your data is locked up inside the platform choices you make, at both the hardware and software level. If your systems are based on Oracle, Microsoft, you are very unlikely to move in a hurry. Data likes to stay where it is, and tends to attract more data as you build systems around it. Production systems are expensive to replace.

So, the vendors of your software platforms of choice also get long term value from your data. For this reason, it’s hard to bet against existing enterprise application platform incumbents in the big data world. Big data, for most of today’s organizations, is an additive phenomenon, not a challenge to the core of IT.

We’ll see more large platforms ensuring nobody needs to move away. Examples include SAP adding HANA, in order to enable their existing customers for big data, or Amazon Web Services’ addition of their data warehouse Redshift, to ensure that the entire data needs of a company can be met on their platform.

Finally, it would be lunacy to bet against either Oracle, who have been portentously quiet in the big data world over the last year, or Microsoft, who in Excel have the world’s most popular data manipulation environment.

It is all business as usual?

I’m not saying there is no new opportunity in the big data industry. What I am saying is that, as an additive technology, big data is unlikely to enable anybody to challenge Oracle or Microsoft for the throne.

There will be change, though, and it will bring both winners and losers.

The area of value we haven’t yet looked at yet is the point where data interacts with the actual mechanism of a business. That’s where it transfers value to you and enables you to leverage data to get ahead. This breaks down into several areas of opportunity for big data innovators.

Domain specific: tools that enable the manipulation and exploitation of data in a way that’s specific to a business segment. We see initial evidence of this market opportunity in the evolution of web and customer analytics products.
Machine learning: more data means more to understand, and the only way an organization can realistically do this is with the aid of computers. Machine learning helps automate many parts of the data wrangling process. Furthermore, cross-company data sharing can significantly boost the effectiveness of machine-learning, creating the opportunity for companies gaining early market share. A recent example of this is Sift Science, a fraud detection application.
Tools for exploration: human interaction with data is a requirement that’s hard to abstract away. Tableau has a head start in this market for big data, filling the role of “Microsoft Office for Data”, but the field is ripe for new innovation, especially with the increasing power of graphical capabilities and new device formats.
Data agility: speed-to-decision is a critical factor in business competitiveness. Any solution that removes laborious steps has an advantage: a particularly problematic area here is data integration, the loading of both internal and external data sources ready for analytics. Most of today’s big data solutions are frankensteined combinations of layers: there’s a great opportunity for vertically integrated solutions that removes needless impedance to data manipulation.

Areas of risk

What are those riskier big data options? If a solution isn’t scoring high in the categories above, it’s not likely to be around for the long run.

The biggest risk is with solutions that address only a single horizontal part of the data architecture. Time is against companies in this game. By selling just part of a complete solution, they’re working against the rising tide of commoditization. Customers will demand standardization (e.g. Hadoop compatibility) in order to feel safe adopting such solutions, but that prevents the lock-in that will protect that business. It’s not Oracle’s relational database that cements their position: it’s their vertical position up and down the application stack.

Therefore, it’s not surprising to see the pure Hadoop distribution companies making partnerships, and branching out into other vertical layers. In the long term, it’s a tough road they’ve chosen.

One company working actively to solve this problem is DataStax, who have pivoted from being seen as the corporate backer for the Cassandra NoSQL database—a risky horizontal play—to selling an integrated stack of Cassandra, Hadoop and Solr, intended as a complete platform for building enterprise applications. They’re going after some of the platform business: being involved in the actual use of data to drive business value.

For some start-ups, not having long term big data viability might be just fine as a strategy. As the bigger enterprise companies lumber into the arena, they’ll prove handy acquisitions. But given the crowded space and progressive commoditization, this isn’t a certain future. And certainly for their customers, the risks are growing.

More controversially, another area at risk is that of traditional ETL. Or in its broader sense, data integration. This is a hard, hard, problem. It’s not easy to integrate data retrospectively, and it’s not all certain that the integration players from the data warehouse world will be able to translate that success into the big data world. Many early adopters of Hadoop were motivated by the fact that existing ETL solutions couldn’t meet their needs.

In the long term, data integration is likely to be best served by entire new architectures that don’t try to separate and tame the data in the first place. It’s a lot easier to build truly integrated data infrastructure as greenfield.

For those who crack the integration problem, the potential rewards are high. But so is the risk.

Conclusion

In the long term, the additive nature of big data, combined with inertia, makes it a safe bet that current enterprise IT incumbents will continue their reign, as long as they move to embrace big data in their architectures.

There is plenty of opportunity though, especially in greenfield and cloud scenarios. Expect to see increasing returns for those who provide integrated solutions and do a good job of equipping human decision makers.

There’s long term value in metal, and value in data. About everything else, it’s worth thinking carefully.

Source: Wikipedia./Forbes

Sunday, 20 January 2008

Smart Bridge Trading Solutions Neeraj Nathani Picking Winners In Big Data

Picking Winners In Big Data

As I am often asked my opinion about big data companies, I thought I’d share some of the principles I use to help me think about the industry.

Where’s the value?

Compute and storage

Data

Value resting in data is the more subtle of the two axes of big data success.

Big data is ultimately about the smart use of data to drive a business. There are two kinds of data: data about your business, and data external to your business that you can create value from.

It is all business as usual?

There will be change, though, and it will bring both winners and losers.

Domain specific: tools that enable the manipulation and exploitation of data in a way that’s specific to a business segment. We see initial evidence of this market opportunity in the evolution of web and customer analytics products.
Machine learning: more data means more to understand, and the only way an organization can realistically do this is with the aid of computers. Machine learning helps automate many parts of the data wrangling process. Furthermore, cross-company data sharing can significantly boost the effectiveness of machine-learning, creating the opportunity for companies gaining early market share. A recent example of this is Sift Science, a fraud detection application.
Tools for exploration: human interaction with data is a requirement that’s hard to abstract away. Tableau has a head start in this market for big data, filling the role of “Microsoft Office for Data”, but the field is ripe for new innovation, especially with the increasing power of graphical capabilities and new device formats.
Data agility: speed-to-decision is a critical factor in business competitiveness. Any solution that removes laborious steps has an advantage: a particularly problematic area here is data integration, the loading of both internal and external data sources ready for analytics. Most of today’s big data solutions are frankensteined combinations of layers: there’s a great opportunity for vertically integrated solutions that removes needless impedance to data manipulation.

Areas of risk

What are those riskier big data options? If a solution isn’t scoring high in the categories above, it’s not likely to be around for the long run.

For those who crack the integration problem, the potential rewards are high. But so is the risk.

Conclusion

There’s long term value in metal, and value in data. About everything else, it’s worth thinking carefully.

Source: Wikipedia./Forbes

Wednesday, 16 January 2008

Smart Bridge Trading Solutions Neeraj Nathani Revenue Assurance

Revenue Assurance

Revenue assurance (RA) is a business activity most commonly undertaken within businesses that provide telecommunication services. The activity is the use of data quality and process improvement methods that improve profits, revenues and cash flows without influencing demand. This was defined by a TM Forum working group based on research documented in its Revenue Assurance Technical Overview [1]. In many telecommunications service providers, revenue assurance is led by a dedicated revenue assurance department.

Overview

"Revenue assurance" is a broad umbrella term. It is used both to describe an activity performed within telecommunications service providers, and is a common name for a small business unit associated with that activity. Revenue assurance is a practical response to perceived or actual issues with operational underperformance, most commonly relating to billing and collection of revenue. Some of the procedures associated with identifying, remedying or preventing errors may be undertaken by a dedicated Revenue Assurance department, though responsibility for revenue assurance is often diffuse and varies greatly with the organizational structure of the provider. Assuming a provider with a typical organizational split, responsibilities for revenue assurance primarily sit between the Finance and Technology directorates, however, revenue assurance initiatives are often started in a business unit or marketing group.

The relevance to Finance rests with the responsibility for financial control, audit and reporting, whilst the subject matter would be network and IS systems as implemented or operated by the Technology side of the business. Marketing groups and / or business units (e.g. wholesale or retail business lines) will often embark on revenue assurance projects in an effort to improve product line margins. Furthermore, marketing and business units are pivotal in providing input into the "should be" state of customer bills and products.

The sphere of influence described by revenue assurance varies greatly between telecommunications service providers, but is usually closely related to back office functions where small errors may have a disproportionately large impact on revenues or costs. The processing of transaction data in modern telecommunications providers exhibits many attributes akin to a complex system. However, there is significant disagreement about the ultimate aims and legitimate scope of revenue assurance teams. This is in part caused by:

(1) the cross-functional nature of the activity and the consequent need for a variety of skills from IT, marketing, finance, et al.;

(2) the difficulty of generalizing across businesses with different objectives and business models;

(3) political infighting within each telco about responsibility for revenue leakages and assurance; and

(4) the difficulty in reliably measuring the value added by revenue assurance as separable from underlying performance.

There is high-level agreement between practitioners about the goals and methods of revenue assurance, though reaching a consensus on defining the boundaries of revenue assurance has proved elusive so far. The goals relate to improving the financial performance by eliminating mistakes in the processing of transaction data. Some take a more encompassing view of what counts as a mistake, which may extend as far as questioning the policy set by executives even when this has been executed correctly. Others take a more open-ended view of the data that is the subject matter. For example, in decreasing order of frequency, revenue assurance may cover:

(1) revenues from retail and corporate sales;

(2) revenues and costs from interconnect and wholesale contracts; and

(3) margins and profitability of investment in networks and information systems.

Other markets have different or more refined priorities. For example, in the U.S.A. management of wholesale contracts has often been the first objective because of the complexity of the domestic market resulting from the Federal Communications Commission's regulatory framework. In contrast, telcos in developing countries may prioritize management of international interconnect arrangements because of the risks posed by fraud and arbitrage. A cable supplier or internet service provider that predominantly offers retail customers flat monthly charges and no limits on usage may be most interested in assuring the profitability of network investments.

Revenue assurance is often regarded by practitioners as a low-cost mechanism to generate significant financial returns for telecommunications service providers. However, the returns are unpredictable as well as being hard to measure, which encourages many executives to take a sceptical view of its worth. Comparable revenue assurance activities do occur in other industries, such as with billing of utilities or with the licensing of software, and there are many parallels with financial and operational control activities undertaken by most large businesses. The rationale for why revenue assurance has come to be considered particularly important in telecommunications, unlike other industries, is disputed. Reasonable conjectures are that:

(1) the fast pace of change and intense commercial competition increase the likelihood of mistakes;

(2) there is significant complexity in determining the combined effect of interacting systems and processes; and

(3) the high-volume, low-value nature of transactions amplifies the financial implications of "small" errors.

Another conjecture is that revenue assurance is a response to changing market conditions. The thinking is that as markets reach saturation and growth potential falls off, so the value of maximizing returns from existing sales increases. This observation has some merit but does not explain the increasing popularity of revenue assurance in telcos serving growth markets. It also in part contradicts the assumption of a compelling costs versus benefits argument for revenue assurance, which would be enhanced in businesses undergoing rapid change. It is also important to recognize that there is a long history of revenue assurance activities in some telcos that predates the coining of the term "revenue assurance".

The revenue assurance techniques applied in practice cover a broad spectrum, from analysis and implementation of business controls to automated data interrogation. At one end of the spectrum, revenue assurance can appear very similar to the kinds of review and process mapping techniques applied for other financial controlling objectives like accounting integrity, as exemplified by those derived from clause 404 of the Sarbanes-Oxley Act. This form of revenue assurance is most commonly promoted by consultancies. The size of such consultancies covers the entire range; the Big 4 all offer some form of revenue assurance consulting, but there are also niche specialist consultancies. At the other end of the spectrum, revenue assurance is treated as a form of reactive automated data interrogation, seeking to find anomalies in transaction data that may indicate errors and potential revenue loss. This form of revenue assurance is most commonly promoted by software houses that aim to provide databases and configurable tools to extract and interrogate a telco's source data. A less popular form of reactive automated assurance involves using both software and specialised hardware as a means of extracting additional data on transactions, for example by creating actual network events or interfacing directly with network elements to replicate dummy events. As with consultancies, IT-oriented revenue assurance solutions are offered by both large vendors like providers of billing and mediation software, and by specialized niche providers.

There is some debate about the relative merits of the different techniques that can be employed in revenue assurance.

As yet, there is no professional body, no qualifications, and no academic research that would help to drive consensus about the purpose or methods of revenue assurance. In part this is addressed by individuals working in the sector through membership and qualification in related fields such as accountancy and information systems audit. Some scientific research in other fields is also applicable to revenue assurance, though most revenue assurance "facts" rely heavily on anecdotes and oft-repeated truisms. Some of the most helpful and progressive initiatives in addressing the problem of consensus and scientific basis are listed below.

The value of revenue assurance

Revenue assurance is usually understood as a means to identify and remedy, and perhaps also to prevent, problems that result in financial under performance without seeking to generate additional sales. The most common metaphor is that of leaking water from a pipe, where water stands in place of revenues or cash flows, and the leaks represent waste. The value of revenue assurance is hence determined by the size of the leaks "plugged", and possibly also those leaks prevented before they occur, although estimating the value of the latter is very problematic. The value added also includes the recovery of "lost" revenues or costs (through issuing additional bills, chasing uncollected payments, renegotiating with suppliers a refund of costs etc.) after the fact. This last form of reactive revenue assurance is the easiest to put a value to, but is in many ways the least efficient form of revenue assurance; effort is directed towards repeatedly addressing the consequences of known flaws, and not on addressing the flaws themselves. This can lead to a parasitical relationship between a Revenue Assurance department or vendor and the wider business, where the department/vendor finds it easiest to justify its ongoing existence/contract by repeatedly fixing symptoms and not the root causes.

The TM Forum conducted a benchmark survey in 2008 that concluded average leakage, not including losses due to fraud, was 1% of the gross revenues for those telcos that took part.^[1] The number of participating telcos was relatively small compared to some other surveys, but the survey technique was more demanding than any comparable survey to date. The survey used the most detailed and prescriptive definition of how to calculate leakage of any survey of its type. The definition was taken from the TM Forum's own standard on how to calculate revenue assurance metrics [2]. To increase confidence that participants calculated their leakages correctly, the TM Forum's benchmark program independently reviewed the results and corroborated them with representatives of the participating companies. The survey's average of 1% leakage of gross revenue, whilst still significant, is notably lower than many other quoted estimates and reported survey findings about average leakage. This may be because the survey used a very strict definition of leakage. The survey measured only actual under-billed and unbilled amounts discovered by the participants; it excluded other types of leakages such as cost leakages and loss of opportunity leakages, and it excluded projected leakage estimations that are commonly used (i.e., what would have been the amount of leakage, if the leakage would not been discovered by revenue assurance activities). It may also reflect a reduction in bias or exaggeration in reported leakages, or at least the exclusion of guesswork. Respondents were given authoritative instructions on how to quantify leakage based on actual data and were instructed to avoid making suppositions in the absence of such data.

The best known estimates of "typical" revenue leakage come from a series of annual surveys conducted by the Analysys consultancy and research business. In these surveys, leakage was commonly estimated as being worth between 5% and 15% of the total revenue of the business. Similar research by other businesses has generated results in the same range, with none concluding leakage of less than 1% of gross revenue, and some suggesting leakage of 20% or more was not uncommon. Reasons to doubt these estimates are as follows:

(1) All the estimates of leakage were derived from the subjective opinions of staff working in service providers;

(2) All the research was conducted by businesses wishing to promote their revenue assurance products;

(3) Increased annual spend on revenue assurance has not resulted in a clear downward trend in estimates;

(4) The estimates were broadly similar even when the criteria for what losses to include varied greatly; and

(5) Genuine losses of this scale should be a severe corporate governance issue in any publicly listed business.

What can be said with some confidence is that revenue assurance practitioners are able to provide a vast number of consistent anecdotes relating to the causes of leakage and means to resolve them. Though there is little objective evidence relating to actual leakages approaching this scale in the public domain as this information per se is highly confidential, there are some indirect measures of data integrity that help give a sense of potential leakage. For example, in reconciling interconnect costs and revenues between telcos, a 5% variance is the common practice to accept before a disputed invoice can lead one party to withhold payment, and a 0.5% variance would be considered industry-leading practice according to best practice advice issued by the UK Revenue Assurance Group.

Revenue Assurance in Telecom

Although Revenue Assurance has always been present in the telecom parlance^[^{citation needed}^] it has recently been brought at the forefront of the top management^[^{citation needed}^]. This is due to several factors including

Profit: Increasing cost pressures and decreasing margins. The high profit days for most telcos are over^[^{citation needed}^]. They all need to find alternative means to squeeze higher margins by effectively tracking their revenue.
Regulatory: New regulatory structure and compliance requirements^[^{citation needed}^] which force the telecom operators to report their revenue accurately.
Technology Innovation: Ensuring new technologies and products are performing as per perceived plans. Keeping up with release of new technologies along with co-existing of legacy systems.
Mergers and Acquisitions: With increase in the number of telco mergers and acquisitions, organizations are finding it very difficult handle multiple BSS systems including Billing, Mediation and Rating together etc.

Revenue Assurance has been a problem for the telecom companies since the very early stages. Tracking of pulses, minutes, counts, bytes etc. has never been more difficult. One would think these would be easy for the tech-savvy telco companies. However, the truth has been just the opposite. In a hurry to release new technologies in the market, the Revenue Assurance systems are always lagging behind. Revenue Assurance in a telco environment covers a wide range of technical and business aspects. An RA operator needs to be aware of both OSS & BSS processes and internal dependencies to accurately decipher the revenue code.

What causes the problem

A telecom organization's revenue chain is usually a very complex set of inter-related technologies and processes providing a seamless set of services to the end consumer. As the set of technologies and business processes grows bigger and more complex, the chance of failure increases in each of its connections. A revenue leakage is typically attributed to when a telco organization is unable to bill correctly for a given service or to receive the correct payment. As the organization grows the probability of revenue leakage only increases.

Where is the problem

The most debated part of revenue assurance is where to start checking, i.e. at the network side, the rating side, the billing side, the interconnect side, the CRM side, etc. However, most surveys and reports state that the maximum leakage happens during the flow of Call Detail Records (CDRs) or Event Detail Records (EDRs) from the Switch to the respective rating / billing engines. Some of the common problem areas are :

Network

* Signaling problems

* CDRs in Switch not sent to Mediation

* CDRs in Mediation not send downstream

* CDRs rejected by rating / billing system

* Wrong duration on the CDRs

* Incorrect Business rules

* Subscriber provisioning

* Incorrect Routing

Rating & Billing

* Incorrect Rejection Logic

* Duplicate CDRs resulting in double charging

* Incorrect tariff plans

* Rating & Billing accuracy errors

* Late rating / billing

* Incorrect configurations – rating minutes instead of seconds

* Incorrect Disconnection

Revenue assurance discipline

Among the disciplines in revenue assurance are:

1. The CORE functions of a revenue assurance group: Monitoring, Baselining, Auditing, Synchronizing, Investigating and Compliance.
2. Decomposing an organization's revenue assurance scope (The Revenue Management Chain).
3. Assessing and minimizing revenue loss risk.

Source: Wikipedia.

Friday, 21 December 2007

Top-down versus bottom-up design methodologies Neeraj nathani-smartbridge.net

Top-down versus bottom-up design methodologies

Bottom-up design

Ralph Kimball, a well-known author on data warehousing,^[8] is a proponent of an approach to data warehouse design which he describes as bottom-up.^[9]
In the bottom-up approach, data marts are first created to provide reporting and analytical capabilities for specific business processes. It is important to note that in Kimball methodology, the bottom-up process is the result of an initial business-oriented top-down analysis of the relevant business processes to be modelled.
Data marts contain, primarily, dimensions and facts. Facts can contain either atomic data and, if necessary, summarized data. The single data mart often models a specific business area such as "Sales" or "Production." These data marts can eventually be integrated to create a comprehensive data warehouse. The integration of data marts is managed through the implementation of what Kimball calls "a data warehouse bus architecture".^[10] The data warehouse bus architecture is primarily an implementation of "the bus", a collection of conformed dimensions and conformed facts, which are dimensions that are shared (in a specific way) between facts in two or more data marts.
The integration of the data marts in the data warehouse is centered on the conformed dimensions (residing in "the bus") that define the possible integration "points" between data marts. The actual integration of two or more data marts is then done by a process known as "Drill across". A drill-across works by grouping (summarizing) the data along the keys of the (shared) conformed dimensions of each fact participating in the "drill across" followed by a join on the keys of these grouped (summarized) facts.
Maintaining tight management over the data warehouse bus architecture is fundamental to maintaining the integrity of the data warehouse. The most important management task is making sure dimensions among data marts are consistent. In Kimball's words, this means that the dimensions "conform".
Some consider it an advantage of the Kimball method, that the data warehouse ends up being "segmented" into a number of logically self-contained (up to and including The Bus) and consistent data marts, rather than a big and often complex centralized model. Business value can be returned as quickly as the first data marts can be created, and the method gives itself well to an exploratory and iterative approach to building data warehouses. For example, the data warehousing effort might start in the "Sales" department, by building a Sales-data mart. Upon completion of the Sales-data mart, the business might then decide to expand the warehousing activities into the, say, "Production department" resulting in a Production data mart. The requirement for the Sales data mart and the Production data mart to be integrable, is that they share the same "Bus", that will be, that the data warehousing team has made the effort to identify and implement the conformed dimensions in the bus, and that the individual data marts links that information from the bus. Note that this does not require 100% awareness from the onset of the data warehousing effort, no master plan is required upfront. The Sales-data mart is good as it is (assuming that the bus is complete) and the Production-data mart can be constructed virtually independent of the Sales-data mart (but not independent of the Bus).
If integration via the bus is achieved, the data warehouse, through its two data marts, will not only be able to deliver the specific information that the individual data marts are designed to do, in this example either "Sales" or "Production" information, but can deliver integrated Sales-Production information, which, often, is of critical business value.

Top-down design

Bill Inmon, one of the first authors on the subject of data warehousing, has defined a data warehouse as a centralized repository for the entire enterprise.^[10] Inmon is one of the leading proponents of the top-down approach to data warehouse design, in which the data warehouse is designed using a normalized enterprise data model. "Atomic" data, that is, data at the lowest level of detail, are stored in the data warehouse. Dimensional data marts containing data needed for specific business processes or specific departments are created from the data warehouse. In the Inmon vision, the data warehouse is at the center of the "Corporate Information Factory" (CIF), which provides a logical framework for delivering business intelligence (BI) and business management capabilities.
Inmon states that the data warehouse is:

Subject-oriented

The data in the data warehouse is organized so that all the data elements relating to the same real-world event or object are linked together.

Non-volatile

Data in the data warehouse are never over-written or deleted — once committed, the data are static, read-only, and retained for future reporting.

Integrated

The data warehouse contains data from most or all of an organization's operational systems and these data are made consistent.

Time-variant

For An operational system, the stored data contains the current value. The data warehouse, however, contains the history of data values.

The top-down design methodology generates highly consistent dimensional views of data across data marts since all data marts are loaded from the centralized repository. Top-down design has also proven to be robust against business changes. Generating new dimensional data marts against the data stored in the data warehouse is a relatively simple task. The main disadvantage to the top-down methodology is that it represents a very large project with a very broad scope. The up-front cost for implementing a data warehouse using the top-down methodology is significant, and the duration of time from the start of project to the point that end users experience initial benefits can be substantial. In addition, the top-down methodology can be inflexible and unresponsive to changing departmental needs during the implementation phases.^[10]

Hybrid design

Data warehouse (DW) solutions often resemble the hub and spokes architecture. Legacy systems feeding the DW/BI solution often include customer relationship management (CRM) and enterprise resource planning solutions (ERP), generating large amounts of data. To consolidate these various data models, and facilitate the extract transform load (ETL) process, DW solutions often make use of an operational data store (ODS). The information from the ODS is then parsed into the actual DW. To reduce data redundancy, larger systems will often store the data in a normalized way. Data marts for specific reports can then be built on top of the DW solution.
It is important to note that the DW database in a hybrid solution is kept on third normal form to eliminate data redundancy. A normal relational database however, is not efficient for business intelligence reports where dimensional modelling is prevalent. Small data marts can shop for data from the consolidated warehouse and use the filtered, specific data for the fact tables and dimensions required. The DW effectively provides a single source of information from which the data marts can read, creating a highly flexible solution from a BI point of view. The hybrid architecture allows a DW to be replaced with a master data management solution where operational, not static information could reside.
The Data Vault Modeling components follow hub and spokes architecture. This modeling style is a hybrid design, consisting of the best practices from both 3rd normal form and star schema. The Data Vault model is not a true 3rd normal form, and breaks some of the rules that 3NF dictates be followed. It is however, a top-down architecture with a bottom up design. The Data Vault model is geared to be strictly a data warehouse. It is not geared to be end-user accessible, which when built, still requires the use of a data mart or star schema based release area for business purposes.

Data warehouses versus operational systems

Operational systems are optimized for preservation of data integrity and speed of recording of business transactions through use of database normalization and an entity-relationship model. Operational system designers generally follow the Codd rules of database normalization in order to ensure data integrity. Codd defined five increasingly stringent rules of normalization. Fully normalized database designs (that is, those satisfying all five Codd rules) often result in information from a business transaction being stored in dozens to hundreds of tables. Relational databases are efficient at managing the relationships between these tables. The databases have very fast insert/update performance because only a small amount of data in those tables is affected each time a transaction is processed. Finally, in order to improve performance, older data are usually periodically purged from operational systems.

Confirmed Dimensions, Junk Dimensions, and Degenerated Dimensions

Conformed Dimensions (CD): these dimensions are something that is built once in your model and can be reused multiple times with different fact tables. For example, consider a model containing multiple fact tables, representing different data marts. Now look for a dimension that is common to these facts tables. In this example let’s consider that the product dimension is common and hence can be reused by creating short cuts and joining the different fact tables.Some of the examples are time dimension, customer dimensions, product dimension. §

Junked Dimensions (JD): When you consolidate lots of small dimensions and instead of having 100s of small dimensions, that will have few records in them, cluttering your database with these mini ‘identifier’ tables, all records from all these small dimension tables are loaded into ONE dimension table and we call this dimension table Junk dimension table. (Since we are storing all the junk in this one table) For example: a company might have handful of manufacture plants, handful of order types, and so on, so forth, and we can consolidate them in one dimension table called junked dimension table.

Degenerated Dimension (DD): An item that is in the fact table but is stripped off of its description, because the description belongs in dimension table, is referred to as Degenerated Dimension. Since it looks like dimension, but is really in fact table and has been degenerated of its description, hence is called degenerated dimension. Now coming to the slowly changing dimensions (SCD) and Slowly Growing Dimensions (SGD): I would like to classify them to be more of an attributes of dimensions its self.

Although other might disagree to this view but Slowly Changing Dimensions are basically those dimensions whose key value will remain static but description might change over the period of time. For example, the product id in a companies, product line might remain the same, but the description might change from time to time, hence, product dimension is called slowly changing dimension.

Lets consider a customer dimension, which will have a unique customer id but the customer name (company name) might change periodically due to buy out / acquisitions, Hence, slowly changing dimension, as customer number is static but customer name is changing, However, on the other hand the company will add more customers to its existing list of customers and it is highly unlikely that the company will acquire astronomical number of customer over night (wouldn’t the company CEO love that) hence, the customer dimension is both a Slowly changing as well as slowly growing dimension

Source: Wikipedia.