Smart Bridge Trading Solutions Neeraj Nathani Revenue Assurance

 Revenue Assurance

Revenue assurance (RA) is a business activity most commonly undertaken within businesses that provide telecommunication services. The activity is the use of data quality and process improvement methods that improve profits, revenues and cash flows without influencing demand. This was defined by a TM Forum working group based on research documented in its Revenue Assurance Technical Overview [1]. In many telecommunications service providers, revenue assurance is led by a dedicated revenue assurance department.

Overview

"Revenue assurance" is a broad umbrella term. It is used both to describe an activity performed within telecommunications service providers, and is a common name for a small business unit associated with that activity. Revenue assurance is a practical response to perceived or actual issues with operational underperformance, most commonly relating to billing and collection of revenue. Some of the procedures associated with identifying, remedying or preventing errors may be undertaken by a dedicated Revenue Assurance department, though responsibility for revenue assurance is often diffuse and varies greatly with the organizational structure of the provider. Assuming a provider with a typical organizational split, responsibilities for revenue assurance primarily sit between the Finance and Technology directorates, however, revenue assurance initiatives are often started in a business unit or marketing group.

The relevance to Finance rests with the responsibility for financial control, audit and reporting, whilst the subject matter would be network and IS systems as implemented or operated by the Technology side of the business. Marketing groups and / or business units (e.g. wholesale or retail business lines) will often embark on revenue assurance projects in an effort to improve product line margins. Furthermore, marketing and business units are pivotal in providing input into the "should be" state of customer bills and products.

The sphere of influence described by revenue assurance varies greatly between telecommunications service providers, but is usually closely related to back office functions where small errors may have a disproportionately large impact on revenues or costs. The processing of transaction data in modern telecommunications providers exhibits many attributes akin to a complex system. However, there is significant disagreement about the ultimate aims and legitimate scope of revenue assurance teams. This is in part caused by:

(1) the cross-functional nature of the activity and the consequent need for a variety of skills from IT, marketing, finance, et al.;

(2) the difficulty of generalizing across businesses with different objectives and business models;

(3) political infighting within each telco about responsibility for revenue leakages and assurance; and

(4) the difficulty in reliably measuring the value added by revenue assurance as separable from underlying performance.

There is high-level agreement between practitioners about the goals and methods of revenue assurance, though reaching a consensus on defining the boundaries of revenue assurance has proved elusive so far. The goals relate to improving the financial performance by eliminating mistakes in the processing of transaction data. Some take a more encompassing view of what counts as a mistake, which may extend as far as questioning the policy set by executives even when this has been executed correctly. Others take a more open-ended view of the data that is the subject matter. For example, in decreasing order of frequency, revenue assurance may cover:

(1) revenues from retail and corporate sales;

(2) revenues and costs from interconnect and wholesale contracts; and

(3) margins and profitability of investment in networks and information systems.

Other markets have different or more refined priorities. For example, in the U.S.A. management of wholesale contracts has often been the first objective because of the complexity of the domestic market resulting from the Federal Communications Commission's regulatory framework. In contrast, telcos in developing countries may prioritize management of international interconnect arrangements because of the risks posed by fraud and arbitrage. A cable supplier or internet service provider that predominantly offers retail customers flat monthly charges and no limits on usage may be most interested in assuring the profitability of network investments.

Revenue assurance is often regarded by practitioners as a low-cost mechanism to generate significant financial returns for telecommunications service providers. However, the returns are unpredictable as well as being hard to measure, which encourages many executives to take a sceptical view of its worth. Comparable revenue assurance activities do occur in other industries, such as with billing of utilities or with the licensing of software, and there are many parallels with financial and operational control activities undertaken by most large businesses. The rationale for why revenue assurance has come to be considered particularly important in telecommunications, unlike other industries, is disputed. Reasonable conjectures are that:

(1) the fast pace of change and intense commercial competition increase the likelihood of mistakes;

(2) there is significant complexity in determining the combined effect of interacting systems and processes; and

(3) the high-volume, low-value nature of transactions amplifies the financial implications of "small" errors.

Another conjecture is that revenue assurance is a response to changing market conditions. The thinking is that as markets reach saturation and growth potential falls off, so the value of maximizing returns from existing sales increases. This observation has some merit but does not explain the increasing popularity of revenue assurance in telcos serving growth markets. It also in part contradicts the assumption of a compelling costs versus benefits argument for revenue assurance, which would be enhanced in businesses undergoing rapid change. It is also important to recognize that there is a long history of revenue assurance activities in some telcos that predates the coining of the term "revenue assurance".

The revenue assurance techniques applied in practice cover a broad spectrum, from analysis and implementation of business controls to automated data interrogation. At one end of the spectrum, revenue assurance can appear very similar to the kinds of review and process mapping techniques applied for other financial controlling objectives like accounting integrity, as exemplified by those derived from clause 404 of the Sarbanes-Oxley Act. This form of revenue assurance is most commonly promoted by consultancies. The size of such consultancies covers the entire range; the Big 4 all offer some form of revenue assurance consulting, but there are also niche specialist consultancies. At the other end of the spectrum, revenue assurance is treated as a form of reactive automated data interrogation, seeking to find anomalies in transaction data that may indicate errors and potential revenue loss. This form of revenue assurance is most commonly promoted by software houses that aim to provide databases and configurable tools to extract and interrogate a telco's source data. A less popular form of reactive automated assurance involves using both software and specialised hardware as a means of extracting additional data on transactions, for example by creating actual network events or interfacing directly with network elements to replicate dummy events. As with consultancies, IT-oriented revenue assurance solutions are offered by both large vendors like providers of billing and mediation software, and by specialized niche providers.

There is some debate about the relative merits of the different techniques that can be employed in revenue assurance.

As yet, there is no professional body, no qualifications, and no academic research that would help to drive consensus about the purpose or methods of revenue assurance. In part this is addressed by individuals working in the sector through membership and qualification in related fields such as accountancy and information systems audit. Some scientific research in other fields is also applicable to revenue assurance, though most revenue assurance "facts" rely heavily on anecdotes and oft-repeated truisms. Some of the most helpful and progressive initiatives in addressing the problem of consensus and scientific basis are listed below.

The value of revenue assurance

Revenue assurance is usually understood as a means to identify and remedy, and perhaps also to prevent, problems that result in financial under performance without seeking to generate additional sales. The most common metaphor is that of leaking water from a pipe, where water stands in place of revenues or cash flows, and the leaks represent waste. The value of revenue assurance is hence determined by the size of the leaks "plugged", and possibly also those leaks prevented before they occur, although estimating the value of the latter is very problematic. The value added also includes the recovery of "lost" revenues or costs (through issuing additional bills, chasing uncollected payments, renegotiating with suppliers a refund of costs etc.) after the fact. This last form of reactive revenue assurance is the easiest to put a value to, but is in many ways the least efficient form of revenue assurance; effort is directed towards repeatedly addressing the consequences of known flaws, and not on addressing the flaws themselves. This can lead to a parasitical relationship between a Revenue Assurance department or vendor and the wider business, where the department/vendor finds it easiest to justify its ongoing existence/contract by repeatedly fixing symptoms and not the root causes.

The TM Forum conducted a benchmark survey in 2008 that concluded average leakage, not including losses due to fraud, was 1% of the gross revenues for those telcos that took part.^[1] The number of participating telcos was relatively small compared to some other surveys, but the survey technique was more demanding than any comparable survey to date. The survey used the most detailed and prescriptive definition of how to calculate leakage of any survey of its type. The definition was taken from the TM Forum's own standard on how to calculate revenue assurance metrics [2]. To increase confidence that participants calculated their leakages correctly, the TM Forum's benchmark program independently reviewed the results and corroborated them with representatives of the participating companies. The survey's average of 1% leakage of gross revenue, whilst still significant, is notably lower than many other quoted estimates and reported survey findings about average leakage. This may be because the survey used a very strict definition of leakage. The survey measured only actual under-billed and unbilled amounts discovered by the participants; it excluded other types of leakages such as cost leakages and loss of opportunity leakages, and it excluded projected leakage estimations that are commonly used (i.e., what would have been the amount of leakage, if the leakage would not been discovered by revenue assurance activities). It may also reflect a reduction in bias or exaggeration in reported leakages, or at least the exclusion of guesswork. Respondents were given authoritative instructions on how to quantify leakage based on actual data and were instructed to avoid making suppositions in the absence of such data.

The best known estimates of "typical" revenue leakage come from a series of annual surveys conducted by the Analysys consultancy and research business. In these surveys, leakage was commonly estimated as being worth between 5% and 15% of the total revenue of the business. Similar research by other businesses has generated results in the same range, with none concluding leakage of less than 1% of gross revenue, and some suggesting leakage of 20% or more was not uncommon. Reasons to doubt these estimates are as follows:

(1) All the estimates of leakage were derived from the subjective opinions of staff working in service providers;

(2) All the research was conducted by businesses wishing to promote their revenue assurance products;

(3) Increased annual spend on revenue assurance has not resulted in a clear downward trend in estimates;

(4) The estimates were broadly similar even when the criteria for what losses to include varied greatly; and

(5) Genuine losses of this scale should be a severe corporate governance issue in any publicly listed business.

What can be said with some confidence is that revenue assurance practitioners are able to provide a vast number of consistent anecdotes relating to the causes of leakage and means to resolve them. Though there is little objective evidence relating to actual leakages approaching this scale in the public domain as this information per se is highly confidential, there are some indirect measures of data integrity that help give a sense of potential leakage. For example, in reconciling interconnect costs and revenues between telcos, a 5% variance is the common practice to accept before a disputed invoice can lead one party to withhold payment, and a 0.5% variance would be considered industry-leading practice according to best practice advice issued by the UK Revenue Assurance Group.

Revenue Assurance in Telecom

Although Revenue Assurance has always been present in the telecom parlance^{[citation needed]} it has recently been brought at the forefront of the top management^{[citation needed]}. This is due to several factors including

• Profit: Increasing cost pressures and decreasing margins. The high profit days for most telcos are over^{[citation needed]}. They all need to find alternative means to squeeze higher margins by effectively tracking their revenue.
• Regulatory: New regulatory structure and compliance requirements^{[citation needed]} which force the telecom operators to report their revenue accurately.
• Technology Innovation: Ensuring new technologies and products are performing as per perceived plans. Keeping up with release of new technologies along with co-existing of legacy systems.
• Mergers and Acquisitions: With increase in the number of telco mergers and acquisitions, organizations are finding it very difficult handle multiple BSS systems including Billing, Mediation and Rating together etc.

Revenue Assurance has been a problem for the telecom companies since the very early stages. Tracking of pulses, minutes, counts, bytes etc. has never been more difficult. One would think these would be easy for the tech-savvy telco companies. However, the truth has been just the opposite. In a hurry to release new technologies in the market, the Revenue Assurance systems are always lagging behind. Revenue Assurance in a telco environment covers a wide range of technical and business aspects. An RA operator needs to be aware of both OSS & BSS processes and internal dependencies to accurately decipher the revenue code.

What causes the problem

A telecom organization's revenue chain is usually a very complex set of inter-related technologies and processes providing a seamless set of services to the end consumer. As the set of technologies and business processes grows bigger and more complex, the chance of failure increases in each of its connections. A revenue leakage is typically attributed to when a telco organization is unable to bill correctly for a given service or to receive the correct payment. As the organization grows the probability of revenue leakage only increases.

Where is the problem

The most debated part of revenue assurance is where to start checking, i.e. at the network side, the rating side, the billing side, the interconnect side, the CRM side, etc. However, most surveys and reports state that the maximum leakage happens during the flow of Call Detail Records (CDRs) or Event Detail Records (EDRs) from the Switch to the respective rating / billing engines. Some of the common problem areas are :

• Network

  * Signaling problems

  * CDRs in Switch not sent to Mediation       

  * CDRs in Mediation not send downstream      

  * CDRs rejected by rating / billing system

  * Wrong duration on the CDRs

  * Incorrect Business rules

  * Subscriber provisioning

  * Incorrect Routing

• Rating & Billing

  * Incorrect Rejection Logic

  * Duplicate CDRs resulting in double charging

  * Incorrect tariff plans

  * Rating & Billing accuracy errors

  * Late rating / billing

  * Incorrect configurations – rating minutes instead of seconds

  * Incorrect Disconnection

Revenue assurance discipline

Among the disciplines in revenue assurance are:

1. The CORE functions of a revenue assurance group: Monitoring, Baselining, Auditing, Synchronizing, Investigating and Compliance.
2. Decomposing an organization's revenue assurance scope (The Revenue Management Chain).
3. Assessing and minimizing revenue loss risk.

Source: Wikipedia.

Top-down versus bottom-up design methodologies Neeraj nathani-smartbridge.net

Top-down versus bottom-up design methodologies

Bottom-up design

Ralph Kimball, a well-known author on data warehousing,^[8] is a proponent of an approach to data warehouse design which he describes as bottom-up.^[9]
In the bottom-up approach, data marts are first created to provide reporting and analytical capabilities for specific business processes. It is important to note that in Kimball methodology, the bottom-up process is the result of an initial business-oriented top-down analysis of the relevant business processes to be modelled.
Data marts contain, primarily, dimensions and facts. Facts can contain either atomic data and, if necessary, summarized data. The single data mart often models a specific business area such as "Sales" or "Production." These data marts can eventually be integrated to create a comprehensive data warehouse. The integration of data marts is managed through the implementation of what Kimball calls "a data warehouse bus architecture".^[10] The data warehouse bus architecture is primarily an implementation of "the bus", a collection of conformed dimensions and conformed facts, which are dimensions that are shared (in a specific way) between facts in two or more data marts.
The integration of the data marts in the data warehouse is centered on the conformed dimensions (residing in "the bus") that define the possible integration "points" between data marts. The actual integration of two or more data marts is then done by a process known as "Drill across". A drill-across works by grouping (summarizing) the data along the keys of the (shared) conformed dimensions of each fact participating in the "drill across" followed by a join on the keys of these grouped (summarized) facts.
Maintaining tight management over the data warehouse bus architecture is fundamental to maintaining the integrity of the data warehouse. The most important management task is making sure dimensions among data marts are consistent. In Kimball's words, this means that the dimensions "conform".
Some consider it an advantage of the Kimball method, that the data warehouse ends up being "segmented" into a number of logically self-contained (up to and including The Bus) and consistent data marts, rather than a big and often complex centralized model. Business value can be returned as quickly as the first data marts can be created, and the method gives itself well to an exploratory and iterative approach to building data warehouses. For example, the data warehousing effort might start in the "Sales" department, by building a Sales-data mart. Upon completion of the Sales-data mart, the business might then decide to expand the warehousing activities into the, say, "Production department" resulting in a Production data mart. The requirement for the Sales data mart and the Production data mart to be integrable, is that they share the same "Bus", that will be, that the data warehousing team has made the effort to identify and implement the conformed dimensions in the bus, and that the individual data marts links that information from the bus. Note that this does not require 100% awareness from the onset of the data warehousing effort, no master plan is required upfront. The Sales-data mart is good as it is (assuming that the bus is complete) and the Production-data mart can be constructed virtually independent of the Sales-data mart (but not independent of the Bus).
If integration via the bus is achieved, the data warehouse, through its two data marts, will not only be able to deliver the specific information that the individual data marts are designed to do, in this example either "Sales" or "Production" information, but can deliver integrated Sales-Production information, which, often, is of critical business value.

Top-down design

Bill Inmon, one of the first authors on the subject of data warehousing, has defined a data warehouse as a centralized repository for the entire enterprise.^[10] Inmon is one of the leading proponents of the top-down approach to data warehouse design, in which the data warehouse is designed using a normalized enterprise data model. "Atomic" data, that is, data at the lowest level of detail, are stored in the data warehouse. Dimensional data marts containing data needed for specific business processes or specific departments are created from the data warehouse. In the Inmon vision, the data warehouse is at the center of the "Corporate Information Factory" (CIF), which provides a logical framework for delivering business intelligence (BI) and business management capabilities.
Inmon states that the data warehouse is:

Subject-oriented

The data in the data warehouse is organized so that all the data elements relating to the same real-world event or object are linked together.

Non-volatile

Data in the data warehouse are never over-written or deleted — once committed, the data are static, read-only, and retained for future reporting.

Integrated

The data warehouse contains data from most or all of an organization's operational systems and these data are made consistent.

Time-variant

For An operational system, the stored data contains the current value. The data warehouse, however, contains the history of data values.

The top-down design methodology generates highly consistent dimensional views of data across data marts since all data marts are loaded from the centralized repository. Top-down design has also proven to be robust against business changes. Generating new dimensional data marts against the data stored in the data warehouse is a relatively simple task. The main disadvantage to the top-down methodology is that it represents a very large project with a very broad scope. The up-front cost for implementing a data warehouse using the top-down methodology is significant, and the duration of time from the start of project to the point that end users experience initial benefits can be substantial. In addition, the top-down methodology can be inflexible and unresponsive to changing departmental needs during the implementation phases.^[10]

Hybrid design

Data warehouse (DW) solutions often resemble the hub and spokes architecture. Legacy systems feeding the DW/BI solution often include customer relationship management (CRM) and enterprise resource planning solutions (ERP), generating large amounts of data. To consolidate these various data models, and facilitate the extract transform load (ETL) process, DW solutions often make use of an operational data store (ODS). The information from the ODS is then parsed into the actual DW. To reduce data redundancy, larger systems will often store the data in a normalized way. Data marts for specific reports can then be built on top of the DW solution.
It is important to note that the DW database in a hybrid solution is kept on third normal form to eliminate data redundancy. A normal relational database however, is not efficient for business intelligence reports where dimensional modelling is prevalent. Small data marts can shop for data from the consolidated warehouse and use the filtered, specific data for the fact tables and dimensions required. The DW effectively provides a single source of information from which the data marts can read, creating a highly flexible solution from a BI point of view. The hybrid architecture allows a DW to be replaced with a master data management solution where operational, not static information could reside.
The Data Vault Modeling components follow hub and spokes architecture. This modeling style is a hybrid design, consisting of the best practices from both 3rd normal form and star schema. The Data Vault model is not a true 3rd normal form, and breaks some of the rules that 3NF dictates be followed. It is however, a top-down architecture with a bottom up design. The Data Vault model is geared to be strictly a data warehouse. It is not geared to be end-user accessible, which when built, still requires the use of a data mart or star schema based release area for business purposes.

Data warehouses versus operational systems

Operational systems are optimized for preservation of data integrity and speed of recording of business transactions through use of database normalization and an entity-relationship model. Operational system designers generally follow the Codd rules of database normalization in order to ensure data integrity. Codd defined five increasingly stringent rules of normalization. Fully normalized database designs (that is, those satisfying all five Codd rules) often result in information from a business transaction being stored in dozens to hundreds of tables. Relational databases are efficient at managing the relationships between these tables. The databases have very fast insert/update performance because only a small amount of data in those tables is affected each time a transaction is processed. Finally, in order to improve performance, older data are usually periodically purged from operational systems.

Confirmed Dimensions, Junk Dimensions, and Degenerated Dimensions 

  Conformed Dimensions (CD): these dimensions are something that is built once in your model and can be reused multiple times with different fact tables.   For example, consider a model containing multiple fact tables, representing different data marts.  Now look for a dimension that is common to these facts tables.  In this example let’s consider that the product dimension is common and hence can be reused by creating short cuts and joining the different fact tables.Some of the examples are time dimension, customer dimensions, product dimension. §         

Junked Dimensions (JD):  When you consolidate lots of small dimensions and instead of having 100s of small dimensions, that will have few records in them, cluttering your database with these mini ‘identifier’ tables, all records from all these small dimension tables are loaded into ONE dimension table and we call this dimension table Junk dimension table.  (Since we are storing all the junk in this one table) For example: a company might have handful of manufacture plants, handful of order types, and so on, so forth, and we can consolidate them in one dimension table called junked dimension table. 

 Degenerated Dimension (DD):  An item that is in the fact table but is stripped off of its description, because the description belongs in dimension table, is referred to as Degenerated Dimension.  Since it looks like dimension, but is really in fact table and has been degenerated of its description, hence is called degenerated dimension. Now coming to the slowly changing dimensions (SCD) and Slowly Growing Dimensions (SGD):  I would like to classify them to be more of an attributes of dimensions its self.   

Although other might disagree to this view but Slowly Changing Dimensions are basically those dimensions whose key value will remain static but description might change over the period of time.  For example, the product id in a companies, product line might remain the same, but the description might change from time to time, hence, product dimension is called slowly changing dimension.  

 Lets consider a customer dimension, which will have a unique customer id but the customer name (company name) might change periodically due to buy out / acquisitions, Hence, slowly changing dimension, as customer number is static but customer name is changing,  However, on the other hand the company will add more customers to its existing list of customers and it is highly unlikely that the company will acquire astronomical number of customer over night (wouldn’t the company CEO love that) hence, the customer dimension is both a Slowly changing as well as slowly growing dimension

Source: Wikipedia.

Neeraj Nathani PERFORMANCE BOOSTERS FOR ETL PROGRAMS:

PERFORMANCE BOOSTERS FOR ETL PROGRAMS:

Challenges

ETL processes can involve considerable complexity, and significant operational problems can occur with improperly designed ETL systems.

The range of data values or data quality in an operational system may exceed the expectations of designers at the time validation and transformation rules are specified. Data profiling of a source during data analysis can identify the data conditions that will need to be managed by transform rules specifications. This will lead to an amendment of validation rules explicitly and implicitly implemented in the ETL process.

Data warehouses are typically assembled from a variety of data sources with different formats and purposes. As such, ETL is a key process to bring all the data together in a standard, homogeneous environment.

Design analysts should establish the scalability of an ETL system across the lifetime of its usage. This includes understanding the volumes of data that will have to be processed within service level agreements. The time available to extract from source systems may change, which may mean the same amount of data may have to be processed in less time. Some ETL systems have to scale to process terabytes of data to update data warehouses with tens of terabytes of data. Increasing volumes of data may require designs that can scale from daily batch to multiple-day microbatch to integration with message queues or real-time change-data capture for continuous transformation and update

Performance

ETL vendors benchmark their record-systems at multiple TB (terabytes) per hour (or ~1 GB per second) using powerful servers with multiple CPUs, multiple hard drives, multiple gigabit-network connections, and lots of memory. The fastest ETL record is currently held by Syncsort,^[1] Vertica and HP at 5.4TB in under an hour which is more than twice as fast as the earlier record held by Microsoft and Unisys.

In real life, the slowest part of an ETL process usually occurs in the database load phase. Databases may perform slowly because they have to take care of concurrency, integrity maintenance, and indices. Thus, for better performance, it may make sense to employ:

• Direct Path Extract method or bulk unload whenever is possible (instead of querying the database) to reduce the load on source system while getting high speed extract
• most of the transformation processing outside of the database
• bulk load operations whenever possible.

Still, even using bulk operations, database access is usually the bottleneck in the ETL process. Some common methods used to increase performance are:

• Partition tables (and indices). Try to keep partitions similar in size (watch for null values which can skew the partitioning).
• Do all validation in the ETL layer before the load. Disable integrity checking (disable constraint ...) in the target database tables during the load.
• Disable triggers (disable trigger ...) in the target database tables during the load. Simulate their effect as a separate step.
• Generate IDs in the ETL layer (not in the database).
• Drop the indices (on a table or partition) before the load - and recreate them after the load (SQL: drop index ...; create index ...).
• Use parallel bulk load when possible — works well when the table is partitioned or there are no indices. Note: attempt to do parallel loads into the same table (partition) usually causes locks — if not on the data rows, then on indices.
• If a requirement exists to do insertions, updates, or deletions, find out which rows should be processed in which way in the ETL layer, and then process these three operations in the database separately. You often can do bulk load for inserts, but updates and deletes commonly go through an API (using SQL).

Whether to do certain operations in the database or outside may involve a trade-off. For example, removing duplicates using distinct may be slow in the database; thus, it makes sense to do it outside. On the other side, if using distinct will significantly (x100) decrease the number of rows to be extracted, then it makes sense to remove duplications as early as possible in the database before unloading data.

A common source of problems in ETL is a big number of dependencies among ETL jobs. For example, job "B" cannot start while job "A" is not finished. You can usually achieve better performance by visualizing all processes on a graph, and trying to reduce the graph making maximum use of parallelism, and making "chains" of consecutive processing as short as possible. Again, partitioning of big tables and of their indices can really help.

Another common issue occurs when the data is spread between several databases, and processing is done in those databases sequentially. Sometimes database replication may be involved as a method of copying data between databases - and this can significantly slow down the whole process. The common solution is to reduce the processing graph to only three layers:

• Sources
• Central ETL layer
• Targets

This allows processing to take maximum advantage of parallel processing. For example, if you need to load data into two databases, you can run the loads in parallel (instead of loading into 1st - and then replicating into the 2nd).

Of course, sometimes processing must take place sequentially. For example, you usually need to get dimensional (reference) data before you can get and validate the rows for main "fact" tables.

Parallel processing

A recent^[update] development in ETL software is the implementation of parallel processing. This has enabled a number of methods to improve overall performance of ETL processes when dealing with large volumes of data.

ETL applications implement three main types of parallelism:

• Data: By splitting a single sequential file into smaller data files to provide parallel access.
• Pipeline: Allowing the simultaneous running of several components on the same data stream. For example: looking up a value on record 1 at the same time as adding two fields on record 2.
• Component: The simultaneous running of multiple processes on different data streams in the same job, for example, sorting one input file while removing duplicates on another file.

All three types of parallelism usually operate combined in a single job.

An additional difficulty comes with making sure that the data being uploaded is relatively consistent. Because multiple source databases may have different update cycles (some may be updated every few minutes, while others may take days or weeks), an ETL system may be required to hold back certain data until all sources are synchronized. Likewise, where a warehouse may have to be reconciled to the contents in a source system or with the general ledger, establishing synchronization and reconciliation points becomes necessary.

Source: Wikipedia.

Category management Neeraj nathani

Category management

Category management is a retailing and purchasing concept in which the range of products purchased by a business organization or sold by a retailer is broken down into discrete groups of similar or related products; these groups are known as product categories (examples of grocery categories might be: tinned fish, washing detergent, toothpastes). It is a systematic, disciplined approach to managing a product category as a strategic business unit.^[1] The phrase "category management" was coined by Brian F. Harris.^{[n/a 1]}

Category management in a retail context

Each category is run as a "mini business" (business unit) in its own right, with its own set of turnover and/or profitability targets and strategies. Introduction of Category Management in a business tends to alter the relationship between retailer and supplier: instead of the traditional adversarial relationship, the relationship moves to one of collaboration, with exchange of information, sharing of data and joint business building.

The focus of all supplier negotiations is the effect on turnover of the category as whole, not just the sales of individual products. Suppliers are expected, indeed in many cases mandated, to only suggest new product introductions, a new planogram or promotional activity if it is expected to have a beneficial effect on the turnover or profit of the total category and be beneficial to the shoppers of that category.

The concept originated in grocery (mass merchandising) retailing, and has since expanded to other retail sectors such as DIY, cash and carry, pharmacy, and book retailing.^[2]

Definition of category management (retail)

Category management lacks a single definition thus leading to some ambiguity even among industry professionals as to its exact function. Three comparative mainstream definitions are as follows:
Category management is a process that involves managing product categories as business units and customizing them [on a store by store basis] to satisfy customer needs. (Nielsen)^[3]
The strategic management of product groups through trade partnerships which aims to maximize sales and profit by satisfying consumer and shopper needs (Institute of Grocery Distribution)^[4]
.. marketing strategy in which a full line of products (instead of the individual products or brands) is managed as a strategic business unit (SBU). (Business Dictionary)^[5]
The Nielsen definition, published in 1992, was a little ahead of its time in that customising product offerings on a store by store basis is logistically difficult and is now not considered a necessary part of category management; it is a concept now referred to as micromarketing. Nevertheless, most grocery retailers will segment stores at least by size, and select product assortments accordingly. Wal*Mart's Store of the Community, implemented in North America is one of the few examples of where product offerings are tailored right down to the specific store.^[6]

Definition of a category

The Nielsen definition of a category, used as the basic definition across the industry is that the products should meet a similar consumer need, or that the products should be inter-related or substitutable.^[8] The Nielsen definition also includes a provision that products placed together in the same category should be logistically manageable in store (for example there may be issues in having room-temperature and chilled products together in the same category even though the initial two conditions are met).

However, this definition does not explain how the process often works in practical retailing situations, where demographic or marketing considerations take precedence.

The category management 8-step process (retail)

The category management 8-step process

The industry standard model for category management in retail is the 8-step process, or 8-step cycle developed by the Partnering Group.^[9] The eight steps are shown in the diagram on the right; they are :

1. Define the category (i.e. what products are included/excluded).
2. Define the role of the category within the retailer.
3. Assess the current performance.
4. Set objectives and targets for the category.
5. Devise an overall Strategy.
6. Devise specific tactics.
7. Implementation.
8. The eighth step is one of review which takes us back to step 1.

The 8-step process, whilst being very comprehensive and thorough has been criticized for being rather too unwieldy and time-consuming in today's fast-moving sales environment; in one survey only 9% of supplier companies stated they used the full 8-step process.^[10] The current industry trend is for supplier companies to use the standard process as a basis to develop their own more streamlined processes, tailored to their own particular products^[11]

Market research company Nielsen has a similar process based on only 5 steps : reviewing the category, targeting consumers, planning merchandising, implementing strategy, evaluating results

Category captains

It is commonplace for one particular supplier into a category to be nominated by the retailer as a category captain. The category captain will be expected to have the closest and most regular contact with the retailer and will also be expected to invest time, effort, and often financial investment into the strategic development of the category within the retailer.
In return, the supplier will gain a more influential voice with the retailer. The category captain is often the supplier with the largest turnover in the category. Traditionally the job of category captain is given to a brand supplier, but in recent times the role has also gone to particularly switched-on private label suppliers.^[12]
In order to do the job effectively, the supplier may be granted access to a greater wealth of data-sharing, e.g. more access to an internal sales database such as Walmart's Retail Link

Category management in purchasing

Category management can also be applied to purchasing within an organisation. Although the term is the same and there are many similarities with elements of retail category management including the use of similar tools and techniques applied in reverse, the methodology is fundamentally different. Applying Category Management in purchasing benefits organisations by providing an approach to reduce the cost of buying goods and services, reduce risk in the supply chain, increase overall value from the supply base and gain access to more innovation from suppliers. It is a strategic approach that focuses on the vast majority of organisational spend. If applied effectively throughout an entire organisation the results can be significantly greater than traditional transactional based purchasing negotiations.

The concept of Category Management in purchasing originated in the late 80's. There is no single founder or originator but the methodology first appeared in the automotive sector and has since been developed and adopted by organisations worldwide. Today Category Management is considered by many global companies as an essential strategic purchasing approach. Category Management has been defined as “an evolving methodology that drives sourcing strategy in progressive organisations today”.^[15]

The Chartered Institute of Purchasing & Supply defines Category Management as:

"organising the resources of the procurement team in such a way as to focus externally onto the supply markets of an organisation (as against having a focus on the internal customers or on internal Procurement departmental functions) in order to fully leverage purchasing decisions”.^[16]

Jonathan O'Brien, author of Category Management in Purchasing, defines Category Management as:

"the practice of segmenting the main areas of organisational spend on bought-in goods and services into discrete groups of products and services according to the function of those goods or services and, most importantly, to mirror how individual marketplaces are organised. Using this segmentation organisations work cross functionally on individual categories, examining the entire category spend, how the organisation uses the products or services within the category, the marketplace and individual suppliers.",^[17]

Peter Hunt, partner at ADR International, writes

“the term category management can mean different things to different people, so a working definition is needed. A ‘category’ is the logical grouping of similar expenditure items, such as spend on advertising agency services or IT hardware. Category management is the sourcing process used to manage these categories to satisfy business needs while maximising the value delivered from the supply base”.^[18]

Many public sector organisations have recently adopted category management as a strategic transformation tool. Sir Philip Green, in his “Efficiency Review” of UK government spending, recommended that “centralised procurement [should be] mandated for common categories to leverage this buying power and achieve best practice”.^[19]

Source: Wikipedia.