Smart Bridge Trading Solutions Neeraj Nathani Picking Winners In Big Data

Picking Winners In Big Data

Big data solutions are picking up speed in the IT industry. There’s a Cambrian explosion of interesting start-ups, and all the database and business intelligence incumbents have moved to create big data offerings, or rebrand into the new universe.

The key to seeing the value of big data is understanding that it’s a business problem, not a matter of picking the right tools and waiting for the magic to happen. That said, technology choices still need to be made in an increasingly crowded and confused market.

The biggest question for anybody wanting to invest or adopt technology in this area is how to pick the winner? Glancing through marketing materials will do little to help you: everybody claims relevance to big data.

As I am often asked my opinion about big data companies, I thought I’d share some of the principles I use to help me think about the industry.

Where’s the value?

We need to understand where the actual value is in the data world. For the most part, this value doesn’t lie purely in the software. Over the past decade we have seen a rising tide of commoditization of the software stack: from operating system, to relational databases, to Hadoop itself. In fact, without this, we wouldn’t have the big data revolution as we know it.

As Hadoop has become a de facto standard, so has the notion of building on top of it with open source. There is some advantage in software innovation, but it is momentary. Once something is known to be possible, there are enough smart programmers out there that reproducing it becomes straightforward. (Because of this, I gloomily predict no shortage of patent battles in the not-too-distant future.)

Despite the flux in the software world, two things about big data remain constant: the need for compute and storage, and data itself. It’s ultimately to ownership of one or both of these factors that IT industry value will gravitate.

Compute and storage

The ever growing need for computing power and storage bodes well for companies providing the basics. These fall into two categories: hardware manufacturers, and cloud infrastructure providers. Not that these two markets are immune to their own fluctuations, thanks to standardization and commoditization, but fundamentally, getting paid for use of metal is the name of the game.

In this respect, it’s not too much of a mystery why storage company EMC has plowed so early and so deep into the big data world. Neither is it hard to see the reasoning behind Intel creating its own optimized Hadoop distribution.

Data

Value resting in data is the more subtle of the two axes of big data success.

Big data is ultimately about the smart use of data to drive a business. There are two kinds of data: data about your business, and data external to your business that you can create value from.

It’s easy to see who might get success from the latter, external data. We can expect that massive data owners such as Google, Facebook, Thomson Reuters, Bloomberg will experience ongoing success for as long as they are able to create product from their data.

Who owns your data, though? The obvious answer, you, isn’t the only answer. In fact, your data is locked up inside the platform choices you make, at both the hardware and software level. If your systems are based on Oracle, Microsoft, you are very unlikely to move in a hurry. Data likes to stay where it is, and tends to attract more data as you build systems around it. Production systems are expensive to replace.

So, the vendors of your software platforms of choice also get long term value from your data. For this reason, it’s hard to bet against existing enterprise application platform incumbents in the big data world. Big data, for most of today’s organizations, is an additive phenomenon, not a challenge to the core of IT.

We’ll see more large platforms ensuring nobody needs to move away. Examples include SAP adding HANA, in order to enable their existing customers for big data, or Amazon Web Services’ addition of their data warehouse Redshift, to ensure that the entire data needs of a company can be met on their platform.

Finally, it would be lunacy to bet against either Oracle, who have been portentously quiet in the big data world over the last year, or Microsoft, who in Excel have the world’s most popular data manipulation environment.

It is all business as usual?

I’m not saying there is no new opportunity in the big data industry. What I am saying is that, as an additive technology, big data is unlikely to enable anybody to challenge Oracle or Microsoft for the throne.

There will be change, though, and it will bring both winners and losers.

The area of value we haven’t yet looked at yet is the point where data interacts with the actual mechanism of a business. That’s where it transfers value to you and enables you to leverage data to get ahead. This breaks down into several areas of opportunity for big data innovators.

• Domain specific: tools that enable the manipulation and exploitation of data in a way that’s specific to a business segment. We see initial evidence of this market opportunity in the evolution of web and customer analytics products.
• Machine learning: more data means more to understand, and the only way an organization can realistically do this is with the aid of computers. Machine learning helps automate many parts of the data wrangling process. Furthermore, cross-company data sharing can significantly boost the effectiveness of machine-learning, creating the opportunity for companies gaining early market share. A recent example of this is Sift Science, a fraud detection application.
• Tools for exploration: human interaction with data is a requirement that’s hard to abstract away. Tableau has a head start in this market for big data, filling the role of “Microsoft Office for Data”, but the field is ripe for new innovation, especially with the increasing power of graphical capabilities and new device formats.
• Data agility: speed-to-decision is a critical factor in business competitiveness. Any solution that removes laborious steps has an advantage: a particularly problematic area here is data integration, the loading of both internal and external data sources ready for analytics. Most of today’s big data solutions are frankensteined combinations of layers: there’s a great opportunity for vertically integrated solutions that removes needless impedance to data manipulation.

Move up http://i.forbesimg.com t Move down

Areas of risk

What are those riskier big data options? If a solution isn’t scoring high in the categories above, it’s not likely to be around for the long run.

The biggest risk is with solutions that address only a single horizontal part of the data architecture. Time is against companies in this game. By selling just part of a complete solution, they’re working against the rising tide of commoditization. Customers will demand standardization (e.g. Hadoop compatibility) in order to feel safe adopting such solutions, but that prevents the lock-in that will protect that business. It’s not Oracle’s relational database that cements their position: it’s their vertical position up and down the application stack.

Therefore, it’s not surprising to see the pure Hadoop distribution companies making partnerships, and branching out into other vertical layers. In the long term, it’s a tough road they’ve chosen.

One company working actively to solve this problem is DataStax, who have pivoted from being seen as the corporate backer for the Cassandra NoSQL database—a risky horizontal play—to selling an integrated stack of Cassandra, Hadoop and Solr, intended as a complete platform for building enterprise applications. They’re going after some of the platform business: being involved in the actual use of data to drive business value.

For some start-ups, not having long term big data viability might be just fine as a strategy. As the bigger enterprise companies lumber into the arena, they’ll prove handy acquisitions. But given the crowded space and progressive commoditization, this isn’t a certain future. And certainly for their customers, the risks are growing.

More controversially, another area at risk is that of traditional ETL. Or in its broader sense, data integration. This is a hard, hard, problem. It’s not easy to integrate data retrospectively, and it’s not all certain that the integration players from the data warehouse world will be able to translate that success into the big data world. Many early adopters of Hadoop were motivated by the fact that existing ETL solutions couldn’t meet their needs.

In the long term, data integration is likely to be best served by entire new architectures that don’t try to separate and tame the data in the first place. It’s a lot easier to build truly integrated data infrastructure as greenfield.

For those who crack the integration problem, the potential rewards are high. But so is the risk.

Conclusion

In the long term, the additive nature of big data, combined with inertia, makes it a safe bet that current enterprise IT incumbents will continue their reign, as long as they move to embrace big data in their architectures.

There is plenty of opportunity though, especially in greenfield and cloud scenarios. Expect to see increasing returns for those who provide integrated solutions and do a good job of equipping human decision makers.

There’s long term value in metal, and value in data. About everything else, it’s worth thinking carefully.

Source: Wikipedia./Forbes

Smart Bridge Trading Solutions Neeraj Nathani Revenue Assurance

 Revenue Assurance

Revenue assurance (RA) is a business activity most commonly undertaken within businesses that provide telecommunication services. The activity is the use of data quality and process improvement methods that improve profits, revenues and cash flows without influencing demand. This was defined by a TM Forum working group based on research documented in its Revenue Assurance Technical Overview [1]. In many telecommunications service providers, revenue assurance is led by a dedicated revenue assurance department.

Overview

"Revenue assurance" is a broad umbrella term. It is used both to describe an activity performed within telecommunications service providers, and is a common name for a small business unit associated with that activity. Revenue assurance is a practical response to perceived or actual issues with operational underperformance, most commonly relating to billing and collection of revenue. Some of the procedures associated with identifying, remedying or preventing errors may be undertaken by a dedicated Revenue Assurance department, though responsibility for revenue assurance is often diffuse and varies greatly with the organizational structure of the provider. Assuming a provider with a typical organizational split, responsibilities for revenue assurance primarily sit between the Finance and Technology directorates, however, revenue assurance initiatives are often started in a business unit or marketing group.

The relevance to Finance rests with the responsibility for financial control, audit and reporting, whilst the subject matter would be network and IS systems as implemented or operated by the Technology side of the business. Marketing groups and / or business units (e.g. wholesale or retail business lines) will often embark on revenue assurance projects in an effort to improve product line margins. Furthermore, marketing and business units are pivotal in providing input into the "should be" state of customer bills and products.

The sphere of influence described by revenue assurance varies greatly between telecommunications service providers, but is usually closely related to back office functions where small errors may have a disproportionately large impact on revenues or costs. The processing of transaction data in modern telecommunications providers exhibits many attributes akin to a complex system. However, there is significant disagreement about the ultimate aims and legitimate scope of revenue assurance teams. This is in part caused by:

(1) the cross-functional nature of the activity and the consequent need for a variety of skills from IT, marketing, finance, et al.;

(2) the difficulty of generalizing across businesses with different objectives and business models;

(3) political infighting within each telco about responsibility for revenue leakages and assurance; and

(4) the difficulty in reliably measuring the value added by revenue assurance as separable from underlying performance.

There is high-level agreement between practitioners about the goals and methods of revenue assurance, though reaching a consensus on defining the boundaries of revenue assurance has proved elusive so far. The goals relate to improving the financial performance by eliminating mistakes in the processing of transaction data. Some take a more encompassing view of what counts as a mistake, which may extend as far as questioning the policy set by executives even when this has been executed correctly. Others take a more open-ended view of the data that is the subject matter. For example, in decreasing order of frequency, revenue assurance may cover:

(1) revenues from retail and corporate sales;

(2) revenues and costs from interconnect and wholesale contracts; and

(3) margins and profitability of investment in networks and information systems.

Other markets have different or more refined priorities. For example, in the U.S.A. management of wholesale contracts has often been the first objective because of the complexity of the domestic market resulting from the Federal Communications Commission's regulatory framework. In contrast, telcos in developing countries may prioritize management of international interconnect arrangements because of the risks posed by fraud and arbitrage. A cable supplier or internet service provider that predominantly offers retail customers flat monthly charges and no limits on usage may be most interested in assuring the profitability of network investments.

Revenue assurance is often regarded by practitioners as a low-cost mechanism to generate significant financial returns for telecommunications service providers. However, the returns are unpredictable as well as being hard to measure, which encourages many executives to take a sceptical view of its worth. Comparable revenue assurance activities do occur in other industries, such as with billing of utilities or with the licensing of software, and there are many parallels with financial and operational control activities undertaken by most large businesses. The rationale for why revenue assurance has come to be considered particularly important in telecommunications, unlike other industries, is disputed. Reasonable conjectures are that:

(1) the fast pace of change and intense commercial competition increase the likelihood of mistakes;

(2) there is significant complexity in determining the combined effect of interacting systems and processes; and

(3) the high-volume, low-value nature of transactions amplifies the financial implications of "small" errors.

Another conjecture is that revenue assurance is a response to changing market conditions. The thinking is that as markets reach saturation and growth potential falls off, so the value of maximizing returns from existing sales increases. This observation has some merit but does not explain the increasing popularity of revenue assurance in telcos serving growth markets. It also in part contradicts the assumption of a compelling costs versus benefits argument for revenue assurance, which would be enhanced in businesses undergoing rapid change. It is also important to recognize that there is a long history of revenue assurance activities in some telcos that predates the coining of the term "revenue assurance".

The revenue assurance techniques applied in practice cover a broad spectrum, from analysis and implementation of business controls to automated data interrogation. At one end of the spectrum, revenue assurance can appear very similar to the kinds of review and process mapping techniques applied for other financial controlling objectives like accounting integrity, as exemplified by those derived from clause 404 of the Sarbanes-Oxley Act. This form of revenue assurance is most commonly promoted by consultancies. The size of such consultancies covers the entire range; the Big 4 all offer some form of revenue assurance consulting, but there are also niche specialist consultancies. At the other end of the spectrum, revenue assurance is treated as a form of reactive automated data interrogation, seeking to find anomalies in transaction data that may indicate errors and potential revenue loss. This form of revenue assurance is most commonly promoted by software houses that aim to provide databases and configurable tools to extract and interrogate a telco's source data. A less popular form of reactive automated assurance involves using both software and specialised hardware as a means of extracting additional data on transactions, for example by creating actual network events or interfacing directly with network elements to replicate dummy events. As with consultancies, IT-oriented revenue assurance solutions are offered by both large vendors like providers of billing and mediation software, and by specialized niche providers.

There is some debate about the relative merits of the different techniques that can be employed in revenue assurance.

As yet, there is no professional body, no qualifications, and no academic research that would help to drive consensus about the purpose or methods of revenue assurance. In part this is addressed by individuals working in the sector through membership and qualification in related fields such as accountancy and information systems audit. Some scientific research in other fields is also applicable to revenue assurance, though most revenue assurance "facts" rely heavily on anecdotes and oft-repeated truisms. Some of the most helpful and progressive initiatives in addressing the problem of consensus and scientific basis are listed below.

The value of revenue assurance

Revenue assurance is usually understood as a means to identify and remedy, and perhaps also to prevent, problems that result in financial under performance without seeking to generate additional sales. The most common metaphor is that of leaking water from a pipe, where water stands in place of revenues or cash flows, and the leaks represent waste. The value of revenue assurance is hence determined by the size of the leaks "plugged", and possibly also those leaks prevented before they occur, although estimating the value of the latter is very problematic. The value added also includes the recovery of "lost" revenues or costs (through issuing additional bills, chasing uncollected payments, renegotiating with suppliers a refund of costs etc.) after the fact. This last form of reactive revenue assurance is the easiest to put a value to, but is in many ways the least efficient form of revenue assurance; effort is directed towards repeatedly addressing the consequences of known flaws, and not on addressing the flaws themselves. This can lead to a parasitical relationship between a Revenue Assurance department or vendor and the wider business, where the department/vendor finds it easiest to justify its ongoing existence/contract by repeatedly fixing symptoms and not the root causes.

The TM Forum conducted a benchmark survey in 2008 that concluded average leakage, not including losses due to fraud, was 1% of the gross revenues for those telcos that took part.^[1] The number of participating telcos was relatively small compared to some other surveys, but the survey technique was more demanding than any comparable survey to date. The survey used the most detailed and prescriptive definition of how to calculate leakage of any survey of its type. The definition was taken from the TM Forum's own standard on how to calculate revenue assurance metrics [2]. To increase confidence that participants calculated their leakages correctly, the TM Forum's benchmark program independently reviewed the results and corroborated them with representatives of the participating companies. The survey's average of 1% leakage of gross revenue, whilst still significant, is notably lower than many other quoted estimates and reported survey findings about average leakage. This may be because the survey used a very strict definition of leakage. The survey measured only actual under-billed and unbilled amounts discovered by the participants; it excluded other types of leakages such as cost leakages and loss of opportunity leakages, and it excluded projected leakage estimations that are commonly used (i.e., what would have been the amount of leakage, if the leakage would not been discovered by revenue assurance activities). It may also reflect a reduction in bias or exaggeration in reported leakages, or at least the exclusion of guesswork. Respondents were given authoritative instructions on how to quantify leakage based on actual data and were instructed to avoid making suppositions in the absence of such data.

The best known estimates of "typical" revenue leakage come from a series of annual surveys conducted by the Analysys consultancy and research business. In these surveys, leakage was commonly estimated as being worth between 5% and 15% of the total revenue of the business. Similar research by other businesses has generated results in the same range, with none concluding leakage of less than 1% of gross revenue, and some suggesting leakage of 20% or more was not uncommon. Reasons to doubt these estimates are as follows:

(1) All the estimates of leakage were derived from the subjective opinions of staff working in service providers;

(2) All the research was conducted by businesses wishing to promote their revenue assurance products;

(3) Increased annual spend on revenue assurance has not resulted in a clear downward trend in estimates;

(4) The estimates were broadly similar even when the criteria for what losses to include varied greatly; and

(5) Genuine losses of this scale should be a severe corporate governance issue in any publicly listed business.

What can be said with some confidence is that revenue assurance practitioners are able to provide a vast number of consistent anecdotes relating to the causes of leakage and means to resolve them. Though there is little objective evidence relating to actual leakages approaching this scale in the public domain as this information per se is highly confidential, there are some indirect measures of data integrity that help give a sense of potential leakage. For example, in reconciling interconnect costs and revenues between telcos, a 5% variance is the common practice to accept before a disputed invoice can lead one party to withhold payment, and a 0.5% variance would be considered industry-leading practice according to best practice advice issued by the UK Revenue Assurance Group.

Revenue Assurance in Telecom

Although Revenue Assurance has always been present in the telecom parlance^{[citation needed]} it has recently been brought at the forefront of the top management^{[citation needed]}. This is due to several factors including

• Profit: Increasing cost pressures and decreasing margins. The high profit days for most telcos are over^{[citation needed]}. They all need to find alternative means to squeeze higher margins by effectively tracking their revenue.
• Regulatory: New regulatory structure and compliance requirements^{[citation needed]} which force the telecom operators to report their revenue accurately.
• Technology Innovation: Ensuring new technologies and products are performing as per perceived plans. Keeping up with release of new technologies along with co-existing of legacy systems.
• Mergers and Acquisitions: With increase in the number of telco mergers and acquisitions, organizations are finding it very difficult handle multiple BSS systems including Billing, Mediation and Rating together etc.

Revenue Assurance has been a problem for the telecom companies since the very early stages. Tracking of pulses, minutes, counts, bytes etc. has never been more difficult. One would think these would be easy for the tech-savvy telco companies. However, the truth has been just the opposite. In a hurry to release new technologies in the market, the Revenue Assurance systems are always lagging behind. Revenue Assurance in a telco environment covers a wide range of technical and business aspects. An RA operator needs to be aware of both OSS & BSS processes and internal dependencies to accurately decipher the revenue code.

What causes the problem

A telecom organization's revenue chain is usually a very complex set of inter-related technologies and processes providing a seamless set of services to the end consumer. As the set of technologies and business processes grows bigger and more complex, the chance of failure increases in each of its connections. A revenue leakage is typically attributed to when a telco organization is unable to bill correctly for a given service or to receive the correct payment. As the organization grows the probability of revenue leakage only increases.

Where is the problem

The most debated part of revenue assurance is where to start checking, i.e. at the network side, the rating side, the billing side, the interconnect side, the CRM side, etc. However, most surveys and reports state that the maximum leakage happens during the flow of Call Detail Records (CDRs) or Event Detail Records (EDRs) from the Switch to the respective rating / billing engines. Some of the common problem areas are :

• Network

  * Signaling problems

  * CDRs in Switch not sent to Mediation       

  * CDRs in Mediation not send downstream      

  * CDRs rejected by rating / billing system

  * Wrong duration on the CDRs

  * Incorrect Business rules

  * Subscriber provisioning

  * Incorrect Routing

• Rating & Billing

  * Incorrect Rejection Logic

  * Duplicate CDRs resulting in double charging

  * Incorrect tariff plans

  * Rating & Billing accuracy errors

  * Late rating / billing

  * Incorrect configurations – rating minutes instead of seconds

  * Incorrect Disconnection

Revenue assurance discipline

Among the disciplines in revenue assurance are:

1. The CORE functions of a revenue assurance group: Monitoring, Baselining, Auditing, Synchronizing, Investigating and Compliance.
2. Decomposing an organization's revenue assurance scope (The Revenue Management Chain).
3. Assessing and minimizing revenue loss risk.

Source: Wikipedia.