Big data Neeraj Nathani smart bridge

Big data

Big data^[1][2] is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage,^[3] search, sharing, transfer, analysis,^[4] and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions."^[5][6][7]

As of 2012^[update], limits on the size of data sets that are feasible to process in a reasonable amount of time were on the order of exabytes of data.^[8][9] Scientists regularly encounter limitations due to large data sets in many areas, including meteorology, genomics,^[10] connectomics, complex physics simulations,^[11] and biological and environmental research.^[12] The limitations also affect Internet search, finance and business informatics. Data sets grow in size in part because they are increasingly being gathered by ubiquitous information-sensing mobile devices, aerial sensory technologies (remote sensing), software logs, cameras, microphones, radio-frequency identification readers, and wireless sensor networks.^[13][14] The world's technological per-capita capacity to store information has roughly doubled every 40 months since the 1980s;^[15] as of 2012^[update], every day 2.5 quintillion (2.5×10¹⁸) bytes of data were created.^[16] The challenge for large enterprises is determining who should own big data initiatives that straddle the entire organization.^[17]

Big data is difficult to work with using most relational database management systems and desktop statistics and visualization packages, requiring instead "massively parallel software running on tens, hundreds, or even thousands of servers".^[18] What is considered "big data" varies depending on the capabilities of the organization managing the set, and on the capabilities of the applications that are traditionally used to process and analyze the data set in its domain. "For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration."^[19].

Definition

Big data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012^[update] ranging from a few dozen terabytes to many petabytes of data in a single data set. With this difficulty, new platforms of "big data" tools are being developed to handle various aspects of large quantities of data.
In a 2001 research report^[20] and related lectures, META Group (now Gartner) analyst Doug Laney defined data growth challenges and opportunities as being three-dimensional, i.e. increasing volume (amount of data), velocity (speed of data in and out), and variety (range of data types and sources). Gartner, and now much of the industry, continue to use this "3Vs" model for describing big data.^[21] In 2012, Gartner updated its definition as follows: "Big data are high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization."^[22

Examples

Examples include Big Science, web logs, RFID, sensor networks, social networks, social data (due to the social data revolution), Internet text and documents, Internet search indexing, call detail records, astronomy, atmospheric science, genomics, biogeochemical, biological, and other complex and often interdisciplinary scientific research, military surveillance, forecasting drive times for new home buyers, medical records, photography archives, video archives, and large-scale e-commerce.

Big science

The Large Hadron Collider experiments represent about 150 million sensors delivering data 40 million times per second. There are nearly 600 million collisions per second. After filtering and refraining from recording more than 99.999% of these streams, there are 100 collisions of interest per second.^[23][24][25]

• As a result, only working with less than 0.001% of the sensor stream data, the data flow from all four LHC experiments represents 25 petabytes annual rate before replication (as of 2012). This becomes nearly 200 petabytes after replication.
• If all sensor data were to be recorded in LHC, the data flow would be extremely hard to work with. The data flow would exceed 150 million petabytes annual rate, or nearly 500 exabytes per day, before replication. To put the number in perspective, this is equivalent to 500 quintillion (5×10²⁰) bytes per day, almost 200 times higher than all the other sources combined in the world.

Science and research

• When the Sloan Digital Sky Survey (SDSS) began collecting astronomical data in 2000, it amassed more in its first few weeks than all data collected in the history of astronomy. Continuing at a rate of about 200 GB per night, SDSS has amassed more than 140 terabytes of information. When the Large Synoptic Survey Telescope, successor to SDSS, comes online in 2016 it is anticipated to acquire that amount of data every five days.^[5]
• Decoding the human genome originally took 10 years to process; now it can be achieved in one week.^[5]
• Computational social science — Tobias Preis et al. used Google Trends data to demonstrate that Internet users from countries with a higher per capita gross domestic product (GDP) are more likely to search for information about the future than information about the past. The findings suggest there may be a link between online behaviour and real-world economic indicators.^[26][27][28] The authors of the study examined Google queries logs made by Internet users in 45 different countries in 2010 and calculated the ratio of the volume of searches for the coming year (‘2011’) to the volume of searches for the previous year (‘2009’), which they call the ‘future orientation index’.^[29] They compared the future orientation index to the per capita GDP of each country and found a strong tendency for countries in which Google users enquire more about the future to exhibit a higher GDP. The results hint that there may potentially be a relationship between the economic success of a country and the information-seeking behavior of its citizens captured in big data.

Government

• In 2012, the Obama administration announced the Big Data Research and Development Initiative, which explored how big data could be used to address important problems facing the government.^[30] The initiative was composed of 84 different big data programs spread across six departments.^[31]
• Big data analysis played a large role in Barack Obama's successful 2012 re-election campaign.^[32]
• The United States Federal Government owns six of the ten most powerful supercomputers in the world.^[33]
• The NASA Center for Climate Simulation (NCCS) stores 32 petabytes of climate observations and simulations on the Discover supercomputing cluster.^[34]
• The Utah Data Center is a data center currently being constructed by the United States National Security Agency. When finished, the facility will be able to handle yottabytesinformation collected by the NSA over the Internet.^[35][36]

Private sector

• Amazon.com handles millions of back-end operations every day, as well as queries from more than half a million third-party sellers. The core technology that keeps Amazon running is Linux-based and as of 2005 they had the world’s three largest Linux databases, with capacities of 7.8 TB, 18.5 TB, and 24.7 TB.^[37]
• Walmart handles more than 1 million customer transactions every hour, which is imported into databases estimated to contain more than 2.5 petabytes (2560 terabytes) of data – the equivalent of 167 times the information contained in all the books in the US Library of Congress.^[5]
• Facebook handles 50 billion photos from its user base.
• FICO Falcon Credit Card Fraud Detection System protects 2.1 billion active accounts world-wide.^[38]
• The volume of business data worldwide, across all companies, doubles every 1.2 years, according to estimates.^[39]
• Infosys has also launched the BigDataEdge to analyse the Big data.^[40][41]
• Windermere Real Estate uses anonymous GPS signals from nearly 100 million drivers to help new home buyers determine their typical drive times to and from work throughout various times of the day ^[42]

International development

Following decades of work in the area of the effective usage of information and communication technologies for development (or ICT4D), it has been suggested that Big Data can make important contributions to international development.^[43][44] On the one hand, the advent of Big Data delivers the cost-effective prospect to improve decision-making in critical development areas such as health care, employment, economic productivity, crime and security, and natural disaster and resource management.^[45] On the other hand, all the well-known concerns of the Big Data debate, such as privacy, interoperability challenges, and the almighty power of imperfect algorithms, are aggravated in developing countries by long-standing development challenges like lacking technological infrastructure and economic and human resource scarcity. "This has the potential to result in a new kind of digital divide: a divide in data-based intelligence to inform decision-making."^[45]

Technologies

File:DARPA’s Topological Data Analysis program seeks the fundamental structure of massive data sets and is developing the tools to exploit that knowledge.tiff

DARPA’s Topological Data Analysis program seeks the fundamental structure of massive data sets.

Big data requires exceptional technologies to efficiently process large quantities of data within tolerable elapsed times. A 2011 McKinsey report^[49] suggests suitable technologies include A/B testing, association rule learning, classification, cluster analysis, crowdsourcing, data fusion and integration, ensemble learning, genetic algorithms, machine learning, natural language processing, neural networks, pattern recognition, anomaly detection, predictive modelling, regression, sentiment analysis, signal processing, supervised and unsupervised learning, simulation, time series analysis and visualisation. Multidimensional big data can also be represented as tensors, which can be more efficiently handled by tensor-based computation,^[50] such as multilinear subspace learning.^[51] Additional technologies being applied to big data include massively parallel-processing (MPP) databases, search-based applications, data-mining grids, distributed file systems, distributed databases, cloud based infrastructure (applications, storage and computing resources) and the Internet.^{[citation needed]}

Some but not all MPP relational databases have the ability to store and manage petabytes of data. Implicit is the ability to load, monitor, back up, and optimize the use of the large data tables in the RDBMS.^[52]

DARPA’s Topological Data Analysis program seeks the fundamental structure of massive data sets and in 2008 the technology went public with the launch of a company called Ayasdi.

The practitioners of big data analytics processes are generally hostile to slower shared storage^{[citation needed]}, preferring direct-attached storage (DAS) in its various forms from solid state disk (SSD) to high capacity SATA disk buried inside parallel processing nodes. The perception of shared storage architectures—SAN and NAS—is that they are relatively slow, complex, and expensive. These qualities are not consistent with big data analytics systems that thrive on system performance, commodity infrastructure, and low cost.

Real or near-real time information delivery is one of the defining characteristics of big data analytics. Latency is therefore avoided whenever and wherever possible. Data in memory is good—data on spinning disk at the other end of a FC SAN connection is not. The cost of a SAN at the scale needed for analytics applications is very much higher than other storage techniques.

There are advantages as well as disadvantages to shared storage in big data analytics, but big data analytics practitioners as of 2011^[update] did not favour it.^[53]

Source: Wikipedia.

FMCG Market Smart Bridge Neeraj Nathani Smart Bridge

                                                                FMCG Market

Fast-moving consumer goods (FMCG) or consumer packaged goods (CPG) are products that are sold quickly and at relatively low cost. Examples include non-durable goods such as soft drinks, toiletries, and grocery items.^[1][2] Though the absolute profit made on FMCG products is relatively small, they are generally sold in large quantities, and so the cumulative profit on such products can be substantial.
Fast-moving consumer electronics are a type of FMCG and are typically low priced generic or easily substitutable consumer electronics, including lower end mobile phones, MP3 players, game players, and digital cameras, which have a short usage life, typically a year or less, and as such are disposable. Cheap FMCG electronics are often retained even after immediate failure, as the purchaser rationalizes the decision to not return the goods on the basis that the goods were cheap to begin with, and that the cost of return relative to the low cost of purchase is high. Thus low-quality electronic FMCG goods can be highly profitable for the vendors.
The term FMCGs refers to those retail goods that are generally replaced or fully used up over a short period of days, weeks, or months, and within one year. This contrasts with durable goods or major appliances such as kitchen appliances, which are generally replaced over a period of several years
FMCG have a short shelf life, either as a result of high consumer demand or because the product deteriorates rapidly. Some FMCGs—such as meat, fruits and vegetables, dairy products, and baked goods—are highly perishable. Other goods such as alcohol, toiletries, pre-packaged foods, soft drinks, and cleaning products have high turnover rates. An excellent example is a newspaper—every day's newspaper carries different content, making one useless just one day later, necessitating a new purchase every day.

The following are the main characteristics of FMCGs:^[1]

• From the consumers' perspective:
• Frequent purchase
• Low involvement (little or no effort to choose the item – products with strong brand loyalty are exceptions to this rule)
• Low price
• From the marketers' angle:
• High volumes
• Low contribution margins
• Extensive distribution networks
• High stock turnover

Source: Wikipedia.

Neeraj Nathani smart Bridge RETAILING

RETAILING

Retail is the sale of goods and services from individuals or businesses to the end-user. Retailers are part of an integrated system called the supply chain. A retailer purchases goods or products in large quantities from manufacturers directly or through a wholesale, and then sells smaller quantities to the consumer for a profit. Retailing can be done in either fixed locations like stores or markets, door-to-door or by delivery. Retailing includes subordinated services, such as delivery. The term "retailer" is also applied where a service provider services the needs of a large number of individuals, such as a public. Shops may be on residential streets, streets with few or no houses or in a shopping mall. Shopping streets may be for pedestrians only. Sometimes a shopping street has a partial or full roof to protect customers from precipitation. Online retailing, a type of electronic commerce used for business-to-consumer (B2C) transactions and mail order, are forms of non-shop retailing.
Shopping generally refers to the act of buying products. Sometimes this is done to obtain necessities such as food and clothing; sometimes it is done as a recreational activity. Recreational shopping often involves window shopping (just looking, not buying) and browsing and does not always result in a purchase

Types of retail outlets

A marketplace is a location where goods and services are exchanged. The traditional market square is a city square where traders set up stalls and buyers browse the merchandise. This kind of market is very old, and countless such markets are still in operation around the whole world.

In some parts of the world, the retail business is still dominated by small family-run stores, but this market is increasingly being taken over by large retail chains.

Retail is usually classified by type of products as follows:

• Food products
• Hard goods or durable goods ("hardline retailers") - appliances, electronics, furniture, sporting goods, etc. Goods that do not quickly wear out and provide utility over time.
• Soft goods or consumables - clothing, apparel, and other fabrics. Goods that are consumed after one use or have a limited period (typically under three years) in which you may use them.

There are the following types of retailers by marketing strategy:

• Department stores - very large stores offering a huge assortment of "soft" and "hard goods; often bear a resemblance to a collection of specialty stores. A retailer of such store carries variety of categories and has broad assortment at average price. They offer considerable customer service.
• Discount stores - tend to offer a wide array of products and services, but they compete mainly on price offers extensive assortment of merchandise at affordable and cut-rate prices. Normally retailers sell less fashion-oriented brands.
• Warehouse stores - warehouses that offer low-cost, often high-quantity goods piled on pallets or steel shelves; warehouse clubs charge a membership fee;
• Variety stores - these offer extremely low-cost goods, with limited selection;
• Demographic - retailers that aim at one particular segment (e.g., high-end retailers focusing on wealthy individuals).
• Mom-And-Pop : is a retail outlet that is owned and operated by individuals. The range of products are very selective and few in numbers. These stores are seen in local community often are family-run businesses. The square feet area of the store depends on the store holder.
• Specialty stores: A typical speciality store gives attention to a particular category and provides high level of service to the customers. A pet store that specializes in selling dog food would be regarded as a specialty store. However, branded stores also come under this format. For example if a customer visits a Reebok or Gap store then they find just Reebok and Gap products in the respective stores.
• General store - a rural store that supplies the main needs for the local community;
• Convenience stores: is essentially found in residential areas. They provide limited amount of merchandise at more than average prices with a speedy checkout. This store is ideal for emergency and immediate purchases as it often works with extended hours, stocking everyday;
• Hypermarkets: provides variety and huge volumes of exclusive merchandise at low margins. The operating cost is comparatively less than other retail formats.
• Supermarkets: is a self-service store consisting mainly of grocery and limited products on non food items. They may adopt a Hi-Lo or an EDLP strategy for pricing. The supermarkets can be anywhere between 20,000 and 40,000 square feet (3,700 m²). Example: SPAR supermarket.
• Malls: has a range of retail shops at a single outlet. They endow with products, food and entertainment under a roof.
• Category killers or Category Specialist: By supplying wide assortment in a single category for lower prices a retailer can "kill" that category for other retailers. For few categories, such as electronics, the products are displayed at the centre of the store and sales person will be available to address customer queries and give suggestions when required. Other retail format stores are forced to reduce the prices if a category specialist retail store is present in the vicinity.
• E-tailers: The customer can shop and order through internet and the merchandise are dropped at the customer's doorstep. Here the retailers use drop shipping technique. They accept the payment for the product but the customer receives the product directly from the manufacturer or a wholesaler. This format is ideal for customers who do not want to travel to retail stores and are interested in home shopping. However it is important for the customer to be wary about defective products and non secure credit card transaction. Example: Amazon, Pennyful and eBay.
• Vending Machines: This is an automated piece of equipment wherein customers can drop the money in the machine and acquire the products.

Some stores take a no frills approach, while others are "mid-range" or "high end", depending on what income level they target.

Other types of retail store include:

• Automated Retail stores are self-service, robotic kiosks located in airports, malls and grocery stores. The stores accept credit cards and are usually open 24/7. Examples include ZoomShops and Redbox.
• Big-box stores encompass larger department, discount, general merchandise, and warehouse stores.

Retailers can opt for a format as each provides different retail mix to its customers based on their customer demographics, lifestyle and purchase behaviour. A good format will lend a hand to display products well and entice the target customers to spawn sales.

Global Top Five Retailers

Worldwide Top Five Retailers[2]
Retail Sales Rank	Company	Country of Origin	2010 group revenue (US $mil)
1	Walmart	US	$421,849
2	Carrefour	France	$121,519
3	Tesco	UK	$94,244
4	Metro AG	Germany	$89,311
5	Kroger	US	$82,189

Operations

Retail pricing

The pricing technique used by most retailers is cost-plus pricing. This involves adding a markup amount (or percentage) to the retailer's cost. Another common technique is suggested retail pricing. This simply involves charging the amount suggested by the manufacturer and usually printed on the product by the manufacturer.
In Western countries, retail prices are often called psychological prices or odd prices. Often prices are fixed and displayed on signs or labels. Alternatively, when prices are not clearly displayed, there can be price discrimination, where the sale price is dependent upon who the customer is. For example, a customer may have to pay more if the seller determines that he or she is willing and/or able to. Another example would be the practice of discounting for youths, students, or senior citizens..

Staffing

Because patronage at a retail outlet varies flexibility in scheduling is desirable. Employee scheduling software is sold which, using known patterns of customer patronage, more or less reliably predicts the need for staffing for various functions at times of the year, day of the month or week, and time of day. Usually needs vary widely. Conforming staff utilization to staffing needs requires a flexible workforce which is available when needed but does not have to be paid when they are not, part-time workers; as of 2012 70% of retail workers in the United States were part-time. This may result in financial problems for the workers, who while they are required to be available at all times if their work hours are to be maximized, may not have sufficient income to meet their family and other obligations.^[3]

Transfer mechanisms

There are several ways in which consumers can receive goods from a retailer:

• Counter service, where goods are out of reach of buyers and must be obtained from the seller. This type of retail is common for small expensive items (e.g. jewelry) and controlled items like medicine and liquor. It was common before the 1900s in the United States and is more common in certain countries like India.^[which?]
• Delivery, where goods are shipped directly to consumer's homes or workplaces. Mail orderfrom a printed catalog was invented in 1744 and was common in the late 19th and early 20th centuries. Ordering by telephone is now common, either from a catalog, newspaper, television advertisement or a local restaurant menu, for immediate service (especially for pizza delivery). Direct marketing, including telemarketing and television shopping channels, are also used to generate telephone orders. started gaining significant market share in developed countries in the 2000s.
• Door-to-door sales, where the salesperson sometimes travels with the goods for sale.
• Self-service, where goods may be handled and examined prior to purchase

Source: Wikipedia.