Big Data Analytics

Use of Big Data is increasing by the hour and so is the investment in this area for Data Analytics. There are  slew of startups and existing Analytical players like SAS, SPSS, R, RStat, E-Views, TreeNet who are building deep algorithms, forecasting and mining tools that can provide business insights from Big Data.

With a deluge of data that flows from Web and other sources, we are now entangled with much wider and deeper silos of information at our disposal. Mining this information with analytics is the need of the hour for many companies to stay competitive and also to improve their processes and overcome their structural inefficiencies.

Today’s Big Data analytics provides a slew of statistical techniques and usage of algorithms to provides for  State of the current business and assessment and prediction on future growth models for their products and services either on Cloud or their SMAQ clustered of servers.

A host of Statistical techniques and algorithms are being customized and build with various Visualization techniques for Segmentation of profiles be it homogeneous or heterogeneous data sets (K-Means, Discriminant Analysis, Bayesian Belief Network etc) Forecasting future events which are both qualitative and quantitative (Monte Carlo simulation, Markov Chain), Predicting modeling with probabilities on outcomes (Linear Regression, Bayesian techniques),  Descriptive modeling on trends on population (Structured Equation Modeling)  are some of Statistical techniques that may sound Greek and Latin to most common readers who are not Data Scientists ( A new term getting in vogue for folks who have good grasp of understanding and usage of Statistical techniques and Econometric modeling which were earlier called as Operational Research folks).

Data driven decisions are not new in many enterprises be it in CPG (Consumer Packaging Group), Financial Services, Healthcare, Retail, Media and so forth,  but they had only scratched the surface in the past because these techniques required special silos for storage, computation and analysis.

Big Data Analytics are used to correlate and identify patterns to study Customer behaviors, Brand preferences, Loyalties and Reward program usages to strategize and position their products and services appropriately on a ongoing basis. Business mining, Data Analytics, Visualization will become a key arsenal and core requirements in their Managers job profiles in understanding and implementation them.

Business leaders of many US Companies are proactively taking this discerning note from McKinsey Consulting group that by 2018, there will be a shortage of 1.5 million managers who would not have the skills to utilize Analytics in their business repertoire in their job profile. Many companies are planning to reposition and retrain their employees to meet the new challenging job environment.

On the other hand, IT Consulting companies are already gearing up with resources and processes of this positive note that by 2018, BI Analytics space alone would generate an additional US$20. 3 billions. Major Educational institutions to keep up with growing demand  in the Global market have already started offering Business Analytical courses as a core requirement in both Graduate and Undergraduate degrees. Not to be outdone, many Institutes have created Certification Courses on Business Analytics.

I believe the mantra today on Big Data appears to be how business wants  model and build their data requirements, similar to looking at Glass that is either half full or half empty philosophy.

“We capture what we model or We model what we capture”

Conclusion: Imagine building a Eureqa like program, which keeps iterating with different data points until it finds an equation that matches its relationship. Human mind is not capable of modeling or figuring it out or just like Einstein's e= mc² what 'c' is. Often many complexities outrun minds capability to understand it. Big Data analytics comes handy to unravel and demystify it.

Big Data Profile

I am sure, most IT Managers have got some grounding on Big Data in terms of what it is, how to integrate with traditional enterprise data warehouses. Big IT vendors like IBM, Oracle, HP, Teradata, Microsoft to name a few have matured their offerings to integrate Big data with their traditional products, while others have perfected their niche products and services in their offerings for Big Data on Infrastructure, Integration Services and Analytical areas. Even there are specialized niche consulting companies that provide Bigdata solutions, while other major consulting companies are catching upon this exciting environment and certainly seems to have mastered the challenges and opportunities this new paradigm provides to the enterprise of all sizes.

·         Today, social media is an integral part to study Customer usage, experience on their products and service which can also be monitored on daily based on their sentiments.  A Bigdata platform facilitates understanding this sentiment by integrating to read the semi and unstructured data on their comments and tweets. Klout score is a perfect example that leverages Social media platforms to provides individuals social online influence using Bigdata.

·         A wind farm manufacture was able to identify patterns for a given terrain to install wind mills across 80 countries  by deploying Big data to churn petabytes of weather data in 15 minutes for analysis which otherwise would have taken 3 to 4 weeks and many lost opportunities.

·         A major patient caregiver was able to provide better patient care with 360 degree view of the patient and able to avoid readmitting losses and get timely analysis from patients past and current records for key insights by marrying both their RDBM data warehouse with Big data analytics platform.

·         A University built in-stream Bigdata computing platform to monitor in real time round the clock  signal streams for its many medical devices from patients to detect and respond before things become critical and avoid potential complications.

Bigdata is now considered as the next new phenomenon only to “Internet” and many companies are vying with each other to stay ahead of the curve as they sit on golden mound of common sources. These sources are best suited for Bigdata to extract content and the context to provide them with appropriate analytics for competitive edge and better decision making abilities in the current ecosystem.

Here are some examples which I believe we are all too familiar.

• Market research reports
• Consumer research reports
• Survey data
• Call center voice calls
• Emails
• Social media data
• Excel spreadsheets from multiple business units
• Data from interactive web channels

Even, GPS  data (time, location), Machine data (activity logs) consists of usage and behavior content and deciphering, studying and predicting for patterns from these sensors, assembly line robotic arms machines can be automated in Bigdata.

Note, these sources existed even before, but what is fueling today is the availability of commodity infrastructure with automated data processing capabilities that is extremely fast with scalability exceeding the limitations that existed in the mindset of business and IT managers. Innovation has transformed the way business is done today to meet ever increasing personalized services with speed, agility and flexibility.

Now lets look at the some obvious questions like how to build one and what are the key components of the Bigdata solution.

A Bigdata namely has  four dimensions namely Volume (terabytes running into petabytes), Velocity (Speed at which data can change dramatically) Variety( machine processed logs, posts, tweets, blogs coming in different formats and structures and data types including text, videos, audio, images etc ) and veracity (trust worthiness of the data).

For Bigdata  Infrastructure, the scalability needs to be linear to scale, with a high throughput to mandate the  velocity dimension requirements, a fault tolerant system (automatic recovery with no manual intervention), with a high degree of parallelism to load and processes across multiple machines with its own copy with distributed data.

To meet the above features, Bigdata relies on Hadoop framework to support data intensive process that can support thousands of nodes and petabytes of data and being fault tolerant. Hadoop is an Open source software that supports distributed processing on very large scale and was inspired by Google's MapReduce and Google File System (GFS) . Besides processing large of amounts of data, Hadoop can crank intensive computational problems that are deep like Portfolio evaluation analysis for Finance Investment Company.

Some of the common processing tools of Bigdata Infrastructure for Hadoop (An Open Source):

Hadoop HDFS	A DFS Storage system that distributes files across multiple machines for scalability, greater throughput and fault tolerance
MapReduce	Programming interface that consists of Map and Reduce functionality which follows divide and conquer strategy for distributing an extremely large problem on extremely large computing cluster.
Chukwa	Hadoop's large-scale log collection and analysis tool.
Hive	Data warehouse infrastructure built on top of Hadoop for providing data summarization and analysis with SQL like queries for structured data.
HBase	Distributed column-oriented NoSQL database built on top of HDFS. HBase supports billions of rows and millions of columns on thousands of nodes.(Some NoSQL DB’s like CouchDB, MongoDB, Riak have built in MapReduce functionality)
Pig	A high level data flow programming language for Hadoop
Mahout	Open source library which implements several scalable machine learning algorithms.
Zookeeper	Helps with coordination between Hadoop nodes efficiently
Oozie	Oozie is a workflow scheduler system to manage Hadoop jobs

Four steps of data processing steps in Bigdata data

·         Gather data. A landing zone holds data received from different sources with file name changes accommodated in this stage.

·         Load data. Metadata is applied to map context to the raw source while creating small chunk of files which are partitioned either horizontally or vertically based requirements.

·         Transform data. Apply business rules and build intermediary data sets at each processing steps as key-value-pair and associate metadata and associated metrics..

·         Extract data. In this stage the result data set can be extracted for further processing including analytics, operational reporting, data warehouse integration, and visualization purposes.

In Bigdata, the data processing is essentially batch oriented but can also be real-time for steam processing like for trending analysis with some latency for computation. Data transformation is built as multistep derivations and complexity is kept to minimal within each step.

Bigdata works on un-modeled unstructured data on its commodity servers and can be hooked with Flume an interface for streaming data into Hadoop HDFS. Tools like Sqoop provides bidirectional API interface for data between Hadoop and RDBMS for data interchange. Bigdata can be associated with NoSQL (Not only SQL) namely for Graphs DBMS for social graph analytics (Social media networks). So you get the point. You can create a Hybrid infrastructure by mixing and matching various options to solve and integrate different types of data characteristics and analytics workloads.

 There are also several Commercial products with Storage, MapReduce and Query(SMAQ) based Bigdata solutions in the market to choose from and those can be integrated with RDBMS.

• Greenplum utilizes open source PostgressSQL DBMS runs on clusters of distributed hardware or on EMC with MapReduce functions expressed in Perl or Python.
• Teradata Aster nCluster with SQL MapReduce written in C++, C#, Java, R or Python can intermingle with Teradata.
• Vertica of HP has built- in connectors with Hadoop on their column oriented MPP database
• Neteeza utilizes Cloudera’s Sqoop connector to interface its appliance with Hadoop
• Oracle Bigdata solution integrates Hadoop with Cloudera’s connector and its family of products
• Microsft SQL Server provides Hadoop connector and family of products from HortonWorks  as part of Bigdata solutions

The capabilities of Bigdata Analytical solutions be extended to data at  “rest” or  “in- motion” with appropriate configuration to churn huge quantities of data. In day to day life, we leave data trail every where whether we are surfing the web or making purchases in a super market or data being picked up from sensors on vehicles that we drive or we fly could all be collected, and analyzed to provide timely decisions for quicker customer gratification and business products and better services.

In 1950’s, a 5 MB data storage from Hitachi was of a size of large refrigerator and weighed more than 500 pounds compared today’s SD Ram holding more than 32GB weighing just under 0.5 grams.  Based on Moore’s law trend, we can certainly see more and more foot print of data being captured, stored and processed in today’s world as we enter a new age of “data consumerism” with a deluge of “Data Products” available in the market.

Finally, what is the Bigdata definition. What I have read so far says, “Bigdata is when the size of the data itself becomes part of the problem”.

Federated Data Mart & UDBA’s

Federated Data Mart & UDBA’s

In many businesses today, users do not find all the data that they need in one datamart or EDW, but distributed and spread across different silos and technology platforms. Some of the challenges faced by businesses are;

·         Required data sets are in different platforms and difficult to access and have to navigate on the thin line of permission issues and security concerns

·         Formats of these data sets makes it difficult be it structured, semi-structured or unstructured data

·         Grain of the data differs, often stored at higher level rather than at atomic level

·         Even partially satisfied data sets lack utility due to its relevance, accuracy and timely availability

·         Data sizes are too large to be accommodated in their own databases

Businesses have built UDBA’s (User defined database) stores just to source these datasets to circumvent the challenges even though they are difficult to maintain and manage. These data stores are not scalable besides their reusability is very negligible other than for whom these are build and as a result there are 100’s of different versions being created to satisfy specific business needs.

The issues and concerns are real, and IT managers are aware of it, and are trying to pitch in with solutions, but with varying success. There are companies which are rewriting their existing legacy EDW’s either with a top down or as bottom up approach so business can get to access the atomic grain data in normalized structures as well as providing data in dimensional model for reporting and analytical needs.

 One such solution is to build Federated data model architecture as a bottom up approach across multiple data mart systems. A successful federated model should share and build very high level of common data content (Core), metadata, and metrics in the Core to be accessible by various marts. For example, combining Digital Marketing information with Customer information can certainly provide proper customer, product/service segmentation information and even reliable demographics assessment. A Financial company for their Auto Finance LOB can, by combining Origination, Servicing, Collection and Recovery stack could potential be able build various risk tier propensity models.

Bringing Core data of Employees, Customers, Products, Services, Marketing data assets in a Federated model would be easily accessible by different data marts of LOB (Line of Businesses) with their specific line of data for their operational and analytical reporting needs. Besides the Core, common data sets among the LOB stacks can also be layered in the Core in different data grains with access privileges for building Mashup BI Analytics at enterprise levels. The key requirements of a Federated Data mart are Shared keys, Global Metadata and capable of running distributed queries

One of the true benefits of this design is to the UDBA’s (User Defined Databases). As indicated earlier UDBA’s are decentralized data stores, build to meet the customized business requirements of a LOB. The proliferation of these data stores in enterprises today is due to lack of access to Core data as a single source platform in atomic grain level and multitude of customized individual group business rules and user routines that needs to be applied to the data sets for their reporting requirements.

These UDBA’s come in different forms, shapes, sizes and complexity. There are ones that pull data via ODBC drivers built into applications like Excel and run complicated macros to produce reporting requirements, while some actually have small databases of their own that extracts data from different sources and apply business transformation logic with user routines, functions, stored procedures, triggers to provide business reporting requirements. To top it all, there are 100’s of versions of specific business requirements and these UDBA’s processes are duplicated, cloned to meet each individual group within the LOB’s.

In a typical Financial institution, an Investment Fund manager managing a portfolio (equities, bonds and other investments) has to comply daily, hundreds of rules and regulations as inscribed in the prospectus as its underlying stock/bonds changes its prices in the stock exchanges along with Corporate Actions (CA) that potentially alters underlying portfolio. The Rules, terms for the rules and metadata that defines the decision of the rules all differ from one portfolio to the other. Fund managers have different data source requirements, but most need standard core data sources like Security master data, Stock prices, CA, Fund Index to name a few.

In the absence of a Common Core data sources and Common denomination of core rules construction engine, each Investment Fund manger would have to build UDBA’s under their desk to manage and comply with SEC requirements.

Conclusion: First identify, define and profess the need of common data sources to IT folks to build them in Core of a Federated Data Mart. Second, commoditize and build a scalable user driven rules engine and routines independent of database platform technology as a persistent semantic layer with Global metadata.

In a nutshell, there are many benefits of building a Federated data mart with flexible Core and UDBA is one right candidate up the sleeve.

Quantitative Easing (QE-2) by FED

Breaking the psychological barrier of eleven thousand in the DJIA index this week, surge in commodity markets, weakening of US Dollar globally against major currencies, pressure on global economies to stabilize their currency rates, emerging markets finding easy-money policies in the developed world a challenge, fear of hyperinflation at home are all results of several fiscal measures taken by major economies and notably one that is taken by Federal Reserve (FED) called Quantitative Easing in Short (QE) and another one anticipated in early November, 2010 estimated between $500 billions to $1.2 trillions all in the game to come out of recession and possibly to avoid double dip.

What is Quantitative Easing (QE)
Financial Regulators have several arsenal of tools at their disposal to stimulate and regulate the markets. QE is a monetary policy used by Central Banks to stimulate the economy by increasing the money supply when interest rates are near zero and need another tool to stimulate the economy.

How does it work
Central Banks first credits its own account with money from nothing,  and then purchases bonds from financial institutions, government treasuries, corporate bonds thus infusing money supply in the hands of financial institutions. The new purchases by way of new account deposits,  provides the 'excess reserve' required for them to create new money in a multiplier factor in the fractional reserve banking system (For example 10% reserve requirement means for every $10,000 created by QE can potentially create $100,000).

Central banks controls "Interest rates"  or "Reserve Requirements" to indirectly influence the supply of money in the economy inflation or deflation situations. Some times keeping the interest low may not just be sufficient and that may lead them to resort to Quantitative Easing. "Quantitative" because they specify a number to work which is generally large and "Easing" refers to reduction of pressure on banks.

FED is not the first one which adopted this technique. It was first used by Japan in early 2000 to fight domestic deflation, while Bank of England used this method at the same time as US Federal Reserve in 2008, to stimulate and alleviate its economy. The QE positive impact of flooding with excess liquidity enables commercial banks to continue to provide private lending while still maintaining and retaining excess reserves to avoid any liquidity shortage durig these tough periods. The depressing interest yield on Government bonds, and similar instruments, make it cheaper to raise capital. Besides, Investors could potentially swap their investment strategies favorably towards equities and thus boosting the share prices and thus revive towards increasing the wealth in the economy.

On negative side, the major concern is about hyperinflation with currency loosing its value in the Global market, which is often measured in much shorter intervals like months rather than a year. Some economists also believe that once this scenario sets in,  it is tough to control in the short interval and could take years to stabilize as it happened with Japan. Also, Banks may opt to sit on the excess reserves rather than lend if they think its much more riskier to lend as it happened earlier in 2008. (QE-1 of Fed's credit easing produced $670 billion of excess reserves at banks and financial institutions which works out to around 4.7% of the nation's annualized GDP). We may also see flight of currency to newer destinations like Brazil to earn higher bond yields than US, Europe and Japan because of rock bottom rates.(Brazil this week to stop the inflow of foreign currency doubled its Tax rate on foreign investment in certain Brazilian bonds and increased the amount its Central Banks can intervene in Currency markets).

In the coming months we could see turmoil and pressure on major currencies of  China's Yuan, Israel's Sheqel, South Koreans Won, Brazil's Real, South African Rand, Indonesia Rupiah, Australian Dollar, Swiss Franc against US dollar and worsening tensions among Euro currency and of other 24 Emerging and developing nations with each trying to keep their currencies weak to keep exports competitive because of Quantitative Easing by US.

QE's current impact

• Last week dollar fell 7% against major basket of currencies for total fall of 21% this year.
• Investors are moving into Equity market resulting in 11,000+ DJIA 
• Investors are moving to commodities mainly metal resulting in all time high on Gold & Silver prices creating a bubble scenario.
• Flight of currencies to other countries like South Africa, Indonesia, Brazil, Israel because of near zero interest rates at home.
• QE-1's trillions of dollar has not yet been burned and the likely infusion of another trillion in QE-2 by FED has market up in arms.

The "Currency War" as described by Brazil's finance minister Guido Mantega last week may not be correct description but perhaps if global leaders continue to take unilateral and unconditional actions one may see some potential out of control situations in the foreign exchange market. Remember it was not long ago, we saw major economic issues with European countries of Greek, Spain that needed bailout,  and EU and IMF intervened to pump in  €110b to stabilize their economies and we are continuing to see other countries like Ireland all needing a coordinated help to ease pressure in the world market.

Bond Market: Is there a bubble lurking?

Everyone these days hear about Double dip recession, Euro dollar crisis, Greek debt crisis, Irish instability, Poland debt  and now bond bubble. Lets understand some basis on Bond and why some economists and investors think that we may see similar trends as we saw in the housing bubble..

Availability of easy resources with no constraints on the borrowers coupled with greed on the part of investing companies to package the same debt into exotic and yet risky products and funnel it across up & down the financial industry chain and not realizing the simple truth that if section of the majority of borrowers were to default the cascading impact could swallow global companies like Lehman & Brothers, AIG, Bear Stearns into bankruptcy are yet again seeing the potential seeds being sown for the history in making.

Bond & How it works:
Countries, Corporations, Charities, Cities, Counties all need resources to fund their operations and to beef up the resources they could either take loans from Financial Institutions or issue bonds (its just a loan with contractual terms) in return for an interest to be paid for the term of loan period plus principal in layman terms. But the packaging of contractual terms and features coupled with market conditions and risks can make bonds very complicated.

The contract terms relates to Interest rate (aka Coupon rate), Maturity, Callability, Convertibility, Secured, Principal and are traded just like Equities either for discount or premium depending upon the quality of the Issuer and risk associated with this loan. All bonds quality, risk and returns are rated by major agencies like Moody, Standard & Poor and Fitch based on issuer's background and contract terms to enable customer to pick Bonds that suit his/her preference on risk & rewards. So in general higher the risk, higher is the yield an investor seeks for a given set of contractual terms. Note these ratings change over the life of the bond and that makes it complicated in a dynamic free market. The market pricing of the bond has an inverse relationship to interest rates.

These are also named as debt, fixed income or credit in the financial institutions.The size of the global bond market as of 2009 is estimated at $91 trillions and US leads the bond market with $32 trillions followed by Japan. Majority of the $899 billions of the daily trading volume happens between broker dealers and large institutions.

Individuals can invest in bond funds and an array of indices are available for managing the portfolio and measuring performance like Barclays Aggregate, Citigroup BIG and Merrill Lynch Domestic Master.

Junk Bonds:
These are high risk, low rated debt with a promise of high yields issued by corporations to entice the public to subscribe to their bonds with no guarantees. This type of bond is subject matter for this discussion.

With Interest rates being very low for a good period of time, investors are still not bracing the equity market for the fear of double dip recession but however are seeking steady returns by investing in these high yield junk bonds while ignoring early warnings reminiscent of credit bubble mainly credit worthiness.

In the last nine months, companies have already sold $172 billion in junk bonds as per Dealogic a data provider. Factors for this scenario appears to be,  the declining rate of corporate defaults, corporations in general have a good balance sheet. Also, Interest rate in near future does not seem to rise based on assumption that Fed would unleash with a tidal wave of freshly printed US-dollars to avoid double dip recession and alleviate US Job market. This act by Fed is already put in full play by major countries like China, Japan, Russia & South Korea who have already printed additional their respective sovereign paper currencies in the market to make sure their currencies are not overvalued that would have negative impact on their exports.

More and more Corporation are seeking this route both with strong and not so strong ratings on their bonds. Companies with weak credit ratings are paying just 6.2% points above Treasury's down from 20% points in 2008.  Companies with fewer protection covenants, provisions in their bonds but with high yield are gobbled up by the investors and this echoes similar symptoms we saw at the start of housing bubble where Financial Institutions were more than willing to provide loans even when Customer credit rating was not favourable or downright unworthy.

Investors with their appetite to find niche markets to plough and park their funds for returns hopefully have not yet forgotten the recent fresh memories of the credit bubble which are still lingering for majority of them.

Use Oracle Hints or Not

There has been many a heated discussions about this subject among developers and DBA's and at times taking pretty ugly turn with one section saying "Yes" and others saying "No" as if to emphasize that we only live in Yes & No world. Some of our tuning experts who were brought onsite at a client to resolve performance issues brought their own arsenal of repertoire of "Do's" and "Don'ts" and their first option of tweaking the code was with hints in the queries. This I hilariously call "Injecting of Hints".

Is this a right approach? Did we understand what was root cause before put in a hint. Will this hint support the same level of execution speed should data in the underlying Join tables change? Will this support if the client moves on to different version of the Database? What happens if the hint is deprecated? These are some of the valid questions developers, DBA's should take into account before including one in the queries.

Oracle CBO optimizer is built to provide the best execution plan for a Query based on the statistics that it has at the time of execution. Should the characteristics of the underlying data change drastically and statistics on those changed objects is not captured then it is worth putting a hint around the query to execute it faster. But this type injecting a hint in the query should be temporary fix and not a permanent one.

Does this mean we should not use hints in the first place? No the questions begs for more details on Approaches, Goals, Complexity, Database Type (OLTP, OLAP, MOLAP, ROLAP), Query nature (Adhoc or Tuned), Data Maintenance (One time upload),  any known issues with database, special query requirements that otherwise wouldn't work etc.

Oracle has broadly built different types of hints to address the above and many other scenarios and these hints as a Swiss knife like tool for any issues at Database, Session or for individual query requirements. (Remember Swiss knife is not equivalent to butcher knife if that is what you need in the first place)

• Hints for Optimization Approaches and Goals (All Rows, First Rows etc)
• Hints for Access Paths (Cluster, Full, Index, Hash, RowID etc)
• Hints for Join Orders (Leading, Ordered etc)
• Hints for Join Operations (NL, Merge, Hash etc)
• Hints for Parallel Execution (Parallel, No Parallel etc)
• Additional Hints (Anti Join, No Cache, Materialize etc)

 As I had indicated earlier, Oracle Optimizer relies on Statistics and collection of the same becomes one of the key constraint for CBO to work effectively. This is a Metadata for the Optimizer holding information about size of the table, distribution of data within columns of these tables, cardinality of data among combined columns, function based data columns population etc. Having Stats (Short for Statistics) makes the Optimizer to pick the right execution plan rather than having to pick one based on guess work.

This is one such area, Oracle has been improving upon constantly and in 11g has come up methods of collecting extended statistics. Collection of Stats using the Oracle DBMS_STATS package is results in  manifold performance improvement over init.ora changes even by Oracle Gurus. This package has become the core heart for collection of stats and surprisingly many developers have just skimmed the surface of this package. Besides, using this package one could transfer Statistics from Production server to say Build server to compare the execution plan with varying degrees of data in the build environment.

Oracle recommends using the Automatic Optimizer Statistics method for Stats collection in a appropriate maintainence window for ALL objects in the database for which statistics are absent or stale and this avoids any need for manual intervention. Timing and setting up of the maintenance window is critical so it is run when processing is idle and data for the current or previous period are already loaded to avoid manual re-run for stats.

Oracle provides several flexible features for collection of Stats and user needs to understand these and take several important considerations while using this package.

• Statistics Gathering Using Sampling (Estimate Percent, Auto Sample Size etc)
• Parallel Statistics Gathering (Degree of Parallelism)
• Statistics on Partitioned Objects (Local/Global, Granularity, Incremental etc)
• Column Statistics and Histograms (Understand skewness of data at column level)
• Extended Statistics (Column grouping for accurate selectivity within a group, Expression Stats)
• Determining Stale Statistics (Monitor Staleness on Inserts/Updates/Deletes, Truncation etc)
• User-Defined Statistics (Stats on User defined Indexes, Functions)
• Set Preferences on Stats collection package DBMS_STATS (Cascade, Degree, Granularity etc)

I prefer to take the route of rewriting the queries first rather than 'Injecting Hints'. Majority of the time I try to spot issues with queries in the first or second glance and simplify to make sure my cost and execution plan appears to be good. I try to eliminate multiple round trips to the data from same table, improvise the filter conditions to pick the right amount of data with appropriate filter clauses, avoid predicates which inhibit optimizer from picking indexes, eliminate the generous usage of features like 'distinct', 'group by' 'union', rewrite where possible with Oracle's Analytical functions and so forth. To be honest, writing queries is an Art and mastering it takes time like any other Art.

Secondly I prefer to keep on top of the subject on software upgrades,  improvements and implement these features as I go along on the development side rather than keep singing and humming that "old is gold" or "old was gold" rhymes.

Digitizing Is the Way..

Is it a million dollar question today whether eBooks (digitized version of hard books) will become the dominant force and place the last nail on the coffin named "Demise of Brick & Mortar based companies" such as Border, Barnes & Noble and a host of publishing houses? You bet, going by the last quarter earnings from Amazon for this year are any indication and direction in which this industry is moving. Amazon reported that digitized version of their books outsold hard cover copies for the first time and research folks all agree that this is not a blimp but sure indication of things to follow in the market.

Today (September, 22, 2010) Wall Street Journal carried a title “Web Start-Up Values Soar” providing a picture of overcrowded manufacturers of tablet computers from makers such as Apple (Ipad), Dell (Streak), Samsung (Galaxy Tab), RIM, ACER, Asus (Eeepad), Lenova(LePad), Cisco(Clus) who are potentially trying to catch up this very important paradim shift in consumer habit & behavior.

Also, eReader manufacturers (SONY, Amazon, Barnes & Noble , EBS Technology, Condor Technology Associates, Kogan Technologies, Spring Designs, Kobo Books, iRex Technologies, Wolder Electronics, Onyx International, , iPapyrus, Hanvon) are also playing a big role in accelerating this movement of digital era world wide with different flavors and support level.

One can pick & choose a eReader based on its file format support capabilities  ( .epub,.pdb ,.arg ,.azw ,.djvu ,.html ,.lbr ,.lit ,.mobi ,.opf ,.pdf ,.pdg ,.tr3 ,.txt ,.xeb ,.mp3) preference features such as (Size,Weight, Screen Resolution,Shades,Operating System,Touchscreen,Wifi,Text-to-Speech,Integrated dictionary directory organization capability,Amount of memory Card,Reader user replaceable battery,Web Browser).

This change will alter the dynamic relationship between Authors, Publishers and retailers along the lines on profits in the $40 billion a year retail book industry and digital publishing globally which is around $480 billion. The direct sale to end customer will eliminates the need for shipping, warehousing, returns, inventory of stock and out-off print scenarios resulting in lower cost to a consumer.

Authors now have the ultimate freedom and opportunity to show case their work while avoiding agents and long approval process at reputed publishing houses. Rather websites like http://www.scribd.com/ would provide window for any author to publish their work and earn 80% of the sales when compared to 7 to 15 percent royalty that would otherwise be paid.

There are still many challenges that needs to be addressed before it makes hard covers/books obsolete. These newer gadgets (eReaders/Tablets) need to be cost effective and should be able to download any article from any publishing source for a price. There should be an exchange market made available to exchange one article with the others to proliferate the usage of digital articles. 

Next Trends:
Innovative digitizing mainly for Text book/Fictional ones with appropriate drill downs with 3D features on Computer tablets could be the next wave that customer can't wait to see...