Great news - SPARC T5-8 Delivers Oracle OLAP World Record Performance with near real-time analytic capability

Our SPARC and OLAP development teams have published some fantastic performance figures on their blog: https://blogs.oracle.com/BestPerf/entry/20130326_sparc_t5_8_olap

Using Oracle's SPARC T5-8 server the teams delivered world record query performance with near real-time analytic capability using the Oracle OLAP Perf Version 3 workload running Oracle Database 11g Release 2 on Oracle Solaris 11. The workload uses a set of realistic BI queries that run against an OLAP cube based on a 4 billion row fact table of sales data. The 4 billion rows are partitioned by month spanning 10 years. The combination of the Oracle Database with the Oracle OLAP option running on a SPARC T5-8 server supports live data updates occurring concurrently with minimally impacted user query executions.

This is a great evolution of the development work we have been doing with our in-database OLAP features. The details about "real-time" analytics reminded me of the work with did with BNP Paribas to develop a daily time-series data and analysis solution based around the in-database OLAP features. This customer story was covered in the press and in the January 2007 and April 2008 editions of the OLAP Newsletter:

• http://www.oracle.com/technetwork/database/enterprise-edition/downloads/oracleolapnewsletter-jan07-100293.html
• http://www.oracle.com/technetwork/database/enterprise-edition/oracleolapnewsletter-apr08-086784.html

The BNP Paribas solution stores over 3 billion data values within multiple OLAP cubes across several geographical locations. The solution was based on Oracle 10g utilising Oracle OLAP Option and data access tools for Traders and Quant Analysts. Most of the OLAP calculations, written in DML, have a query time ranging from sub-second to just a few seconds - even for the most complex analytics. The business users access their OLAP cubes through a custom web services front end as well as customised spreadsheet applications. The system supports the retrieval of market data, as well as stored formula, customised functions and complex user generated formula. At the time of the article, BNP Paribas was processing over 10 million rows of data are loaded and continuously aggregating cubes throughout the day.  Overall the system was processing over 20 million data requests per day from over 4000 users.

Cyrus Kapadia, manager of the implementation team at BNP Paribas states, ‘The advantages of Oracle 10g’s scalability, reliability, ease of maintenance, performance and an industry standard analytical query language made it the natural choice to replace our existing system’.

This latest benchmark result from our development team show how we are continuing to develop our OLAP features to deliver groundbreaking performance to support complex data warehouse workloads. In my recent article about in-database analytics I pointed out the importance of having high performance in-database analytics. Therefore, this benchmark is important for OLAP queries but it is also very important in a wider analytical context because it will increase the performance of analytical mashups where OLAP data is mixed and mashed with other key analytical features such as spatial analytics, data mining, time series analytics and/or statistical analytics.  I am hoping to publish a whitepaper on this topic in the next few weeks.

Visit the OLAP Option home page on OTN for more information about Oracle OLAP Option, including how-to videos, presentations, webcasts etc: http://www.oracle.com/technetwork/database/options/olap/index.html

Oracle SQL Developer Data Modeler 3.3 now available

http://www.oracle.com/technetwork/developer-tools/datamodeler/overview/index.html

Attention all data modellers - we are pleased to announce the release of SQL Developer Data Modeler 3.3. This release includes a new search, reports can be generated from search results, extended Excel import and export capabilities and more control and flexibility in generating your DDL. Here are a few links to get you started:

• For downloads see here: http://www.oracle.com/technetwork/developer-tools/datamodeler/downloads/index.html
• For information about new features see here: http://www.oracle.com/technetwork/developer-tools/datamodeler/datamodeler-33-newfeatures-1869156.html
• The release notes are available here: http://www.oracle.com/technetwork/developer-tools/datamodeler/data-modeler-releasenotes33-1869517.html
• The documentation can be found here: http://www.oracle.com/pls/topic/lookup?ctx=E29463_01&id=homepage

For data warehouse data modellers there are some very important new features around logical models, multi-dimensional models and physical models. For example:

• Support for surrogate keys during engineering to relational model which can be set on each entity. 
• More flexible transformation to relational model with mixed engineering strategies based on “engineer” flag and subtypes setting for each entity in the hierarchy
• Export to “Oracle AW” now supports Oracle 11g OLAP
• Support for role playing dimensions in export to Oracle AW.
• Level descriptive attributes can be created without mapping to attribute in logical model.
• Multidimensional model can be bound directly to relational model. 
• Support EDITIONING option on views, and support for invisible indexes in Oracle 11g physical model.

Lots of great features that will make life a lot easier for data warehouse teams.

Predictive Analytics Is Red Hot so get reading!

According to an article by Forrester analyst Mike Gualtieri, predictive analytics is a red hot topic! There are many examples across a broad range of industries where predictive analytics is driving better decisions:

• President Obama's campaign team used sophisticated uplift modeling to target and influence swing voters.
• Telecom firms that use predictive analytics to help prevent customer churn. 
• Police departments that use it to reduce crime.

The list goes on and on and on. Let us quickly look at two of these uses cases because we have Oracle customers who have already implemented these types of solutions:

1) How telcos are using predictive analytics to reduce churn 

a) Telecom Italia Lab - Churn is a critical problem in the telecommunications industry because losing customers is a drain on resources. Finding new customers to replace the ones that have defected to a competitor is an expensive process. Consequently companies go to great lengths to reduce the churn of their customer base because it saves them money.  We have a number of case studies in this area and the best one for demonstrating just how natural it is for Oracle Data Mining to analyze a data warehouse star schema has been documented in a whitepaper and a number of related blog posts. These articles explain the business problem and provide an overview of this particular churn model was developed:

• Oracle Data Mining 11g Release 2 Mining Star Schemas A Telco Churn Case Study whitepaper, click here
• Blog posts on the Data Mining blog are here 

To help telco companies speed up the process of creating and deploying predictive analytics we have incorporated this type of advanced analytics directly into our Oracle Communications Data Model by delivering pre-built and automated churn and sentiment analysis. There is an excellent blog post that on this topic on the Oracle Data Mining blog, see here.

b) Turkcell - The presentation linked below provides an overview of how Turkcell, the leading GSM operator in Turke, developed a churn prediction model using Oracle Advanced Analytics. The presentation from OpenWorld 2010 starts with an overview of the churn prediction problem and the existing data mining system in Turkcell so that the participants have some knowledge about the churn problem and the existing solution to this problem. It the looks at the advantages of in-database data mining and Oracle Data Mining followed by a review of the prepaid churn model which was created by using Oracle Data Mining. The last section covers the results and the overall performance. The presentation (PPT file) can be download via this link: http://ndenizh.files.wordpress.com/2010/09/prepaid_churn_model.pptx or you can view the presentation via Google Quick View.

2) Crime fighting with Oracle Spatial and Oracle Advanced Analytics

Italian Government fights crime with technology using a new Business and Location Intelligence system—The Department of Public Security of the Ministry of the Interior to combat crime through the SIGR (Integrated System for the Georeferencing of Crimes) platform, developed by Iconsulting using Oracle Business Intelligence and Geospatial technology.

• Press release: Italian Government fights crime with technology using a new Business and Location Intelligence system, click here.
• Presentation: Location Intelligence and Spatial What-If Analysis - Government: A New Way of Fighting Petty Crime, click here.

How to get started with predictive analytics

While many Oracle customers are already working with predictive analytics according to Forrester it appears that many businesses do not really understand the what, why, and how of predictive analytics. If you want to understand more about predictive analytics and what it can do for your business then Mike has produced a recommended reading list to help get you started on your journey to deep insight. The list is available here: http://blogs.forrester.com/mike_gualtieri/13-03-20-intro_to_predictive_analytics_reading_list.

Where to get more information about Oracle's in-database predictive analytics

For more information about how our customers are already using Advanced Analytics please follow this link to "Oracle Data Mining Customer Successes" page on OTN. 

For more information about Oracle's Advanced Analytics (Data Mining and R Enterprise) visit the home page on OTN: http://www.oracle.com/technetwork/database/options/advanced-analytics/index.html

For more information about Oracle Spatial visit the home page on OTN: http://www.oracle.com/technetwork/database-options/spatialandgraph/overview/spatialfeatures-1902020.html

Don't forget to join our LinkedIn groups:

• Oracle Data Warehousing: http://www.linkedin.com/groups?gid=2129659
• Oracle Big Data: http://www.linkedin.com/groups/BigData-Oracle-4124374

Personal data usage: what your car really says about you

Stumbled across this article on the motoring page of the Guardian newspaper.

Cars will soon be so linked into wireless networks they will be like giant rolling smartphones. Ford has already started to integrate web services such as Spotify into their dashboard systems and this looks like the first step to introducing a real-time data collection service to your car. This new vision of in-car big data goes far beyond the current engine management monitoring that garages use when you take your car in for servicing.

A few examples of this brave new world:

- Low on fuel? Soon a petrol station app may know before you do.
- Tyres need rotating? Your car may wirelessly alert your dealership when it's time.
- Ready for a lunch break? Your car can make a reasonable guess based on the hour.

Are these realistic? Well, according to the article "…more than 60% of vehicles worldwide will be connected directly to the internet by 2017, up from 11% last year, predicts ABI Research. In North America and Europe, that is likely to reach 80%.". That means a lot of new data streams to be harvested by car dealers, manufacturers, insurance companies, finance companies etc. Not forgetting law enforcement agencies! Our every movement within our car could be tracked and interpreted by all sorts of agencies. 

As always the issue is going to be how has access to all this information because at the moment few laws exist to govern who can see and use such data.

The article is here: http://www.guardian.co.uk/technology/2013/mar/12/cars-internet-data-privacy-debate.

Gartner Positions Oracle in Leaders Quadrant for Data Warehouse Database Management Systems

The latest Data Warehouse Magic Quadrant is out and we (Oracle) has been positioned in the leaders quadrant. How do you get into the leaders quadrant? This is how Gartner defines a leader:

vendors in the Leaders quadrant, “demonstrate the greatest support for data warehouses of all sizes, with large numbers of concurrent users and management of mixed data warehousing workloads. These vendors lead in data warehousing by consistently demonstrating customer satisfaction and strong support, as well as longevity in the data warehouse DBMS market, with strong hardware alliances. 

Here is a great quote from my boss, Cetin Ozbutun, vice president, Data Warehousing and Big Data Technologies:

“We believe Oracle's position in Gartner's Magic Quadrant for Data Warehouse Database Management Systems recognizes the outstanding performance of the Oracle Exadata Database Machine, as well as Oracle’s delivery of innovative, integrated solutions for big data with the Big Data Appliance and Big Data Connectors. Oracle's unique ability to execute enables our customers to stay ahead of their most demanding data warehouse requirements."

The Oracle press release for this announcement is here: http://www.oracle.com/us/corporate/press/1905789

One thing to remember when looking at these market reports from Gartner (MQ) and Forrester (Wave) is that, I'm my opinion, their view of the "data warehouse" market is too narrowly defined. Both companies focus exclusively on a specific capabilities of the database software and general market vision of each data warehouse vendor. In real life, a data warehouse project requires a much wider set of features such as: analytics, data loading, data quality, security, content management and data visualisation. Each of these areas gets treated in isolation by Gartner and Forrester because this is how they make their money!

The good news is that if you are planning a data warehouse project then Oracle is in the leaders segments across all the relevant areas that make up the real world view of data warehousing, as shown below:

Oracle truly is a leader!

OTN Developer Day - Free Oracle Big Data Workshop

If you want to get hands-on time with Oracle's big data technology stack then we are rolling out, in collaboration with OTN, a new complementary one-day hands-on workshop - yes it really is FREE! The date will is Wednesday, February 20, 2013 at Oracle's HQ in Redwood Shores, California.

The workshop will be based around the MoviePlex demonstration application that was first shown at last year's OpenWorld and you will get hands-on time with the source code and we will show you how to develop some of the key features of this application.

What is Oracle MoviePlex?
The Oracle MoviePlex is a fictional on-line movie streaming company and like many other on-line stores, they needed a cost effective approach to tackle their “big data” challenges. They recently implemented Oracle’s Big Data Platform to better manage their business, identify key opportunities and enhance customer satisfaction. The key challenge for Oracle Movieplex is how to manage and process massive volumes of unstructured data flowing in to their environment.

The business users at Oracle MoviePlex want to use their big data platform to help them explore some new business opportunities:

• Make the right movie offers at the right time?
• Better understand the viewing trends of various customer segments?
• Optimize marketing spend by targeting customers with optimal promotional offers?
• Minimize infrastructure spend by understanding bandwidth usage over time?

Below is a slideshow that will help provide some background and more general information about this application: (Thanks to Marty Gubar, Director of Product Management Big Data, for overseeing the development of the MoviePlex application, putting together the workshop and for creating the slides below)


What will you learn at this workshop?
The overall objective for this workshop is to show how you can develop a low latency, personalized recommendations environment that leverages Oracle's advanced analytic capabilities. During the workshop you will learn from technical experts who will demonstrate the following core technical topics:

• Write MapReduce on Oracle’s Big Data Platform
• Manage a Big Data environment
• Access Oracle NoSQL Database
• Manage Oracle NoSQL DB Cluster
• Use data from a Hadoop Cluster with Oracle
• Develop analytics on big data

Don't delay - register today to learn these key big data skills which you can immediately put to use within your organization.Space is limited so do not delay clicking on this link [here] to register.

2013 - The year of In-Database Analytics!

Many of our customers spent 2012 kick-starting big data projects. Based on the number of analyst reports, news articles and general chatter on the web I think it is fair to say that 2012 was the Year of Big Data. To help our customers plan and coordinate their big data projects we developed a very simple high level five step workflow for big data (Stream-Acquire-Organize-Analyze-Decide) as shown here:

 

Many of the projects that our customers stated in 2012 have at least completed the first three steps (with some having completed stages 4 and 5 and now looping back to stage 1 to kick-start the next round of data enrichment and knowledge discovery):

1) STREAM - all the relevant data streams have been identified and the APIs coded to collect the data on a regular basis

2) ACQUIRE - these data streams are increasing landing on our Big Data Appliance and that is great news!

3) ORGANIZE - many project teams have completed the first round of light-touch transformations and made their new data sets available for analysis. Typically this is being done using R, Java and our Big Data Connectors

We can see that during 2012 an awful lot was achieved. For 2013 we are going to see everyone focusing on the ANALYZE phase of the workflow and my prediction for 2013 is that this will be the year of "Big Data Analytics". The good news for Oracle customers is that you do not have to wait until 2015 to analyse your Hadoop data.

The most important thing is that customers must learn the key lesson from those old days of data warehousing There are two ways to manage ANALYZE the analyse phase:

1. Use subject-specific specialised analytic engines and take the data to the analysis
2. Use an analytically rich database and take the analysis to the data

1) Taking the data to the analysis

The danger for many organizations is that in delivering this type solution to the business they simply create “analytic silos” that are designed to resolve specific business problems. These analytic silos create unnecessary cost, they increase complexity and cause increased levels of data movement across the network as data is pushed in and results pulled out from each silo. This continuous movement of data creates time delays in being able to view results because business questions usually require multiple levels of analysis using lots of different types of analytical functions. The longer it takes to arrive at an answer the greater the chance you will make the wrong business decisions or completely miss out on an significant opportunity.

As before each of these specialised platforms have their own proprietary engine, tools and languages and this makes it difficult to kick-start and grow a project. Broadening the analysis can be complex and in many cases impossible. There has to be a better way to analyse data!

2) Taking the analysis to the data with In-Database Analytics

The end game for all big data projects (whether they realise it or not when the project first starts out) is to deliver an environment that offers the following: a broad range of analytical tools that can analyse all types of data and be accessed using existing skills and tools.  What you need is a single place to run your analysis against all your data so your business users can easily apply layer after layer of analysis. This ensures that the process of transforming data into insight delivers the right data at the right time, to the right person and on the right device. 

To get the most from your data streams it is important that your analysis has the ability to incorporate the broadest range of analytical functions and that these can be applied to all your data steams with results delivered in a real-time. The ANALZYZE phase in our workflow leverages the Oracle Database. The Oracle Database started with a rich set of built-in SQL analytic features that allowed developers to process data directly inside the database using standard SQL syntax rather than having to move data to a separate platform, process it using a proprietary language and then return the results to the database.  Oracle's approach to analytics is to take the "analysis to the data", i.e. Oracle provides in-database analytics. In-database analytics offers some important advantages over the three other options outlined above:

• Reduced latency - data can be analyzed in-place
• Reduced risk – a single set of data security policies can be applied across all types of analysis
• Increased reusability – all data types are accessible via SQL making it very easy to re-use analysis and analytical workflows across many tools: ETL, business intelligence reports and dashboards and operational applications

With each release of the database the types of data supported by the in-database analytics and the types of SQL analytics available to process that data continues to expand. The latest release of the Oracle Database covers a wide variety of data types and structures, including: numerical, text, images, videos, voice, XML, network, spatial, semantic and multi-dimensional. There is a wide a range of analytical features to support these data types that allow users to explore and layer their analysis. The picture below provides an overview of the rich analytical ecosystem within the Oracle Database:

Why is in-database analytics so important?

By making all the data available from inside the data warehouse and moving the analytics to the data Oracle is able to provide a wide range of analytical functions and features and ensure data governance and security is enforced at every stage of the analytic process, while at the same time providing timely delivery of analytical results. Oracle’s single integrated analytic platform offers many advantages.

To help explain those advantages  I am busy writing a whitepaper on precisely this topic and it should be released shortly. The whitepaper will explore the points I have highlighted above and will explain how Oracle can help you transform all your data into real actionable insight. When you change the way you analyse your data by moving the algorithms to the data rather than the traditional approach of extracting the data and moving it to the algorithms for analysis, it CHANGES EVERYTHING, including your business. You can know more about your products, your operations and your customers by having that insight delivered at the right time, to the right person and on the right device.

As part of the paper I will outline how some of Oracle’s industry leading customers are already using our in-database analytics today to help them understand more about their products, operations and customers and how many of them are layering different types of analysis one on top of the other, something that we believe is only possible with Oracle: if you want to enhance spatial analytics with data mining or drive multi-dimensional analysis using data mining or use semantics to help manage your data warehouse metadata then Oracle is the answer. There is a better way to do analysis in 2013

 

Which in-database analytical features do you use?

I have started a poll on LinkedIn to capture information about which in-database features people use the most. If you want to vote in the poll then click [here] - if you are not a member of the group then click the "request to join" button and I will approve you as soon as I get the email alert from LinkedIn

 As soon as the whitepaper is ready I will post a note on my blog.