Image courtesy of pixabay.com

In my previous post I reviewed some reasons why people seem reluctant to accept approximate results as being correct and useful. The general consensus is that approximate results are wrong which is very strange when you consider how often we interact with approximations as part of our everyday life.

Most of the use cases in my first post on this topic covered situations where distinct counts were the primary goal - how many click throughs did an advert generate, how many unique sessions were recorded for a web site etc. The use cases that I outlined provided some very good reasons for using approximations of distinct counts. As we move forward into the era of Analytics-of-Things the use of approximations in queries will expand and this approach to processing data will become an accepted part of our analytical workflows.

To support Analytics-of-Things, Database 12c Release 12.2 includes even more approximate functions. In this release we have added approximations for median and percentile computations and support for aggregating approximate results (counts, median and percentiles).

What is a median and percentile?

A quick refresher course….according to wikipedia a percentile is:

a measure used in statistics indicating the value below which a given percentage of observations in a group of observations fall. For example, the 20th percentile is the value (or score) below which 20 percent of the observations may be found.

Percentiles are prefect for locating outliers in your data set. In the vast majority of cases you can start with the assumption that a data set exhibits a normal distribution. Therefore if you take the data around the 0.13th and 99.87th percentiles (i.e. outside 3 standard deviations from the mean) then you get the anomalies. Percentiles are great for allowing you to quickly eyeball the distribution of a data set so that you can check for skew or bimodalities etc. Probably, the most common use case is around monitoring service levels where these anomalies are the values of most interest.  

On the other hand, a median is:

the number separating the higher half of a data sample, a population, or a probability distribution, from the lower half. 

Why would you use median rather than the mean? In other words, what are the use cases that require median? Median is great at removing the impact of outliers because the data is sorted and then the middle value is extracted. The average is susceptible to be skewed by outliers. A great use case for median is in resource planning. If you want to know how many staff you should assign to manage your web-store application you might create a metric based on number of sessions during the year. With a web-store the number of sessions will peak around key dates such as July 4th and Thanksgiving. Calculating the average number of sessions over the year will be skewed by these two dates and you will probably end-up with too many staff looking after your application. Using the median removes these two spikes and will return a more realistic figure for the number of sessions per day during the year.

But before you start to consider where, when, how or even if you want to consider using approximate calculations you need to step back for a moment and think about the accuracy of your existing calculations, which I am guessing you think are 100% accurate!

Is your data accurate anyway?

Most business users work on the assumption that the data set they are using is actually 100% accurate and for the vast majority of operational sources flowing into the data warehouse this is probably true although there will always parentless dimension values and in some cases “Other” bucket dimension members to create some semblance of accuracy.

As we start to explore big data related sources pulled from untrusted external sources and IoT sensor streams, which typically are inherently “noisy”, then the level of “accuracy” within the data warehouse starts to become a range rather than a single specific value.

Let’s quickly explore the three key ways that noise gets incorporated into data sets:

1) Human errors

Human input errors: probably the most obvious. It affects both and internal and external sources that rely on human input or interpretation of manually prepared data. Free format fields on forms create all sorts of problems because the answers need to be interpreted. Good examples are insurance claim forms, satisfaction surveys, crime reports, sales returns forms etc 

2) Coding errors

ETL errors: Just about every data source feeding a data warehouse goes through some sort of ETL process. Whilst this is sort of linked to the first group of errors it does fall into this group simply because of the number of steps involved in most ETL jobs. There are some many places where errors can be introduced

Rounding and conversion errors: When an ETL job takes source data, converts it and then aggregates it before pushing it into the warehouse it will always be difficult to back trace the aggregated numbers down to the source data because of inherent rounding errors. When dealing with currency exchange rates it can be a little difficult to tie-back source data in one currency to the aggregated data in the common currency dues to tiny rounding errors.

3) Data Errors

Missing data points:  Data always get lost in translation somewhere down the line or is simply just out of date. In many cases this is the biggest source of errors. For example, one bank recently put together a marketing campaign to stop customer churn. Before they launched they campaign one of their data scientists did some deeper analysis and discovered that the training data for the model included customers who were getting divorced and this was being flagged as a lost customer. Including this group ended up skewing the results. The data about changes to marital status was not being pushed through fast enough to the data warehouse.

Meaningless or distracting data points: with the growth in interest in the area of IoT it is likely that this type of “noise” will become more prevalent in data sets. Sensor data is rarely 100% accurate mainly because in many cases it does not need to deliver that level of accuracy. The volume of data being sent from the sensor will allow you to easily remove or flag meaningless or distracting data. With weblogs it is relatively easy to ignore click-events where a user clicks on an incorrect link and immediately clicks the back-button.

In other words, in many situations, getting precise answers is nothing but an illusion: even when you process your entire data, the answer is still an approximate one. So why not use approximation to your computational advantage and in a way where the trade off between accuracy and efficiency is controlled by you? 

Use cases for these new features

There are a lot of really good use cases for these types of approximations but here are my two personal favorites:

Hypothesis testing— a good example of this is A/B testing which is most commonly used in conjunction with website design and ads design to select the page design or ad that generates the best response. With this type of analysis it is not vital that you have accurate, precise values .What is needed is the ability to reliably compare results and approximations are good normally enough.

Ranking — How does your ISP calculate your monthly usage so they can bill you fairly for your usage? They use a percentile calculation where they will remove the top 5% - 2%, of your bandwidth peaks. and then use that information to calculate your bill. By using data below the 95th-98th percentile they can ignore the infrequent peaks when say your are downloading the lasted update to your Android or iOS device. Again, having precise numbers for this percentile cut-off is not really necessary. A good enough approximation of  the 95th percentile is usually going to be sufficient because it implies that approximately 95% of the time, your usage is below the data volume identified around that percentile. An conversely the remaining 5% of the time, your usage creeps above that amount. 

Of course all the use cases that we considered for distinct counts in the first posts are also valid:

Discovery analytics: data analysts often slice and dice their dataset in their quest for interesting trends, correlations or outliers. If your application falls into this type of explorative analytics, getting an approximate answer within a second is much better compared to waiting twenty minutes for an exact answer. In fact, research on human-computer interaction has shown that, to keep business users engaged and productive, the response times for queries must be below 10 seconds. In particular, if the user has to wait for the answer to their query for more than a couple of seconds then their level of analytical thinking can be seriously impaired.

Market testing: most common use case for market testing is around serving ads on websites. This is where two variants of a specific ad (each with a group of slightly different attributes such as animations or colour schemes) are served up to visitors during a session. The objective is to measure which version generates a higher conversion rate (i.e. more click-throughs). The analytics requires counting the number of clicks per ad with respect to the number of times each ad was displayed. Using an approximation of the number of click-throughs is perfectly acceptable. This is similar to the crowd-counting problem where it is not really necessary to report exactly how many people joined a rally or turned up to an event.

Root cause analysis:  contrary to perceived wisdom, this can in fact be accomplished using approximations. Typically RCA follows a workflow model where results from one query trigger another query, which in turn triggers another related query. Approximations are used to speed up that the decision as to whether or not to continue with a specific line of analysis. Of course you need to incorporate the likelihood of edge cases within your thinking process because there is the danger that the edge values will get lost within the general hashing process.

however, in these examples we usually end up merging or blending the first two use cases with the three above to gain a deeper level of insight so now let’s look at the new approximate statistical functions introduced in Database 12.2

Approximate median and percentile

With Database 12c Release 2 we have added two new approximate functions:

APPROX_PERCENTILE(%_number [DETERMINISTIC], [ERROR_RATE|CONFIDENCE]) WITHIN GROUP (ORDER BY expr [ DESC | ASC ])

This function takes three input arguments. The first argument is numeric type ranging from 0% to 100%. The second parameter is optional. If ‘DETERMINISTIC’ argument is provided, it means user requires deterministic results. If it is not provided, it means deterministic results are not mandatory.  The input expression for the function is derived from the expr in the ORDER BY clause.  

The approx_median function has the following syntax:

APPROX_MEDIAN(expr [DETERMINISTIC], [ERROR_RATE|CONFIDENCE]

)  

We can use these functions separately or together as shown here using the SH schema:

SELECT
  calendar_year,
  APPROX_PERCENTILE(0.25) WITHIN GROUP (ORDER BY amount_sold ASC) as "p-0.25",
  TRUNC(APPROX_PERCENTILE(0.25, 'ERROR_RATE') WITHIN GROUP (ORDER BY amount_sold ASC),2) as "p-0.25-er",
  TRUNC(APPROX_PERCENTILE(0.25, 'CONFIDENCE') WITHIN GROUP (ORDER BY amount_sold ASC),2) as "p-0.25-ci",
  APPROX_MEDIAN(amount_sold deterministic) as "p-0.50",
  TRUNC(APPROX_MEDIAN(amount_sold deterministic, 'ERROR_RATE'),2) as "p-0.50-er",
  TRUNC(APPROX_MEDIAN(amount_sold deterministic, 'CONFIDENCE'),2) as "p-0.50-ci",
  APPROX_PERCENTILE(0.75 deterministic) WITHIN GROUP (ORDER BY amount_sold ASC) as "p-0.75",
  TRUNC(APPROX_PERCENTILE(0.75, 'ERROR_RATE') WITHIN GROUP (ORDER BY amount_sold ASC),2) as "p-0.75-er",
  TRUNC(APPROX_PERCENTILE(0.75, 'CONFIDENCE') WITHIN GROUP (ORDER BY amount_sold ASC),2) as "p-0.75-ci"
FROM sales s, times t
WHERE s.time_id = t.time_id
GROUP BY calendar_year
ORDER BY calendar_year

The results from the above query are shown below

Note that for the APPROX_MEDIAN function I have included the keyword “DETERMINISTIC”.  What does this actually mean? 

Due to the nature of computing approximate percentiles and medians it is not possible to provide a specific and constant value for the error rate or the confidence interval. However, when we have used a large scale real world customer data set (manufacturing use case) we saw an error range of around 0.1 - 1.0%. Therefore, in broad general terms, accuracy will not be a major concern.

Error rates and confidence intervals

 How closely an approximate answers matches the precise answer is gauged by two important statistics:

• margin of error
• confidence level.

These two pieces of information tell us how well the approximation represents the precise value. For example, a result may have a margin of error of plus or minus 3 percent at a 95 percent level of confidence. These terms simply mean that if the analysis were conducted 100 times, the data would be within a certain number of percentage points above or below the percentage reported in 95 of the 100 runs.

In other words, Company X surveys customers and finds that 50 percent of the respondents say its customer service is “very good.” The confidence level is cited as 95 percent plus or minus 3 percent. This information means that if the survey were conducted 100 times, the percentage who say service is “very good” will range between 47% and 53% most (95%) of the time (for more information see here: https://www.isixsigma.com/tools-templates/sampling-data/margin-error-and-confidence-levels-made-simple/).

Please note that if you search for more information about error rates and confidence levels then a lot of results will talk about sample size and working back from typical or expected error rates and confidence levels to determine the sample size needed. With approximate query process we do not sample the source data. We always read all the source values, there is no sampling!

Performance - how much faster is an approximate result?

 As a test against a real world schema we took a simple query from the customer that computed a number of different median calculations:

SELECT count(*) FROM (SELECT /*+ NO_GBY_PUSHDOWN */ b15, median(b4000), median(b776), median(e), median(f), median(n), median(z) FROM mdv group by b15);

As you can see from the real-time monitoring page, the query accessed 105 million rows and the calculations generated 11GB of temp. That’s a lot of data for one query to spill to disk!

Now if we convert the above query to use the approx_median function and rerun the query we can see below that we get a very different levels of resource usage: 

Looking closely at the resource usage you can see that the query is 13x faster, uses considerably less memory (830Kb vs 1GB) but most importantly there is no usage of temp: 

 Summary

One of the most important take-aways from this post relates to the fact that we always read all the source data. The approximate functions in Database 12c Release 2 do not using sampling as a way to increase performance. These new features are significantly faster and use fewer resources which means more resources are available for other queries - allowing you to do more with the same level of resources.

Technorati Tags: Analytics, Big Data, Data Warehousing, Database 12c, Oracle Database 12c, SQL, Statistics

Image courtesy of wikipedia

This post is the result of reviewing a post on stackoverflow.com: http://stackoverflow.com/questions/41649178/getting-error-ora-00918-when-using-match-recognize. Here is my version of the code which includes the same issues/errors as the original, however, I am using the TICKER schema table that I always use for the tutorials that I post on liveSQL :

SELECT symbol, tstamp, price 
FROM ticker 
MATCH_RECOGNIZE(
 PARTITION BY symbol
 ORDER BY symbol, tstamp
 MEASURES
   a.symbol AS a_symbol,
   a.tstamp AS a_date,
   a.price AS a_price
 ALL ROWS PER MATCH
 PATTERN(A B*)
 DEFINE
   B AS (price < PREV(price))
);

The above example will not run because of the following error:

ORA-00918: column ambiguously defined

00918. 00000 - "column ambiguously defined"
*Cause: 
*Action:
Error at Line: 1 Column: 8

So what is wrong with our code? As MATHGUY pointed out in his reply on stackoverflow.com - quite a lot actually! Let’s start by differentiating between “won’t run” and “wrong”. The ORA-918 error is easy to resolve if you stare long enough at the ORDER BY clause! It’s pointless to include the SYMBOL column as the partition by key and the order by key. If we change the ORDER BY clause as shown here then the code will run:

SELECT symbol, tstamp, price 
FROM ticker 
MATCH_RECOGNIZE(
 PARTITION BY symbol
 ORDER BY tstamp
 MEASURES
   a.symbol AS a_symbol,
   a.tstamp AS a_date,
   a.price AS a_price
 ALL ROWS PER MATCH
 PATTERN(A B*)
 DEFINE
   B AS (price < PREV(price))
); 

which returns the following resultset (all 60 rows from our source ticker table):

No MEASURE clause

Okay so our code is running now what? If you look at the output you will notice that it contains the same rows and columns as the source table. What happens if we omit the MEASURE clause? Well it’s optional so the code should still run…

SELECT symbol, tstamp, price 
FROM ticker 
MATCH_RECOGNIZE(
 PARTITION BY symbol
 ORDER BY tstamp
 ALL ROWS PER MATCH
 PATTERN(A B*)
 DEFINE
   B AS (price < PREV(price))
); 

and sure enough we get the same resultset (all 60 rows from our source ticker table):

ALL ROWS PER MATCH vs. ONE ROW

from the above we can assume that there is no need to list the source columns from your input table in the MEASURE clause because they are automatically included in the output. BUT this is ONLY true when you use ALL ROWS PER MATCH. If we change the output control to ONE ROW PER MATCH:

SELECT symbol, tstamp, price
FROM ticker
MATCH_RECOGNIZE(
PARTITION BY symbol
ORDER BY tstamp
ONE ROW PER MATCH
PATTERN(A B*)
DEFINE
B AS (price < PREV(price))
); 

you will now get an error:

ORA-00904: "PRICE": invalid identifier

00904. 00000 - "%s: invalid identifier"
*Cause:
*Action:
Error at Line: 19 Column: 24

 

because when using ONE ROW PER MATCH the only columns that are automatically returned are those listed in the PARTITION BY clause. Therefore, we need to use either “SELECT * FROM …..” or “SELECT symbol FROM…” to get a working version of our code. Using “SELECT * FROM…” as follows:

SELECT * 
FROM ticker 
MATCH_RECOGNIZE(
 PARTITION BY symbol
 ORDER BY tstamp
 ONE ROW PER MATCH
 PATTERN(A B*)
 DEFINE
   B AS (price < PREV(price))
); 

actually returns only one column (symbol) from the ticker table:

So what should we include in the MEASURE clause?

Based on the query that was in the original post I think the following syntax would make it easier to understand what is happening within the pattern matching process and provide useful information about the data that matches the pattern:

SELECT symbol, tstamp, price, first_date, last_date, first_price, last_price, m_n, classi 
FROM ticker 
MATCH_RECOGNIZE(
 PARTITION BY symbol
 ORDER BY tstamp
 MEASURES
FIRST(b.tstamp) AS first_date,
LAST(b.tstamp) AS last_date,
FIRST(b.price) AS first_price,
LAST(b.price) AS last_price,
match_number() AS m_n,
 classifier() AS classi
 ALL ROWS PER MATCH
 PATTERN(A B*)
 DEFINE
   B AS (price < PREV(price))
); 

which gives us the following output based on ALL ROWS PER MATCH:

and if we want to switch to using ONE ROW PER MATCH then we need to remove references to the columns tstamp and price and replace them with references to the pattern variable specific versions, or we can just remove the references all together. In this case as we only have two pattern variables we can NVL the references to return the required data:

SELECT symbol, o_tstamp, o_price, first_date, last_date, first_price, last_price, m_n, classi 
FROM ticker 
MATCH_RECOGNIZE(
 PARTITION BY symbol
 ORDER BY tstamp
 MEASURES
 nvl(a.tstamp, b.tstamp) as o_tstamp,
 nvl(a.price, b.price) as o_price,
   FIRST(b.tstamp) as first_date,
   LAST(b.tstamp) as last_date,
   FIRST(b.price) as first_price,
   LAST(b.price) as last_price,
   match_number() as m_n,
   classifier() as classi
 ONE ROW PER MATCH
 PATTERN(A B*)
 DEFINE
   B AS (price < PREV(price))
);

which generates roughly the same output as the previous statement except that this time we are referencing specific instances of tstamp and price.

Summary

What have we learned:

Point 1: Check your PARTITION BY and ORDER BY clauses to ensure they make sense!

Point 2: there is no need to list the source columns from your input table in the MEASURE clause because they are automatically included BUT ONLY when you use ALL ROWS PER MATCH.

Point 3: Decide on your output method and match the columns listed in the SELECT clause with those returned by either ALL ROWS PER MATCH or ONE ROW PER MATCH.

Point 4: Always a good idea to check your pattern is being applied correctly by using the built-in  MATCH_NUMBER and CLASSIFIER() measures.

For more general information about how to get started with MATCH_RECOGNIZE follow these links to previous blog posts:

• SQL Pattern Matching Deep Dive - Part 1, The basics
• SQL Pattern Matching Deep Dive - Part 2, using MATCH_NUMBER() and CLASSIFIER()

and checkout the growing library of tutorials on liveSQL.oracle.com. Hope this helps to throw some light on the workings of the MEASURE clause within MATCH_RECOGNIZE.

Technorati Tags: Analytics, Database 12c, Pattern Matching, SQL, SQL Analytics

Oracle RDBMS 11gR2 introduced the LISTAGG function for working with string values. It can be used to aggregate values from groups of rows and return a concatenated string where the values are typically separated by a comma or semi-colon - you can determine this yourself within the code by supplying your own separator symbol.

Based on the number of posts across various forums and blogs, it is widely used by developers. However, there is one key issue that has been highlighted by many people: when using LISTAGG on data sets that contain very large strings it is possible to create a list that is too long. This causes the following overflow error to be generated:

ORA-01489: result of string concatenation is too long.

Rather annoyingly for developers and DBAs, it is very difficult to determine ahead of time if the concatenation of the values within the specified LISTAGG measure_expr will cause an ORA-01489 error. Many people have posted workarounds to resolve this problem - including myself. Probably the most elegant and simple solution has been to use the 12c MATCH_RECOGNIZE feature, however, this required use of 12c Release 1 which was not always available to all DBAs and/or developers.

If you want to replicate the problem and you have access to the sample SH schema then try executing this query:

SELECT
g.country_region,
LISTAGG(c.cust_first_name||''||c.cust_last_name, ',') WITHIN GROUP (ORDER BY c.country_id) AS Customer
FROM customers c, countries g
WHERE g.country_id = c.country_id
GROUP BY country_region
ORDER BY country_region;

All the samples in this post use our sample SH schema. Once we release the on-premise version of 12.2 you will be able to download the Examples file for your platform from the database home page on OTN. I do have a tutorial ready-to-go on LiveSQL, however, there is currently a technical issue with using very very very long strings - essentially, running my LISTAGG workshop generates a ‘Data value out of range’ error so as soon as it’s fixed I will update this blog post with a link to the tutorial.

What have we changed in 12.2?

One way of resolving ORA-01489 errors is to simply increase the size of VARCHAR2 objects.

Larger object sizes

The size limit for VARCHAR2 objects is determined by the database parameter MAX_STRING_SIZE. You can check the setting in your database using the following command:

show parameter MAX_STRING_SIZE

in my demo environment this returns the following:

NAME            TYPE   VALUE
--------------- ------ --------
max_string_size string STANDARD

Prior to Oracle RDBMS 12.1.0.2 the upper limit for VARCHAR2 was 4K. With Oracle RDBMS 12.1.0.2 this limit has been raised to 32K. This increase may solve a lot of issues but it does require a change to the database parameter MAX_STRING_SIZE. By setting MAX_STRING_SIZE = EXTENDED this enables the new 32767 byte limit.

ALTER SYSTEM SET max_string_size=extended SCOPE= SPFILE;

However, with the increasing interest in big data sources it is clear that there is still considerable potential for ORA-01489 errors as you use the LISTAGG feature within queries against extremely large data sets.

What is needed is a richer syntax within the LISTAGG function and this has now been implemented as part of Database 12c Release 2.

Better list management

With 12.2 we have made it easier to manage to lists that are likely to generate an error because they are too long. There are a whole series of new keywords that can be used:

• ON OVERFLOW ERROR
• ON OVERFLOW TRUNCATE
• WITH COUNT vs. WITHOUT COUNT

Let’s look a little closer at each of these features….. 

1. Keeping Pre-12.2 functionality

If you want to your existing code to continue to return an error if the string is too long then the great news is that this is the default behaviour. When the length of the LISTAGG string exceeds the VARCHAR2 limit then the standard error will be returned:

ERROR at line xxx:
ORA-01489: result of string concatenation is too long

However, where possible I would recommend adding “ON OVERFLOW ERROR” to your LISTAGG code to make it completely clear that you are expecting an error when an overflow happens:

SELECT 
g.country_region,
LISTAGG(c.cust_first_name||’ ‘||c.cust_last_name, ‘,' ON OVERFLOW ERROR) WITHIN GROUP (ORDER BY c.country_id) AS Customer
FROM customers c, countries g
WHERE g.country_id = c.country_id
GROUP BY country_region
ORDER BY country_region;

So it’s important to note that by default the truncation features are disabled and you will need to change any existing code if you don’t want to raised an error. 

2. New ON OVERFLOW TRUNCATE… keywords

If you want to truncate the list of values at the 4K or 32K boundary then you need to use the newly added keywords ON OVERFLOW TRUNCATE as shown here:

SELECT 
g.country_region,
LISTAGG(c.cust_first_name||’ ‘||c.cust_last_name, ‘,' ON OVERFLOW TRUNCATE) WITHIN GROUP (ORDER BY c.country_id) AS Customer
FROM customers c, countries g
WHERE g.country_id = c.country_id
GROUP BY country_region
ORDER BY country_region;

when truncation occurs we will truncate back to the next full value at which point you can control how you tell the user that the list has been truncated. By default we append three dots ‘…’ to the string as indicator that truncation has occurred but you can override this as follows:

SELECT 
g.country_region,
LISTAGG(c.cust_first_name||’ ‘||c.cust_last_name, ‘,’ ON OVERFLOW TRUNCATE ‘***’) WITHIN GROUP (ORDER BY c.country_id) AS Customer
FROM customers c, countries g
WHERE g.country_id = c.country_id
GROUP BY country_region
ORDER BY country_region;

If you want to keep the existing pre-12.2 behaviour where we return an error if the string is too long then you can either rely on the default behaviour or explicitly state that an error should be returned (always a good idea to avoid relying on default behaviour in my opinion) by using the keywords:

SELECT 
g.country_region,
LISTAGG(c.cust_first_name||’ ‘||c.cust_last_name, ‘,' ON OVERFLOW ERROR) WITHIN GROUP (ORDER BY c.country_id) AS Customer
FROM customers c, countries g
WHERE g.country_id = c.country_id
GROUP BY country_region
ORDER BY country_region;

which will now generate the normal error message - i.e. replicates the pre-12.2 behaviour:

ORA-01489: result of string concatenation is too long
01489. 00000 - "result of string concatenation is too long"
*Cause: String concatenation result is more than the maximum size.
*Action: Make sure that the result is less than the maximum size.

 of course you can simply omit the new keywords and get the same behaviour:

SELECT 
g.country_region,
LISTAGG(c.cust_first_name||’ ‘||c.cust_last_name, ‘,') WITHIN GROUP (ORDER BY c.country_id) AS Customer
FROM customers c, countries g
WHERE g.country_id = c.country_id
GROUP BY country_region
ORDER BY country_region;

which, as before, generates the normal error message - i.e. replicates the pre-12.2 behaviour:

ORA-01489: result of string concatenation is too long
01489. 00000 - "result of string concatenation is too long"
*Cause: String concatenation result is more than the maximum size.
*Action: Make sure that the result is less than the maximum size.

3. How many values are missing?

If you need to know how many values were removed from the list to make it fit into the available space then you can use the keywords ‘WITH COUNT’ - this is the default behaviour. Alternatively if you don’t want a count at the end of the truncated string you can use the keywords ‘WITHOUT COUNT’, which is the default behaviour.

SELECT 
g.country_region,
LISTAGG(c.cust_first_name||’ ‘||c.cust_last_name, ‘,’ ON OVERFLOW TRUNCATE ‘***’ WITH COUNT) WITHIN GROUP (ORDER BY c.country_id) AS Customer
FROM customers c, countries g
WHERE g.country_id = c.country_id
GROUP BY country_region
ORDER BY country_region;

4. Do we split values when truncation occurs?

No. When determining where to force the truncation we take into account the full length of each value. Therefore, if you consider the example that we have been using which creates a list of customer names within each country we will always include the customer full name  “Keith Laker” (i.e. first name + last name). There has to be enough space to add the complete string (first+last name) to the list otherwise the whole string, “Keith Laker” is removed and the truncation indicator is inserted. It’s not possible for the last value in the string to be only the first name where the last name has been truncated/removed. 

5. How do we calculate the overall length of the string values?

The characters to indicate that an overflow has occurred are appended at the end of the list of values, which in this case if the default value of three dots “. . .”.  The overflow functionality traverses backwards from the maximum possible length to the end of the last complete value in the LISTAGG clause, then it adds the user-defined separator followed by the user defined overflow indicator, followed by output from the ’WITH COUNT’ clause which adds a counter at the end of a truncated string to indicate the number of values that have been removed/truncated from the list. 

Summary

With Database 12c Release 2 we have tackled the ORA-01489 error in two ways: 1) increased the the size of VARCHAR2 objects to 32K and 2) extended functionality of LISTAGG to allow greater control over the management of extremely long lists. Specifically there are several new keywords:

• ON OVERFLOW TRUNCATE
• ON OVERFLOW ERROR (default behaviour)
• WITH COUNT
• WITHOUT COUNT  (default behaviour)

Hopefully this new functionality will mean that all those wonderful workarounds for dealing with “ORA-01489: result of string concatenation is too long“ errors that have been created over the years can now be replaced by standard SQL functionality. 

Technorati Tags: Analytics, Database 12c, SQL, SQL Analytics

The growth of low-cost storage platforms has allowed many companies to actively seeking out new external data sets and combine them with internal historical data that goes back over a very long time frame. Therefore, as both the type of data and the volume of data continue to grow the challenge for many businesses is how to process this every expanding pool of data and at the same time, make timely decisions based on all the available data.

(Image above courtesy of http://taneszkozok.hu/)

In previous posts I have discussed whether an approximate answer just plain wrong and if approximate answers are the best way to analyze big data. As with the vast majority of data analysis at some point there is going to be a need to aggregate a data set to get a higher level view across various dimensions. When working with approximation result sets, dealing with aggregations can get a little complicated because it is not possible to “reuse” approximate a result set to aggregate data to higher levels across the various dimensions of the original query. To obtain a valid approximate result set requires a query to rescan the source data and compute the required analysis for the given combination of levels. Just because I have a result set that contains a count of the number of unique products sold this week at the county level does not mean that I can simply reuse that result set to determine the number of distinct products sold this week at the state level. In many cases you cannot just rollup aggregations of aggregations..

Until now….

With Database 12c Release 2 we have introduced a series of new functions to deal with this specific issue - the need to create reusable aggregated results that can be “rolled-up” to higher aggregate levels. So at long last you can now intelligently aggregate approximations! Here is how e do it….Essentially there are three parts that provide the “intelligence”:

• APPROX_xxxxxx_DETAIL
• APPROX_xxxxxx_AGG
• TO_APPROX_xxxxxx

Here is a quick overview of each function:

APPROX_xxxxxx_DETAIL

This function takes a numeric expression and builds summary result set containing results for all dimensions in GROUP BY clause. The output from this function is a column containing BLOB data. As with other approximate functions the results can be deterministic or non-deterministic depending on your requirements.

APPROX_xxxxxx_AGG 

This function builds a higher level summary table based on results from the _DETAIL function. This means that it is not necessary to re-query base fact table in order to derive new aggregates results. As with the _DETAIL function, the results are returned as a blob.

TO_APPROX_xxxxxx

Returns results from the _AGG and _DETAIL functions in user readable format.

A worked example

Let’s build a working example using the sample SH schema. Our product marketing team wants to know within each year the approximate number of unique customers within each product category. Thinking ahead we know that once they have this result set we expect them to do further analysis such as drilling on the time and product dimension levels to get deeper insight. The best solution is to build a reusable aggregate approximate result set using the new functions in 12.2.

SELECT
 t.calendar_year,
 t.calendar_quarter_number AS qtr,
 p.prod_category_desc AS category_desc,
 p.prod_subcategory_desc AS subcategory_desc,
 APPROX_COUNT_DISTINCT_DETAIL(s.cust_id) as acd_agg
FROM sales s, products p, times t
WHERE s.time_id= t.time_id
AND p.prod_id = s.prod_id
GROUP BY t.calendar_year, t.calendar_quarter_number, p.prod_category_desc, p.prod_subcategory_desc
ORDER BY t.calendar_year, t.calendar_quarter_number, p.prod_category_desc, p.prod_subcategory_desc;

this returns my result set as a blob as shown below and this blob contains the various tuple combinations from my GROUP BY clause. As a result I can reuse this result set to answer new questions based around higher levels of aggregation.

and the explain plan for this query shows the new sort keywords (GROUP BY APPROX) which tells us that approximate processing has been used as part of this query.

If we want to convert the BLOB data into a readable format we can transform it by using the TO_APPROX_xxx function as follows:

SELECT
t.calendar_year,
t.calendar_quarter_number AS qtr,
p.prod_category_desc AS category_desc,
p.prod_subcategory_desc AS subcategory_desc,
TO_APPROX_COUNT_DISTINCT(APPROX_COUNT_DISTINCT_DETAIL(s.cust_id)) as acd_agg
FROM sales s, products p, times t
WHERE s.time_id= t.time_id
AND p.prod_id = s.prod_id
GROUP BY t.calendar_year, t.calendar_quarter_number, p.prod_category_desc, p.prod_subcategory_desc
ORDER BY t.calendar_year, t.calendar_quarter_number, p.prod_category_desc, p.prod_subcategory_desc;

this creates the following results

alternatively we coult create a table using the above query and then simply pass the BLOB column directly into the TO_APPROX function as follows:

CREATE TABLE agg_cd AS
SELECT
 t.calendar_year,
 t.calendar_quarter_number AS qtr,
 p.prod_category_desc AS category_desc,
 p.prod_subcategory_desc AS subcategory_desc,
 APPROX_COUNT_DISTINCT_DETAIL(s.cust_id) as acd_agg
FROM sales s, products p, times t
WHERE s.time_id= t.time_id
AND p.prod_id = s.prod_id
GROUP BY t.calendar_year, t.calendar_quarter_number, p.prod_category_desc, p.prod_subcategory_desc
ORDER BY t.calendar_year, t.calendar_quarter_number, p.prod_category_desc, p.prod_subcategory_desc;

using this table we can simplify our query to return the approximate number of distinct customers directly from the above table:

SELECT
 calendar_year,
 qtr,
 category_desc,
 subcategory_desc,
 TO_APPROX_COUNT_DISTINCT(acd_agg)
FROM agg_cd
ORDER BY calendar_year, qtr, category_desc, subcategory_desc;

which returns the same results as before - as you would expect!

using the aggregated table as our source we can now change the levels that we wish to calculate without having to go back to the original source table and again scan all the rows. However, to extract the new aggregations we need to introduce the third function APPROX_COUNT_DISTINCT_AGG to our query and wrap this within the TO_APPROX_COUNT_DISTINCT function to see the results:

SELECT
 calendar_year,
 subcategory_desc,
 TO_APPROX_COUNT_DISTINCT(APPROX_COUNT_DISTINCT_AGG(acd_agg))
FROM agg_cd
GROUP BY calendar_year, subcategory_desc
ORDER BY calendar_year, subcategory_desc;

will return the following results based only on the new combination of levels included in the GROUP BY clause:

Summary

This post has reviewed the three new functions that we have introduced in Database 12c Release 2 that allow you to reuse aggregated approximate result sets:

• APPROX_xxxxxx_DETAIL
• APPROX_xxxxxx_AGG
• TO_APPROX_xxxxxx

 Database 12c Release 2 makes it possible to intelligently aggregate approximations. In the next post I will explore how you can combine approximate processing with existing query rewrite functionality so you can have intelligent approximate query rewrite. 

Technorati Tags: Analytics, Database 12c, Oracle Database 12c, SQL, SQL Analytics

Database 12c Release 2 available for download 

Yes, it’s the moment the world has been waiting for: the latest generation of the world’s most popular database, Oracle Database 12c Release 2 (12.2) is now available everywhere - in the Cloud and on-premises. You can download this latest version from the database home page on OTN - click on the Downloads tab.

So What’s New in 12.2 for Data Warehousing?

This latest release provides some incredible new features for data warehouse and big data. If you attended last year’s OpenWorld event in San Francisco then you probably already know all about the new features that we have added to 12.2 - checkout my blog post from last year for a comprehensive review of #oow16:

Blog: The complete review of data warehousing and big data content from Oracle OpenWorld 2016

 If you missed OpenWorld and if you are a data warehouse architect, developer or DBA then here are the main feature highlights 12.2 with links to additional content from OpenWorld and my data warehouse blog: 

General Database Enhancements

1) Partitioning

Partitioning: External Tables
Partitioned external tables provides both the functionality to map partitioned Hive tables into the Oracle Database ecosystem as well as providing declarative partitioning on top of any Hadoop Distributed File System (HDFS) based data store.

Partitioning: Auto-List Partitioning
The database automatically creates a separate (new) partition for every distinct partition key value of the table.

Auto-list partitioning removes the management burden from the DBAs to manually maintain a list of partitioned tables for a large number of distinct key values that require individual partitions. It also automatically copes with the unplanned partition key values without the need of a DEFAULT partition.

Partitioning: Read-Only Partitions
Partitions and sub-partitions can be individually set to a read-only state. This then disables DML operations on these read-only partitions and sub-partitions. This is an extension to the existing read-only table functionality. Read-only partitions and subpartitions enable fine-grained control over DML activity. 

Partitioning: Multi-Column List Partitioning
List partitioning functionality is expanded to enable multiple partition key columns. Using multiple columns to define the partitioning criteria for list partitioned tables enables new classes of applications to benefit from partitioning.

For more information about partitioning see:

#OOW16 - Oracle Partitioning: Hidden Old Gems and Great New Tricks, by Hermann Baer, Senior Director Product Management

Partitioning home page on OTN

2) Parallel Execution

Parallel Query Services on RAC Read-Only Nodes
Oracle parallel query services on Oracle RAC read-only nodes represents a scalable parallel data processing architecture. The architecture allows for the distribution of a high number of processing engines dedicated to parallel execution of queries.

For more information about parallel execution see: 

#OOW16 - The Best Way to Tune Your Parallel Statements: Real-Time SQL Monitoring by Yasin Baskan, Senior Principal Product Manager

Parallel Execution home page on OTN

Schema Enhancements

Dimensional In-Database analysis with Analytic Views

Analytic views provide a business intelligence layer over a star schema, making it easy to extend the data set with hierarchies, levels, aggregate data, and calculated measures. Analytic views promote consistency across applications. By defining aggregation and calculation rules centrally in the database, the risk of inconsistent results in different reporting tools is reduced or eliminated.

The analytic view feature includes the new DDL statements, such as CREATE ATTRIBUTE DIMENSION, CREATE HIERARCHY and CREATE ANALYTIC VIEW, new calculated measure expression syntax, and new data dictionary views.These analytic views allow data warehouse and BI  developers to extend the star schema with time series and other calculations eliminating the need to define calculations within the application. Calculations can be defined in the analytic view and can be selected by including the measure name in the SQL select list.

For more information about Analytic Views see:

#OOW16 - Analytic Views: A New Type of Database View for Simple, Powerful Analytics by Bud Endress, Director, Product Management

SQL Enhancements

Cursor-Duration Temporary Tables Cached in Memory
Complex queries often process the same SQL fragment (query block) multiple times to answer a question. The results of these queries are stored internally, as cursor-duration temporary tables, to avoid the multiple processing of the same query fragment. With this new functionality, these temporary tables can reside completely in memory avoiding the need to write them to disk. Performance gains are the result of the reduction in I/O resource consumption.

Enhancing CAST Function With Error Handling
The existing CAST function is enhanced to return a user-specified value in the case of a conversion error instead of raising an error. This new functionality provides more robust and simplified code development.

New SQL and PL/SQL Function VALIDATE_CONVERSION
The new function, VALIDATE_CONVERSION, determines whether a given input value can be converted to the requested data type. The VALIDATE_CONVERSION function provides more robust and simplified code development.

Enhancing LISTAGG Functionality
LISTAGG aggregates the values of a column by concatenating them into a single string. New functionality is added for managing situations where the length of the concatenated string is too long. Developers can now control the process for managing overflowing LISTAGG aggregates. This increases the productivity and flexibility of this aggregation function.

Approximate Query Processing
This release extends the area of approximate query processing by adding approximate percentile aggregation. With this feature, the processing of large volumes of data is significantly faster than the exact aggregation. This is especially true for data sets that have a large number of distinct values with a negligible deviation from the exact result.

Approximate query aggregation is a common requirement in today's data analysis. It optimizes the processing time and resource consumption by orders of magnitude while providing almost exact results. Approximate query aggregation can be used to speed up existing processing.

Parallel Recursive WITH Enhancements
Oracle Database supports recursive queries through the use of a proprietary CONNECT BY clause, and an ANSI compliant resursive WITH clause. The parallel recursive WITH clause enables this type of query to run in parallel mode. These types of queries are typical with graph data found in social graphs, such as Twitter graphs or call records and commonly used in transportation networks (for example, for flight paths, roadways, and so on).

Recursive WITH ensures the efficient computation of the shortest path from a single source node to single or multiple destination nodes in a graph. Bi-directional searching is used to ensure the efficient computation of the shortest path from a single source node to single or multiple destination nodes in a graph. A bi-directional search starts from both source and destination nodes, and then advancing the search in both directions.

For more information about the new data warehouse SQL enhancements see:

In addition to the above features, we have made a lot of enhancements and added new features to the Optimizer and there is a comprehensive review by Nigel Bayliss, senior principle product manager, available on the optimizer blog. Obviously, the above is my take on what you need to know about for 12.2 and it’s not meant to be an exhaustive list of all the data warehouse and big data features. For the complete list of all the new features in 12.2 please refer to the New Features Guide in the database documentation set.

I would really like to thank my amazing development team for all their hard work on the above list of data warehouse features and the all the time they have spent proof-reading and fact-checking my blog posts on these new features. 

Enjoy using this great new release and checkout all the 12.2 tutorials and scripts on livesql!

Technorati Tags: Analytics, Database 12c, Oracle Database 12c, SQL, SQL Analytics

after a match is found I would like match_recognize to stop searching - I want at most one match per partition. I don’t want to filter by MATCH_NUMBER() in an outer query - that is too wasteful (or, in some cases, I may know in advance that there is at most one match per partition, and I don’t want match_recognize to waste time searching for more matches which I know don't exist).

SELECT 
 symbol,
 tstamp,
 price,
 match_number,
 classifier,
 first_x,
 last_y 
FROM ticker
MATCH_RECOGNIZE (
 PARTITION BY symbol
 ORDER BY tstamp
 MEASURES
   FIRST(x.tstamp) AS first_x,
   LAST(y.tstamp) AS last_y,
   MATCH_NUMBER() AS match_number,
 CLASSIFIER() AS classifier
 ALL ROWS PER MATCH WITH UNMATCHED ROWS
 AFTER MATCH SKIP PAST LAST ROW
 PATTERN (strt X+ Y)
 DEFINE 
   X AS (price <= PREV(price)),
   Y AS (price >= PREV(price))
)
ORDER BY symbol, match_number, tstamp asc;

Returning only the 1st match

SELECT 
 symbol,
 tstamp,
 price,
 match_number,
 classifier,
 first_x,
 last_y 
FROM ticker
MATCH_RECOGNIZE (
 PARTITION BY symbol
 ORDER BY tstamp
 MEASURES
   FIRST(x.tstamp) AS first_x,
   LAST(y.tstamp) AS last_y,
   MATCH_NUMBER() AS match_number,
   CLASSIFIER() AS classifier
 ALL ROWS PER MATCH WITH UNMATCHED ROWS
 AFTER MATCH SKIP PAST LAST ROW
 PATTERN (strt X+ Y)
 DEFINE 
  X AS (price <= PREV(price)),
  Y AS (price >= PREV(price))
)
WHERE match_number = 1
ORDER BY symbol, match_number, tstamp asc;

SELECT 
 symbol,
 tstamp,
 price,
 match_number,
 classifier,
 first_x,
 last_y 
FROM ticker
MATCH_RECOGNIZE (
 PARTITION BY symbol
 ORDER BY tstamp
 MEASURES
   FIRST(x.tstamp) AS first_x,
   LAST(y.tstamp) AS last_y,
   MATCH_NUMBER() AS match_number,
   CLASSIFIER() AS classifier
 ALL ROWS PER MATCH
 AFTER MATCH SKIP PAST LAST ROW
 PATTERN (strt X+ Y c*)
 DEFINE 
   X AS (price <= PREV(price)),
   Y AS (price >= PREV(price))
)
ORDER BY symbol, match_number, tstamp asc;

SELECT 
 symbol,
 tstamp,
 price,
 match_number,
 classifier,
 first_x,
 last_y 
FROM ticker
MATCH_RECOGNIZE (
 PARTITION BY symbol
 ORDER BY tstamp
 MEASURES
   FIRST(x.tstamp) AS first_x,
   LAST(y.tstamp) AS last_y,
   MATCH_NUMBER() AS match_number,
   CLASSIFIER() AS classifier
 ALL ROWS PER MATCH
 AFTER MATCH SKIP PAST LAST ROW
 PATTERN (strt X+ Y {-c*-})
 DEFINE 
   X AS (price <= PREV(price)),
   Y AS (price >= PREV(price))
)
ORDER BY symbol, match_number, tstamp asc;

Returning only the 2nd match

…and the final answer is: Enhancement Request

• Demo: Simplified SQL for fraud detection: livesql.oracle.com - if you have never used LiveSQL then you can get more information here, it’s a fantastic service and it’s completely FREE
• Slides:Simplified and fast fraud detection is now available on the OTN SQL Analytics “Learn More” page.

Above image courtesy of wikimedia.org

What is a cursor duration temp table?

• WITH Clause and parallel recursive WITH
• Grouping Sets
• Star Transformation
• Frequent Item Set Counting
• XLATE 

So what is an In-Memory cursor duration temp table?

Will all my queries use this new feature?

Do I need to license database in-memory option?

Where does the memory come from?

Sample Schema

Example: GROUP BY with GROUPING SETS

SELECT /*+MONITOR */
 p.prod_category, 
 x.promo_id,
 c.channel_class,
 t.calendar_quarter_desc,
 s.time_id,
 SUM(s.amount_sold)
FROM sales s, products p, promotions x, channels c, times t
WHERE s.time_id 
      BETWEEN to_date('1998-01-01', 'YYYY-MM-DD') 
      AND to_date('2000-12-31', 'YYYY-MM-DD')
AND p.prod_id = s.prod_id
AND x.promo_id = s.promo_id
AND c.channel_id = s.channel_id
AND t.time_id = s.time_id
GROUP BY GROUPING SETS 
 ((p.prod_category, x.promo_id), 
  (c.channel_class, t.calendar_quarter_desc), 
  (p.prod_category, t.calendar_quarter_desc), 
  (s.time_id, t.calendar_quarter_desc))
ORDER BY 1,2,3,4,5

Part A, Pre 12-2:  using cursor duration temp tables

If you want more information the temp table transformation step then there is an excellent post on the optimizer blog: https://blogs.oracle.com/optimizer/entry/star_transformation. From the plan below you can see that our temp table is then reused during the rest of the query where we reuse the temp table to construct the various totals for each combination of dimension  members. 

Part B, 12.2: using in-memory cursor duration temp tables

LOAD AS SELECT ((CURSOR DURATION MEMORY)) 

Summary

• a simple way to ingest data
• access to languages that help with data discovery and data analytics
• some basic data visualization tools
• a set of collaboration services for sharing notebooks (collections of reports)

Use Cases

%osql

select * from ext_bank_data

Building Dashboards 

Visualizations

Quick Summary

What’s next?

• Installing Zeppelin
• Configuring Zeppelin
• What is an interpreter
• Finding and installing the Oracle JDBC drivers
• Setting up a connection to an Oracle PDB

Installing Zeppelin

Installing Zeppelin - Version 0.6.2

Installing Zeppelin - Version 0.7.x

Configuring Zeppelin (0.6.2 and 0.7.x)

Configuring the Zeppelin http port

<property>
<name>zeppelin.server.port</name>
<value>8080</value>
<description>Server port.</description>
</property>

<property>
<name>zeppelin.server.port</name>
<value>7081</value>
<description>Server port.</description>
</property>

Starting Zeppelin

. ../my-zeppelin/bin/zeppelin-daemon.sh start

• ZEPPELIN_MEM: amount of memory available to Zeppelin. The default setting is - -Xmx1024m -XX:MaxPermSize=512m
• ZEPPELIN_INTP_MEM: amount of memory available to the Zeppelin Interpreter (connection) engine and the default setting is derived from the setting of ZEPPELIN_MEM
• ZEPPELIN_JAVA_OPTS: simply lists any additional JVM options

set ZEPPELIN_MEM=-Xms1024m -Xmx4096m -XX:MaxPermSize=2048m

set ZEPPELIN_INTP_MEM=-Xms1024m -Xmx4096m -XX:MaxPermSize=2048m

set ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g -Dspark.cores.max=16"

. ../my-zeppelin/bin/zeppelin-daemon.sh start

 Zeppelin start                                             [  OK  ]

Connecting to Zeppelin

What is an Interpreter?

Configuring your JDBC files

Creating an Interpreter for Oracle Database

Process for creating a connection using version 0.6.2

• osql.driver
• osql.password
• osql.url
• osql.user

Process for creating a connection using version 0.7.x

• default.driver: driver entry point into the Oracle JDBC driver
• default.password: Oracle user password
• default.url: JDBC connection string to access the Oracle database/pDB 
• default.user: Oracle username

Testing your new interpreter

SELECT sysdate FROM dual

Summary

• How to install Zeppelin
• How configure and start Zeppelin
• Finding and installing the correct Oracle JDBC drivers
• Set up a connection to an Oracle PDB and tested the connection

It’s here - at last! I have just pushed my usual must-see guide to the Apple iBooks Store.

Page 8 - On-Demand Videos	Page 17 - Justify Your trip	Page 19 - Key Presenters	Page 41 - Must See Sessions	Page 90 - Useful Maps

Chapter 1 - Introduction to the must-see guide. 

Chapter 2 - A guide to the key the highlights from last year’s conference so you can relive the experience or see what you missed. Catch the most important highlights from last year's OpenWorld conference with our on demand video service which covers all the major keynote sessions. Sit back and enjoy the highlights. The second section explains why you need to attend this year’s conference and how to justify it to your company. 

Chapter 3 - Full list of Oracle Product Management and Development presenters who will be at this year’s OpenWorld. Links to all their social media sites are included alongside each profile. Read on to find out about the key people who can help you and your teams build the FUTURE using Oracle’s Data Warehouse and Big Data technologies. 

Chapter 4 - List of the “must-see” sessions at this year’s OpenWorld by category. It includes all the sessions and hands-on labs by the Oracle Product Management and Development teams along with key customer sessions. Read on for the list of the best, most innovative sessions at Oracle OpenWorld 2016. 

Chapter 5 - Details of all the links you need to keep up to date on Oracle’s strategy and products for Data Warehousing and Big Data. This covers all our websites, blogs and social media pages. 

Chapter 6 -  Details of our exclusive web application for smartphones and tablets provides you with a complete guide to everything related to data warehousing and big data at OpenWorld 2016. 

Chapter 7 - Information to help you find your way around the area surrounding the Moscone Conference Center this section includes some helpful maps.

….and now it’s here in PDF format as well!

Page 8 - On-Demand Videos	Page 17 - Justify Your trip	Page 19 - Key Presenters	Page 41 - Must See Sessions	Page 90 - Useful Maps

Chapter 1 - Introduction to the must-see guide.
Chapter 2 - A guide to the key the highlights from last year’s conference so you can relive the experience or see what you missed. Catch the most important highlights from last year's OpenWorld conference with our on demand video service which covers all the major keynote sessions. Sit back and enjoy the highlights. The second section explains why you need to attend this year’s conference and how to justify it to your company.
Chapter 3 - Full list of Oracle Product Management and Development presenters who will be at this year’s OpenWorld. Links to all their social media sites are included alongside each profile. Read on to find out about the key people who can help you and your teams build the FUTURE using Oracle’s Data Warehouse and Big Data technologies.
Chapter 4 - List of the “must-see” sessions at this year’s OpenWorld by category. It includes all the sessions and hands-on labs by the Oracle Product Management and Development teams along with key customer sessions. Read on for the list of the best, most innovative sessions at Oracle OpenWorld 2016.
Chapter 5 - Details of all the links you need to keep up to date on Oracle’s strategy and products for Data Warehousing and Big Data. This covers all our websites, blogs and social media pages.
Chapter 6 -  Details of our exclusive web application for smartphones and tablets provides you with a complete guide to everything related to data warehousing and big data at OpenWorld 2016.
Chapter 7 - Information to help you find your way around the area surrounding the Moscone Conference Center this section includes some helpful maps.

MATCH_RECOGNIZE and predicates

1. PARTITION BY column
2. ORDER BY column
3. All other columns

1. Predicates on the PARTITION BY column

select * from ticker
MATCH_RECOGNIZE(
  PARTITION BY symbol ORDER BY tstamp
  MEASURES match_number() as mn
  ALL ROWS PER MATCH
PATTERN (strt)
 DEFINE strt as 1=1
);

select * from ticker
MATCH_RECOGNIZE(
  PARTITION BY symbol ORDER BY tstamp
  MEASURES match_number() as mn
  ALL ROWS PER MATCH
PATTERN (strt+)
 DEFINE strt as 1=1
)
WHERE symbol = ‘ACME';

Conclusion - predicates on the PARTITION BY column.

2. Predicates on the ORDER BY column

select * from ticker
MATCH_RECOGNIZE(
 PARTITION BY symbol ORDER BY tstamp
 MEASURES match_number() as mn
 ALL ROWS PER MATCH
 PATTERN (strt)
 DEFINE strt as 1=1
)
WHERE symbol='ACME'
AND tstamp BETWEEN '01-APR-11' AND '10-APR-11';

select * from ticker
MATCH_RECOGNIZE(
 PARTITION BY symbol ORDER BY tstamp
 MEASURES match_number() as mn
 ALL ROWS PER MATCH
 PATTERN (strt)
 DEFINE strt as 1=1
)
WHERE tstamp BETWEEN ’01-APR-11' AND '10-APR-11';

Conclusion

3.Using other columns

select * from ticker
MATCH_RECOGNIZE(
  PARTITION BY symbol ORDER BY tstamp
  MEASURES match_number() as mn
  ALL ROWS PER MATCH
PATTERN (strt)
 DEFINE strt as 1=1
)
WHERE symbol = ‘ACME';

select * from ticker
MATCH_RECOGNIZE(
  PARTITION BY symbol ORDER BY tstamp
  MEASURES match_number() as mn
  ALL ROWS PER MATCH
PATTERN (strt)
 DEFINE strt as 1=1
)
WHERE symbol = ‘ACME'
AND day_name in (‘MONDAY’, ’TUESDAY’, ‘WEDNESDAY’, ’THURSDAY’, ‘FRIDAY’);

select * from ticker
MATCH_RECOGNIZE(
 PARTITION BY symbol ORDER BY tstamp
 MEASURES match_number() as mn,
prev(day_name) as prev_day
 ALL ROWS PER MATCH
 PATTERN (strt)
 DEFINE strt as 1=1
)
WHERE symbol='ACME'
AND day_name in ('MONDAY', 'WEDNESDAY', 'FRIDAY');

One last example

select symbol, tstamp, mn, price, day_name, prev_day, total_rows from ticker
MATCH_RECOGNIZE(
 PARTITION BY symbol ORDER BY tstamp
 MEASURES match_number() as mn,
          prev(day_name) as prev_day,
          count(*) as total_rows 
 ALL ROWS PER MATCH
 PATTERN (strt+)
 DEFINE strt as 1=1
)
WHERE symbol='ACME'
AND day_name in ('MONDAY', 'WEDNESDAY', 'FRIDAY');

select symbol, tstamp, mn, price, day_name, prev_day, total_rows, avg_price, max_price 

from ticker
MATCH_RECOGNIZE(
 PARTITION BY symbol ORDER BY tstamp
 MEASURES match_number() as mn,
 prev(day_name) as prev_day,
 count(*) as total_rows,
 trunc(avg(price),2) as avg_price,
 max(price) as max_price
 ALL ROWS PER MATCH
PATTERN (strt+)
 DEFINE strt as 1=1
)
WHERE symbol=‘ACME';

select symbol, tstamp, mn, price, day_name, prev_day, total_rows, avg_price, max_price from ticker
MATCH_RECOGNIZE(
 PARTITION BY symbol ORDER BY tstamp
 MEASURES match_number() as mn,
   prev(day_name) as prev_day,
   count(*) as total_rows,
   trunc(avg(price),2) as avg_price,
   max(price) as max_price
 ALL ROWS PER MATCH
PATTERN (strt+)
 DEFINE strt as 1=1
)
WHERE symbol='ACME'
AND day_name in ('MONDAY', 'WEDNESDAY', 'FRIDAY');

Conclusion

** NEW ** Chapter 5

*** UPDATED *** Must-See Guide now available as PDF and via Apple iBooks Store

Page 8 - On-Demand Videos	Page 11 - Justify Your trip	Page 18 - Key Presenters	Page 39 - Must See Sessions	Page 83 - Must See Day-by-Day	Page 150 - Useful Maps

Chapter 1 - Introduction to the must-see guide. 

Chapter 2 - A guide to the key the highlights from last year’s conference so you can relive the experience or see what you missed. Catch the most important highlights from last year's OpenWorld conference with our on demand video service which covers all the major keynote sessions. Sit back and enjoy the highlights. The second section explains why you need to attend this year’s conference and how to justify it to your company. 

Chapter 3 - Full list of Oracle Product Management and Development presenters who will be at this year’s OpenWorld. Links to all their social media sites are included alongside each profile. Read on to find out about the key people who can help you and your teams build the FUTURE using Oracle’s Data Warehouse and Big Data technologies. 

Chapter 4 - List of the “must-see” sessions and hands-on labs at this year’s OpenWorld by category. It includes all the sessions and hands-on labs by the Oracle Product Management and Development teams along with key customer sessions. Read on for the list of the best, most innovative sessions at Oracle OpenWorld 2017. 

Chapter 5 - Day-by-Day “must-see” guide. It includes all the sessions and hands-on labs by the Oracle Product Management and Development teams along with key customer sessions. Read on for the list of the best, most innovative sessions at Oracle OpenWorld 2017. 

Chapter 6 - Details of all the links you need to keep up to date on Oracle’s strategy and products for Data Warehousing and Big Data. This covers all our websites, blogs and social media pages. 

Chapter 7 -  Details of our exclusive web application for smartphones and tablets provides you with a complete guide to everything related to data warehousing and big data at OpenWorld 2017. 

Chapter 8 - Information to help you find your way around the area surrounding the Moscone Conference Center this section includes some helpful maps. 

• Data Warehousing and Cloud
• Analytics and Machine Learning
• Unstructured and development
• Big Data
• Hands-on labs

Landing page which includes video for #oow17	Main page app with links to the day-by-day session guides

SUNDAY'S MUST-SEE GUIDE

Day 2, Monday, is here and this is my definitive list of Must-See sessions for today. The list is packed full of sessions and labs that follow on from yesterday's (Sunday) big announcements around Oracle Autonomous Database and Oracle Autonomous Data Warehouse Cloud. These sessions are what Oracle OpenWorld is all about: the chance to learn about the latest technology from the real technical experts.

MONDAY's MUST-SEE GUIDE

Don't worry if you are not able to join us in San Francisco for this year's conference because I will be providing a comprehensive review after the conference closes on Thursday.

The review will include links to download the presentations for each of my Must-See sessions and links to any hands-on lab content as well.

Have a great conference.

If you are here in San Francisco then enjoy the conference - it's going to be an awesome conference this year.

Don't forget to make use of our Big DW #oow17 smartphone app which you can access by pointing your phone at this QR code:

MONDAY's MUST-SEE GUIDE

WEDNESDAY'S MUST-SEE GUIDE