In case of an Issue that occured in the past, while most of the experts call for AWR reports and rely on ASH data available, but in my opinion, the analysis and recommendation based on AWR and ASH reports does not guarantee 100% accuracy. To prove this point, I have a real life scenario to share.

One of my customers added another node to an existing 2 node RAC production setup. Post completion of this activity, the system was released to the end users. In this case, end users are the customers across the globe. In the afternoon, severe slowness was reported and before anyone could analyze the issue, all the Application Servers / Web Servers Crashed and had to be restarted. There was a severe escalation and everyone pointed the finger of blame on the addition of a 3rd Node, as this was the only change in the entire production system. The Database Administrators also mentioned that while investigating the issue, they could observe severe “Cursor:pin” related waits. There were around 400+ sessions waiting for the “Cursor:pin” and the queries that were stuck were running successfully on other 2 nodes. Therefore, it was assumed to be a BUG or some issue related to 3rd Node. Since the Application architecture involved connections thru connection pool, when all the connections pool got exhausted, severe slowness and then a crash was reported.

A detailed investigation was initiated the next day. AWR report for the problem period and the ASH reports also revealed that almost 90% of the time the sessions were waiting on “Cursor:Pin” and the other 2 nodes the AWR was clean. While we were still investigating the issue, one of the DBA also mentioned that all other queries were working absolutely fine and, the problem only existed with the two tables as the queries on these 2 objects were waiting on the Mutexes and the executions of these queries were ZERO. The executions of the queries on these 2 objects were 1000+. Further, he also mentioned that they tried running the queries on these 2 objects from SQL*Plus, even these hung and had to be cancelled using CTRL-C. This raised some suspicion, as why the problem was only limited to 2 Objects and therefore, we started concentrating more these 2 Objects.

Analysis

These 2 Objects were actually a Synonym to 2 remote tables and 90% of the application modules get data from the remote database, using Database Link. Does it mean that during the problem period, there was some connectivity issue between the two Servers ? Let us simulate this with an example.

To simulate this, I created a database link between my 11g and 10g instance and then created a Synonym on EMP table, which resides in 10g Database. Further, to simulate production load, I created a database procedure that selects a row from this remote table and scheduled around 1000 sessions using DBMS_JOB. With the 10g Listener UP and Running, the job runs perfectly fine, therefore, I stopped the listener of 10g Database, as if there was some problem with the connectivity and re-scheduled the job.

connect vivek/vivek
create database link o10g connect to vivek identified by vivek using 'orcl10g';
create synonym emp_remote for emp@o10g;

create or replace procedure test_load as
e_name  varchar2(30);
l_eno	number;
begin
  l_eno:=round(dbms_random.value(7000,8000),0);
  select ename into e_name from emp_remote where empno=l_eno;
end;
/

alter session set nls_date_format='Dd-MM-YYYY HH24:MI:SS';

select sysdate from dual;

SYSDATE
-------------------
15-11-2011 00:45:42

## I Scheduled the Jobs to run at 00 Hours & 47 Minutes

declare
l_job	number;
begin
   for i in 1..1000
     loop

       dbms_job.submit(l_job,'test_load;',trunc(sysdate)+00/24+47/(24*60));
       commit;
     end loop;
end;
/

Once the Jobs started, the wait events and the execution of the query hung on “Cursor:Pin” are as under :

       SID EVENT                                  P1         P2         P3 SQL_ID
---------- ------------------------------ ---------- ---------- ---------- -------------
         1 pmon timer                            300          0          0
         4 DIAG idle wait                          5          1       1000
        10 Streams AQ: qmn coordinator id          0          0          0
        13 Streams AQ: qmn slave idle wai          1          0          0
        15 Streams AQ: waiting for time m          0          0          0
        26 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
        28 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
        30 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
        32 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
        34 library cache lock             1269367872 1235709864          2 f20gyq4m5fh3c
        35 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
        42 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
        47 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
        49 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
        51 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
        52 library cache lock             1269367872 1235628808          2 f20gyq4m5fh3c
        54 library cache lock             1269367872 1269117072          2 f20gyq4m5fh3c
       125 VKRM Idle                               0          0          0
       126 VKTM Logical Idle Wait                  0          0          0
       127 DIAG idle wait                          5          1       1000
       131 smon timer                            300          0          0
       136 Space Manager: slave idle wait          0          0          0
       139 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
       142 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
       143 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
       149 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
       153 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
       154 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
       155 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
       156 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
       170 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
       172 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c
       175 library cache lock             1269367872 1234978972          2 f20gyq4m5fh3c
       176 cursor: pin S wait on X         643252332   10682368     327680 f20gyq4m5fh3c

SQL> @sql_sqlid
Enter value for sql: f20gyq4m5fh3c
old   1: select sql_id, sql_text, executions, buffer_gets from v$sqlarea where sql_id='&sql'
new   1: select sql_id, sql_text, executions, buffer_gets from v$sqlarea where sql_id='f20gyq4m5fh3c'

SQL_ID        SQL_TEXT                                           EXECUTIONS BUFFER_GETS
------------- -------------------------------------------------- ---------- -----------
f20gyq4m5fh3c SELECT ENAME FROM EMP_REMOTE WHERE EMPNO=:B1                0           0

This was the exact behaviour that was reported on the 3rd Node production setup. I compared the alert.log of my Database and the production 3rd Node and the common observation was an entry in both the environment (this is from my database). In production, the error was related to tns timeout.

Fatal NI connect error 12541, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost.localdomain)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=orcl10g)(CID=(PROGRAM=oracle)(HOST=localhost.localdomain)(USER=oracle))))

  VERSION INFORMATION:
        TNS for Linux: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production
  Time: 15-NOV-2011 01:01:30
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12541

TNS-12541: TNS:no listener
    ns secondary err code: 12560
    nt main err code: 511

TNS-00511: No listener
    nt secondary err code: 111
    nt OS err code: 0

At this point, when we discussed this with the customer, one of the Application team member admitted that as a part of the checklist, they missed out on Opening of a Port from the database server to the Server hosting the remote database and this was realized in the afternoon. They were not aware of the implication it had on the production setup.

In this case, the AWR & ASH reports revealed “Cursor:Pin” waits but were not sufficient in revealing the actual cause of the issue. It required further analysis, therefore, I usually insist on production access to diagnose a Performance Issue whether it was in the past or is an ongoing issue.

A possible way to get these is 10046 traces. I used these traces to get the Queries that were causing these failures to occur. Yesterday, I was working at another customer site, where I could see some failures happening. These are again around 25% to 30% of the total Hard Parses. Therefore, if we eliminate these, total hard parses / second would come down. Following query helps me check, whether the statistics related to hard parses are alarming.

set linesize 132
alter session set nls_date_format='DD-MM-YYYY HH24:MI:SS';
select sysdate, name, value from v$sysstat where name like 'parse%';
exec dbms_lock.sleep(10);
select sysdate, name, value from v$sysstat where name like 'parse%';

I execute a query against v$sysstat, wait for 10 seconds and re-execute the same. Therefore, the incremental figure is for 10 seconds duration. From this, it is very easy to compute the parses/second ratio. Following is the output of the query taken yesterday :

SQL> @hard_parse

Session altered.

Elapsed: 00:00:00.00

SYSDATE             NAME                                                                  VALUE
------------------- ---------------------------------------------------------------- ----------
20-09-2011 10:28:14 parse time cpu                                                       117213
20-09-2011 10:28:14 parse time elapsed                                                   390923
20-09-2011 10:28:14 parse count (total)                                                54294681
20-09-2011 10:28:14 parse count (hard)                                                   766713
20-09-2011 10:28:14 parse count (failures)                                               177555
20-09-2011 10:28:14 parse count (describe)                                                  204

6 rows selected.

Elapsed: 00:00:00.03

PL/SQL procedure successfully completed.

Elapsed: 00:00:10.00

SYSDATE             NAME                                                                  VALUE
------------------- ---------------------------------------------------------------- ----------
20-09-2011 10:28:24 parse time cpu                                                       117216
20-09-2011 10:28:24 parse time elapsed                                                   390969
20-09-2011 10:28:24 parse count (total)                                                54303954
20-09-2011 10:28:24 parse count (hard)                                                   767011
20-09-2011 10:28:24 parse count (failures)                                               177637
20-09-2011 10:28:24 parse count (describe)                                                  204

6 rows selected.

Elapsed: 00:00:00.02
SQL> select 767011-766713, 177637-177555 from dual;

767011-766713 177637-177555
------------- -------------
          298            82

In this case, the Hard Parses are 29.8 per second and failures are 8.2, which is around 28% of the total hard parses. To check for the queries causing these, I generated enabled 10046 trace at system level, during less concurrency time (non-peak hours), and issued grep -i “PARSE ERROR” *.trc. This gave me the name of the trace files and from these traces, it was very easy to figure out the queries that were failing. In the trace file, search for PARSE ERROR and you will get the query, alongwith the reason for its failure, which is ora-942 (table or view does not exists), as can be seen in err column.

grep -i "PARSE ERROR" *.trc
RAC1_ora_1966688.trc:PARSE ERROR #1:len=157 dep=0 uid=38 oct=3 lid=38 tim=2611876001420 err=942
RAC1_ora_1966688.trc:PARSE ERROR #1:len=157 dep=0 uid=38 oct=3 lid=38 tim=2611876029670 err=942
RAC1_ora_1966688.trc:PARSE ERROR #5:len=95 dep=1 uid=0 oct=3 lid=0 tim=2611876430154 err=942

Initial investigation revealed that with Automatic Shared Memory Management enabled, the Shared Pool had grown upto 40GB. This is pretty high. I have not seen any configuration that requires such a huge Shared Pool. This was surely a cause of concern and therefore, this was further drilled down to one of a serious bug in the application. This was Hard Parse.

An Application or Database flooded with Unshared Application queries is considered as an Unscalable Application. Slowly or Gradually, as the User Load increase, the performance issues are bound to come. In this case, the Hard Parse / Second ratio was around 180 per second. While there were many queries that were using Literals, the most surprising part was the Hard Parse failures, which were almost 68% of the total Hard Parses. Therefore, once a patch to fix the queries that were using Literals was applied, there was no significant improvement.

The most challenging part of this assignment was the reluctance from the Application Team to accept the Hard Parses or the Parse failure to be an Issue. The debate from the Application Team was that reducing the Hard Parses would not yield any benefit as they do not see any Library Cache related latches on top. They have been taught to see the AWR reports and look to the top events. Reducing the number of waits of the top most component seemed to be on top of the list for Application team. While this is true, to some extent, but in case of Hard Parses, this would not work.

This is the reason, the title of my blog is “Do we really need to take care of Hard Parses only when Library Cache related Latches are on top ?”. In this case, the Hard Parses can be easily related to the contention that we observed on the Connection Pool. Usually, the response time of a Connection Pool or a Shared Server Architecture should be less than a Second. If this goes high, we might see a slowness or contention on Connection Pool, as other users may have to wait for the pool to be released. This again explains, why the number of connections pools increased as the load increases.

Hard Parses are a performance killer, also called as a Silent Killer. Many a times, Hard Parses does not cause significant waits on Library Cache latches. It depends on the resources available at the Database Server. In this case, it was a 2 Node RAC with 56 CPU’s on each machine. The Hardware configuration was enough and therefore, the contention was not that significant to cause Latch Free waits. As we all know, Hard Parse takes more time than a Soft Parse or a No Parse and this resulted in a signifcant waits or contention on Connection Pool. Also, as mentioned above, Hard Parses are Silent Killer. With that said, if these are not taken care of at a initial stage, this can bring down the Production System (as it did in the case of my Customer).

To summarize, Hard Parses are to be addressed even if these are not visible as a Latch Contention. In case of my customer, as soon as the Hard Parses and Parse failures were fixed, the performance of the production system improved drastically and was consistent through out the day. The users were able to enter their transactions in the business hours. Further, the contention on the connection pool came down as well.

In this blog, I will answer to the Question and Comment mentioned above. This would be one of the case that can trigger a sudden degradation. The test case will be around Optimizer Statistics. I have been asked this question by the Customers and concerns were raised over the way Optimizer works or use these Statistics. Some of the Concerns raised are :

1. Why do I generate fresh Stats when I am Satisfied by the Application Performance on current Stats ?
2. I generated Statistics over the weekend and suddenly the performance degraded. Finally, I reverted back the statistics and everything came back to normalcy.
3. What is the cause of this Sudden Degradation ?

While lot has been written on Concern 1 (the way Optimizer works and the change in Optimizer behaviour in 10g, I will answer Concern 2, and Concern 3 will be automatically answered. As usual, all this will be well supported and demonstrated by way of a Test Case.

SCENARIO

A Customer gathered Statistics on all the production tables. As per the policy, these statistics are gathered every weekend since last couple of years. On Monday, suddenly the Users started complaining about the slowness. The Onsite DBA’s had a look at the system and complained about the I/O issue. Further investigations revealed that the Execution plans for some of the critical application queries has changed from Index Scan to Full Table Scan. What has changed between Friday and Monday? The DBA’s realised that the only change is Optimizer Statistics. In such cases, if it is possible to identify the suspected table and its stats, then it is very easy to revert the stats of this table else the option left out is to revert the entire statistics and bring back the stats of Friday. Both the fallback plan has its own Implications. After reverting the Statistics, the Application Performance comes back to normalcy. Based on the cause of the Issue and the change to get back to normalcy, it is quite evident that the finger of blame would be on Oracle Optimizer. The Customer then decided not to gather fresh statistics unless the Optimizer Issue is resolved.

While there are various reasons that can cause a Slowness post-fresh statistics generation, one of the top most cause is Inefficient Indexing. If the Indexing Strategy of an Application is not optimal, the performance of that application would show a sudden degradation. Usually, Performance drop is not linear. It is always exponential, when the utilization reaches a threshold where it is difficult to handle the extra (minor) growth, this minor growth is enough to cause a sudden degradation.

TEST CASE & EXPLANATION

To Demonstrate the impact of In-efficient Indexing, we need to make certain assumption. In this case, let us assume that the Production System has just gone live and the production tables were without any rows, except the Master Tables. For our demonstration, I will create a table and populate some data (initially 10 days) and run an application query against that table. The table is setup to create 10 Customer ID’s and each of these customers have 5 transactions a day. We will run an application query to get the product_name and sale_value for a Customer for a day. This query should return 5 rows from initial 500 rows (10 Days x 5 Transactions a day x 10 Customers).

Next, as the system become 60 days old, the same application query is run. In our case, even if the volume increase, the number of rows return by the query remains unchanged. We shall see the significant impact of this Increase in Volume on the Performance and this impact was all of a sudden.

Table Creation Script - Inserts 10 days Data, which means the production is 10 days old

exec dbms_random.seed(0);
drop table vivek_test;
create table vivek_test as
with main as
(select mod(rownum,10) custid,
        round(dbms_random.value(1000,7000),0) sale_value,
        dbms_random.string('A',20) product_name
from    all_objects a
where rownum<=50)
select custid, s_date sale_date, sale_value, product_name
from    main, (select sysdate - level s_date from dual connect by level<=10) ;

Next we Create an Index on CustID and Gather Statistics

create index vivek_test_idx on vivek_test(custid);
exec dbms_stats.gather_table_stats(user,'VIVEK_TEST',cascade=>true,method_opt=>'for all columns size 1');

Once this is done, we execute our Application Query to get data for 09th June 2011

variable b1 varchar2(20);
exec :b1:='06-09-2011';

select /*+ vivek */ custid, product_name, sale_value
from  vivek_test
where custid=1
and   sale_date between to_date(:b1,'MM-DD-YYYY HH24:MI:SS') and to_date(:b1,'MM-DD-YYYY HH24:MI:SS')+1-1/(24*60*60);

    CUSTID PRODUCT_NAME         SALE_VALUE
---------- -------------------- ----------
         1 rMLTDXxxqXOZnqYRJwIn       1380
         1 AOcitwrWNXCxPHzIIIxw       3850
         1 hpalFVIprLxIHDrRgqHb       2287
         1 vkfKdaFoCaHgBCQAFgCY       4364
         1 DAlVYefHAurNoryMikSJ       1125

Runtime Plan

select sql_id, sql_text, buffer_gets, executions, version_count
from v$sqlarea where sql_text like 'select /*+ vivek%';

SQL_ID        SQL_TEXT                                           BUFFER_GETS EXECUTIONS VERSION_COUNT
------------- -------------------------------------------------- ----------- ---------- -------------
2t8mwg94j89h8 select /*+ vivek */ custid, product_name, sale_val          10          2             1
              ue from  vivek_test where custid=1 and   sale_date
               between to_date(:b1,'MM-DD-YYYY HH24:MI:SS') and
              to_date(:b1,'MM-DD-YYYY HH24:MI:SS')+1-1/(24*60*60

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------
SQL_ID  2t8mwg94j89h8, child number 0
-------------------------------------
select /*+ vivek */ custid, product_name, sale_value from  vivek_test where custid=1
and   sale_date between to_date(:b1,'MM-DD-YYYY HH24:MI:SS') and
to_date(:b1,'MM-DD-YYYY HH24:MI:SS')+1-1/(24*60*60)

Plan hash value: 2623397998

-----------------------------------------------------------------------------------------------
| Id  | Operation                    | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                |       |       |     4 (100)|          |
|*  1 |  FILTER                      |                |       |       |            |          |
|*  2 |   TABLE ACCESS BY INDEX ROWID| VIVEK_TEST     |     1 |    35 |     4   (0)| 00:00:01 |
|*  3 |    INDEX RANGE SCAN          | VIVEK_TEST_IDX |    50 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter(TO_DATE(:B1,'MM-DD-YYYY HH24:MI:SS')=TO_DATE(:B1,'MM-DD-YYYY HH24:MI:SS') AND
              "SALE_DATE"<=TO_DATE(:B1,'MM-DD-YYYY HH24:MI:SS')+1-.0000115740740740740740740740740740
              7407407407))
   3 - access("CUSTID"=1)

The Runtime Plan in this case shows that the Application Query has used an Index on Custid, which is usually an appropriate approach in an OLTP system. The Interesting fact here is a Throwaway, which is, 50 Rows were fetched from an Index and only 1 row was fetched from the Table Access. In this case, 49 rows were THROWNAWAY. The Actual THROWAWAY rows are 45, but again, 45 rows out of 50 rows is 90% of the rows are discarded and this is very bad.

THE REAL PERFORMANCE ISSUE

User Continue to use the system and it is now 60 days old. As per the policy, after the 60th day, fresh Statistics were gathered. Further, the DBA’s also take the backup of the old stats before generating new. As the data is entered, users also continue to execute the application query.

truncate table vivek_test;
exec dbms_random.seed(0);

insert into vivek_test
with main as
(select mod(rownum,10) custid,
        round(dbms_random.value(1000,7000),0) sale_value,
        dbms_random.string('A',20) product_name
from    all_objects a
where rownum<=50)
select custid, s_date sale_date, sale_value, product_name
from    main, (select sysdate - level s_date from dual connect by leveltrue,method_opt=>'for all columns size 1');

exec dbms_stats.create_stat_table(user,'STATS');
exec dbms_stats.export_table_stats(user,'VIVEK_TEST',stattab=>'stats');

variable b1 varchar2(20);
exec :b1:='06-09-2011';

select /*+ vivek */ custid, product_name, sale_value
from  vivek_test
where custid=1
and   sale_date between to_date(:b1,'MM-DD-YYYY HH24:MI:SS') and to_date(:b1,'MM-DD-YYYY HH24:MI:SS')+1-1/(24*60*60);

Run time Plan
Plan hash value: 2623397998
-----------------------------------------------------------------------------------------------
| Id  | Operation                    | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                |       |       |    18 (100)|          |
|*  1 |  FILTER                      |                |       |       |            |          |
|*  2 |   TABLE ACCESS BY INDEX ROWID| VIVEK_TEST     |    15 |   525 |    18   (0)| 00:00:01 |
|*  3 |    INDEX RANGE SCAN          | VIVEK_TEST_IDX |   300 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------------

Check for the Throwaway percentage now, which has gone up considerably. Let us pump in 5 (for 5 working days) days data, and as per the policy, generate fresh statistics. We already have a backup of the previously generated Statistics in STATS table.

truncate table vivek_test;
exec dbms_random.seed(0);

insert into vivek_test
with main as
(select mod(rownum,10) custid,
        round(dbms_random.value(1000,7000),0) sale_value,
        dbms_random.string('A',20) product_name
from    all_objects a
where rownum<=50)
select custid, s_date sale_date, sale_value, product_name
from    main, (select sysdate - level s_date from dual connect by leveltrue,method_opt=>'for all columns size 1');

variable b1 varchar2(20);
exec :b1:='06-09-2011';

select /*+ vivek */ custid, product_name, sale_value
from  vivek_test
where custid=1
and   sale_date between to_date(:b1,'MM-DD-YYYY HH24:MI:SS') and to_date(:b1,'MM-DD-YYYY HH24:MI:SS')+1-1/(24*60*60);

Run time Plan
Plan hash value: 1579208797

---------------------------------------------------------------------------------
| Id  | Operation          | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |            |       |       |    18 (100)|          |
|*  1 |  FILTER            |            |       |       |            |          |
|*  2 |   TABLE ACCESS FULL| VIVEK_TEST |    15 |   525 |    18   (0)| 00:00:01 |
---------------------------------------------------------------------------------

In just 4-5 days, the plan of the query changed from Index Scan to Full Table Scan. Further, the I/O’s of the query also increased from 10 (when it started) to 22 and then now to 63 per execution. The Cost of the Query remains same. In high concurrency, these Full table Scan’s would surely have an impact on the overall performance. Let us now revert the statistics from the backup table and check for the query performance.

Current Stats
new   2: where table_name='VIVEK_TEST'

OWNER                          PAR   NUM_ROWS     BLOCKS LAST_ANAL GLO
------------------------------ --- ---------- ---------- --------- ---
VIVEK                          NO        3250         60 13-JUN-11 YES

Elapsed: 00:00:00.45
SQL> exec dbms_stats.import_table_stats(user,'VIVEK_TEST',stattab=>'STATS');

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.04

Reverted Stats
OWNER                          PAR   NUM_ROWS     BLOCKS LAST_ANAL GLO
------------------------------ --- ---------- ---------- --------- ---
VIVEK                          NO        3000         60 13-JUN-11 YES

variable b1 varchar2(20);
exec :b1:='06-09-2011';

select /*+ vivek */ custid, product_name, sale_value
from  vivek_test
where custid=1
and   sale_date between to_date(:b1,'MM-DD-YYYY HH24:MI:SS') and to_date(:b1,'MM-DD-YYYY HH24:MI:SS')+1-1/(24*60*60);

Runtime Plan
Plan hash value: 2623397998

-----------------------------------------------------------------------------------------------
| Id  | Operation                    | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                |       |       |    18 (100)|          |
|*  1 |  FILTER                      |                |       |       |            |          |
|*  2 |   TABLE ACCESS BY INDEX ROWID| VIVEK_TEST     |    15 |   525 |    18   (0)| 00:00:01 |
|*  3 |    INDEX RANGE SCAN          | VIVEK_TEST_IDX |   300 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------------

After the Statistics were reverted, the application query regenerated the Better plan. This was enough to raise a doubt over the behaviour of Optimizer. On my test database, the actual issue occured on the 61st day. In your case, this may not be the case. For this example, I will also demonstrate you a way to find out on which day the Optimizer will change the plan.

The test case and demonstration clearly explains that the issue here is not because of the Optimizer Statistics, but is because of Inefficient Indexing, which is also evident from the Throwaway percentage mentioned above. To prove this point, I will create a composite index on Custid, sale_date. I will generate a fresh statistics, which will for 65 days. Once the Application query is run, I will pump in 365 days worth of data, regenerate fresh statistics and check for the Execution Plan (and throwaway percentage).

drop index vivek_test_idx;
create index vivek_test_idx on vivek_test(custid, sale_date);
exec dbms_stats.gather_table_stats(user,'VIVEK_TEST',cascade=>true,method_opt=>'for all columns size 1');
variable b1 varchar2(20);
exec :b1:='06-09-2011';

select /*+ vivek */ custid, product_name, sale_value
from  vivek_test
where custid=1
and   sale_date between to_date(:b1,'MM-DD-YYYY HH24:MI:SS') and to_date(:b1,'MM-DD-YYYY HH24:MI:SS')+1-1/(24*60*60);

SQL_ID        SQL_TEXT                                           BUFFER_GETS EXECUTIONS VERSION_COUNT
------------- -------------------------------------------------- ----------- ---------- -------------
2t8mwg94j89h8 select /*+ vivek */ custid, product_name, sale_val           6          1             1
              ue from  vivek_test where custid=1 and   sale_date
               between to_date(:b1,'MM-DD-YYYY HH24:MI:SS') and
              to_date(:b1,'MM-DD-YYYY HH24:MI:SS')+1-1/(24*60*60
              )

Plan hash value: 2623397998
-----------------------------------------------------------------------------------------------
| Id  | Operation                    | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                |       |       |     5 (100)|          |
|*  1 |  FILTER                      |                |       |       |            |          |
|   2 |   TABLE ACCESS BY INDEX ROWID| VIVEK_TEST     |    15 |   525 |     5   (0)| 00:00:01 |
|*  3 |    INDEX RANGE SCAN          | VIVEK_TEST_IDX |    16 |       |     2   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------------

The Production is now a year old, with new Index
truncate table vivek_test;
exec dbms_random.seed(0);

insert into vivek_test
with main as
(select mod(rownum,10) custid,
        round(dbms_random.value(1000,7000),0) sale_value,
        dbms_random.string('A',20) product_name
from    all_objects a
where rownum<=50)
select custid, s_date sale_date, sale_value, product_name
from    main, (select sysdate - level s_date from dual connect by leveltrue,method_opt=>'for all columns size 1');

select /*+ vivek */ custid, product_name, sale_value
from  vivek_test
where custid=1
and   sale_date between to_date(:b1,'MM-DD-YYYY HH24:MI:SS') and to_date(:b1,'MM-DD-YYYY HH24:MI:SS')+1-1/(24*60*60);

SQL_ID        SQL_TEXT                                           BUFFER_GETS EXECUTIONS VERSION_COUNT
------------- -------------------------------------------------- ----------- ---------- -------------
2t8mwg94j89h8 select /*+ vivek */ custid, product_name, sale_val           6          1             1
              ue from  vivek_test where custid=1 and   sale_date
               between to_date(:b1,'MM-DD-YYYY HH24:MI:SS') and
              to_date(:b1,'MM-DD-YYYY HH24:MI:SS')+1-1/(24*60*60
              )

Plan hash value: 2623397998
-----------------------------------------------------------------------------------------------
| Id  | Operation                    | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                |       |       |     5 (100)|          |
|*  1 |  FILTER                      |                |       |       |            |          |
|   2 |   TABLE ACCESS BY INDEX ROWID| VIVEK_TEST     |    15 |   525 |     5   (0)| 00:00:01 |
|*  3 |    INDEX RANGE SCAN          | VIVEK_TEST_IDX |    15 |       |     2   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------------

With the New Index, the performance is consistent and the performance matrix (I/O’s and throwaway percentage) are stable too. This answers the concern raised by the Customers “What is the cause of Sudden Degration ?”. In most of the cases, the change in behaviour is due to Application. You may notice that the change in number of rows from 3000 to 3250 (a difference of 250 rows) could cause the performance degradation. In this case, the number of blocks remained unchanged. As mentioned earlier, in my test case, it was the 61st day that triggered the plan change, which means, addition of just 50 rows. On an actual OLTP production system, the number of rows inserted on a day into a table is huge.

You may run the following pl/sql block on your test database to check the day on which the execution plan of above query would change.

drop index vivek_test_idx;
create index vivek_test_idx on vivek_test(custid);
This is for 120 Days and Statement Id from plan_table corresponds to the day.

declare
s_date	varchar2(30):=to_char(trunc(sysdate)-1,'MM-DD-YYYY');
e_date  varchar2(30):=to_char(trunc(sysdate)+1-1/(24*60*60),'MM-DD-YYYY');
l_stat	varchar2(1000);
begin
  execute immediate 'delete from plan_table';
  for i in 1..120
  loop
   execute immediate 'truncate table vivek_test';
   insert into vivek_test
   with main as
   (select mod(rownum,10) custid,
           round(dbms_random.value(1000,7000),0) sale_value,
           dbms_random.string('A',20) product_name
   from    all_objects a
   where rownum<=50)
   select custid, s_date sale_date, sale_value, product_name
   from    main, (select sysdate - level s_date from dual connect by leveltrue,method_opt=>'for all columns size 1');
   l_stat:= 'explain plan set statement_id '||''''||i||''''||' for select custid, sale_value, sale_date from vivek_test where custid=1
            and   sale_date between to_date(:b2,'||''''||'MM-DD-YYYY HH24:MI:SS'||''''||')
            and to_date(:b3,'||''''||'MM-DD-YYYY HH24:MI:SS'||''''||')';
    execute immediate l_stat using s_date, e_date;
--    dbms_output.put_line(l_stat);
   commit;
  end loop;
end;
/
column operation for a20
column options for a20
select statement_id, operation, options from plan_table
where operation='TABLE ACCESS'
order by to_number(statement_id);

STATEMENT_ID                   OPERATION            OPTIONS
------------------------------ -------------------- --------------------
1                              TABLE ACCESS         BY INDEX ROWID
2                              TABLE ACCESS         BY INDEX ROWID
3                              TABLE ACCESS         BY INDEX ROWID

56                             TABLE ACCESS         BY INDEX ROWID
57                             TABLE ACCESS         BY INDEX ROWID
58                             TABLE ACCESS         BY INDEX ROWID
59                             TABLE ACCESS         BY INDEX ROWID
60                             TABLE ACCESS         BY INDEX ROWID
61                             TABLE ACCESS         FULL <---- From 61st Day, the Plan Changed
62                             TABLE ACCESS         FULL
63                             TABLE ACCESS         FULL
64                             TABLE ACCESS         FULL

120                            TABLE ACCESS         FULL

120 rows selected.

One of my customer is evaluating Oracle Exadata and the mandate, in this case, is clearly increasing Market competition and sustenance. While the team were busy preparing a presentation, we felt, it would be a good idea to present the customer with the artifacts of the benefits in the actual production queries from their own production system. Post running these scripts a text file was presented to the customer. The customer was amazed to see the performance benefits.

Before we get into the API’s, a brief explanation on Exadata. EXADATA is a fully loaded Server, that comes in different configuration and is plug-and-play machine. It is basically an storage aware software and this intelligence is built in Storage Cells. These Cells store the data they contain and therefore, it is easier to get into the data block that contain the relevant rows directly. Unlike traditional storage that returns all the blocks to the Database Server, Exadata Storage Software returns only the relevant blocks that contain the data. Therefore, the main performance gain observed is due to the amount of data that travels between the Storage System and Database Server. Usually, the I/O bottleneck is observed in the production system if the amount of data that travels is enormous. With Exadata, these unwanted data is filtered at the storage level itself and relevant data traverses from the I/O subsystem to the Database Server. Further, its also reduces the amount of Latching, that would have otherwise required, to protect the blocks reads from the Storage into the SGA. Reduction in Latch Gets means a Scalable Application.

Following are the Scripts that were executed at the Customer site to simulate Exadata Performance and get the Performance Matrix of Application Queries. As mentioned earlier, Exadata Storage Cells provide the intelligence, these scripts use SQL Peformance Analyzer (SPA) to test cell storage in simulation mode and the matrix that is compared is IO_INTERCONNECT_BYTES. This is the amount of data, in bytes, sent by the Storage to the Database Server.

We start by creating SQL Tuning Sets. Once the STS is created, we need to capture all the Cursors (SQL Queries) in the Cache for a certain period, which is speficied in seconds. In my case, I captured the Queries for 180 Seconds. On a production system, this should be enough as the Shared Pool is usually warm and most of the queries are already found in the cache. If this value is increased, the Analysis (simulation) would take time. Once the Queries are captured, the Simulation is initiated, first with CELL_SIMULATION_ENABLED set to FALSE and then with CELL_SIMULATION_ENABLED set to TRUE. When the simulation is run, the queries are internally executed to generate and capture the io_interconnect_bytes value. Finally, it is time to compare the matrix and generate a TEXT or HTML file.

/* Create Tuning SQLSet */

begin
  dbms_sqltune.create_sqlset(
    sqlset_name => 'Trad_2_Exadata_Simulation',
    description => 'SQL Tuning Set for Exadata Simulation');
end;
/

/* time_limit is the time in number of seconds the Queries will be captured. */
/* In this case, it is 180 Seconds */
begin
  dbms_sqltune.capture_cursor_cache_sqlset (
    sqlset_name     => 'Trad_2_Exadata_Simulation',
    time_limit      => 180,
    repeat_interval => 10
  );
end;
/

variable Ret_Val clob
set long 999999
begin
  :Ret_Val :=
       dbms_sqlpa.create_analysis_task(
	sqlset_name       => 'Trad_2_Exadata_Simulation',
	task_name         => 'Trad_2_Exadata_Simulation_SPA');
end;
/

begin
   dbms_sqlpa.set_analysis_task_parameter ('Trad_2_Exadata_Simulation_SPA','TIME_LIMIT','UNLIMITED');
   dbms_sqlpa.set_analysis_task_parameter ('Trad_2_Exadata_Simulation_SPA','LOCAL_TIME_LIMIT',300);
   :Ret_Val := dbms_sqlpa.execute_analysis_task (
	task_name        => 'Trad_2_Exadata_Simulation_SPA',
	execution_type   => 'test execute',
	execution_name   => 'TRADITIONAL',
	execution_params => DBMS_ADVISOR.arglist ('cell_simulation_enabled','FALSE'),
	execution_desc   => 'Exadata simulation disabled'
    );
end;
/

begin
   :Ret_Val := dbms_sqlpa.execute_analysis_task (
	task_name        => 'Trad_2_Exadata_Simulation_SPA',
	execution_type   => 'test execute',
	execution_name   => 'EXADATA',
	execution_params => dbms_advisor.arglist ('cell_simulation_enabled','TRUE'),
	execution_desc   => 'Exadata simulation enabled'
    );
end;
/

begin
   :Ret_Val := dbms_sqlpa.execute_analysis_task (
	task_name       => 'Trad_2_Exadata_Simulation_SPA',
	execution_type  => 'compare performance',
	execution_params=> dbms_advisor.arglist('comparison_metric','io_interconnect_bytes')
  );
end;
/

set lines 300
set pages 0
set trimspool on
set long 999999
spool report.txt
select dbms_sqlpa.report_analysis_task ('Trad_2_Exadata_Simulation_SPA','TEXT','TYPICAL','ALL') from dual;
spool off

Oracle also provides with a script tcellsim.sql under $ORACLE_HOME/rdbms/admin. This can also be used to run the simulation. The output from the production enviroment is pasted below. I have trimmed the output to make this short.

General Information
---------------------------------------------------------------------------------------------

 Task Information:                              Workload Information:

 ---------------------------------------------  ---------------------------------------------
  Task Name    : Trad_2_Exadata_Simulation_SPA   SQL Tuning Set Name        : Trad_2_Exadata_Simulation
  Task Owner   : SYS                             SQL Tuning Set Owner       : SYS
  Description  :                                 Total SQL Statement Count  : 1335

Execution Information:
---------------------------------------------------------------------------------------------
  Execution Name             : EXEC_10626             Started             : 12/30/2010 14:34:36
  Execution Type             : COMPARE PERFORMANCE    Last Updated        : 12/30/2010 14:38:18
  Description                :                        Global Time Limit   : UNLIMITED
  Scope                      : COMPREHENSIVE          Per-SQL Time Limit  : 300

  Status                     : COMPLETED              Number of Errors    : 7

  Number of Timeouts         : 4

  Number of Unsupported SQL  : 77

SQL Details:
-----------------------------
 Object ID            : 3160

 Schema Name          : F_LEA

 SQL ID               : 5um8xff95jn6x

 Execution Frequency  : 266122

 SQL Text             : SELECT 'x' FROM L_AGREEMENT_DTL WHERE AGREEMENTID =

                      :b1 AND STATUS IN ( 'A','L','O','X','C' )

Execution Statistics:
-----------------------------
----------------------------------------------------------------------
|                       | Impact on | Value      | Value    | Impact |
| Stat Name             | Workload  | Before     | After    | on SQL |
----------------------------------------------------------------------
| elapsed_time          |           |  57.165548 |          |        |
| parse_time            |           |    .000424 |          |        |
| cpu_time              |           |      14.94 |          |        |
| user_io_time          |           | 1181.09798 |          |        |
| buffer_gets           |           |    1360874 |          |        |
| cost                  |        0% |          2 |        2 |     0% |
| reads                 |           |     340101 |          |        |
| writes                |           |          0 |          |        |
| io_interconnect_bytes |    84.13% | 2786107392 | 12337536 | 99.56% |
| rows                  |           |          1 |        1 |        |
----------------------------------------------------------------------
SQL Details:
-----------------------------
 Object ID            : 2839

 Schema Name          : F_LEA

 SQL ID               : 1xny80dtnbf6f

 Execution Frequency  : 157981

 SQL Text             : SELECT INSTALTYPE,EMI FROM L_AGREEMENT_DTL WHERE

                      AGREEMENTID = :b1

Execution Statistics:
-----------------------------
---------------------------------------------------------------------
|                       | Impact on | Value     | Value    | Impact |
| Stat Name             | Workload  | Before    | After    | on SQL |
---------------------------------------------------------------------
| elapsed_time          |           |  3.045475 |          |        |
| parse_time            |           |   .000509 |          |        |
| cpu_time              |           |       .65 |          |        |
| user_io_time          |           | 65.463246 |          |        |
| buffer_gets           |           |     57237 |          |        |
| cost                  |        0% |         2 |        2 |     0% |
| reads                 |           |     54172 |          |        |
| writes                |           |         0 |          |        |
| io_interconnect_bytes |     7.77% | 443777024 | 12243136 | 97.24% |
| rows                  |           |         1 |        1 |        |
---------------------------------------------------------------------

From this Simulation, it is clearly visible that the performance benefit using Exadata is tremendous. To make this shoirt, I have pasted only two comparisons, but there were other queries as well that showed a significant amount of performance benefit. In the first query above, the performance benefit on the overall system load is around 85% and the amout of data that traverses between the storage to database server has dropped by 99.56% (from 2.7 GB to 12 MB). This is significant. In the second query as well, since the table is same, the drop is from 443 MB to 12MB, a saving of 97%.

http://www.rampant-books.com/book_1001_advanced_techniques_oem_grid_control.htm

The Book Synopsis is as follows:

Oracle’s Enterprise Manager Grid Control is recognized as the IT Industry’s leading Oracle database administration and management tool. It is unrivalled in its ability to monitor, manage, maintain and report on entire enterprise grids that comprise hundreds (if not thousands) of Oracle databases and servers following an approach that is consistent and repeatable.

However, Enterprise Manager Grid Control may seem daunting even to the most advanced Oracle Administrator. The problem is you know about the power of Enterprise Manager but how do you unleash that power amongst what initially appears to be a maze of GUI-based screens that feature a myriad of links to reports and management tasks that in turn lead you to even more reports and management tasks?

This book shows you how to unleash that power.

Based on the Author’s considerable and practical Oracle database and Enterprise Manager Grid Control experience you will learn through illustrated examples how to create and schedule RMAN backups, generate Data Guard Standbys, clone databases and Oracle Homes and patch databases across hundreds and thousands of databases. You will learn how you can unlock the power of the Enterprise Manager Grid Control Packs, PlugIns and Connectors to simplify your database administration across your company’s database network, as also the management and monitoring of important Service Level Agreements (SLAs), and the nuances of all important real-time change control using Enterprise Manager.

There are other books on the market that describe how to install and configure Enterprise Manager but until now they haven’t explained using a simple and illustrated approach how to get the most out of your Enterprise Manager. This book does just that.

This book covers everything on Enterprise Manager Grid Control 11g. I am sure that this book should help you in managing and maintaining a large Enterprise site using Grid Control.