Fancy new Disk array technology

Well first off, I need to put a big disclaimer down.  These are my opinions, and my opinions only.  These to do not reflect the opinions of my employer, my spouse or my dog.

I was watching some twitter updates go by and this blogpost caught my eye. 
http://chucksblog.emc.com/chucks_blog/2011/10/shifts-happen.html

This blog was talking about new disk technology, and part of the covered the idea of FAST technology.  If you haven't heard of FAST (this is the EMC name, I'm sure other vendors have their own flavors), it is disk technology that moves blocks to the best tier of storage automagically. Really !  The idea is that you buy an array with 3 different tiers of disk.  Flash, Fibre channel, and Sata.  The disk array learns the patterns for the data access, and moves the data to the appropriate tiers.  Sounds great right  ?  It does make sense.. 
Let take an example...  Let say that you are a supplier and you supply parts for 100,000 small businesses.  You keep historical data their orders for 5 years for reference.  Whenever they place a new order you reference their latest orders to find patterns.

So following this workload you can guess what happens.. The current data for your customers stays in fiber channel (everything starts in fibre channel),  The old data gets migrated to sata, and your customer master data will most likely go the Flash.  All well and good.  Even though customers only order every month, their recent activity gets moved to a higher tier disk, and all that old history gets moved to Sata.  

Now lets throw in a physical standby, dataguard .

With dataguard, we are writing the new blocks of history, and they are not accessed (this is cold standby).  If you mix this data with other applications that are busy, all your data for the standby database is surely going to end up in Sata over time.. This makes perfect sense to the algorithms for the array.  This historical data (or even current data) isn't accessed. For your standby sata it is !!

Bang... Sinking feeling.... wham.. You do a failover. 

Now lets see what happens.. All your data is in Sata.  You are now accessing, and trying to give your customers the same performance they are used to.  You system is slow.  You have 100,000 business, that access data over the course of the month.  How long do you think it takes to move all the data from SATA to Flash or Fibre ?  It could take quite a while for your system to learn the new patterns, and during this time your old primary (now standby) has it's data pattern getting changed. The data is getting migrated to SATA.  You stay in your alternative site for a month, fail back, and guess what.. WHAM again.  The disk array has to learn the pattern again.

As I said, this is all conjecture, and solely my opinion.

Configuring an Exadata (follow up)

Now that we have the exadata and it is up and running, we are working on getting it configured for ease of maintenance.  I know there are some notes from metalink that can be helpful.

The first thing I wanted to do was get the machine (and all the hardware) configured with OCM (oracle Configuration manager).  Like most things with the exadata, there is a special configuration piece for this called the "mass deployment Kit"..  Here is a link for the latest information on it.

 On MOS [ID 1319476.1]

I am still in the process of getting this configured by using the Oracle support Hub (or repeater).  A lot of this information is contained in the PDF mentioned in the My Oracle Support note.  As you can imagine, the exadata is usually installed in a companies core infrastructure, far within any firewalls.  Connecting directly out to the internet isn't always possible, so setting up a repeater (like a proxy) as part of grid (or separately) will help get your configuration information sent up to oracle support.

The second item is Grid/Cloud 12c.  I have to say that I set up Cloud 12c for the exadata about 48 hours after it came out.  It was relatively easy.  You just add the database nodes (and push out the agents), then once the database nodes are done, you use the tools with grid to walk you through discovering all the components (by starting with one of the database nodes). It all worked well, and there are some notes now on this.  Oracle Enterprise Manager Cloud Control 12c Setup Automation kit for Exadata

So the exadata is close to be set up.. I believe setting up OCM is one of the most challenging things.  One of the first steps is to create a spreadsheet with the configuration information.. Following this is the steps from the documentation.  The one complaint I would state is that a lot of the information for the OCM configuration is the same information provided to the "one" script.  I am hoping down the road the ACS group (or whoever does the configuration), also configures OCM, or at least provides the input for it.  OCM isn't necessary, but I think having it configured will save a lot of time when we need to open up an SR.



From the manual......

**************************************************************

Use your favorite spreadsheet editor to create the input csv file. To facilitate the use of the input

file, the Mass Deployment document contains a template for you to use in providing the field
values (ocm_companion/distributions/ocm/md/sample_input.csv). See Section 2.4.5 “Input File”
in Mass Deployment documentation for details on the input file format.


Much of the information required as input into Mass Deployment can be retrieved from the
Exadata Database Machine configuration worksheets. Please see Appendix A for examples.


1. Copy/rename the sample_input.csv file (e.g., getinfo_exadata_csi_input.csv). This file
can be used as a template for entering the data for each host on which OCM will be
deployed and/or configured. Add information for all the compute nodes as listed below.


a. Action: Set this column to “get_info” to retrieve information about the state of the
OCM collector in all the Exadata Database (compute node) Oracle homes.


b. Host-Name: Host name of the node.


c. Host-User: OS user that owns the Oracle home.


d. Host-Password: Password for the OS user - set to “__PROMPT__” (two
underscores before and after). See Section 2.4.4 Credential s in the Mass
Deployment Documentation or Appendix B of this document for secure ways of
providing the password. If the same credentials are being used for multiple
hosts, another option is to use a password group name in the password.csv file
as described in Section 2.4.1 of the Oracle Configuration Manager Companion
Distribution Guide .


e. Oracle Home: Oracle Database home location.

f. Db SID: Set the Database SID for the last database host in the input file. This is
required for Mass Deployment to instrument the database for configuration data
collections. This script need only be run on one of the database hosts, but must
be run after the last server is installed.


g. DB Type: Set to 'db' for the last database host in the input file.Specify only for
Install and Instrumentation actions.


h. ML-User: Enter the customer's MOS Account username (email address).


i. ML-CSI: This field holds the Exadata Hardware Customer Support Identifier
(CSI) an can be used in conjunction with the ML-User field to authenticate OCM
uploads. If the CSI is not know, see Appendix B.


j. ML-Pwd: Leave it as blank (should only be used if the CSI is not known).


k. DB-user: Database username required to instrument the database.


l. DB-Pwd: Database user password

Grid control 12c

I've been spending my week playing with Grid Control 12c. I know it has only been out just over a week, but I was very excited to see if it is that much better than grid 11g. My company is currently rolling out Grid 11g, and I wanted to see if we should be pushing for grid 12c right on it's heels.

I am extremely impressed with this product, so much so that I set up a virtual environment with Grid 12c to check it out.


I've spent the last couple of days getting my exadata configured in grid 12c. After a couple of false starts (and reinstalling of the agent) I finally got it up and running. These are my lessons learned

• First discover your database nodes, and make sure the name you use is the default fully qualified name.
• Add the database machine as a target, and make sure you have all the passwords including the nm2user on the IB switches (password is changeme), and you also need the id and password for the PDU (admin/admin).

Once you get these all set, Grid 12c will recognize your machine, and you will see wonderous things. Here are 2 example screens from an exadata..

The first one shows the IB traffic through the switch,

The second one shows the combined load on the Storage cells.

Even if you don't have an exadata, here is my favorite ASH analytics. Notice the timeframe is very small.

It is definately worth checking out.

10 x 10=100. Larryisnm's from oow11

I wanted to put down my impressions on the big announcements at openworld this year. First is Exalytics.. Analyitics at the speed of throught. This is an intriguing product, and I can definately see the uses for applications where real time analytics is key. I think for most of us, this appliance is going to be out of our range. I know I don't know of any business cases. No prices was mentioned either. Second was EM grid 12c. Now this was pretty impressive. I was surprised on all the enhancements that was put into it. It really seems to do a nice job of centralized for Cloud support. I was especially impressed with the virtualization pieces. The provisioning, and support of virtual environments is a great component. It is also a very big carrot for those companies turning to virtualization, and aren't sure whether to choose VMware or Oracle VM. Big Data Appliance. - This one I am waiting to see more specs on. it looks interesting, but what is the licensing model ? I can't believe that there is no software licensing (other than OS pieces). All these are interesting announcments, but I think the 12c features was the most interesting to me. IF only the documentation was available I would install it right now.

partitioning Local vs Global

In my last post I talked about creating a function based index on a GTT after my query plans went to hell after partitioning.  Someone asked me to elaborate why my query plans went south.

Well to start with, I deal with very large tables.. Not terribly large (250g 2 billion rows).  We are in the processess of partitioning this data, so we can purge it.. The performance on the data is very good, but we keep eating up disk space.

Seems simple enough right ? partition by date ranges, with some hash partitions thrown in on the column used the most for lookups.  Nice and neat. At this point we have 116 partitions.  Smaller is better right.

Since the whole reason for doing this is being able to purge, we created local indexes on almost all the columns except for the primary key.  Being able to maintain the partitions is critical.

Doing all this I assumed we would be OK with local indexes. The application does index lookups, and the ones that don't use the primary key (or the hash partitioned key) are close to unique.

How long can a lookup take with an index and number of distict values = num_rows.  Easy..

Then the dbreplay came, and the queries were slower.. much slower.. plan was similar but buffer gets was off.

% Total Gets Gets per Exec #Executions Exec Time (ms) per Exec CPU Time (ms) per Exec I/O Time (ms) per Exec Physical Reads per Exec #Rows Processed per Exec #Plans   
SQL Id 1st 1st Total 2nd 2nd Total Diff 1st 2nd 1st 2nd 1st 2nd 1st 2nd 1st 2nd 1st 2nd 1st 2nd 1st/2nd/Both SQL Text 
d76xhcfh5dsrs 0.71 1.42 25.46 50.96 24.75 12,401.18 875,159.54 2,389 2,518 347 2,819 135 2,130 203 351 23.43 82.44 9.34 9.48 1/ 1/ 2 SELECT vpcyd_wrkr_id, vpcyd_cl... 
.

It's hard to see above, but the 2 critical values are 12,401 buffer gets vs 875,159 buffer gets.. and 347 ms/exec vs 2,819 ms/exex

Buffer gets was making a huge difference with my partitioned tables.. Now to dig into the trace file.

Here is the part that really stood out..147,157 cr (buffer block reads), to get 54 rows of data.. wow..

       545        545        545                     PARTITION RANGE ALL PARTITION: 1 29 (cr=147157 pr=19 pw=0 time=1142973 us cost=232 size=0 card=1)
       545        545        545                      PARTITION HASH ALL PARTITION: 1 4 (cr=147157 pr=19 pw=0 time=1126073 us cost=232 size=0 card=1)
       545        545        545                       INDEX RANGE SCAN PIDX_CUST_ID PARTITION: 1 116 (cr=147157 pr=19 pw=0 time=1032020 us cost=232 size=0 card=

I isolated this lookup, and found that it was a "unique" key (it had no duplicate values).. Why would 545 rows of data take all that time? (this was where the time was going).

I created a small query, and did a index lookup for one row and compared partitioned vs non-partitioned.

                SQL_ID       PLAN_HASH_VALUE BUFFER_GETS EXECUTIONS CPU_TIME ELAPSED_TIME AVG_HARD_PARSE_TIME APPLICATION_WAIT_TIME CONCURRENCY_WAIT_TIME 
Partitioned     gz67xt981w53p  3,540,849,128       7,323          6  472,928      642,455  625,819       0 168,862 
Non-Partitioned gz67xt981w53p    791,655,517          32          6    3,999        4,473    2,847       0 0 


 

Comparing the index partitioned vs non-partitioned, (with 116 subpartitions), you can see the difference. 3,999 ms vs 472,928 ms .  What caused me the biggest issue is that I didn't realize it was doing a nested loop, 54 times.. this made the difference 36,000 ms vs 4,256,352 ms.  116x longer with a local partion vs global. 

Lesson learned was that with partitioning you need to balance performance with maintainability.. Local indexes can be very expensive.  Especially with nested loops.

Why are the developers using functions ?

Hi all,

  I have been working all week on trying to figure out why a query went to hell when we partitioned the tables.  I dug into it, and found one good fix.. But  I can't implement it.

The detail on what happened in my last post.. Keep in mind I found that issue, but working through this one, and moving the bottleneck.

Here is the problem.. They are joining to a GTT (global temporary table), but they are using a function on the column in the table. ARGH.. They are making it impossible for the optimizer to find the best plan.

Here is an example of what's happening...

First here the GTT

CREATE GLOBAL TEMPORARY TABLE my_temp_table (
  tmp_strt_dt  date,
  tmp_end_dt   date
) ON COMMIT preserve ROWS;

Here is table and lets load 128 rows of data into it.

create table test_table
(  strt_dt   date,
   end_dt    date,
   col1      varchar(1));

insert into test_table values(sysdate-1000,sysdate+1000,'Y');
insert into test_table select * from test_table;
insert into test_table select * from test_table;
insert into test_table select * from test_table;
insert into test_table select * from test_table;
insert into test_table select * from test_table;
insert into test_table select * from test_table;
insert into test_table select * from test_table;
commit;

Now lets insert into the temporary table, and analyze both tables.

insert into my_temp_table values(sysdate,sysdate);

exec dbms_stats.gather_table_stats(ownname=> null, tabname=> 'MY_TEMP_TABLE',estimate_percent=>null, cascade=>true, method_opt=> 'FOR ALL COLUMNS SIZE 1');
exec dbms_stats.gather_table_stats(ownname=> null, tabname=> 'TEST_TABLE',estimate_percent=>null, cascade=>true, method_opt=> 'FOR ALL COLUMNS SIZE 1');

Now for my query ..

select * from my_temp_table ,test_table 
where "END_DT">=TRUNC("TMP_STRT_DT") AND                                                                              
      "STRT_DT"<=TRUNC("TMP_END_DT");

and the explain plan.. Notice the cardinality of 1, though there are 128 rows that match

Plan hash value: 1231029307

------------------------------------------------------------------------------------
| Id  | Operation          | Name          | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |               |     1 |    34 |     4   (0)| 00:00:01 |
|   1 |  NESTED LOOPS      |               |     1 |    34 |     4   (0)| 00:00:01 |
|   2 |   TABLE ACCESS FULL| MY_TEMP_TABLE |     1 |    16 |     2   (0)| 00:00:01 |
|*  3 |   TABLE ACCESS FULL| TEST_TABLE    |     1 |    18 |     2   (0)| 00:00:01 |
------------------------------------------------------------------------------------


PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------

   3 - filter("END_DT">=TRUNC(INTERNAL_FUNCTION("TMP_STRT_DT")) AND
              "STRT_DT"<=TRUNC(INTERNAL_FUNCTION("TMP_END_DT")))

So what to do ??? I removed the trunc function, and the cardinality was right...

select * from my_temp_table ,test_table 
where "END_DT">="TMP_STRT_DT" AND                                                                              
      "STRT_DT"<="TMP_END_DT";

Plan hash value: 1231029307

------------------------------------------------------------------------------------
| Id  | Operation          | Name          | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |               |   128 |  4352 |     4   (0)| 00:00:01 |
|   1 |  NESTED LOOPS      |               |   128 |  4352 |     4   (0)| 00:00:01 |
|   2 |   TABLE ACCESS FULL| MY_TEMP_TABLE |     1 |    16 |     2   (0)| 00:00:01 |
|*  3 |   TABLE ACCESS FULL| TEST_TABLE    |   128 |  2304 |     2   (0)| 00:00:01 |
------------------------------------------------------------------------------------

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------

   3 - filter("END_DT">="TMP_STRT_DT" AND "STRT_DT"<="TMP_END_DT")

Unfortunately, I can't change the code.. How do I get the optimizer to get the right cardinality ?? Function based indexes to the rescue. Here is what I did.. First create the indexes on the 2 columns.

create index my_temp_table_fbi1 on my_temp_table(TRUNC("TMP_STRT_DT"));
create index my_temp_table_fbi2 on my_temp_table(TRUNC("TMP_END_DT"));

Next insert into the table, and gather stats.. Notice that I am using "hidden" column clause.

insert into my_temp_table values(sysdate,sysdate);

exec dbms_stats.gather_table_stats(ownname=>null, tabname=> 'MY_TEMP_TABLE',estimate_percent=>null, cascade=>true, method_opt=> 'FOR ALL HIDDEN COLUMNS SIZE 1');

Now to run my query and look at the cardinality.

elect * from my_temp_table ,test_table 
where "END_DT">=TRUNC("TMP_STRT_DT") AND                                                                              
      "STRT_DT"<=TRUNC("TMP_END_DT");

Plan hash value: 1231029307

------------------------------------------------------------------------------------
| Id  | Operation          | Name          | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |               |   128 |  6400 |     4   (0)| 00:00:01 |
|   1 |  NESTED LOOPS      |               |   128 |  6400 |     4   (0)| 00:00:01 |
|   2 |   TABLE ACCESS FULL| MY_TEMP_TABLE |     1 |    32 |     2   (0)| 00:00:01 |
|*  3 |   TABLE ACCESS FULL| TEST_TABLE    |   128 |  2304 |     2   (0)| 00:00:01 |
------------------------------------------------------------------------------------

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------

   3 - filter("END_DT">=TRUNC(INTERNAL_FUNCTION("TMP_STRT_DT")) AND
              "STRT_DT"<=TRUNC(INTERNAL_FUNCTION("TMP_END_DT")))

Notice that the index is not used for the query plan, but by having the index, and gathering statistics, the optimizer is able to figure out the correct cardinality even though a function is used for the column. Problem solved without changing the query.

As always, you can find my script here

My LIO silly little benchmark

I have been working on a benchmark for LIO.  I know there are TPC and TPH transactions numbers that are published, on CPU speeds, but how much does that directly releate to LIO's, the heart of an Oracle database ?

  To help benchmark, I wrote a little pl/sql package.  This packages takes the Zip Code database, and randomly picks  some rows with a cursor (about 1 % of the table).  This package is then called by swing bench, and I put a "think time" in it for each execution of the package.

Ideally, I try to execute it up to what the Server can handle.. This was especially usefull with the benchmarking I did in a previous post on hyperthreading.

I was interested in what anyone else does ?  I try to do a LIO lookup, and compare numbers between servers.  By doing this I have a pretty good idea how many LIO's an AMD server can handle per second, an Intel server can do, and different architectures (2 socket, 4 socket, and 8 socket).. I even benchmark virutalization to see how much of an overhead is caused from the Software.

This may not be the best way (it excludes what happens with updates (redo logs etc), and how much physical I/O's affect the workload.

Any ideas would be appreciated.  I would love to come up with a nice reproducable benchark, and then maybe create a dbcapute of it, and do a dbreplay on different architectures ? Would that be more accurate.

I know many of you will say the line "well it depends on the workload", maybe the benchmarking that comes with swingbench is good enough ??

I'm just tired of reading server bencharks, and finding that for an oracle database, those benchmarks aren't very meaningful.

I would also love to do some benchmarking with Solaris X86, and RHEL/OEL on an 8 socket box.

I would also love to learn what anyone else has learned ?  I am especially interested how 8 socket intel servers compare with 2 socket. I'm seeing some pretty increadable numbers from 2 socket servers (almost 2x the speed of 8 socket).  I'm wondering if anyone else is seeing some measureable differences.

I'm starting to move to "go wide"  camp rather than go high camp for increasing server power.  The blade servers are being more, and more powerful, and you can have more memory local to the CPU. Increasing CPU sockets just increases hops to get those LIO's done, costing time, waits, latches. etc. etc.

So here is a piece of my LIO benchmark...

CREATE TABLE "KILLER" ("CC_ID" NUMBER(20, 0) NOT NULL ENABLE)  ;

/*  import 55,000 rows of distinct data */
CREATE PROCEDURE          kill_lio IS
   my_count number := 1;
   my_executions number;
   my_buffer_gets number
   my_cpu_time number;
   my_elapsed_time number;

error_code number;

BEGIN
for i in 1..10000 LOOP


select count(distinct cc_id) into my_count from kill_lio.killer;

end loop;

select executions,buffer_gets,cpu_time,elapsed_time into my_executions,my_buffer_gets,my_cpu_time,my_elapsed_time 
from sys.v_$sqlstats where sql_id='2j5tvp5rdzmym';
 
 dbms_output.put_line('exectutions:                          ' || to_char(my_executions,'999,999,999'));"
dbms_output.put_line('buffer gets:                          ' || to_char(my_buffer_gets,'999,999,999'));"
dbms_output.put_line('cpu time:                             ' || to_char(my_cpu_time,'999,999,999'));"
dbms_output.put_line('elapsed time:                         ' || to_char(my_elapsed_time,'999,999,999'));"
dbms_output.put_line('elapsed time per execution(ms)   :      ' ||to_char( my_elapsed_time/my_executions/1000,'999,999.9'));"
dbms_output.put_line('buffer_gets/second:                   ' ||to_char( my_buffer_gets/(my_elapsed_time/1000000),'999,999,999'));"

END;  -- exception handlers

and here is the output I use to compare.  I look at the average elapsed time, and buffer_gets/second to benchark systems.

executions:                                 10,000
buffer gets:                             1,190,000
cpu time:                              190,983,974
elapsed time:                          191,374,061
elapsed time per execution(ms)    :           19.1
buffer_gets/second:                          6,218

Here is the AWR report from the execution

Oracle Database Appliance

I have spent the day on an oracle call, and reading all the subsequent tweets that follow.  I think the best way to describe the appliance is that it is NOT a mini-exadata, but it is a simple rac appliance.

My impression is that it is a nice product for the small to mid market, but those us working with the bigger toys I don't see the gain.

I know, I've spent more days than I care to remember schooling the SA's on how to set up an interconnect, and ensure that all the IP's are correct.  I've worked with Storage administrators on how to present the disks, and make them available to ASM, and I've worked with networking on the ranges of IP's I need for scan, interconnect, etc. etc.  I'm sure you get the picture.

I also think that people like me that work in a big organization and have a team to handle these tasks, are probably going huh ? what is this? 

Personally, I don't see the big deal in this.. I see lots of dissadvantages.

• These Appliances cannot be clustered. What they have in them is all they will ever have in them.
• The 2 database nodes have 96g of memory, not a lot in today server sizes..
• There is no storage software like the exadata. No HCC, no offloading, no infiniband
• This is local disk in the appliance, meaning no cloning, storage virtualization, etc.
• The interconnect is 1ge, not infiniband.
• You CANNOT hook up fiber to this server, ever.
• It runs OEL, NOT redhat linux.. the differences are getting greater over time.
• This is a closed system with specific patch sets that need to be maintained to a short list of acceptable patches.

I know for a small, to midsize, the ideal of creating a new rac system in 2 hours is thing of beauty, but for bigger companies, there isn't a lot there.

Especially without the Exadata candy filling (infiniband, HCC, offloading, storage indexes).

I still think virtualiation is the direction, and this is a step in the opposite direction.  There may be a few takers, but I think companies will realize that virtualization is a better direction than a single closed appliance.

We will see.. just some thoughts.

Delphix, Solaris and ZFS

I was inspired to write a new blog posting after seeing a great presentation on Delphix.  Delphix is an extremely interesting product to help with the managing of many copies of an oracle database.  This product goes hand-in-hand with a virtualization product like VMWARE. What VMWare is servers, Delphix is to storage.

First I want to babble on a bit.. Why ? Because this is my blog.

I have used Solaris for many years and I am a big fan of Sparc/Solaris, and everything that it became over the years.  I know in the last couple of years, it has kind of fallen down, and Linux x86 has been growing leaps and bounds.  The most impressive piece of Solaris however has been ZFS.  ZFS offers snapshotting, cloning, lots of fantastic options built into the OS.  All very easy for the DBA.

I was very excited when ACFS came out (anyone remember).  It was the coolest thing going.  It looked a lot like ZFS built on top of ASM.   It had the potential to become a fantastic tool.  I was really, really, really hoping that down the road ACFS, and ZFS would combine, and replace the cooked file system with an Oracle File System that Rocked.    Well in case you didn't know ACFS has morphed into the "Cloud File System" or Cloud FS.  This is a licensed product used for the sharing of directories in a "cloud" environment.  Also, if you didn't know, ACFS won't run on the Exadata.  2 nails in the Coffin of this fantastic vision that I had in my head.

I was devastated for a while.

Why the background ? Because Delphix picked up where this vision died.  They offer the ability to do snapshots, along with deduplication, and compression to boot.

Delphix was founded by Jedidiah Yueh, who founded Avamar (dedup software that was sold to EMC),  The company also has  Karthik Rau, ad VP of Products who worked on VMware.  Lots of very bright people that were already involved in the founding technology.

Delphix makes the creation of database copies, easy, and it save space to boot.  I will post the presentation that I saw once I can link to it.

Definitely something worth checking out, and to watch for in the infrastructure Arsenal.

My quest for a performance warehouse

I have written on this before, and at openworld last year I've talked to some of the presenters about this.

My goal is to to create a performance data warehouse.  What does this mean ?

I want to take all the AWR data from all the databases in the corporate infrastructure, and create one giant repository. 

Well I just got one step closer.

I have a repository server set up already. It is just an 11.2.0.2 database with lots of space (about 3 tb), running on a Linux blade server.

My first test was to take all the AWR data from an 10g database we just retired running on HPUX. Since this database was moved to Linux (cross endian), transportable tablespaces was used, and all the AWR data was not brought over.

First step.. Get the data out of original source.

In order to do that you can use awrextr.sql in the $ORACLE_HOME/rdbms/admin directory.  I noticed (and found from other blogs) that this just calls sys.dbms_swrf_internal.awr_extract.

So 

STEPS on SOURCE.

1. Create user "awrextract" identified by xxxxx
2. grant connect ot awrextract
3. grant execute on sys.dbms_swrf_internal to awrextract
4. Login as awrextract
5. execute awr_extract procedure

exec sys.dbms_swrf_internal.awr_extract(dmpfile=>'awr_data',dmpdir=>'tmp_dir',bid=>9920,eid=>9925);

Notes

• The procedure automagically puts a .dmp on the dump file name
• You need to know the start and end snapshot you want to export
• The dmpdir needs to be a precreated directory and it has to be a cooked filesystem (non-asm).  I found that it also puts the logfile here, and that's why it has to be cooked.
• You cannot use dbms_file_transfer, especially going across endian types

 6. copy the file to my repository server

STEPS on the Destination

1.  Create schema to import the data into
2. Import the file in

DBMS_SWRF_INTERNAL.AWR_LOAD ( SCHNAME => 'AWR', dmpfile => 'awr_data', dmpdir => 'TMP_DIR');

  3.  Once imported move the awr data to sys catalogs

DBMS_SWRF_INTERNAL.MOVE_TO_AWR ( SCHNAME => 'AWR');

FINALLY

All this worked flawlessly, and then I tried to run COE_PROFILE.

IF you didn't know COE_PROFILE.sql  is part of sqlt and allows you to transfer profiles from one database to another by creating a script to build the profile.. It all works from my repository.

I will write about this again, but I am physched to finally get this off the ground.

UNYOUG meeting. Presentation on Cardinality Feedback

Well I've gotten my presentation in a pdf to take a look at.  Here it is. Short and sweet

Hyperthreading

I am am working in standing up a cluster of 2 database servers that look an aweful lot like an exadata 2-8 (only without the infiniband)..  It is a 2 node cluster of 8 socket, 8 core processrs with 256g of memory.

We started with a standard configuration and hyperthreading was turned on.. This means that Oracle sees the 64 cpu, as 128 cpus.  We were getting periodic reboots every Saturday morning.. The culprit was finally found as the analyze.. Why ??? because we have a 2 node rac, with 128 CPU's per, the degree of parallelization chosen was 512 !  With an analyze running with 512  parallel processors we were running out of memory.. The final diagnoses was to change parallel_max_servers to 64 to cut back on the parallel servers utilized. The second recommendation from Oracle was to turn off Hyperthreading.

We all said Huh ?  Hyperthreading gives you 2x the throughput because it doubles your CPU's right ???

Well lets' see.

I took my 2 node cluster and split it in 1/2  First half (A) is non-hyperthreaded, second half (B) is hyperthreaded.

I then took swingbench and threw a workload at it with lots of logical I/O.

Here is the benchmark from (A)   non-hyperthreaded

Notice we are getting 2062 transactions per minute.




 OK.. Lets see what happens with (B) hyperthreading.

Notice we are getting 1,685 transactions per minute. 

LESS wow lets look at our AWR compare of both nodes (same period) and see what the database says.

Notice  that the gets per execution match for both nodes, and the rows processed. The difference is in the exec time(ms) per exectuion, CPU time and executions.

IT is faster non-hyperthreaded.   The CPU's are doing more work with hypertrheading off, and pushing more work through.

OK... I've shown that for a non-saturating workload (under 64 active processes) on a 64 CPU box, hyperthreading off is faster.

Now let's saturate !!

First hyperthreading off.

Notice I can get the throughput up to 7,489 transactions per minute.

Now lets check the second node with hyperthreading on.

Notice we are getting 6552.. Darn !!

It looks like hypthreading isn't helping us eithor !! 

 

Now lets check the AWR for the 2 nodes.

Notice that once again the rows processed match up and the gets/execution match.

The number of executions is greater, the execution time is less, and the CPU time is way less for for hyperthreading off.

For a Logical I/O intensive workload, it looks like Hyperthreading is not as efficient.

I would love to hear any other experiences/opinions.

I just wanted to add on to this a thought as to why hypertrheading is such a big deal... Virtualization.  With 2x as many "cpus", you can fine tune the CPU a lot more.. You can carve up the server into smaller pieces.



Cardinality Feedback detailed example

Well, I have often written about cardinality feedback, and I have been striving to find out more detail on how it works, how to determine why it was used, and how to determine when it was used.

I have ran through a series of tests that answers a lot of these questions.  There is some information available, but  it isn't complete.  Here a couple of good starting points..

http://orastory.wordpress.com/2011/07/2

http://kerryosborne.oracle-guy.com/2011/07/cardinality-feedback/

http://dioncho.wordpress.com/2009/12/17/trivial-research-on-the-cardinality-feedback-on-11gr2/

They were even able to recreate some examples. Well here is my example.

First, my dataset. I used the zipcode database you can find here.  I used a sqldr script you can find here, to load a table structure you can find here.

First I loaded the data (about 74,000 rows), created a couple of indexes and ran a query.

here are the indexes

 CREATE INDEX "SYSTEM"."INDEX1" ON "SYSTEM"."TEST_DATA"
    (  "ZIPCODE" )
    TABLESPACE "SYSTEM" ;
  
 CREATE INDEX "SYSTEM"."INDEX2" ON "SYSTEM"."TEST_DATA"
    (  "CITY" )
     TABLESPACE "SYSTEM" ;
  
 CREATE INDEX "SYSTEM"."INDEX3" ON "SYSTEM"."TEST_DATA"
    (  "COUNTY" )
     TABLESPACE "SYSTEM" ;  

select /* colorme */ * from test_data where city='ROCHESTER' and county='MONROE';

There are individual indexes on both city and county, and there is a relationship between the 2 items.  This query will return 44 rows of data.

First output is the information from v$sql_shared_cursor.  Notice the column "use_feedback_stats".  I am also supplying the information from v$sqlstats for this query.  Here is the output after the first execution.

v$SQL_SHARED_CURSOR

SQL_ID        CHILD_NUMBER           USE_FEEDBACK_STATS 
------------- ---------------------- ------------------ 
33x0cazrmgrgy 0                      Y                  

V$SQLSTATS

SQL_ID        PLAN_HASH_VALUE        PARSE_CALLS            BUFFER_GETS            ROWS_PROCESSED         
------------- ---------------------- ---------------------- ---------------------- ---------------------- 
33x0cazrmgrgy 757026858              1                      551                    44                     

Notice that the column "use_feedback_stats" is set to 'Y" indicating that the query is marked for cardinality feedback to determine if the first execution is accurate enough.  I also ran a 10053 trace.  You can see from the output the query is rewritten to be executed with dynamic sampling.

SELECT /* OPT_DYN_SAMP */ /*+ ALL_ROWS IGNORE_WHERE_CLAUSE NO_PARALLEL(SAMPLESUB) opt_param('parallel_execution_enabled', 'false') NO_PARALLEL_INDEX(SAMPLESUB) NO_SQL_TUNE */ NVL(SUM(C1),0), NVL(SUM(C2),0), NVL(SUM(C3),0), NVL(SUM(C4),0) 
FROM 
(SELECT /*+ IGNORE_WHERE_CLAUSE NO_PARALLEL("TEST_DATA") FULL("TEST_DATA") NO_PARALLEL_INDEX("TEST_DATA") */ 1 AS C1, 
CASE WHEN "TEST_DATA"."CITY"='ROCHESTER' AND "TEST_DATA"."COUNTY"='MONROE' THEN 1 ELSE 0 END AS C2, 
CASE WHEN "TEST_DATA"."COUNTY"='MONROE' THEN 1 ELSE 0 END AS C3, 
CASE WHEN "TEST_DATA"."CITY"='ROCHESTER' THEN 1 ELSE 0 END AS C4 
FROM "BGRENN"."TEST_DATA" SAMPLE BLOCK (4.972376 , 1) SEED (1) "TEST_DATA") SAMPLESUB

Here is the output from explain plan

 PLAN_TABLE_OUTPUT                                                    
  
 -------------------------------------------------------------------------------------------------------------------------
   Plan hash value: 757026858                                                
   -----------------------------------------------------------------------------------------                
   | Id | Operation          | Name   | Rows | Bytes | Cost (%CPU)| Time   |                
   -----------------------------------------------------------------------------------------                
   |  0 | SELECT STATEMENT      |      |   8 | 1272 |   3  (0)| 00:00:01 |                
   |* 1 | TABLE ACCESS BY INDEX ROWID| TEST_DATA |   8 | 1272 |   3  (0)| 00:00:01 |                
   |* 2 |  INDEX RANGE SCAN     | INDEX2  |  121 |    |   1  (0)| 00:00:01 |                
   -----------------------------------------------------------------------------------------                
   Predicate Information (identified by operation id):                                   
   ---------------------------------------------------                                  
     1 - filter("COUNTY"='MONROE')                                             
     2 - access("CITY"='ROCHESTER')                                            
  
 Note                                                           
   -----                                                          
     - dynamic sampling used for this statement (level=2)                                 
  

Second output is the information from v$sql_shared_cursor (again).  Notice the column "use_feedback_stats" is set to 'N'

SQL_ID        CHILD_NUMBER           USE_FEEDBACK_STATS 
------------- ---------------------- ------------------ 
33x0cazrmgrgy 0                      Y                  
33x0cazrmgrgy 1                      N                  

SQL_ID        PLAN_HASH_VALUE        PARSE_CALLS            BUFFER_GETS            ROWS_PROCESSED         
------------- ---------------------- ---------------------- ---------------------- ---------------------- 
33x0cazrmgrgy 757026858              2                      572                    88                     

I ran a 10053 trace again (as before).  You can see in this case the results from the first execution (with 44 rows), is passed into the second execution using the hint "opt_estimate". You can also see what parts of the query are utilizing cardinality feedback to change the query.

SELECT /*+ OPT_ESTIMATE (TABLE "TEST_DATA" ROWS=44.000000 ) OPT_ESTIMATE (INDEX_SCAN "TEST_DATA" "INDEX2" MIN=69.000000 ) 
OPT_ESTIMATE (INDEX_FILTER "TEST_DATA" "INDEX2" ROWS=69.000000 ) */ 
"TEST_DATA"."ZIPCODE" "ZIPCODE",
"TEST_DATA"."LAT" "LAT",
"TEST_DATA"."CITY" "CITY",
"TEST_DATA"."STATE" "STATE",
"TEST_DATA"."COUNTY" "COUNTY",
"TEST_DATA"."TYPE" "TYPE",
"TEST_DATA"."PREFERRED" "PREFERRED",
"TEST_DATA"."WORLDREGION" "WORLDREGION",
"TEST_DATA"."COUNTRY" "COUNTRY",
"TEST_DATA"."LOCATIONTEXT" "LOCATIONTEXT",
"TEST_DATA"."LOCATION" "LOCATION",
"TEST_DATA"."POPULATION" "POPULATION",
"TEST_DATA"."HOUSINGUNITS" "HOUSINGUNITS"
,"TEST_DATA"."INCOME" "INCOME",
"TEST_DATA"."LANDAREA",
"LANDAREATEST_DATA"."WATERAREA" "WATERAREA",
"TEST_DATA"."DECOMMISIONED" "DECOMMISIONED",
"TEST_DATA"."MILITARYRESTRICTIONCODES" "MILITARYRESTRICTIONCODES" 
FROM "BGRENN"."TEST_DATA" "TEST_DATA" 
WHERE "TEST_DATA"."CITY"='ROCHESTER' AND "TEST_DATA"."COUNTY"='MONROE'

I ran this query until the v$sql_shared_cursor came back clean, and showed a "n" indicating that dynamic feedback was no longer being considered.  There was information from 2 hard parses in the 10053 file.

SQL_ID        CHILD_NUMBER           USE_FEEDBACK_STATS 
------------- ---------------------- ------------------ 
33x0cazrmgrgy 0                      Y                  
33x0cazrmgrgy 1                      N                  

SQL_ID        PLAN_HASH_VALUE        PARSE_CALLS            BUFFER_GETS            ROWS_PROCESSED         
------------- ---------------------- ---------------------- ---------------------- ---------------------- 
33x0cazrmgrgy 757026858              3                      593                    132                    

This answered my first question. How do you tell what caused the cardinality feedback ? You run a 10053, and find the OPT_ESTIMATE cardinality hints that adjust the cardinility and reparse the query.

/***************************************/

For my second test, I used the same result set, and ran statistics against the data.  After running the query from the first test, I found that cardinality was not being utilzed, so I changed my query.

select /* colorme */ * from test_data where city='ROCHESTER' and county='MONROE1' and state='NY';

This returns no rows of data, but the optimizer can't figure this out.

First execution you can see that cardinality feedback was marked for the statement, and notice the plan_hash_value of 7570268858.  In this example I've also included the output from v$SQLSTATS_PLAN_HASH. This information was added after reading the post from Lisa G.

SQL_ID        CHILD_NUMBER           USE_FEEDBACK_STATS 
------------- ---------------------- ------------------ 
88jhv4mryj58p 0                      Y                  

V$SQLSTATS

SQL_ID        PLAN_HASH_VALUE        PARSE_CALLS            BUFFER_GETS            ROWS_PROCESSED         
------------- ---------------------- ---------------------- ---------------------- ---------------------- 
88jhv4mryj58p 757026858              1                      146                    0                      

v$SQLSTATS_PLAN_HASH

SQL_ID        PLAN_HASH_VALUE        PARSE_CALLS            BUFFER_GETS            ROWS_PROCESSED         
------------- ---------------------- ---------------------- ---------------------- ---------------------- 
88jhv4mryj58p 757026858              1                      146                    0                      

Here is the output from the 10053 trace.

SELECT 
    "TEST_DATA"."ZIPCODE" "ZIPCODE",
    "TEST_DATA"."LAT" "LAT",
    "TEST_DATA"."CITY" "CITY",
    "TEST_DATA"."STATE" "STATE",
    "TEST_DATA"."COUNTY" "COUNTY",
    "TEST_DATA"."TYPE" "TYPE",
    "TEST_DATA"."PREFERRED" "PREFERRED",
    "TEST_DATA"."WORLDREGION" "WORLDREGION",
    "TEST_DATA"."COUNTRY" "COUNTRY",
    "TEST_DATA"."LOCATIONTEXT" "LOCATIONTEXT",
    "TEST_DATA"."LOCATION" "LOCATION",
    "TEST_DATA"."POPULATION" "POPULATION",
    "TEST_DATA"."HOUSINGUNITS" "HOUSINGUNITS",
    "TEST_DATA"."INCOME" "INCOME",
    "TEST_DATA"."LANDAREA",
    "LANDAREATEST_DATA"."WATERAREA" "WATERAREA",
    "TEST_DATA"."DECOMMISIONED" "DECOMMISIONED",
    "TEST_DATA"."MILITARYRESTRICTIONCODES" "MILITARYRESTRICTIONCODES" 
FROM "BGRENN"."TEST_DATA" "TEST_DATA" 
WHERE "TEST_DATA"."CITY"='ROCHESTER' AND "TEST_DATA"."COUNTY"='MONROE1'

Here is the first plan

 PLAN_TABLE_OUTPUT                                                    
  
 -------------------------------------------------------------------------------------------------------------------------
  
 Plan hash value: 757026858                                                
  
 -----------------------------------------------------------------------------------------                
  
 | Id | Operation          | Name   | Rows | Bytes | Cost (%CPU)| Time   |                
 -----------------------------------------------------------------------------------------                
 |  0 | SELECT STATEMENT      |      |   1 |  120 |   3  (0)| 00:00:01 |                
 |* 1 | TABLE ACCESS BY INDEX ROWID| TEST_DATA |   1 |  120 |   3  (0)| 00:00:01 |                
 |* 2 |  INDEX RANGE SCAN     | INDEX2  |   2 |    |   1  (0)| 00:00:01 |                
 -----------------------------------------------------------------------------------------                
 Predicate Information (identified by operation id):                                   
 ---------------------------------------------------                                   
   1 - filter("COUNTY"='MONROE1' AND "STATE"='NY')                                    
   2 - access("CITY"='ROCHESTER')                                            
  

Second execution, notice that cardinality feedback will not be used on subsequent executions.

SQL_ID        CHILD_NUMBER           USE_FEEDBACK_STATS 
------------- ---------------------- ------------------ 
88jhv4mryj58p 0                      Y                  
88jhv4mryj58p 1                      N                  

V$SQLSTATS

SQL_ID        PLAN_HASH_VALUE        PARSE_CALLS            BUFFER_GETS            ROWS_PROCESSED         
------------- ---------------------- ---------------------- ---------------------- ---------------------- 
88jhv4mryj58p 2704719303             2                      148                    0                      

v$SQLSTATS_PLAN_HASH

SQL_ID        PLAN_HASH_VALUE        PARSE_CALLS            BUFFER_GETS            ROWS_PROCESSED         
------------- ---------------------- ---------------------- ---------------------- ---------------------- 
88jhv4mryj58p 2704719303             1                      2                      0                      
88jhv4mryj58p 757026858              1                      146                    0                      

Here is the output from the 10053 trace showing the feedback from the first execution using the OPT_ESTIMATE hint again.

SELECT /*+ OPT_ESTIMATE (INDEX_SCAN "TEST_DATA" "INDEX2" MIN=69.000000 ) 
           OPT_ESTIMATE (INDEX_FILTER "TEST_DATA" "INDEX2" ROWS=69.000000 ) */ 
    "TEST_DATA"."ZIPCODE" "ZIPCODE",
    "TEST_DATA"."LAT" "LAT",
    "TEST_DATA"."CITY" "CITY",
    "TEST_DATA"."STATE" "STATE",
    "TEST_DATA"."COUNTY" "COUNTY",
    "TEST_DATA"."TYPE" "TYPE",
    "TEST_DATA"."PREFERRED" "PREFERRED",
    "TEST_DATA"."WORLDREGION" "WORLDREGION",
    "TEST_DATA"."COUNTRY" "COUNTRY",
    "TEST_DATA"."LOCATIONTEXT" "LOCATIONTEXT",
    "TEST_DATA"."LOCATION" "LOCATION",
    "TEST_DATA"."POPULATION" "POPULATION",
    "TEST_DATA"."HOUSINGUNITS" "HOUSINGUNITS",
    "TEST_DATA"."INCOME" "INCOME",
    "TEST_DATA"."LANDAREA",
    "LANDAREATEST_DATA"."WATERAREA" "WATERAREA",
    "TEST_DATA"."DECOMMISIONED" "DECOMMISIONED",
    "TEST_DATA"."MILITARYRESTRICTIONCODES" "MILITARYRESTRICTIONCODES" 
FROM "BGRENN"."TEST_DATA" "TEST_DATA" 
WHERE "TEST_DATA"."CITY"='ROCHESTER' AND "TEST_DATA"."COUNTY"='MONROE1'

But notice something else... notice the plan has value changed from 757026858 to 2704719303.  Fine right ? Cardinality feedback did it's thing, but also notice this is the output from v$sqlstats.  Huh ?? The history for executions of the plan (1 for the first plan, and 1 for the second), got summarized into the second plan.

here is the second plan

 Plan hash value: 2704719303                                                     
  
 -----------------------------------------------------------------------------------------                      
 | Id | Operation          | Name   | Rows | Bytes | Cost (%CPU)| Time   |                      
 -----------------------------------------------------------------------------------------                      
 |  0 | SELECT STATEMENT      |      |    |    |   6 (100)|     |                      
 |* 1 | TABLE ACCESS BY INDEX ROWID| TEST_DATA |   1 |  120 |   6  (0)| 00:00:01 |                      
 |* 2 |  INDEX RANGE SCAN     | INDEX3  |  38 |    |   1  (0)| 00:00:01 |                      
 -----------------------------------------------------------------------------------------                      
  
 Predicate Information (identified by operation id):                                         
 ---------------------------------------------------  
   1 - filter("CITY"='ROCHESTER')                                                  
   2 - access("COUNTY"='MONROE1')                                                  
 PLAN_TABLE_OUTPUT                                                          
 ------------------------------------------------------------------------------------------------------------------------------------
  
 Note                                                                
 -----                                                                
   - cardinality feedback used for this statement         

What does all this mean ????

1) the 10053 trace will reveal where cardinality feedback was used by finding the OPT_ESTIMATE hint.

2) the v$sql_session_cursor view will show you where cardinality feedback was used.

3) There is a new view for looking at the multiple plans created for cardinality feedback called v$SQLSTATS_PLAN_HASH.

And in looking at the DBA_HIST_SQLSTAT, I do see both plans.

SQL_ID        PLAN_HASH_VALUE        PARSE_CALLS_DELTA      BUFFER_GETS_DELTA      ROWS_PROCESSED_DELTA   
------------- ---------------------- ---------------------- ---------------------- ---------------------- 
88jhv4mryj58p 757026858              1                      146                    0                      
88jhv4mryj58p 2704719303             1                      2                      0                      

Another observation I will would make is that when you multiple columns (and you aren't using a composite index), oracle is likely to utilyze cardinality feedback to ensure the cardinality is correct.

Finally, I did try the same example with bind variables (instead of literals).  I found that Cardinality Feedback did not kick in for my example.  Bind variables must tell the optimizer to pick a "generically" better plan, and not look around.

This is all on version 11.2.0.2

I have included a query I'm using to point out where cardinality feedback has been utilized, and sorted the queries by the affect they have on performance, so I can assess the gains (or losses) associated with the use of this feature. Overall my databases have show remarkable improvement, and those fiew cases where there is a regression, the sql_id is highligted so I can create a profile.

I have to note that this query has become invaluable to identify the top queries affected by cardinality feedback.  I have used it to identify where extended stats may help, and I have been able to determine, that cardinality feedback is indeed a good thing most of the time.  A lot of this detail is lost in the AWR reports.

select  query1.sql_id,
query1.plan_hash_value phv1,
query2.plan_hash_value phv2, 
query1.executions exec1,
query2.executions exec2,
trunc((query1.avg_elapsed_time)/1000000,2) aet,
trunc(( query2.avg_elapsed_time)/1000000,2) aet_card,
trunc(( query1.avg_elapsed_time - query2.avg_elapsed_time)/1000000,2) aet_diff,
abs(trunc(( (query1.avg_elapsed_time - query2.avg_elapsed_time)* query2.executions)/1000000,2)) abs_diff,
trunc( ((query1.avg_elapsed_time - query2.avg_elapsed_time) * query2.executions)/1000000,2) tot_diff,
trunc( ((query1.avg_elapsed_time - query2.avg_elapsed_time)/ query1.avg_elapsed_time) * 100,1) card_feedback_perc_difference,query1.sql_text
  from 
(select sql_id,plan_hash_value,last_active_time,executions,(rows_processed/executions) rows_processed,
(elapsed_time/executions) avg_elapsed_time,(cpu_time/executions) avg_cpu_time,(buffer_gets/executions) avg_buffer_gets,
sql_text from V$SQLSTATS_PLAN_HASH a
where exists (select plan_hash_value  from V$SQLSTATS_PLAN_HASH b  where a.sql_id=b.sql_id and a.plan_hash_value <> b.plan_hash_value and executions > 0)
and executions > 0
order by sql_id,last_active_time) query1,
(select sql_id,plan_hash_value,last_active_time,executions,(rows_processed/executions) rows_processed,
(elapsed_time/executions) avg_elapsed_time,(cpu_time/executions) avg_cpu_time,(buffer_gets/executions) avg_buffer_gets,
sql_text from V$SQLSTATS_PLAN_HASH a
where exists (select plan_hash_value  from V$SQLSTATS_PLAN_HASH b  where a.sql_id=b.sql_id and a.plan_hash_value <> b.plan_hash_value and executions > 0)
and executions > 0
order by sql_id,last_active_time) query2
where query1.sql_id=query2.sql_id
and query1.last_active_time < query2.last_active_time 
and exists (select 1 from V$SQL_SHARED_CURSOR c where query1.sql_id=c.sql_id and c.use_feedback_stats='Y') 
--and query1.avg_elapsed_time < query2.avg_elapsed_time
order by abs_diff desc

My 2 favorite queries

For finding out what happened with queries, here are my 2 favorite queries..

For searching the awr and previous executions
select * from table(dbms_xplan.display_awr('3cmh637q9msjs',null,null,'advanced +peeked_binds'));

For looking through the cursor cache
select * from table(dbms_xplan.display_cursor('3cmh637q9msjs',null,'advanced +peeked_binds'));


The output looks something like this...

line 8: SQLPLUS Command Skipped: set linesize 132
line 9: SQLPLUS Command Skipped: set pagesize 0
PLAN_TABLE_OUTPUT                                                                                                                                                                                                                                                                                            
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 
SQL_ID 3j9yx7t5abcyg                                                                                                                                                                                                                                                                                         
--------------------                                                                                                                                                                                                                                                                                         
/* OracleOEM */  SELECT m.tablespace_name,        m.used_percent,                                                                                                                                                                                                                                            
   (m.tablespace_size - m.used_space)*t.block_size/1024/1024 mb_free                                                                                                                                                                                                                                         
FROM  dba_tablespace_usage_metrics m, dba_tablespaces t, v$parameter p                                                                                                                                                                                                                                       
WHERE p.name='statistics_level' and p.value!='BASIC'    AND                                                                                                                                                                                                                                                  
t.tablespace_name = m.tablespace_name                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                             
Plan hash value: 125441316                                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                             
--------------------------------------------------------------------------------------------------------------------------------------------------                                                                                                                                                           
| Id  | Operation                            | Name                         | Rows  | Bytes | Cost (%CPU)| Time     |    TQ  |IN-OUT| PQ Distrib |                                                                                                                                                           
--------------------------------------------------------------------------------------------------------------------------------------------------                                                                                                                                                           
|   0 | SELECT STATEMENT                     |                              |       |       |    19 (100)|          |        |      |            |                                                                                                                                                           
|   1 |  NESTED LOOPS                        |                              |     1 |  2209 |    19  (43)| 00:00:01 |        |      |            |                                                                                                                                                           
|   2 |   NESTED LOOPS                       |                              |     1 |  2196 |    19  (43)| 00:00:01 |        |      |            |                                                                                                                                                           
|   3 |    MERGE JOIN CARTESIAN              |                              |     1 |  2171 |    18  (45)| 00:00:01 |        |      |            |                                                                                                                                                           
|   4 |     HASH JOIN                        |                              |     1 |  2115 |     1 (100)| 00:00:01 |        |      |            |                                                                                                                                                           
|   5 |      FIXED TABLE FULL                | X$KSPPI                      |     1 |    81 |     0   (0)|          |        |      |            |                                                                                                                                                           
|   6 |      FIXED TABLE FULL                | X$KSPPCV                     |     5 | 10170 |     0   (0)|          |        |      |            |                                                                                                                                                           
|   7 |     BUFFER SORT                      |                              |    82 |  4592 |    18  (45)| 00:00:01 |        |      |            |                                                                                                                                                           
|   8 |      VIEW                            | DBA_TABLESPACE_USAGE_METRICS |    82 |  4592 |    17  (42)| 00:00:01 |        |      |            |                                                                                                                                                           
|   9 |       SORT UNIQUE                    |                              |    82 |  7594 |    17  (53)| 00:00:01 |        |      |            |                                                                                                                                                           
|  10 |        UNION-ALL                     |                              |       |       |            |          |        |      |            |                                                                                                                                                           
|  11 |         MERGE JOIN                   |                              |    80 |  5360 |     9  (12)| 00:00:01 |        |      |            |                                                                                                                                                           
|  12 |          TABLE ACCESS CLUSTER        | TS$                          |    31 |   868 |     8   (0)| 00:00:01 |        |      |            |                                                                                                                                                           
|  13 |           INDEX FULL SCAN            | I_TS#                        |     1 |       |     1   (0)| 00:00:01 |        |      |            |                                                                                                                                                           
|  14 |          SORT JOIN                   |                              |   100 |  3900 |     1 (100)| 00:00:01 |        |      |            |                                                                                                                                                           
|  15 |           FIXED TABLE FULL           | X$KTTETS                     |   100 |  3900 |     0   (0)|          |        |      |            |                                                                                                                                                           
|  16 |         HASH GROUP BY                |                              |     1 |    90 |     3  (67)| 00:00:01 |        |      |            |                                                                                                                                                           
|  17 |          NESTED LOOPS                |                              |     1 |    90 |     1   (0)| 00:00:01 |        |      |            |                                                                                                                                                           
|  18 |           FIXED TABLE FULL           | X$KTTEFINFO                  |     1 |    65 |     0   (0)|          |        |      |            |                                                                                                                                                           
|  19 |           TABLE ACCESS CLUSTER       | TS$                          |     1 |    25 |     1   (0)| 00:00:01 |        |      |            |                                                                                                                                                           
|  20 |            INDEX UNIQUE SCAN         | I_TS#                        |     1 |       |     0   (0)|          |        |      |            |                                                                                                                                                           
|  21 |         HASH GROUP BY                |                              |     1 |  2144 |     4  (75)| 00:00:01 |        |      |            |                                                                                                                                                           
|  22 |          NESTED LOOPS                |                              |       |       |            |          |        |      |            |                                                                                                                                                           
|  23 |           NESTED LOOPS               |                              |     1 |  2144 |     2  (50)| 00:00:01 |        |      |            |                                                                                                                                                           
|  24 |            HASH JOIN                 |                              |     1 |  2122 |     1 (100)| 00:00:01 |        |      |            |                                                                                                                                                           
|  25 |             PX COORDINATOR           |                              |     1 |    65 |     0   (0)|          |        |      |            |                                                                                                                                                           
|  26 |              PX SEND QC (RANDOM)     | :TQ10000                     |     1 |    65 |     0   (0)|          |  Q1,00 | P->S | QC (RAND)  |                                                                                                                                                           
|  27 |               VIEW                   | GV$FILESPACE_USAGE           |       |       |            |          |  Q1,00 | PCWP |            |                                                                                                                                                           
|  28 |                FIXED TABLE FULL      | X$KTTEFINFO                  |     1 |    65 |     0   (0)|          |  Q1,00 | PCWP |            |                                                                                                                                                           
|  29 |             PX COORDINATOR           |                              |     1 |  2057 |     1 (100)| 00:00:01 |        |      |            |                                                                                                                                                           
|  30 |              PX SEND QC (RANDOM)     | :TQ20000                     |     1 |  2115 |     1 (100)| 00:00:01 |  Q2,00 | P->S | QC (RAND)  |                                                                                                                                                           
|  31 |               VIEW                   | GV$PARAMETER                 |       |       |            |          |  Q2,00 | PCWP |            |                                                                                                                                                           
|  32 |                HASH JOIN             |                              |     1 |  2115 |     1 (100)| 00:00:01 |  Q2,00 | PCWP |            |                                                                                                                                                           
|  33 |                 FIXED TABLE FULL     | X$KSPPI                      |     1 |    81 |     0   (0)|          |  Q2,00 | PCWP |            |                                                                                                                                                           
|  34 |                 FIXED TABLE FULL     | X$KSPPCV                     |   100 |   198K|     0   (0)|          |  Q2,00 | PCWP |            |                                                                                                                                                           
|  35 |            INDEX UNIQUE SCAN         | I_TS1                        |     1 |       |     0   (0)|          |        |      |            |                                                                                                                                                           
|  36 |           TABLE ACCESS BY INDEX ROWID| TS$                          |     1 |    22 |     1   (0)| 00:00:01 |        |      |            |                                                                                                                                                           
|  37 |    TABLE ACCESS BY INDEX ROWID       | TS$                          |     1 |    25 |     1   (0)| 00:00:01 |        |      |            |                                                                                                                                                           
|  38 |     INDEX UNIQUE SCAN                | I_TS1                        |     1 |       |     0   (0)|          |        |      |            |                                                                                                                                                           
|  39 |   FIXED TABLE FIXED INDEX            | X$KCFISTSA (ind:1)           |     3 |    39 |     0   (0)|          |        |      |            |                                                                                                                                                           
--------------------------------------------------------------------------------------------------------------------------------------------------                                                                                                                                                           
                                                                                                                                                                                                                                                                                                             
Query Block Name / Object Alias (identified by operation id):                                                                                                                                                                                                                                                
-------------------------------------------------------------                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                             
   1 - SEL$CBAA6355                                                                                                                                                                                                                                                                                          
   5 - SEL$CBAA6355 / X@SEL$12                                                                                                                                                                                                                                                                               
   6 - SEL$CBAA6355 / Y@SEL$12                                                                                                                                                                                                                                                                               
   8 - SET$1        / M@SEL$1                                                                                                                                                                                                                                                                                
   9 - SET$1                                                                                                                                                                                                                                                                                                 
  11 - SEL$2                                                                                                                                                                                                                                                                                                 
  12 - SEL$2        / T@SEL$2                                                                                                                                                                                                                                                                                
  13 - SEL$2        / T@SEL$2                                                                                                                                                                                                                                                                                
  15 - SEL$2        / TSTAT@SEL$2                                                                                                                                                                                                                                                                            
  16 - SEL$C8360722                                                                                                                                                                                                                                                                                          
  18 - SEL$C8360722 / X$KTTEFINFO@SEL$5                                                                                                                                                                                                                                                                      
  19 - SEL$C8360722 / T@SEL$3                                                                                                                                                                                                                                                                                
  20 - SEL$C8360722 / T@SEL$3                                                                                                                                                                                                                                                                                
  21 - SEL$6                                                                                                                                                                                                                                                                                                 
  25 - SEL$7        / F@SEL$6                                                                                                                                                                                                                                                                                
  28 - SEL$7        / X$KTTEFINFO@SEL$7                                                                                                                                                                                                                                                                      
  29 - SEL$8        / PARAM@SEL$6                                                                                                                                                                                                                                                                            
  32 - SEL$8                                                                                                                                                                                                                                                                                                 
  33 - SEL$8        / X@SEL$8                                                                                                                                                                                                                                                                                
  34 - SEL$8        / Y@SEL$8                                                                                                                                                                                                                                                                                
  35 - SEL$6        / T@SEL$6                                                                                                                                                                                                                                                                                
  36 - SEL$6        / T@SEL$6                                                                                                                                                                                                                                                                                
  37 - SEL$CBAA6355 / TS@SEL$9                                                                                                                                                                                                                                                                               
  38 - SEL$CBAA6355 / TS@SEL$9                                                                                                                                                                                                                                                                               
  39 - SEL$CBAA6355 / TSATTR@SEL$9                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                             
Outline Data                                                                                                                                                                                                                                                                                                 
-------------                                                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                             
  /*+                                                                                                                                                                                                                                                                                                        
      BEGIN_OUTLINE_DATA                                                                                                                                                                                                                                                                                     
      IGNORE_OPTIM_EMBEDDED_HINTS                                                                                                                                                                                                                                                                            
      OPTIMIZER_FEATURES_ENABLE('11.2.0.1')                                                                                                                                                                                                                                                                  
      DB_VERSION('11.2.0.1')                                                                                                                                                                                                                                                                                 
      OPT_PARAM('_optimizer_fast_pred_transitivity' 'false')                                                                                                                                                                                                                                                 
      ALL_ROWS                                                                                                                                                                                                                                                                                               
      OUTLINE_LEAF(@"SEL$2")                                                                                                                                                                                                                                                                                 
      OUTLINE_LEAF(@"SEL$C8360722")                                                                                                                                                                                                                                                                          
      MERGE(@"SEL$7286615E")                                                                                                                                                                                                                                                                                 
      OUTLINE_LEAF(@"SEL$7")                                                                                                                                                                                                                                                                                 
      OUTLINE_LEAF(@"SEL$8")                                                                                                                                                                                                                                                                                 
      OUTLINE_LEAF(@"SEL$6")                                                                                                                                                                                                                                                                                 
      OUTLINE_LEAF(@"SET$1")                                                                                                                                                                                                                                                                                 
      OUTLINE_LEAF(@"SEL$CBAA6355")                                                                                                                                                                                                                                                                          
      MERGE(@"SEL$9")                                                                                                                                                                                                                                                                                        
      MERGE(@"SEL$E3DF9B48")                                                                                                                                                                                                                                                                                 
      OUTLINE(@"SEL$3")                                                                                                                                                                                                                                                                                      
      OUTLINE(@"SEL$7286615E")                                                                                                                                                                                                                                                                               
      MERGE(@"SEL$5")                                                                                                                                                                                                                                                                                        
      OUTLINE(@"SEL$1")                                                                                                                                                                                                                                                                                      
      OUTLINE(@"SEL$9")                                                                                                                                                                                                                                                                                      
      OUTLINE(@"SEL$E3DF9B48")                                                                                                                                                                                                                                                                               
      MERGE(@"SEL$42DFC41A")                                                                                                                                                                                                                                                                                 
      OUTLINE(@"SEL$4")                                                                                                                                                                                                                                                                                      
      OUTLINE(@"SEL$5")                                                                                                                                                                                                                                                                                      
      OUTLINE(@"SEL$10")                                                                                                                                                                                                                                                                                     
      OUTLINE(@"SEL$42DFC41A")                                                                                                                                                                                                                                                                               
      MERGE(@"SEL$12")                                                                                                                                                                                                                                                                                       
      OUTLINE(@"SEL$11")                                                                                                                                                                                                                                                                                     
      OUTLINE(@"SEL$12")                                                                                                                                                                                                                                                                                     
      FULL(@"SEL$CBAA6355" "X"@"SEL$12")                                                                                                                                                                                                                                                                     
      FULL(@"SEL$CBAA6355" "Y"@"SEL$12")                                                                                                                                                                                                                                                                     
      NO_ACCESS(@"SEL$CBAA6355" "M"@"SEL$1")                                                                                                                                                                                                                                                                 
      INDEX_RS_ASC(@"SEL$CBAA6355" "TS"@"SEL$9" ("TS$"."NAME"))                                                                                                                                                                                                                                              
      FULL(@"SEL$CBAA6355" "TSATTR"@"SEL$9")                                                                                                                                                                                                                                                                 
      LEADING(@"SEL$CBAA6355" "X"@"SEL$12" "Y"@"SEL$12" "M"@"SEL$1" "TS"@"SEL$9" "TSATTR"@"SEL$9")                                                                                                                                                                                                           
      USE_HASH(@"SEL$CBAA6355" "Y"@"SEL$12")                                                                                                                                                                                                                                                                 
      USE_MERGE_CARTESIAN(@"SEL$CBAA6355" "M"@"SEL$1")                                                                                                                                                                                                                                                       
      USE_NL(@"SEL$CBAA6355" "TS"@"SEL$9")                                                                                                                                                                                                                                                                   
      USE_NL(@"SEL$CBAA6355" "TSATTR"@"SEL$9")                                                                                                                                                                                                                                                               
      NO_ACCESS(@"SEL$6" "F"@"SEL$6")                                                                                                                                                                                                                                                                        
      NO_ACCESS(@"SEL$6" "PARAM"@"SEL$6")                                                                                                                                                                                                                                                                    
      INDEX(@"SEL$6" "T"@"SEL$6" ("TS$"."NAME"))                                                                                                                                                                                                                                                             
      LEADING(@"SEL$6" "F"@"SEL$6" "PARAM"@"SEL$6" "T"@"SEL$6")                                                                                                                                                                                                                                              
      USE_HASH(@"SEL$6" "PARAM"@"SEL$6")                                                                                                                                                                                                                                                                     
      USE_NL(@"SEL$6" "T"@"SEL$6")                                                                                                                                                                                                                                                                           
      NLJ_BATCHING(@"SEL$6" "T"@"SEL$6")                                                                                                                                                                                                                                                                     
      USE_HASH_AGGREGATION(@"SEL$6")                                                                                                                                                                                                                                                                         
      FULL(@"SEL$C8360722" "X$KTTEFINFO"@"SEL$5")                                                                                                                                                                                                                                                            
      INDEX(@"SEL$C8360722" "T"@"SEL$3" "I_TS#")                                                                                                                                                                                                                                                             
      LEADING(@"SEL$C8360722" "X$KTTEFINFO"@"SEL$5" "T"@"SEL$3")                                                                                                                                                                                                                                             
      USE_NL(@"SEL$C8360722" "T"@"SEL$3")                                                                                                                                                                                                                                                                    
      USE_HASH_AGGREGATION(@"SEL$C8360722")                                                                                                                                                                                                                                                                  
      INDEX(@"SEL$2" "T"@"SEL$2" "I_TS#")                                                                                                                                                                                                                                                                    
      FULL(@"SEL$2" "TSTAT"@"SEL$2")                                                                                                                                                                                                                                                                         
      LEADING(@"SEL$2" "T"@"SEL$2" "TSTAT"@"SEL$2")                                                                                                                                                                                                                                                          
      USE_MERGE(@"SEL$2" "TSTAT"@"SEL$2")                                                                                                                                                                                                                                                                    
      PX_JOIN_FILTER(@"SEL$2" "TSTAT"@"SEL$2")                                                                                                                                                                                                                                                               
      FULL(@"SEL$7" "X$KTTEFINFO"@"SEL$7")                                                                                                                                                                                                                                                                   
      FULL(@"SEL$8" "X"@"SEL$8")                                                                                                                                                                                                                                                                             
      FULL(@"SEL$8" "Y"@"SEL$8")                                                                                                                                                                                                                                                                             
      LEADING(@"SEL$8" "X"@"SEL$8" "Y"@"SEL$8")                                                                                                                                                                                                                                                              
      USE_HASH(@"SEL$8" "Y"@"SEL$8")                                                                                                                                                                                                                                                                         
      END_OUTLINE_DATA                                                                                                                                                                                                                                                                                       
  */                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                             
Note                                                                                                                                                                                                                                                                                                         
-----                                                                                                                                                                                                                                                                                                        
   - statement not queuable: gv$ statement                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                             

158 rows selected