Twitter Updates 2.2.1: FeedWitter

Thursday, August 18, 2011

Hyperthreading

I am am working in standing up a cluster of 2 database servers that look an aweful lot like an exadata 2-8 (only without the infiniband)..  It is a 2 node cluster of 8 socket, 8 core processrs with 256g of memory.

We started with a standard configuration and hyperthreading was turned on.. This means that Oracle sees the 64 cpu, as 128 cpus.  We were getting periodic reboots every Saturday morning.. The culprit was finally found as the analyze.. Why ??? because we have a 2 node rac, with 128 CPU's per, the degree of parallelization chosen was 512 !  With an analyze running with 512  parallel processors we were running out of memory.. The final diagnoses was to change parallel_max_servers to 64 to cut back on the parallel servers utilized. The second recommendation from Oracle was to turn off Hyperthreading.

We all said Huh ?  Hyperthreading gives you 2x the throughput because it doubles your CPU's right ???

Well lets' see.

I took my 2 node cluster and split it in 1/2  First half (A) is non-hyperthreaded, second half (B) is hyperthreaded.

I then took swingbench and threw a workload at it with lots of logical I/O.


Here is the benchmark from (A)   non-hyperthreaded

Notice we are getting 2062 transactions per minute.



 OK.. Lets see what happens with (B) hyperthreading.

Notice we are getting 1,685 transactions per minute. 


LESS wow lets look at our AWR compare of both nodes (same period) and see what the database says.


Notice  that the gets per execution match for both nodes, and the rows processed. The difference is in the exec time(ms) per exectuion, CPU time and executions.

IT is faster non-hyperthreaded.   The CPU's are doing more work with hypertrheading off, and pushing more work through.

OK... I've shown that for a non-saturating workload (under 64 active processes) on a 64 CPU box, hyperthreading off is faster.

Now let's saturate !!

First hyperthreading off.

Notice I can get the throughput up to 7,489 transactions per minute.

Now lets check the second node with hyperthreading on.

Notice we are getting 6552.. Darn !!


It looks like hypthreading isn't helping us eithor !! 

 
Now lets check the AWR for the 2 nodes.


Notice that once again the rows processed match up and the gets/execution match.

The number of executions is greater, the execution time is less, and the CPU time is way less for for hyperthreading off.


For a Logical I/O intensive workload, it looks like Hyperthreading is not as efficient.

I would love to hear any other experiences/opinions.

I just wanted to add on to this a thought as to why hypertrheading is such a big deal... Virtualization.  With 2x as many "cpus", you can fine tune the CPU a lot more.. You can carve up the server into smaller pieces.








5 comments:

  1. i agree with your findings.
    disabling hyperthreading stabalized our linux/oracle machine.

    before we had random cpu spikes upwards of 500 with hyperthreading, now stead load of 4 sometimes going to load 10.

    ReplyDelete
  2. Hi Bryan,
    thanks for sharing!
    Could you please give the exact specs of your CPU's?
    I read that Intel itself has different implementations of Hyper-Threading.
    According to Kevin Closson (http://kevinclosson.wordpress.com/2009/02/20/intel-hyperthreading-does-little-for-oracle-workloads-who-cares-i-want-my-xeon-5500-cpu-with-simultaneous-multi-threading-smt/), the implementation on Nehalem should provide good performance.

    Cheers

    --
    Kamal

    ReplyDelete
    Replies
    1. Kamal,
      The servers I am testing on are Xeon X7560 @ 2.27GHz chips.

      Since writing this, I have been benchmarking on some new servers. they are 2 socket servers utilzing the Intel(R) Xeon(R) CPU E7- 2870 @ 2.40GHz chips.

      The newer E7 chips handle hyperthreading much better. The throughput increased linearly until I ran out of threads, not just cores.

      Delete
  3. Thanks for the reply.
    Wow that's a beast of a CPU! Anyway it's interesting as they seem to have the same microarchitecture (Nehalem).
    I forgot to ask... what version of Oracle and Linux kernel?

    Cheers

    --
    Kamal

    ReplyDelete
  4. The original testing was 11.2.0.2 on rhel 5. My latest E7 testing was 11.2.0.3 on rhel also. Based on the E7 results i'll do some tests and publish. The results are significant, and this is a popular topic.

    ReplyDelete