Thanks to David Yahalom for flagging this – it doesn’t appear to be covered in metalink/my oracle support.

RAC servers were configured to resolve each others host names with – so ‘racserver1′ resolved to the same IP address as did the fully qualified name ‘racserver1.admin.mycompany.com’.

Client machine (web servers in my case) were not configured in that way. So they could resolve ‘racserver1.admin.mycompany.com’ ok, but could not resolve ‘racserver1′.

You might think that simply specifying the fully qualified hostnames in the client connect string (only) would be enough to resolve this. But no. Server side load balancing will every so often redirect the client to a new hostname. The client has to be able to resolve that new hostname, or it will report an ORA-12545. And that new hostname is supplied by the server, based on what is in the server listener and tnsnames settings. This behaviour is the default (at least in Oracle 11gR1).

This is especially likely with web servers, where security considerations might mean they don’t see exactly the same DNS as other servers do, so is often seen with jdbc, java, php oci8. But really it is a general oracle client issue – sql*plus would do the same from a client which could not resolve all of the server specified host names.

Bottom line – use fully qualified hostnames in RAC installation/DBCA, even when you think you don’t have to.

It just needs two things done to use it:

1) In target database:

SQL> exec dbms_connection_pool.start_pool

2) In client easy connect, add “:pooled” at the end of the easy connect string, so it will look like:

scott/tiger@myserver/mydb:pooled

Or if you use client tnsnames.ora , then add “(server=pooled)” in the “connect_data” part of the “description”, so it will look like:

MYDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = myserver.my.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = POOLED)
      (SERVICE_NAME = mydb.my.com)
    )
  )

3) For PHP to use this, you must be using oci8 v1.3 or above. That comes bundled with PHP v5.3 and above, or can be downloaded separately from http://pecl.php.net/package/oci8. Use phpinfo(); to see what version you are currently using.

This SQL will let you see if the connection pool is working correctly:

col cclass_name form a30
select cclass_name, num_hits, num_misses from gv$cpool_cc_stats ;

Other useful SQL for this:

select * from gv$cpool_stats ;
select * from gv$cpool_cc_stats ;
select program from gv$process where program like 'oracle%(N%)' ;
select * from dba_cpool_info ;

Oracle Documentation on this is excellent, better than the blog or forum posts I saw:
Reference
Net Services Administrator’s Guide
Programmer’s Guide
Database Administrator’s Guide
The Underground PHP and Oracle Manual – Chapter 14. But note that this incorrectly states that PHP 5.3 has not yet been released.

For RAC, load balancing and transparent application failover should also be used. Load balancing is on by default according to the documentation (although I haven’t tested that), but transparent application failover need to be explicitly specified in tnsnames.ora to be used.

Load balancing would have “load_balance = on” in description part of client tnsnames.ora – supposedly that is on by default.

But Transparent Application Failover is not on by default. To use it, again in client tnsnames.ora, add
“(failover=on)” to the description, and “(failover_mode=(type=select)(method=basic))” to the connect_data. Alternatively use “(type=session)” for the more minimal type TAF, which will save some overhead on select operations.

The tnsnames.ora entry will end up looking like:

MYDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = myserver1.my.com)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = myserver2.my.com)(PORT = 1521))
    (LOAD_BALANCE = on)
    (FAILOVER = on)
    (CONNECT_DATA =
      (SERVER = POOLED)
      (SERVICE_NAME = mydb.my.com)
      (FAILOVER_MODE =
        (TYPE = SELECT)
        (METHOD = BASIC)
        (RETRIES = 5)
        (DELAY = 1)
      )
    )
  )

Transparent Application Failover is documented in this part of the Net Services Administrator’s Guide.

Also very good is http://www.oracledba.org/index.html

/etc/init.d/o2cb enable (on both nodes)
mkfs.ocfs2 -L "RAC8LUN" /dev/mapper/RAC8LUN1p1
mkfs.ocfs2 -L "RAC8Q" /dev/mapper/RAC8Qp1
vi /etc/fstab
mkdir /CRS_DATA
mkdir /ORA_DATA
mount /dev/mapper/RAC8LUN1p1
mount /dev/mapper/RAC8Qp1

startup nomount
alter system set cluster_database=false scope=spfile sid='*' ;
shutdown immediate
startup mount
recover database
alter database open ;
alter system set cluster_database=true scope=spfile sid='*' ;
shutdown immediate
srvctl start database -d mydb

alter system set log_archive_dest_1='location=/ORA_DISK/redoarch/arch_' scope=both sid='*' ;
alter system set cluster_database=false scope=spfile sid='mydb1';
srvctl stop database -d mydb
startup mount
alter database archivelog ;
alter system set cluster_database=true scope=spfile sid='mydb1';
shutdown immediate
srvctl start database -d mydb

And switch it into noarchivelog mode:

alter system reset log_archive_dest_1 scope=spfile sid='*' ;
alter system set cluster_database=false scope=spfile sid='mydb1';
srvctl stop database -d mydb
startup mount
alter database noarchivelog ;
alter system set cluster_database=true scope=spfile sid='mydb1';
shutdown immediate
srvctl start database -d mydb

Update 02-Mar-07: dba_ views rather than all_ views have to be used to identify what objects to drop prior to datapump, for reasons explained in this post.

For a nightly process to copy small schemas from one database to another, the normal, or old style, way to do that is with export/import in a cron controlled shell script, which would include a sql-from-sql script to drop all the target schema objects before doing the import.

But that presents problems in RAC, which is what I was working on – which node does the cron script reside on? What happens if that node is down?

An option would be to have the script run on another machine, one unrelated to the RAC cluster, but that introduces more points of failure, and means sending data back and forward over sql*net.

So instead I used the oracle’s 10g scheduler and datapump, as below. This data pumps straight over a database link, without having to write to a dump file in between, which is nice. And because it is all in the database, it ought to be unaffected by particular RAC nodes being down.

Some gotchas:

• The documentation for dbms_datapump isn’t at all clear on what the syntax for SCHEMA_LISTS’s should be. It should be this type of format:

  name => 'SCHEMA_LIST' , value => '''SCHEMA1'',''SCHEMA2'''

  which is a lot of quotes. Thanks to Sunit Joshi for that tip in http://www.webservertalk.com/message1791234.html

• The dbms_datapump documentation has a bug: dbms_datapump.open does not have a parameter called “mode“, it has one called “job_mode” instead.
• When calling dbms_datapump from within a stored procedure, as opposed to an anonymous pl/sql block, the owner of the stored procedure has to have had “create table” privilege granted to them explicitly, rather than through a role. That’s detailed in metalink note 315488.1, but I wasted a lot of time trying to debug that before I went on to metalink. The error reported doesn’t help much:

  ERROR at line 1:
  ORA-31626: job does not exist
  ORA-06512: at "SYS.DBMS_SYS_ERROR", line 79
  ORA-06512: at "SYS.DBMS_DATAPUMP", line 911
  ORA-06512: at "SYS.DBMS_DATAPUMP", line 4330
  ORA-06512: at "NIGHTLY_IMPORT", line 4
  ORA-06512: at line 1

• Jobs are created disabled by default with dbms_scheduler, you have to specify “enabled => TRUE” to get them to run.
• new 13-Feb-07 Setting table_exists_action to ‘replace’ overwrites any existing tables ok, but does not replace view definitions, plsql objects, sequences, and so on. They all have to be explicitly dropped before datapump is called. That is quite a tricky counterintuitive limitation of the table_exists_action replace functionality – why would you want new tables copied over but old sequences left in place? It is suprises like this that make old fashioned (but well understood) exp/imp still worth using.
• new 13-Feb-07 Be aware that the dropped tables will stay on in the recyclebin. There is a good recyclebin article at http://orafaq.com/node/968.

Good information on this is at http://orafaq.com/wiki/Datapump and http://orafaq.com/node/862.

Here’s my code. I included a check against a flag table which allows developers to switch the job on and off:

create table run_copy ( run_copy varchar2(3) ) tablespace users ;
grant select, insert, update, delete, references on run_copy to public ;
create public synonym run_copy for run_copy ;
insert into run_copy ( run_copy ) values ( 'Yes' ) ;
create database link nightly_import_sourcedb connect to username identified by password using 'sourcedb' ;
-- procedure owner needs explicit create table
grant create table to username ;

CREATE OR REPLACE PROCEDURE nightly_import AS
  -- Andrew Fraser 30 January 2007
  run_import NUMBER ;
  hand NUMBER ;
BEGIN
  run_import := 0 ;
  SELECT COUNT(*) INTO run_import FROM run_copy WHERE UPPER(run_copy) IN ('N','NO') ;
  IF run_import > 0
  THEN
    -- dont run import if someone has set NO flag in run_copy table
    NULL ;
  ELSE
    -- first have to drop plsql and other objects that table_exists_action does not deal with
    FOR d1 IN ( select object_type, owner, object_name from all_objects
      where object_type in ('FUNCTION','PACKAGE','PROCEDURE','SEQUENCE','SYNONYM','VIEW')
      and owner in ('BACKUP','GATEKEEPER','REFDATA','TRANSDATA','WHSUSR','WHS_VIEWER') )
    LOOP
      execute immediate 'drop '||d1.object_type||' '||d1.owner||'.'||d1.object_name ;
    END LOOP ;
    -- then run datapump import itself
    hand := dbms_datapump.open (
      operation    => 'IMPORT' ,
      job_mode     => 'SCHEMA' ,
      remote_link  => 'nightly_import_sourcedb' ) ;
    dbms_datapump.metadata_filter (
      handle       => hand ,
      name         => 'SCHEMA_LIST' ,
      value        => '''BACKUP'',''GATEKEEPER'',''REFDATA'',''TRANSDATA'',''WHSUSR'',''WHS_VIEWER''' ) ;
    dbms_datapump.set_parameter (
      handle       => hand ,
      name         => 'TABLE_EXISTS_ACTION' ,
      value        => 'REPLACE' ) ;
    dbms_datapump.start_job(hand) ;
  END IF ;
END ;
/

BEGIN
DBMS_SCHEDULER.CREATE_JOB (
   job_name           =>  'run_nightly_import' ,
   job_type           =>  'STORED_PROCEDURE' ,
   job_action         =>  'nightly_import' ,
   start_date         =>  TRUNC(sysdate+1)+4/24 , /* start 4am tomorrow */
   repeat_interval    =>  'FREQ=DAILY; INTERVAL=1', /* run every night */
   enabled            =>  TRUE ,
   comments           =>  'Andrew Fraser 30 January 2007');
END;
/

Even with the code for this method so short, I’m not sure I would always want to use it over cron and old fashioned export/import. But for RAC, and also for Windows servers (because they lack a decent shell scripting language), this is the way to go.