mtputty directory path title

MtPuTTY multi tabbed putty is an improved variant of putty ssh client, easier to use if working on several servers at the same time.

However it has an annoying tendency to display long directory paths in each tab title.

You can switch that behaviour off with Tools > Settings > Show Display name.

mtputty1

Microsoft Remote Desktop Connection Manager is the equivalent for remote desktop to Windows servers, and WinSCP the equivalent for file transfer. All three are licence free.

Posted in Uncategorized

Oracle date changed user changed columns and batch performance

Most core application tables benefit from columns like these:

ALTER TABLE mytable ADD (
     date_created DATE DEFAULT SYSDATE
   , user_created_by VARCHAR2(30) DEFAULT USER
   , date_changed DATE
   , user_changed_by VARCHAR2(30)
) ;

The default values deal with inserts, for updates a trigger is needed:

CREATE OR REPLACE TRIGGER tu_mytable
BEFORE UPDATE ON mytable
FOR EACH ROW
BEGIN
   :NEW.date_changed := SYSDATE ;
   :NEW.user_changed_by := USER ;
END ;
/

These column can be useful for improving batch processing performance, especially with an index on NVL ( date_changed , date_created ), by allowing selects to be limited to the small set of records changed since the last run.

If you need to record delete statements (instead of the preferable logical deletes that update a status column to ‘deleted’), then a full history table is needed along with a delete trigger.

Posted in Uncategorized

Bash script to check for final closing /data xml tag in file

Bash script to check for final closing </data> xml tag in file:

if [ -f file.xml ]
then
    # if grep -1iq '</data>' file.xml    # simplest syntax
    # if tac file.xml | grep -1iq '</data>'   # faster syntax (tac is reverse cat)
    if tail -100 file.xml | grep -1iq '</data>'   # fastest syntax, assumes at most 100 trailing lines of whitespace/etc. at end of file.
    then
        echo file ready to be loaded
        # code to load and archive file here
    else
        echo file partially transferred, so no action taken.
    fi
else
    echo file not yet transferred, so no action taken.
fi

This could be put into an infinite loop:

while :
do
    # code to do stuff here
    sleep 1  # sleep for 1 second before checking again, to avoid wasting cpu.
done
Posted in Linux

List oracle indexes and their columns with listagg

List indexes on large tables, with listagg to show the indexed columns in a comma separated line, and outer join to include tables with no indexes:

SELECT t.owner
     , t.table_name
     , ROUND ( t.blocks * 8 / 1024 / 1024 ) AS gb  -- assuming each block is default 8kb
     , t.num_rows
     , i.index_name
     , LISTAGG ( ic.column_name || ' , ' ) WITHIN GROUP ( ORDER BY ic.column_position ) AS ind_cols
     , i.index_type
     , i.uniqueness
     , t.partitioned AS table_partitioned
     , i.partitioned AS index_partitioned
  FROM dba_tables t
  LEFT OUTER JOIN dba_indexes i
    ON i.table_owner = t.owner
   AND i.table_name = t.table_name
  LEFT OUTER JOIN dba_ind_columns ic
    ON ic.index_owner = i.owner
   AND ic.index_name = i.index_name
 WHERE t.blocks > 1024 * 1024 / 8  -- 1gb, assuming each block is default 8kb
 GROUP BY t.owner
     , t.table_name
     , t.blocks
     , t.num_rows
     , i.index_name
     , i.index_type
     , i.uniqueness
     , t.partitioned
     , i.partitioned
 ORDER BY t.blocks DESC , t.owner , t.table_name , i.index_name
;
Posted in Performance tuning

Oracle distributed query join over remote database link slow performance leading driving_site hint fix

Oracle Optimizer doesn’t cope so well with distributed queries, joining to tables in different databases.

1) The leading and/or driving_site hints can dramatically improve distributed query performance.

SELECT /*+ leading(tab1) driving_site(tab2) */ ... FROM table1 AS tab1 JOIN table2@db2 AS tab2 ...

Driving_site specifies the database you want doing the bulk of the work, likely the largest database involved. Specify a table alias rather than a database name in the hint.

Leading specifies the table (alias) you want to start the query off at, likely the most restrictive table involved.

For my problem query, I found leading was better for consistent performance than driving_site, and that response time improved from 28 minutes without hint, down to 27 seconds with leading hint. The driving_site hint was very much cache dependent: 2 minutes 20 seconds initially when run from cold, 7 seconds if rerun while all data was still cached. Hinting both ways at the same time had the same effect in my case as just using the driving_site hint, because my leading table was not in the driving database.

For normal, non distributed queries, it would be better to try to improve optimizer stats before resorting to hints. Where possible, avoid adding hints blindly, but use only in specific cases after testing the effects. Re-testing should be done after a major upgrade, because Oracle continually improve the optimizer, and your hint may eventually prevent improvements from taking effect.

2) If you have a lot of distributed queries like this, you should consider merging the databases into a single database (can keep them in separate schemas).

3) Another option, especially for reporting systems, would be to replicate the remote tables, using e.g. materialized views or transportable tablespaces.

Jonathan Lewis has full details on this in an article in Oracle Scene Magazine Spring 2015 Issue 56. The full magazine is here, or just the four pages of his specific article here.

Posted in Performance tuning

Oracle User Group Scotland Conference 2015

Oracle User Group Scotland Conference 2015

1) SQLcl (SQL command line) new enhanced SQL*Plus, easy to output index info, DDL, output/load data in csv, xml. Early adopter (beta) version available for download now, production version expected soon and to be bundled by default into Oracle 12cR2.

More at Kris Rice.

2) Relax and Recover Martin Nash

A free tool to make a bootable .iso file for fast bare metal recovering of any server, onto which the normal backup files could then be restored. Free GPL licence. Linux only.

3) Who changed my Plan? Presanth Kothuri
Read more ›

Posted in Oracle User Group, Performance tuning

execute immediate quote escape oracle pl/sql

To run this via execute immediate:

CREATE TABLE t2 AS SELECT * FROM t1 WHERE col1 = '1' ;

Either use new style quote format:

EXECUTE IMMEDIATE q'[CREATE TABLE t2 AS SELECT * FROM t1 WHERE col1 = '1']' ;

Or older style still works also:

EXECUTE IMMEDIATE 'CREATE TABLE t2 AS SELECT * FROM t1 WHERE col1 = ''1''' ;
Posted in Uncategorized

Partition an existing oracle table example

I used this SQL to convert an existing non-partitioned table to be partitioned. Table was mostly queried on snapshot_id=’Current’, which had the most up to date data, but had much more historic data that was rarely queried against, all flagged with snapshot_id = ‘YYYYMMDD’ format.

-- 1) Create an empty table with one partition
CREATE TABLE a1 TABLESPACE targtabs_big
PARTITION BY RANGE ( snapshot_id ) (
    PARTITION adm_fact_all VALUES LESS THAN ( MAXVALUE ) TABLESPACE targtabs_big
)
AS SELECT * FROM adm_fact WHERE 1 = 2 ;
-- 2) Swap adm_fact into that empty table with one partition
ALTER TABLE a1 EXCHANGE PARTITION adm_fact_all WITH TABLE adm_fact ;
-- 3) Flip adm_fact to this thing we have just built. (beware grants, indexes, constraints, triggers, etc.)
DROP TABLE adm_fact ;  -- make sure to take a note of grants, indexes, constraints, triggers, etc. before doing this!
RENAME a1 TO adm_fact ;
-- 4) Split the partition to put Current by itself
ALTER TABLE adm_fact SPLIT PARTITION adm_fact_all AT ( '29990000' ) INTO (
     PARTITION adm_fact_old TABLESPACE targtabs_big COMPRESS -- AD 2999 = everything except 'Current'
   , PARTITION adm_fact_current TABLESPACE targtabs_big -- 'Current'
) ;
-- 5) Add the indexes back in - they all disappeared at step 3 above (leave out LOCAL if want global non-partitioned index).
CREATE INDEX i_af_pc on adm_fact ( programme_code , programme_discipline_code )
LOCAL tablespace targindx_med ;
CREATE INDEX i_af_si on adm_fact ( snapshot_id )
LOCAL tablespace targindx_med ;
-- 6) Add grants, constraints, triggers, etc. back in - they all disappeared at step 3 above.
GRANT SELECT ON adm_fact TO role ;
-- 7) Gather optimizer stats
EXEC dbms_stats.gather_database_stats_job_proc
-- Or for 10gR2 or older instead use: EXEC dbms_scheduler.run_job ( 'GATHER_STATS_JOB' )
-- Optionally check things are working ok.
SELECT snapshot_id , COUNT(*) FROM adm_fact GROUP BY snapshot_id ORDER BY 1 ;
SELECT snapshot_id , COUNT(*) FROM adm_fact PARTITION ( adm_fact_old ) GROUP BY snapshot_id ORDER BY 1 ;
SELECT snapshot_id , COUNT(*) FROM adm_fact PARTITION ( adm_fact_current ) GROUP BY snapshot_id ORDER BY 1 ;
-- 9) (Optional) by default updating snapshot_id is not allowed, so e.g.:
-- UPDATE adm_fact SET snapshot_id = 'Current' WHERE snapshot_id != 'Current' AND ROWNUM < 11 ;
-- would fail with error ""ORA-14402: updating partition key column would cause a partition change"
-- You can change that behaviour with:
ALTER TABLE adm_fact ENABLE ROW MOVEMENT ;

Dbms_redefinition could be used instead to reduce the outage time to almost nothing, but the above method is fast, was good enough for my outage timescales.

Indexes or their partitions can be compressed as well as table partitions, but often less benefit from that.

Posted in Uncategorized

SQL to display date duration as number of years and months

Convert date duration (here customer age at time of first application) in years and months, e.g. ‘3408’ for 34 years and 8 months old:

  WITH customers AS ( SELECT SYSDATE AS appl_date , TO_DATE ( '01-AUG-1980' , 'DD-MON-YYYY' ) AS date_of_birth FROM DUAL )
SELECT TO_CHAR ( GREATEST ( 0 , LEAST ( 99 , FLOOR ( ( c.appl_date - c.date_of_birth ) / 365.25 ) ) ) , 'fm00' )  -- years
          || TO_CHAR ( GREATEST ( 0 , FLOOR ( 12 * MOD ( ( ( c.appl_date - c.date_of_birth ) / 365.25 ) , 1 ) ) ) , 'fm00' ) AS years_months
     , ( c.appl_date - c.date_of_birth ) / 365.25 AS years
     , 12 * MOD ( ( ( c.appl_date - c.date_of_birth ) / 365.25 ) , 1 ) AS months
  FROM customers c ;
Posted in Uncategorized

Rebuild table indexes lobs in oracle

Rebuilds are beneficial in some rare circumstances such as after a mass delete operation. The below method is safer and faster than export/import.

Example table:

CREATE TABLE tab1 ( col1 VARCHAR2(1) , col2 CLOB ) TABLESPACE ts1 ;
CREATE INDEX tab1i ON tab1 ( col1 ) TABLESPACE ts1 ;

List objects and tablespaces:

SELECT tablespace_name FROM user_tables WHERE table_name = 'TAB1' ;
SELECT tablespace_name FROM user_lobs WHERE table_name = 'TAB1' ;
SELECT tablespace_name , index_name FROM user_indexes WHERE table_name = 'TAB1' ;

Rebuild:

ALTER TABLE tab1 MOVE ;
ALTER TABLE tab1 MOVE LOB ( col2 ) STORE AS ( TABLESPACE ts1 ) ;
ALTER INDEX tab1i REBUILD ;

Update optimizer stats:

EXEC dbms_scheduler.run_job ( 'GATHER_STATS_JOB' )

For large tables, it may be better to rebuild them online using dbms_redefinition.

H/t Anand.

Posted in Uncategorized