Data from MySQL to Oracle with dg4odbc database link or mysqldump

To transfer data from MySQL into Oracle

1) Best method is to use a database link. That way the MySQL table is available in the Oracle database for use in queries, joins, DML, PL/SQL.

Simple guides on how to do to that are:

  1. Shishir Tekade http://www.shishirtekade.com/2014/01/connect-oracle-to-mysql-server-through.html
  2. Pythian http://www.pythian.com/blog/how-to-access-mysql-from-oracle-with-odbc-and-sql/
  3. My Oracle Support 1320645.1 ‘Detailed Overview of Connecting Oracle to MySQL Using DG4ODBC Database Link’ https://support.oracle.com/epmos/faces/DocContentDisplay?id=1320645.1

1b) For a critical/enterprise system it might be worth paying for Oracle’s Hetrogenous Services Database Gateway. That works the same as above but substitutes Oracle’s proprietary driver files in place of free ODBC. The advantage is Oracle Support would fully support this, not pass the buck onto the free ODBC drivers.

2) Alternatively can have a regularly running scheduled task or cron that copies data from MySQL into Oracle. The mysqldump utility used for this is often installed by default with linux, in /usr/bin. To avoid failed copies from leaving empty tables in the target database, a safety check (e.g. count below) is needed before proceeding.

rm -f data.tmp

# 1) Extract list of insert statements from mysql into file data.tmp
mysqldump --host=myhost --user=myuser --password=mypassword --complete-insert --no-create-info --skip-opt --compatible=oracle --skip-quote-names \
    --compact mydatabase mytable > data.tmp

# 2) Change table name from mytable to mytable_tmp in insert statements
sed -i 's/mytable/mytable_tmp/g' data.tmp

# 3) Insert those into Oracle - using temp table and begin/end check to only proceed if reasonable number of rows.
sqlplus -s username/password@database << END_ORACLE
set serverout on
-- Use MySQL datetime format
ALTER SESSION SET nls_date_format='YYYY-MM-DD HH24:MI:SS' ;
DELETE FROM mytable_tmp ;
@data.tmp
-- But only copy this data to the live table if looks to be a reasonable amount of data.
DECLARE
   l_count NUMBER ;
BEGIN
   SELECT COUNT(*) INTO l_count FROM mytable_tmp ;
   IF l_count > 100
   THEN
      dbms_output.put_line ( 'Copying ' || l_count || ' rows from MySQL into mytable' ) ;
      DELETE FROM mytable ;
      INSERT INTO mytable SELECT * FROM mytable_tmp ;
   ELSE
      dbms_output.put_line ( 'Error: insufficient rows loaded, not proceeding with data copy.' ) ;
   END IF ;
END ;
/
END_ORACLE

Removing –no-create-info option from mysqldmp will generate oracle compatible create table DDL statements, useful for first setup.

h/t Slim Backwater.

Posted in Uncategorized | Leave a comment

Analytics row_number to identify first parent row in oracle pl/sql cursor loop

Let’s say you have 10m parent rows and 11m child rows and need to process them row-at-a-time in pl/sql. Two methods of doing that are below.

  • The second method will fire the combined select just the once, picking up all 11m rows in one go.
  • The first method will fire the child select 10m times individually, with a different id value passed into the where clause each time. It uses bind variables for that, so doesn’t hard parse 10m times, but still will take time.

The first method is the simplest to read, maintain and debug, so stick with that unless performance is required.

Method 1 – traditional way, two cursor loops

BEGIN
   FOR r_dept IN ( SELECT deptno , dname FROM dept )
   LOOP
      -- put parent (department) row processing code here.
      FOR r_emp IN ( SELECT empno , ename FROM emp WHERE emp.deptno = r_dept.deptno )
      LOOP
         -- put child (employee) row processing code here.
      END LOOP ;
   END LOOP ;
END ;

Method 2 – single cursor loop

BEGIN
   FOR r_both IN (
      SELECT dept.deptno
           , dept.dname
           , emp.empno
           , emp.ename
           , ROW_NUMBER() OVER ( PARTITION BY dept.deptno ORDER BY dept.deptno , emp.empno ) AS rn
        FROM dept
        LEFT OUTER JOIN emp  -- outer join if you want to process childless parents = departments with no employees assigned. 
          ON dept.deptno = emp.deptpno
   )
   LOOP
      IF r_both.rn = 1  -- starting to work on a different parent (department) record.
      THEN
         -- put parent (department) row processing code here.
      END IF ;
      -- put child (employee) row processing code here.
   END LOOP ;
END ;
Posted in PL/SQL, Uncategorized | Leave a comment

oracle substrb function to chop a string down to its first 4000 bytes

The substrb 1 , 4000 function will chop a string down to its first 4000 bytes, e.g.:

SELECT SUBSTR('1€3',2,1) , SUBSTRB('1€3',1,3) , LENGTH('1€3') , LENGTHB('1€3') FROM dual ;

Needed if using the (non default) option of ‘char’ in column lengths:

CREATE TABLE mytable ( col1 VARCHAR2(4000 CHAR) ) ;

You can change the default setting at session level:

ALTER SESSION SET nls_length_semantics = 'CHAR' ;

But that needs run every time you connect in, it doesn’t remember session settings after you disconnect. Permanently changing that parameter at database level is possible but not advisable.

You can see what is set as default at each of the 3 levels with:

SELECT * FROM nls_session_parameters  WHERE parameter IN ( 'NLS_LENGTH_SEMANTICS' , 'NLS_CHARACTERSET' , 'NLS_NCHAR_CHARACTERSET' ) ORDER BY parameter ;
SELECT * FROM nls_instance_parameters WHERE parameter IN ( 'NLS_LENGTH_SEMANTICS' , 'NLS_CHARACTERSET' , 'NLS_NCHAR_CHARACTERSET' ) ORDER BY parameter ;
SELECT * FROM nls_database_parameters WHERE parameter IN ( 'NLS_LENGTH_SEMANTICS' , 'NLS_CHARACTERSET' , 'NLS_NCHAR_CHARACTERSET' ) ORDER BY parameter ;

The NVARCHAR2 datatype could be useful if you want to use a different character set for a particular column.

Posted in Uncategorized | Leave a comment

Format sql files with sed: tab to space, pad commas with spaces, uppercase match

Shell script to make more readable all sql files in a directory – pads commas and brackets with whitespace, converts tabs to spaces, makes SQL keywords uppercase.

# FUNCTIONS
pad () {
    sed -i "s/\([^ ]\)$1/\1 $1/g" *sql
    sed -i "s/$1\([^ ]\)/$1 \1/g" *sql
}
upper () {
    sed -i "s/\(\<$1\>\)/\U\1/g" *sql
}
# MAIN
dos2unix *sql
# tabs to 4 spaces
sed -i 's/\t/    /g' *sql
# pad brackets commas semicolons
pad '('
pad ')'
pad ','
pad ';'
pad '!='
pad '='
pad '||'
# uppercase
upper nvl
upper decode
upper to_char
upper least
upper greatest
upper when
upper then
upper case
upper end
upper values
upper varchar2
upper in
upper date
upper upper
upper regexp_replace
upper sysdate
upper exists
upper replace
upper else
upper substr
upper count
upper distinct
upper partition
upper over
Posted in Uncategorized | Leave a comment

Laravel Symfony CodeIgniter

Laravel is the new CodeIgniter, a coding framework on top of PHP for simplified/standardised syntax. Complex PHP applications can become un-supportable without a coding framework like those in place.

To install and demo Laravel on WAMP:

(1) follow the steps in http://www.darwinbiler.com/how-to-install-laravel-on-wamp-for-beginners/ (note, uses a different WAMP php.ini file from normal).

(2) File c:\wamp\www\laraveltest\public\.htaccess then needs

RewriteBase /laraveltest/public/

added under the RewriteEngine line, as at http://stackoverflow.com/a/18412482

(3) Using WAMP click on wamp icon > apache > apache modules > scroll and check ‘rewrite_module Restart a LoadModule’ http://stackoverflow.com/a/14490649

(4) Then edit c:\wamp\www\laraveltest\app\routes.php and browse to http://localhost/laraveltest/public/user to see effects.

(5) For database connectivity, edit c:\wamp\www\laraveltest\app\config\database.php to specify database name.

(6) To demo database connectivity, again edit c:\wamp\www\laraveltest\app\routes.php to add code like

Route::pattern('id', '[0-9]+');
Route::get('/user/{id}', function($id)
{
    // Only called if {id} is numeric.
    $user = DB::table('users')->where('id', $id)->first();
    var_dump($user);

    echo '

prev | next

'; });

And browse to http://localhost/laraveltest/public/user/1 to see working.

A table called ‘users’ was used for this demo, with an ‘id’ numeric field plus some text fields

CREATE TABLE IF NOT EXISTS `users` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(250) NOT NULL,
  `email` varchar(400) NOT NULL,
  `description` text NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB  DEFAULT CHARSET=utf8 AUTO_INCREMENT=3 ;
INSERT INTO `users` (`id`, `name`, `email`, `description`) VALUES
(1, 'John Smith', 'john@email.com', 'John''s description field...'),
(2, 'Jane Doe', 'jane@email.com', 'Jane''s description field');

More database code at http://laravel.com/docs/queries#selects, and forms code at http://laravel.com/docs/html.

For debugging, look in file c:\wamp\www\laraveltest\app\storage\logs\laravel.log

Currently Laravel supports four database systems: MySQL, Postgres, SQLite, and Microsoft SQL Server, but does not support Oracle http://laravel.com/docs/database. However the Laravel-OracleDB driver package is avilable as an add on https://github.com/jfelder/Laravel-OracleDB.

An alternative to Laravel is Symfony.

Installation of Symfony requires composer – but that should already have been installed along with Laravel above, which means would now just need:

mkdir c:\wamp\www\symfony
cd c:\wamp\www\symfony
composer create-project symfony/framework-standard-edition myproject/ ~2.4
php app/console server:run

[Source: http://symfony.com/doc/current/quick_tour/the_big_picture.html]

Once installed, browse to http://localhost:8000/ and also http://localhost:8000/demo/hello/YourNameHere.

Posted in php | Leave a comment

PL/SQL bulk collect union

PL/SQL bulk collect will work with UNION ALL when written with an inline view:

DECLARE
   TYPE l_type_assoc IS TABLE OF VARCHAR2(14) INDEX BY BINARY_INTEGER ;
   l_big l_type_assoc ;
BEGIN
     WITH q AS (
          SELECT ename AS name
            FROM emp
           UNION ALL
          SELECT dname AS name
            FROM dept
          )
   SELECT name
     BULK COLLECT INTO l_big
     FROM q
    ORDER BY name ;

   FOR indx IN 1..l_big.COUNT
   LOOP
      DBMS_OUTPUT.put_line ( l_big(indx) ) ;
   END LOOP ;

END ;
/

If the array you are using is not an associative array (that is, does not have the INDEX BY clause in its declaration) then an alternative method is to merge arrays together after they are populated with multiset union:

DECLARE
   TYPE l_type_varray IS TABLE OF VARCHAR2(14) ;
   l_tmp l_type_varray ;
   l_big l_type_varray ;
BEGIN

   SELECT ename BULK COLLECT INTO l_tmp FROM emp ;

   l_big := l_tmp ;  -- copy the array

   SELECT dname BULK COLLECT INTO l_tmp FROM dept ;

   l_big := l_big MULTISET UNION ALL l_tmp ;  -- union the arrays

   FOR indx IN 1..l_big.COUNT
   LOOP
      DBMS_OUTPUT.put_line ( l_big(indx) ) ;
   END LOOP ;

END ;
/

PHP’s oci_bind_array_by_name will only work with associative arrays, returns error ‘ORA-06550: PLS-00306: wrong number or types of arguments in call’ if varrays are used.

Posted in Uncategorized | Leave a comment

Extended Hexadecimal in Oracle

Standard decimal > hexadecimal is done with:

select to_char ( 10 , 'x' ) from dual ;

But I wanted a kind of extended hexadecimal that would keep single letters going up beyond 15/f, so would return ‘g ‘for decimal 16 instead of returning hexadecimal ’10’.

Why did I want that? I needed to squeeze a 2 digit number column into a 1 character column in an extract transform load operation. Most of the source data was less than 16, but there were a few values higher than that, and a simple decimal > hexadecimal translation wouldn’t work for those.

The ascii value for ‘a’ is 97, so using the chr function to turn numbers into their ascii character equivalents gives:

SELECT CASE
           WHEN numcol BETWEEN  0 AND  9 THEN TO_CHAR ( numcol )
           WHEN numcol BETWEEN 10 AND 35 THEN CHR ( 87 + numcol )
       END AS extended_hexadecimal
  FROM mytable ;
Posted in Uncategorized | Leave a comment

srvctl start listener -n nodename

Preferred method to stop and start oracle listener in RAC enivronment:

srvctl config listener -n nodename
srvctl stop listener -n nodename
srvctl start listener -n nodename
Posted in RAC | Leave a comment

Denormalise listagg children on single row comma separated

You can denormalise data using the listagg function, so to display users with their roles on a single line comma separated:

SELECT grantee
     , LISTAGG ( granted_role , ' , ' ) WITHIN GROUP ( ORDER BY granted_role ) AS granted_roles
  FROM dba_role_privs
 GROUP BY grantee
 ORDER BY 1 ;

H/t Adrian Billington. Continue reading

Posted in Uncategorized | Leave a comment

Oracle remove duplicates parent child with listagg

De-duplicating parent – child table sets of data here using the listagg function to merge all the child rows into a single denormalised column.

set serverout on
BEGIN
FOR d1 IN (
     WITH denorm AS (
          SELECT parent.aos_code
               , parent.pre_group_id
               , LISTAGG ( child.aos_code || child.stud_pass ) WITHIN GROUP ( ORDER BY child.aos_code || child.stud_pass ) AS entry_value
            FROM StcAPreList parent
            JOIN StcAPreGroupAos child
              ON parent.pre_group_id = child.pre_group_id
           GROUP BY parent.aos_code , parent.pre_group_id
          )
   SELECT a.pre_group_id
     FROM denorm a
    WHERE a.pre_group_id > (
          SELECT MIN ( b.pre_group_id )
            FROM denorm b
           WHERE a.aos_code = b.aos_code
             AND a.entry_value = b.entry_value )
    ORDER BY 1
   )
LOOP
   dbms_output.put_line ( 'deleting ' || d1.pre_group_id ) ;
   DELETE FROM StcAPreGroupAos child WHERE child.pre_group_id = d1.pre_group_id ;
   DELETE FROM StcAPreList parent WHERE parent.pre_group_id = d1.pre_group_id ;
END LOOP ;
END ;
/
Posted in Uncategorized | Leave a comment