AUTH_COMPATIBILITY value of null causing UNLOAD utility jobs to fail

This week we went to a new fix level that activated the AUTH_COMPATIBILITY in macro DSN6SPRM. The documentation states:

The AUTH_COMPATIBILITY parameter specifies whether current-release of previous-release authorization rules are used for specific Db2 operations.
 
Null (blank)
Indicates that AUTH_COMPATIBILITY has no effect and that there is no authorization check override.
 
SELECT_FOR_UNLOAD
When this value is specified, the UNLOAD utility checks if the user has the SELECT or UNLOAD privilege on the target table. If this value is not specified, the UNLOAD utility checks if the user has the UNLOAD privilege on the target table.

In our case, the zparm was set to null, however, our unload (utility not DSNTIAUL) jobs began to fail. The auth ids executing the unload utilities had SELECT access but not UNLOAD access. The above documentation indicated that the null value should have allowed the jobs to run but we saw failures.

Since this is a dynamic change, we switched from null to SELECT_FOR_UNLOAD and the previous functionality was restored.

Export datapump encounters error ORA-39126 and ORA-01690

We ran across the following error while running a metadata-only export data pump on an 11.2.0.4 database:

ORA-39126: Worker unexpected fatal error in KUPW$WORKER.FETCH_XML_OBJECTS [PROCA
CT_SCHEMA:"SYSMAN"]
ORA-01690: sort area size too small
 
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95
ORA-06512: at "SYS.KUPW$WORKER", line 9715
 
----- PL/SQL Call Stack -----
object line object
handle number name
0x11d4c2bf0 21979 package body SYS.KUPW$WORKER
0x11d4c2bf0 9742 package body SYS.KUPW$WORKER
0x11d4c2bf0 11838 package body SYS.KUPW$WORKER
0x11d4c2bf0 2808 package body SYS.KUPW$WORKER
0x11d4c2bf0 10422 package body SYS.KUPW$WORKER
0x11d4c2bf0 1824 package body SYS.KUPW$WORKER
0x11df95480 2 anonymous block
 
Job "SYS"."SYS_EXPORT_FULL_21" stopped due to fatal error at Wed Dec 23 18:13:11
2020 elapsed 0 00:05:34

According the Oracle Support Doc ID 1358890.1, this is a known bug that occurs after applying patch set 11.2.0.2. There are some solutions in Doc ID 1358890.1 including upgrading to 12.1, applying patch set 11.2.0.4 or an interim patch set. Another solution is to

update sys.metaview$ set properties = properties + 1024 where viewname = 'KU$_PROCACT_SYS_VIEW';

We tried the update option, including increasing the size up to 4096. However, this did not resolve the error. After some trial and error, we were able to complete the export with the below command:

expdp \'/ as sysdba \' directory=exp_dir dumpfile=rpt.dmp exclude=statistics exclude=schema:\"IN \(\'SYSMAN\',\'SYSTEM\',\'VRRPT1NE\',\'SYS\'\)\" full=y content=metadata_only

Changes to SYS user’s password not reflected in column PTIME of SYS.USER$ table

Today I noticed that the changes to the SYS user’s password were not being reflected in the ptime column of SYS.USER$. While this may be trivial, this column can be used to identify passwords that have not been changed recently. This issue is documented by Oracle in “After Changing SYS Password, DBA_USERS.EXPIRY_DATE Not Updated For SYS User (Doc ID 2518310.1)” and Bug 28538439. In summary, after applying the July 2020 patch, you need to apply Patch 28538439 and perform the below steps:

This was performed on a RAC database running Release 12.1.0.2.0.

After changing the SYS user’s password, the SYS.USER$ table displayed the below information:

set line 300
col name format a15
SELECT    name, ctime, ptime
FROM      sys.user$
where     name like 'SYS%'
order by  name
;  
 
NAME            CTIME     PTIME
--------------- --------- ---------
SYS             07-AUG-18 10-JUN-19 <-- Incorrect date

Check to see if patch 28538439 has been applied:

opatch lsinv -all | grep 28538439
     21117072, 22611167, 23182225, 25139545, 
     26196152, 27487279, 28538439 <-- Patch has been applied

The solution is to set the below parameter to “TRUE”:

show parameter _enable_ptime_update_for_sys;
 
No output is displayed, indicating that the 
parameter has not been set

Note that this parameter cannot be set by an ALTER SYSTEM command. It has to be present in the spfile at startup.

Create a temporary pfile from the spfile:

create pfile='/tmp/PRD1_pfile.txt' from spfile='+ORA_DATA_1/PRD1/PARAMETERFILE/spfile';

Edit the temporary pfile and add in the parameter:

vi /tmp/PRD1_pfile.txt
 
add the below line
 
*._enable_ptime_update_for_sys = TRUE

Stop the database and check the status:

srvctl stop database -d PRD1 -o immediate
 
srvctl status database -d PRD1

Start up one node of the RAC in nomount to recreate the spfile from the pfile:

sqlplus / as sysdba
 
startup nomount pfile='/tmp/PRD1_pfile.txt';
 
create spfile='+ORA_DATA_1/PRD1/PARAMETERFILE/spfile' from pfile='/tmp/PRD1_pfile.txt';
 
shutdown immediate;

Start up the RAC (all nodes) and check the status:

srvctl start database -d PRD1
srvctl status database -d PRD1

Check that the parameter to resolve Bug 28538439 has been set:

show parameter _enable_ptime_update_for_sys;
 
Output will be:
 
NAME                         TYPE    VALUE
---------------------------- ------- -----
_enable_ptime_update_for_sys boolean TRUE

This is not retroactive to the previous change to the SYS user’s password. The next time the SYS user’s password is changed, it will be reflected in the SYS.USER$ table:

set line 300
col name format a15
SELECT    name, ctime, ptime
FROM      sys.user$
where     name like 'SYS%'
order by  name
;  
 
NAME            CTIME     PTIME
--------------- --------- ---------
SYS             07-AUG-18 03-DEC-20 <-- Correct date

Oracle GoldenGate – Conditional replication based on the value of a column

A colleague had a requirement to replicate rows from a table only when a column was updated to a specific value and came up with this solution. The table in question was a configuration table which contained parameters and values that was loaded in advance of code deployment. After the code was deployed, the parameter was activated by updating the column REPLICATE_FLAG to a value of “y”.

On the source side, the table is defined in the .prm file as follows:

TABLE TABLE_OWNER.CONFIG_TABLE, WHERE (REPLICATE_FLAG = 'y' OR REPLICATE ='Y');

This causes GoldenGate to replicate only after the row is updated to a “y”. On the target side, the row can either exist in which case GoldenGate performs an update or the row does not exist and GoldenGate performs an insert. In order to handle both conditions, the *.prm file has the parameter insertmissingupdates.

insertmissingupdates
MAP TABLE_OWNER_PROD.CONFIG_TABLE , TARGET TABLE_OWNER_TARGET.CONFIG_TABLE;
noinsertupdates
MAP TABLE_OWNER_PROD.OTHER_TABLE , TARGET TABLE_OWNER_TARGET.OTHER_TABLE

Since you do not want this parameter (i.e. missing updates converted to inserts behavior) to be applied to other tables being replicated, you can either turn it off with the noinsertupdates or put the table that needs the special handling at the end.

This was tested on Oracle GoldenGate Version 19.1.0.0.4 and Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production

Left aligning DSNTEP2 (aka batch SPUFI) output

A short blog about a trivial but annoying item. By default, the DSNTEP2 (aka batch SPUFI) output is center aligned. As I prefer the output to be left aligned for readability and parsing of output, I code the JCL as described below.

By default, the below code will center align the output:

//SYSTSIN   DD *                                         
 DSN SYSTEM(PRN3)                                        
 RUN  PROGRAM(DSNTEP2) PLAN(DSNTEP2)
//*                                                      
//SYSIN    DD  *                                         
  SELECT    COUNT(*)                                     
  FROM      Q.OBJECT_DATA                                
  ;                                                      

And the output looks like:
(Actual display is further to the right.I moved it to the left for readability.)

PAGE    1                                                                     
***INPUT STATEMENT:                                                           
   SELECT    COUNT(*)                                                         
   FROM      Q.OBJECT_DATA                                                    
   ;                                                                          
                                                 +----------------+ 
                                                 |                | 
                                                 +----------------+ 
                                               1_|           4042 | 
                                                 +----------------+ 
SUCCESSFUL RETRIEVAL OF          1 ROW(S)                                     

To left align, code as:

Default (i.e. center) alignment:
//SYSTSIN   DD *                                         
 DSN SYSTEM(PRN3)                                        
 RUN  PROGRAM(DSNTEP2) PLAN(DSNTEP2) PARMS('/ALIGN(LHS)')
//*                                                      
//SYSIN    DD  *                                         
  SELECT    COUNT(*)                                     
  FROM      Q.OBJECT_DATA                                
  ;                                                      

And the output looks like:

PAGE    1                                
***INPUT STATEMENT:                      
   SELECT    COUNT(*)                    
   FROM      Q.OBJECT_DATA               
   ;                                     
       +----------------+                
       |                |                
       +----------------+                
     1_|           4042 |                
       +----------------+                
SUCCESSFUL RETRIEVAL OF          1 ROW(S)

REORG and contention – Forcibly terminating threads?

One of the challenges in performing online REORGs (i.e. SHRLEVEL CHANGE) is successfully completing the switch phase of the process. This problem seems to compound in direct relation to the number of distributed threads that connect to the database objects. I often see a pattern wherein a distributed thread acquires a lock or claim on an object and then does not perform a commit to release the lock or claim. This results in the REORG failing with a page out to the DBA.

I initially coded the REROG jobs with

REORG TABLESPACE   LIST CLUSTTS                            
      LOG           NO                        
      KEEPDICTIONARY                          
      ROWFORMAT BRF                           
      SORTDATA                                
      SORTKEYS                                
      NOSYSREC                                
      COPYDDN      (LPCOPY)                   
      RECOVERYDDN  (RBCOPY)                   
      SHRLEVEL      CHANGE                    
      DRAIN_WAIT    10 RETRY 3  RETRY_DELAY 10
      DEADLINE      NONE                      
      MAXRO         60                        
      DRAIN         WRITERS   <---
      LONGLOG       CONTINUE                  
      DELAY         0                         
      TIMEOUT       TERM                      
      FASTSWITCH    YES                       
      UNLOAD        CONTINUE                  
      STATISTICS    TABLE INDEX ALL           
      UPDATE ALL                

Assuming that this would be the least invasive approach. This was based on the below information from the Utility Guide:

WRITERS
Specifies that Db2 drains only the writers during the 
log phase after the MAXRO threshold is reached and 
then issues DRAIN ALL on entering the switch phase.

I changed the options to include DRAIN ALL FORCE ALL:

REORG TABLESPACE   LIST CLUSTTS                            
      LOG           NO                        
      KEEPDICTIONARY                          
      ROWFORMAT BRF                           
      SORTDATA                                
      SORTKEYS                                
      NOSYSREC                                
      COPYDDN      (LPCOPY)                   
      RECOVERYDDN  (RBCOPY)                   
      SHRLEVEL      CHANGE                    
      DRAIN_WAIT    10 RETRY 3  RETRY_DELAY 10
      DEADLINE      NONE                      
      MAXRO         60 
      DRAIN         ALL    <---                   
      FORCE         ALL    <---           
      LONGLOG       CONTINUE                  
      DELAY         0                         
      TIMEOUT       TERM                      
      FASTSWITCH    YES                       
      UNLOAD        CONTINUE                  
      STATISTICS    TABLE INDEX ALL           
      UPDATE ALL                

The results were as follow:

If the DDF thread had only performed a select without a commit, it was cancelled with the below message in the DB2 master address space:

DSNL027I  -PR01 SERVER DISTRIBUTED AGENT WITH  828                   
           LUWID=JF0C80D3.DC4E.D8C6D35F5B78=129996                   
                                                                     
THREAD-INFO=ABC123Y:67.19.138.241:ABC123y:db2jcc_application:DYNAMIC:
5719:*:<::67.19.138.241.55378.D8C6D35F5B78>                          
           RECEIVED ABEND=04E                                        
           FOR REASON=00E50013

If the DDF thread had performed an update without a commit, it was not cancelled. Instead, the REORG fails with the below message in the job output:

DSNURDRN - RESOURCE NOT AVAILABLE, REASON=X'00C200EA', ON 
DBNAME.TSNAME PROHIBITS PROCESSING

I would have preferred that the DRAIN ALL FORCE ALL options would have cancelled the thread in both cases.

STROBing a transaction

In the last few weeks, we have been challenged with a system that was supposed to process orders was not processing them in a timely fashion. TMON indicated that the process was sitting in an IC wait suggesting that the delay was intentional. However, the concern was that the process was waiting on some unknown resource such as the database. The process consisted of a CICS transaction that called a number of COBOL programs and accessed an ADABAS database.

In an attempt to identify the actual line of code that was causing the delay, we turned on STROBE. Since this is a CICS transaction, you have to run an “ADD ACTIVE” measurement as the CICS region is an executing job. Set the job name to the CICS region, the session duration to 10 minutes, and the target sample size to 20,000 (Change these numbers as required for your context). In addition, select the “Data Collectors” option with a “Y”. On the Data Collectors panel, select the CICS+ option with a “Y”.

On the CICS Data Collector Options, select “Collect Region Data” with a “Y” and provide the name of the transaction in the “Detail by Transaction” section of the panel.

At this point in time, the measurement should be active. Monitor the measurement of completion and then generate a profile in the foreground. You can do this in batch but it is a relatively trivial operation so it can be performed quickly in the foreground.

After the profile is generated, the transaction summary (#TUS) report indicated that 599 seconds of the measurement were attributable to one transaction. Further, all of this time was attribute to a single program. STROBE was able to provide us with the actual CICS command “EXEC CICS DELAY” and the offset of the compile a listing. We were able to take this offset error look into the compile listing and identify that the line of command that was generating the delay was an

EXEC CICS
     DELAY FOR SECONDS (number_of_seconds>
END-EXEC

This proved that the delay in the process was intentional and not the result of a database or other resource contention.

Is it worth turning on compression?

I had a need to turn on auditing of SYS operations and FGA on some tables. Additionally, I preferred to have the audit destination be the DB. I had moved bot the SYS.AUD$ and the SYS.FGA_LOG$ out of SYSAUX to their own tablespace. After collecting some data, I was curious the see if compressing the tables would allow me to store more data. This was on a 12.1.0.2.0 – 64bit Unix environment.

Run the GET_COMPRESSION_RATIO procedures as follows:

SET SERVEROUTPUT ON
DECLARE
  l_blkcnt_cmp    PLS_INTEGER;
  l_blkcnt_uncmp  PLS_INTEGER;
  l_row_cmp       PLS_INTEGER;
  l_row_uncmp     PLS_INTEGER;
  l_cmp_ratio     NUMBER;
  l_comptype_str  VARCHAR2(32767);
BEGIN
  DBMS_COMPRESSION.get_compression_ratio (
    scratchtbsname  => 'USERS',
    ownname         => 'SYS',
    objname         => 'AUD$',
    subobjname      => NULL,
    comptype        => DBMS_COMPRESSION.comp_advanced,
    blkcnt_cmp      => l_blkcnt_cmp,
    blkcnt_uncmp    => l_blkcnt_uncmp,
    row_cmp         => l_row_cmp,
    row_uncmp       => l_row_uncmp,
    cmp_ratio       => l_cmp_ratio,
    comptype_str    => l_comptype_str,
    subset_numrows  => DBMS_COMPRESSION.comp_ratio_allrows,
    objtype         => DBMS_COMPRESSION.objtype_table
  );

  DBMS_OUTPUT.put_line('Number of blocks used (compressed)       : ' ||  l_blkcnt_cmp);
  DBMS_OUTPUT.put_line('Number of blocks used (uncompressed)     : ' ||  l_blkcnt_uncmp);
  DBMS_OUTPUT.put_line('Number of rows in a block (compressed)   : ' ||  l_row_cmp);
  DBMS_OUTPUT.put_line('Number of rows in a block (uncompressed) : ' ||  l_row_uncmp);
  DBMS_OUTPUT.put_line('Compression ratio                        : ' ||  l_cmp_ratio);
  DBMS_OUTPUT.put_line('Compression type                         : ' ||  l_comptype_str);
END;
/

The output looks like:

Number of blocks used (compressed)       : 252
Number of blocks used (uncompressed)     : 459
Number of rows in a block (compressed)   : 70
Number of rows in a block (uncompressed) : 38
Compression ratio                        : 1.8
Compression type                         : "Compress Advanced"

This indicates that we can store about twice the number of rows in the same amount of space.

Setting up Fine Grained Auditing (FGA)

I had a requirement to track the activity on a list of tables and decided to try of Fine Grained Auditing (FGA). This was done in a 12.1.0.2.0 – 64bit Unix environment.

Note that the SYS user is not audited by FGA.

Setting up FGA

Check the location of the audit trail

show parameter audit;

If not set to DB, change as follows (11g):

alter system set audit_trail='DB_EXTENDED' scope=SPFILE sid='*';

In 12c the command seems to have changed to:

alter system set audit_trail=DB,EXTENDED scope=SPFILE sid='*';

Bounce the database

The tables to contain the auditing data are created in SYSAUX. It is good practice to move them to their own table space.

SELECT   table_name
        ,tablespace_name 
FROM     dba_tables 
WHERE    owner='SYS' 
  AND    table_name IN ('AUD$','FGA_LOG$')
;

The standard auditing data is written into SYS.AUD$ but a better-formatted version is in table DBA_AUDIT_TRAIL. Similarily, SYS.FGA_LOG$ is best viewed from DBA_FGA_AUDIT_TRAIL.
Create a tablespace

CREATE TABLESPACE "AUDAUX" DATAFILE 
  '+DG_ORA1/datafile/audaux_1.dbf' SIZE 2G
  LOGGING ONLINE PERMANENT BLOCKSIZE 8192
  EXTENT MANAGEMENT LOCAL UNIFORM SIZE 524288 DEFAULT 
 NOCOMPRESS  SEGMENT SPACE MANAGEMENT AUTO
;

Turn off auto extend as this can fill up fast:

alter database datafile '+DG_ORA1/datafile/audaux_1.dbf' autoextend off;

Move SYS.AUD$ to the new tablespace with

BEGIN
  DBMS_AUDIT_MGMT.set_audit_trail_location(
    audit_trail_type           => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,
    audit_trail_location_value => 'AUDAUX');
END;
/

Move SYS.FGA_LOG$ to the new tablespace with:

BEGIN
  DBMS_AUDIT_MGMT.set_audit_trail_location(
    audit_trail_type           => DBMS_AUDIT_MGMT.AUDIT_TRAIL_FGA_STD,
    audit_trail_location_value => 'AUDAUX');
END;
/

Create an audit policy as follows. In this case, I am auditing inserts, updates, and deletes on this table.

EXEC DBMS_FGA.ADD_POLICY
(
  object_schema=>'BILLING',
  object_name=>'INVENTORY_MASTER',
  policy_name=>'AUDIT_BILLING_INVENTORY_MASTER',
  statement_types=>'insert, update, delete'
)
;

If needed, the policy can be dropped with
EXEC DBMS_FGA.DROP_POLICY
(
object_schema=>’BILLING’,
object_name=>’INVENTORY_MASTER’,
policy_name=>’AUDIT_BILLING_INVENTORY_MASTER’
)
;

Check the current policies and their status

SELECT *
--object_schema,object_name,policy_name,policy_column,enabled,sel,ins,upd,del 
FROM   dba_audit_policies
;

The above example turned on FGA for all columns in the table. FGA can also be used for a specific column as shown below:

EXEC DBMS_FGA.ADD_POLICY
(
  object_schema=>'BILLING',
  object_name=>'INVENTORY_MASTER',
  policy_name=>'AUDIT_BILLING_INVENTORY_MASTER',
  audit_column=>'bonus'
  statement_types=>'insert, update, delete'
)
;

The columns that are part of an FGA policy can be checked with:

SELECT  * 
FROM    dba_audit_policy_columns;

The information being collected by the FGA can be viewed with:

ALTER SESSION SET nls_date_format='dd-Mon-yyyy hh24:mi:ss';

select    *
FROM      DBA_FGA_AUDIT_TRAIL
order by  timestamp
;

Cleaning up FGA
Please read until the end before running these commands:

Check if the DBMS_AUDIT_MGMT.INIT_CLEANUP has already been run on the database:

SET serveroutput ON SIZE 999999
BEGIN
  IF DBMS_AUDIT_MGMT.IS_CLEANUP_INITIALIZED(DBMS_AUDIT_MGMT.AUDIT_TRAIL_DB_STD)
  THEN
    DBMS_output.put_line('True');
  ELSE
    DBMS_output.put_line('False');
  END IF;
END;
/

If the response is “FALSE”, run the DBMS_AUDIT_MGMT.INIT_CLEANUP as follows:

In this example, the cleanup is being set to run every 1 hour

BEGIN
  DBMS_AUDIT_MGMT.init_cleanup(
    audit_trail_type         => DBMS_AUDIT_MGMT.AUDIT_TRAIL_ALL,
    default_cleanup_interval => 1 /* hours */);
END;
/

If you get an error such as:

Error starting at line : 7 in command -
BEGIN
  DBMS_AUDIT_MGMT.init_cleanup(
    audit_trail_type         => DBMS_AUDIT_MGMT.AUDIT_TRAIL_ALL,
    default_cleanup_interval => 24 /* hours */);
END;
Error report -
ORA-46263: The audit trail is already initialized for cleanup
ORA-06512: at "SYS.DBMS_AUDIT_MGMT", line 61
ORA-06512: at "SYS.DBMS_AUDIT_MGMT", line 757
ORA-06512: at line 2

Uninitialize the cleanup with:

BEGIN
 DBMS_AUDIT_MGMT.DEINIT_CLEANUP(
  AUDIT_TRAIL_TYPE  => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD);
END;
/

Followed by the above command to initialize the cleanup.

I spent some time looking into DBMS_SCHEDULER, DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP and DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD etc. but it seemed overly complicated for what the goal was. The same can be accomplished with a cron script:

DELETE 
FROM   SYS.fga_log$ 
WHERE  NTIMESTAMP# < to_date('21-SEP-20 18.42.00','dd-MON-yy hh24:mi:ss')
;

Remember to gather statistics and/or add indices to improve performance

exec dbms_stats.gather_table_stats (ownname=>'SYS', tabname=>'AUD$' , estimate_percent=>10, cascade=>TRUE, degree=>5);

exec dbms_stats.gather_table_stats (ownname=>'SYS', tabname=>'FGA_LOG$' , estimate_percent=>10, cascade=>TRUE, degree=>5);

Storing the result from a v$ table into a variable in a script

Typically while selecting from a V$ table in a Unix script, I have to escape the “$” character as shown below:

sqlplus -s $sqlplus_command << EOF > backup_check_temp.txt
  select  value 
  from    V\$RMAN_CONFIGURATION 
  where   NAME = 'RETENTION POLICY'
  ;  
  exit;
EOF

The above will successfully run the SQL and write the output in the spool file.

However, if instead of the output in a spool file, you want the output in a Unix variable for further processing, you would change the code to

export ret_period=`sqlplus -s $sqlplus_command << EOF
                   select   value 
                   from     V\$RMAN_CONFIGURATION 
                   where    NAME = 'RETENTION POLICY'
                   ;
EOF`
if  [ -z "$ret_period" ]; then
    export ret_period='Not found'
fi    

This will fail with:

+ export 'ret_period=                select value from V where NAME = '\''RETENTION POLICY'\''
                                       *
ERROR at line 1:
ORA-00942: table or view does not exist'

The reason is that the second method of coding results in two levels of interpretation, each of which requires the “$” character to be escaped. To get the code working, it has to be coded as follows (note the two “\\” escape characters):

export ret_period=`sqlplus -s $sqlplus_command << EOF
                   select   value 
                   from     V\\$RMAN_CONFIGURATION 
                   where    NAME = 'RETENTION POLICY'
                   ;
EOF`
if  [ -z "$ret_period" ]; then
    export ret_period='Not found'
fi