ORA-20005: object statistics are locked in AWS/RDS environment

While testing the AWS RDS environment, I ran across the issue that the statistics were locked:

begin 
     DBMS_STATS.GATHER_TABLE_STATS 
       (
          ownname => '"DEAN"',
          tabname => '"table name"',
          estimate_percent => 5
       );
end;

The error was:

Error starting at line : 6 in command -
begin 
     DBMS_STATS.GATHER_TABLE_STATS 
       (
          ownname => '"DEAN"',
          tabname => '"table name"',
          estimate_percent => 5
       );
end;
Error report -
ORA-20005: object statistics are locked (stattype = ALL)
ORA-06512: at "SYS.DBMS_STATS", line 34707
ORA-06512: at line 2

I checked with

select table_name, stattype_locked from dba_tab_statistics where table_name in ( select table_name from dba_tables where owner = 'DEAN');  

The STATTYPE_LOCKED column contained the value of “ALL”. I was able to unlock this with

exec dbms_stats.unlock_table_stats('DEAN', 'table name');

This changed the value in the STATTYPE_LOCKED to “null” after which I was able to gather statistics.

The Fred Factor by Mark Sanborn

The Fred Factor by Mark Sanborn is based on the author’s observations of the work habits of his very memorable postal carrier, Fred Shea. The author was intrigued by the quality of customer service that Fred extended to all the customers on his route despite the constraints of his role as a member of the US Postal Service. Mr. Sanborn’s observations led him to write this book as a guide for both employees and organizations to improve their level of involvement and accomplishment in every aspect of their work.
The Fred Factor

An overview of DevOps from The Phoenix Project

The below Power Point is my review of the “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win” by Gene Kim, Kevi Behr and George Spafford. The book is a fictional account as experienced by Bill Palmer. The Company, Parts Unlimited, is facing challenges, both external and internal. On a Tuesday morning, Bill is called into the CEOs office and informed that the executive hierarchy of the IT department have decided to “pursue other opportunities” and that he is the new VP of IT.
Phoenix Project DevOps

Cleaning up the sys.aud$ table

I inherited a non-production database that had been auditing to the DB (audit_trail=DB) without any cleanup process in place. The sys.aud$ table had over 8 million rows of data. Before I set up the cleanup jobs, I decided to remove all the data in the table. I issued a truncate:

Error starting at line : 16 in command -
truncate table sys.aud$
Error report -
SQL Error: ORA-00942: table or view does not exist
00942. 00000 -  "table or view does not exist"
*Cause:    
*Action:

A select on dba_objects indicated that sys.aud$ was a synonym

select owner, object_name, object_type from dba_objects where object_name = 'AUD$';
 
OWNER    OBJECT_NAME   OBJECT_TYPE
-------- -----------   -----------
SYS      AUD$          SYNONYM
SYSTEM   AUD$          TABLE

The table is SYSTEM.AUD$, but I thought I would check anyway:

select table_owner, table_name from dba_synonyms where owner = 'SYS' and synonym_name = 'AUD$';
 
TABLE_OWNER                    TABLE_NAME
------------------------------ ------------------------------
SYSTEM                         AUD$

Truncated the base table:

truncate table system.aud$;

Oracle 11.2.0.4 and obfuscated (identified by values) passwords

In versions of Oracle older than 11.2.0.4, I have always migrated DB links with dbms_metadata.get_ddl to get the ddl for a db link as I do not have the passwords for the user at the remote site. This provides the ddl as shown below with an obfuscated password:

CREATE DATABASE LINK "db_link_name" 
CONNECT TO "remote_user" 
IDENTIFIED BY VALUES '06AD92A1C4E671B083FE5A long sting here' 
USING '(DESCRIPTION=
(ADDRESS=                                                            
(PROTOCOL=TCP)(HOST=remote.server.com)(PORT=remote port))
    (CONNECT_DATA=                      
      (SERVER=dedicated)
      (SERVICE_NAME=remote service name)
    )                                       
)'                                                                              
;                                                                                

Oracle 11.2.0.4 has a security enhancements to prevent the use of obfuscated passwords (See Doc ID 1905221.1). So you have 2 options:
a. Know the password and provide it explicitly
b. Use expdp/impdp to copy it over

Very often, option a. is not available as we do not know that password of the user at the remote site.

So I tried option b which works and creates the ddl as:

CREATE DATABASE LINK "db_link_name" 
CONNECT TO "remote_user" 
IDENTIFIED BY VALUES ':1'
USING '(DESCRIPTION=
(ADDRESS=                                                            
(PROTOCOL=TCP)(HOST=remote.server.com)(PORT=remote port))
    (CONNECT_DATA=                      
      (SERVER=dedicated)
      (SERVICE_NAME=remote service name)
    )                                       
)'                                                                              
;                                                                                

This can be directly imported via impdp.

One way to identify who modified a file

I was attempting to identify who had modified one of my script files. I used the stat command to identify when the modifications occurred:

[prodsrvr-PRODDB01] stat check_status.ksh
  File: `check_status.ksh'
  Size: 1117            Blocks: 8          IO Block: 4096   regular file
Device: 17c0fc01h/398523393d    Inode: 473         Links: 1
Access: (0644/-rw-r--r--)  Uid: (24718/  oracle)   Gid: (24718/     dba)
Access: 2017-01-11 19:03:35.972217538 -0500
Modify: 2016-12-19 15:10:07.556675538 -0500
Change: 2016-12-19 15:10:07.000000000 -0500

The above indicates that the modification occurred on December 12, 2016 at 15:10.

I then issued the last command and grepped for Dec to see who was logged on at that date/time:

[prodsrvr-PRODDB01] last | grep -i Dec
user01   pts/1        144.28.20.203    Tue Dec 20 19:18 - 19:33  (00:15)    
user02   pts/0        10.69.96.57      Tue Dec 20 19:10 - 22:22 (1+03:11)   
user03   pts/0        139.49.5.29      Mon Dec 19 15:06 - 21:41  (06:34)    
user05   pts/0        144.28.21.98     Mon Dec 19 11:54 - 13:20  (01:26)    

This indicates a very strong possibility that user03 performed the modification. While not 100% accurate it is a reasonable way to narrow the list of users who may have made the modification.

One approach to identifying the cause of ORA-01017

If you have a user account that is repeatedly getting locked out as a result of exceeding the number of failed password attempts (ORA-01017) and you are unable to identify who or what is causing it, this approach may be helpful:

Create a table as follows to hold the failed login information:

create table dean_logon_errors
(
login_date date,
os_user varchar(15),
ip_address varchar(50),
AUTHENTICATED_IDENTITY varchar(50),
AUTHENTICATION_METHOD varchar(50),
CLIENT_IDENTIFIER varchar(50), 
CLIENT_INFO varchar(50),
terminal varchar(50)
)
TABLESPACE name_of_table_space
;

Create a trigger that will be fired when an ORA-01017 error occurs:

CREATE OR REPLACE TRIGGER dean_logon_failures
AFTER SERVERERROR
ON DATABASE
BEGIN
  IF (IS_SERVERERROR(1017)) THEN
    INSERT INTO dean_logon_errors values
    ( sysdate
     ,sys_context('USERENV','OS_USER')
     ,SYS_CONTEXT('USERENV','IP_ADDRESS')
     ,SYS_CONTEXT('USERENV','AUTHENTICATED_IDENTITY')
     ,SYS_CONTEXT('USERENV','AUTHENTICATION_METHOD')
     ,SYS_CONTEXT('USERENV','CLIENT_IDENTIFIER')
     ,SYS_CONTEXT('USERENV','CLIENT_INFO')
     ,SYS_CONTEXT('USERENV','TERMINAL')
);
  END IF;
END logon_failures;
/

Wait for some time to allow the trigger to collect information and then execute the below SQLs:

alter session set nls_date_format = 'dd-MON-yy hh24.mi.ss';
select * from dean_logon_errors order by login_date desc;

Output will look like:

LOGIN_DATE         OS_USER         IP_ADDRESS      AUTHENTICATE AUTHENTICATION_ CLIENT_IDENTIFI CLIENT_INFO     TERMINAL
------------------ --------------- --------------- ------------ --------------- --------------- --------------- ---------------
13-JAN-17 20.50.00 john_doe_os     110.170.248.345 user_acct    PASSWORD
13-JAN-17 20.50.00 john_doe_os     110.170.248.345 user_acct    PASSWORD

Checking the use of undo space in the past

We had a circumstance in which we had to delete 4 billion rows from a 5 billion row table. This was done programmatically as we could not take an outage on the system. After the deletes were complete we were attempting to reclaim space on the index on a table, we were attempting to reclaim the space via an online rebuild of the index when we ran out of undo space: Continue reading “Checking the use of undo space in the past”