Dropping a RAC database without DBCA

One way to drop a RAC database without using DBCA.

This happened to be a smaller non-production database and I had the luxury of taking an export before I dropped the database. All the users had already moved to the new database.

Create a data pump directory (if you do not already have one).

CREATE or replace DIRECTORY dean_data_pump AS '/backup/UATDB/DATAPUMP';

Perform an export with the command:

cd /backup/UATDB/DATAPUMP
nohup expdp \'/ as sysdba \' directory=DEAN_DATA_PUMP dumpfile=UATDB_full_2017_09_18.dmp logfile=UATDB_full_2017_09_18.log full=yes &

Monitor the nohup.out file for errors.

Check the current status of the database. This is a 2 node RAC and is currently running on both nodes

[uatsrvr2-UATDB] srvctl status database -d UATDB
Instance UATDB1 is running on node uatsrvr1
Instance UATDB2 is running on node uatsrvr2

Stop all nodes of the RAC

srvctl stop database -d UATDB

Repeat the status check

[uatsrvr2-UATDB] srvctl status database -d UATDB
Instance UATDB1 is not running on node uatsrvr1
Instance UATDB2 is not running on node uatsrvr2

Start SQL*Plus to change the cluster_database parameter

[uatsrvr2-UATDB2] sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Sep 18 10:35:26 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 4810256384 bytes
Fixed Size                  226136 bytes
Variable Size            13421810 bytes
Database Buffers         3456106 bytes
Redo Buffers                97070 bytes
Database mounted.
SQL> alter system set cluster_database=FALSE scope=spfile sid='*';

System altered.

SQL> shutdown immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.

Bring up the database in exclusive mode with restricted session

SQL> startup mount exclusive restrict;

ORACLE instance started.

Total System Global Area 4810256384 bytes
Fixed Size                  2261368 bytes
Variable Size            1342181000 bytes
Database Buffers         3456106496 bytes
Redo Buffers                9707520 bytes
Database mounted.

Check if the database is in restricted session and cluster is off and then exit from SQL*Plus

SQL> select logins,parallel from v$instance;

LOGINS     PAR
---------- ---
RESTRICTED NO

SQL> exit

Connect to the database via RMAN

[uatsrvr2-UATDB2] rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Mon Sep 18 10:43:07 2017

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: UATDB2 (DBID=407684853, not open)

Drop the database, including all the backups

RMAN> DROP DATABASE INCLUDING BACKUPS;

database name is "UATDB" and DBID is 407684853

Do you really want to drop all backups and the database (enter YES or NO)? yes

using target database control file instead of recovery catalog
allocated channel: ORA_1
channel ORA_DISK_1: SID=2 device type=DISK

List of Backup Pieces
BP Key  BS Key  Pc# Cp# Status      Device Type Piece Name
------- ------- --- --- ----------- ----------- ----------
3535    3535    1   1   AVAILABLE   DISK        /backup01/UATDB/backups/UATDB_db_20170911_eise5da4_1_1.bus
.
.
Many more lines of output
.
.
database name is "UATDB" and DBID is 407684853
database dropped

RMAN> 

Remove the configuration from SRVCTL

[uatsrvr2-UATDB] srvctl remove database -d UATDB
Remove the database UATDB? (y/[n]) y
[uatsrvr2-UATDB] 

The following steps will have to be performed on all nodes of the RAC. Edit the /etc/oratab and remove the entry for the database.

If auditing is turned on, delete the audit files from the audit destination.

Delete any CRON entries, trace files, password files, etc.

Installing the Oracle Client 12c on an AWS Linux EC2 – Silent Mode

After the EC2 is built, run the below commands as root. Depending on the network security, you may have to either set or unset proxy variables.

/usr/bin/yum install -y binutils.x86_64 compat-libcap1.x86_64 compat-libstdc++-33.i686 \
compat-libstdc++-33.x86_64 gcc.x86_64 gcc-c++.x86_64 glibc-devel.i686 glibc-devel.x86_64 glibc.i686 \
glibc.x86_64 libXtst.x86_64 libXtst.i686 ksh.x86_64 libaio.x86_64 libaio.i686 libaio-devel.i686 \
libaio-devel.x86_64 libgcc.i686 libgcc.x86_64 libstdc++.x86_64 libstdc++.i686 libstdc++-devel.i686 \
libstdc++-devel.x86_64 libXi.i686 libXi.x86_64 make.x86_64 sysstat.x86_6

The above command will install all of the prerequisites required for the Oracle Client 12c .

Create the install group

/usr/sbin/groupadd oinstall

Create the dba group

/usr/sbin/groupadd dba

Create the oracle user and add to the groups

/usr/sbin/useradd -g oinstall -G dba oracle

Set the oracle password

passwd oracle

Respond to the above prompt with the oracle password that you want to use.

Make a directory for the software

mkdir /apps/opt/oracle

Change ownership of the directory to the oracle user and the oinstall group

chown oracle:oinstall /apps/opt/oracle

Change to oracle user

su - oracle

Copy the software into the above directory via ftp, sftp, scp etc. The directory will contain file

linuxamd64_12102_client.zip

Unzip the foftware

unzip linuxamd64_12102_client.zip

Change directory to the client directory that was unzipped.

cd /apps/opt/oracle/client

Run the installer in silent mode

./runInstaller -silent \
 -responseFile /apps/opt/oracle/client/response/client_install.rsp    \
   oracle.install.client.installType=Runtime           \
   UNIX_GROUP_NAME=dba                                 \
   INVENTORY_LOCATION=/apps/opt/oracle/oraInventory            \
   SELECTED_LANGUAGES=en                               \
   ORACLE_HOME=/apps/opt/oracle/base/rtclient64            \
   ORACLE_BASE=/apps/opt/oracle/base                           \
   waitForCompletion=true

At the end of the installation, you should see the message

The installation of Oracle Client 12c was successful.
Please check '/apps/opt/oracle/oraInventory/logs/silentInstall2017-08-30_06-00-25PM.log' for more details

Exit from oracl back to root

exit

As a root user, execute the following script

/apps/opt/oracle/oraInventory/orainstRoot.sh

The above script will display messages similar to

Changing permissions of /apps/opt/oracle/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /apps/opt/oracle/oraInventory to dba.
The execution of the script is complete.

Set up environment variables for the oracle user. Go to the oracle user’s home

su - oracle
cd

Edit the oracle users

vi .bash_profile

Setup the below variable to make it easier for the users to access sqlplus etc.

# Oracle client variables
export ORACLE_HOME=/apps/opt/oracle/base/rtclient64
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
PATH=$ORACLE_HOME/bin:$PATH
which sqlplus

Test sqlplus

sqlplus user_id

SQL*Plus: Release 12.1.0.2.0 Production on Wed Aug 30 18:23:27 2017

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Enter password: 

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,
OLAP, Data Mining and Real Application Testing options

Moving a database to a new server while attempting to use the old remote catalog

As part of a hardware upgrade, I moved a database to a new RAC. I was attempting to use my backup scripts from the old RAC. The scripts contained connections to a remote catalog that I wanted to reuse. On the new RAC, I attempted to register and resync the catalog and got the below error messages:

[new_server-DBNAME2] rman catalog=rman_cat/rman_password@remote_catalog target=/ 

Recovery Manager: Release 11.2.0.4.0 - Production on Fri Sep 8 10:47:33 2017

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: DBNAME (DBID=599767446)
connected to recovery catalog database

RMAN> register database;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of register command on default channel at 09/08/2017 10:47:44
RMAN-20002: target database already registered in recovery catalog

RMAN> RESYNC CATALOG;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of resync command on default channel at 09/08/2017 10:47:51
RMAN-20020: database incarnation not set

To resolve this issue, I executed the commands:

RMAN> unregister database;

database name is "DBNAME" and DBID is 599767446

Do you really want to unregister the database (enter YES or NO)? yes
database unregistered from the recovery catalog

After the database was unregistered, I was able to register the database:

RMAN> register database;

database registered in recovery catalog
starting full resync of recovery catalog
full resync complete

RMAN> RESYNC CATALOG;

starting full resync of recovery catalog
full resync complete

The resync catalog command may be redundant, but it never hurts to make sure.

AWS – EC2 with Oracle for a I/O intensive application – Part 02

This week one of our applications is attempting to benchmark the performance of an Oracle database running on an Exadata machine versus running in the AWS cloud on an x1.16xlarge EC2. Our initial thought was to define the storage as two ASM disk groups, each consisting of four 400 gb disks at 6,000 IOPS each per ASM disk group. Assuming that a given data file was spread across all four disks in a disk group, we could have speeds approaching 20,000 IOPS.

During testing, Amazon recommended that the ASM data be placed on single disks running at 20,000 IOPS. The current maximum IOPS that can be provisioned is 20,000 IOPS. Additionally, AWS requires that the IOPS:gb ratio be maintained at a maximum of 50:1. In order to meet these requirements, we decided to create two mount points, each of 1000 gb at 20,000 IOPS. An additional 1000 gb at 20,000 IOPS was also added for the archive logs.

We created a json template to spin up the three disks and mounted them to the existing EC2. Before mounting, make sure that the mount points are not already being used by currently attached disks. After the disks are mounted, we attached them to ASM and waited for the rebalance to complete. After the rebalance was completed, we dropped the old disks that were provisioned at 6000 IOPS. The application saw a 67% decrease in elapsed time on some of the queries.