Out-of-Place Patching - Switch ORACLE_HOME

Important
Out-of-Place Patching is supported for Single Instance primary databases without and with Oracle Restart.
The switch of RAC and RAC One Node databases is not supported.
The Grid Infrastructure home of Oracle Restart and of Grid Infrastructure (cluster) 26ai can be switched as well, see Switch of the Grid Infrastructure home (Oracle Restart and cluster).
Terms (Oracle Restart, Grid Infrastructure, RAC): see Featurelist.
The guide describes the concept, all dependencies and the gaps in the current implementation.

Out-of-Place Patching switches a database from its current ORACLE_HOME to another, already installed and patched ORACLE_HOME. The installation and patching of the new ORACLE_HOME is not part of Out-of-Place Patching. It is done before with other roles (oraswdb_install, oraswdb_manage_patches or from a Golden Image).

Advantages compared to In-Place Patching with opatch apply:

  • The new ORACLE_HOME is installed and patched before the switch without downtime, because no database is running from it.

  • The downtime is reduced to a restart of the database and datapatch.

  • Fallback is easy: the old ORACLE_HOME is still available.

Note
A Golden Image is recommended, but not required.
The new ORACLE_HOME can also be installed and patched with ansible-oracle on the database server. The switch of the database is independent from the way the new ORACLE_HOME was created. See Preparation of the new ORACLE_HOME.

1. Requirements

  • Oracle Database 19c or newer

  • Read-Only Oracle Homes (readonly_home: true) are recommended, but not required

    • Configuration files like SPFile and password file are not part of the ORACLE_HOME and are shared between all Read-Only `ORACLE_HOME`s.

    • The switch copies SPFile, PFile and password file, when the location differs between old and new ORACLE_HOME (e.g. switch from a Read-Write into a Read-Only ORACLE_HOME).

    • readonly_home must not be changed for an installed ORACLE_HOME.

  • Each Read-Only ORACLE_HOME needs 2 attributes in db_homes_config:

    db_homes_config:
      db1932-si-ee:
        readonly_home: true
        oracle_home_name: db1932siee
  • oracle_home_name is mandatory:

    • It cannot be derived from the key in db_homes_config. Oracle allows only characters, numbers and _ in the name of an ORACLE_HOME in the Oracle Inventory. YAML keys are much less restrictive (e.g. db1932-si-ee is a valid key, but not a valid oracle_home_name).

    • It must be unique over all ORACLE_HOME`s on a server. It defines the `ORACLE_BASE_HOME ($ORACLE_BASE/homes/<oracle_home_name>), which must not be shared between the old and the new ORACLE_HOME.

  • Enough disk space for at least 2 `ORACLE_HOME`s at the same time

2. Read-Only Oracle Home

A Read-Only Oracle Home separates the software from the configuration. The locations are reported by orabasehome and orabaseconfig.

Table 1. Locations in a Read-Only Oracle Home
Location Path Shared between ORACLE_HOMEs Content

ORACLE_HOME

e.g. /u01/app/oracle/product/19/db19_2610

no

Software only

ORACLE_BASE_CONFIG

$ORACLE_BASE/dbs

yes

SPFile, password file, hc_*.dat, lk*

ORACLE_BASE_HOME

$ORACLE_BASE/homes/<oracle_home_name>

no

network/admin (listener.ora, sqlnet.ora, tnsnames.ora), log, rdbms/log

Important
ORACLE_BASE_CONFIG is shared, but ORACLE_BASE_HOME is not.
All files in $ORACLE_BASE/homes/<oracle_home_name>/network/admin must exist in the new ORACLE_HOME before the switch.

3. Process overview

  1. Prepare the new ORACLE_HOME (no downtime)

    1. Install the new ORACLE_HOME with oraswdb_install / oraswdb_manage_patches (not part of Out-of-Place Patching), one of:

      1. from a Golden Image (recommended)

      2. from the base release, followed by opatch apply for all patches

    2. Create the network configuration in the new ORACLE_BASE_HOME

  2. Switch the database (downtime) - Out-of-Place Patching with oradb_manage_db

    1. Stop the database (and listener) from the old ORACLE_HOME

    2. Change the ORACLE_HOME in /etc/oratab or in Oracle Clusterware (srvctl)

    3. Start the listener and database from the new ORACLE_HOME

    4. Execute datapatch

  3. Post tasks

    1. APEX, time zone, statistics, etc. (optional)

    2. Remove the old ORACLE_HOME after a successful test period

4. Preparation of the new ORACLE_HOME

The new ORACLE_HOME can be prepared in 2 ways. Both are supported by ansible-oracle and both lead to the same result for the switch.

Table 2. Ways to prepare the new ORACLE_HOME
Topic Golden Image Install and patch on the server

Source

Golden Image (imagename or filesource in db_homes_config)

Base release (e.g. 19.3) + patches from oracle_sw_patches

Patching

Included in the Golden Image. Additional patches are possible like for the base release.

oraswdb_manage_patches with opatch/opatchauto from db_homes_config

Duration on the database server

Short (unzip + runInstaller)

Long (opatch apply for each patch)

Identical `ORACLE_HOME`s on all servers

Guaranteed by the Golden Image

Depends on identical patch lists in the inventory

Software needed on the server

Golden Image only

Base release, OPatch and all patches

Relink options (e.g. Unified Auditing, dNFS)

Included in the Golden Image, checked by oraswdb_install

Set by oraswdb_install

Downtime

None

None - no database is running from the new ORACLE_HOME, stop_processes is not needed

Recommendation: download the Golden Image created by Oracle with the role autoupgrade (gold_image in autoupgrade_download_patches or autoupgrade_download_gold_image). This is the way with the least effort, because no ORACLE_HOME has to be installed and patched for creating the Golden Image.

Alternative: create the Golden Image once from an ORACLE_HOME installed and patched by ansible-oracle (role oraswdb_golden_image), e.g. when one-off patches are needed, which are not part of the Golden Image from Oracle.

Use the Golden Image for all servers.

Important
The implementation of the switch must not depend on the way the new ORACLE_HOME was prepared.
The only requirement for the switch is an installed and fully patched ORACLE_HOME.

5. Dependencies

5.1. Registration of the ORACLE_HOME

Table 3. Registration of the ORACLE_HOME for a database
Environment Registration Switch

Single Instance without Oracle Restart

/etc/oratab

Replace the line <SID>:<old ORACLE_HOME>:<Y/N> with the new ORACLE_HOME.

Oracle Restart

Oracle Clusterware (OCR/OLR)

srvctl modify database -db <db_unique_name> -oraclehome <new ORACLE_HOME>
/etc/oratab is not maintained correctly by Oracle Clusterware. ansible-oracle writes the new ORACLE_HOME into /etc/oratab (flag N), but never reads it for the state of the database.

Grid Infrastructure (cluster): RAC, RAC One Node

Oracle Clusterware (OCR)

Rolling switch node by node. Not supported.

The following parts of ansible-oracle read the ORACLE_HOME from /etc/oratab and follow the switch automatically:

  • systemd service / manage_oracle_rdbms_procs.sh (start/stop of databases, without Oracle Restart only)

  • ocenv / .profile_<ORACLE_SID>

  • rman_backup.sh from role oradb_rman (with Oracle Restart from the database resource, crsctl)

5.2. Stop and Start

  • The database must be stopped from the old ORACLE_HOME.

  • With Oracle Restart the stop must be done with srvctl from the old ORACLE_HOME, before srvctl modify database is executed.

  • The start is done from the new ORACLE_HOME.

5.3. Listener

  • A listener running from the database ORACLE_HOME must be restarted from the new ORACLE_HOME.

  • All databases served by this listener must be switched together. Recommendation: run the listener from the Grid Infrastructure home of Oracle Restart or from a dedicated ORACLE_HOME.

  • Static registrations in SID_LIST_<LISTENER> contain the ORACLE_HOME:

    • Data Guard Broker (<db_unique_name>_DGMGRL)

    • extproc

    • Static service entries

5.4. Network configuration and wallets

  • sqlnet.ora, tnsnames.ora and listener.ora must exist in $ORACLE_BASE/homes/<new oracle_home_name>/network/admin. Alternative: a central TNS_ADMIN.

  • WALLET_LOCATION or ENCRYPTION_WALLET_LOCATION in sqlnet.ora must not point into the old ORACLE_HOME.

  • TDE with the init parameter WALLET_ROOT is independent from the ORACLE_HOME.

  • local_listener must not be an alias from tnsnames.ora of the old ORACLE_HOME. DBCA sets local_listener to LISTENER_<DB_NAME> and writes this alias into tnsnames.ora of the ORACLE_HOME. The alias is missing in the new ORACLE_HOME, the instance does not register at the listener (ORA-12514). orasw_meta requires local_listener with EZConnect in init_parameters of Single Instance databases, e.g. //localhost:1521 (see breaking_changes.adoc). Other entries from DBCA in tnsnames.ora of the old ORACLE_HOME are not copied.

5.5. datapatch

  • datapatch is mandatory after the start from the new ORACLE_HOME.

  • datapatch must be executed for the CDB and all PDBs.

  • OJVM patches may need a startup in upgrade mode, depending on the release.

  • Data Guard: datapatch is executed on the primary database only.

5.6. Directory objects

Directory objects can point into the old ORACLE_HOME. The OPatch directories are changed by datapatch, own directories are not.

Find directory objects in the old ORACLE_HOME
select directory_name, directory_path
from dba_directories
where directory_path like '/u01/app/oracle/product/19/db19_2601%';

5.7. Content of the ORACLE_HOME

The new ORACLE_HOME must contain all customizations of the old ORACLE_HOME:

  • Relink options (e.g. Unified Auditing, dNFS) - included in a Golden Image or set by oraswdb_install

  • One-off patches - included in a Golden Image or defined in db_homes_config[].opatch for the new ORACLE_HOME

  • oranfstab

  • Custom DBCA templates

5.8. Data Guard

  • The standby databases are switched first.

  • The primary database is switched afterwards, followed by datapatch.

  • Static listener registrations for the Data Guard Broker must be changed (see Listener).

5.8.1. Physical Standby

Important
Not implemented yet.
The current implementation opens every database after the switch and executes datapatch.

A physical standby database must be handled differently from a primary database:

  • The standby database is started in MOUNT only. Opening the database while Redo Apply is running uses Active Data Guard, which needs an extra license. This is valid even when the standby database was opened READ ONLY WITH APPLY before the switch.

  • datapatch must not be executed on a standby database. The SQL changes are applied with Redo Apply from the primary database.

The role of the database is read after startup mount from the controlfile:

startup mount
select database_role from v$database;
Table 4. Actions after startup mount depending on DATABASE_ROLE
DATABASE_ROLE Open database datapatch

PRIMARY

yes (+ all PDBs)

yes

PHYSICAL STANDBY

no, remains in MOUNT

no

Redo Apply after the switch:

  • With Data Guard Broker (dg_broker_start=TRUE) the broker restarts Redo Apply automatically.

  • Without Data Guard Broker Redo Apply must be restarted manually. ansible-oracle does not start Redo Apply, because it does not manage Data Guard configurations.

    alter database recover managed standby database disconnect;

5.9. External dependencies

  • Enterprise Manager and other monitoring targets

  • Scripts and crontabs with a hard coded ORACLE_HOME

  • Applications with a local ORACLE_HOME (e.g. Bequeath connections)

6. Implementation in ansible-oracle

6.1. Existing features

Out-of-Place Patching itself is the switch of the ORACLE_HOME in oradb_manage_db. The other features prepare the new ORACLE_HOME and are used before the switch.

Table 5. Existing features used for Out-of-Place Patching
Feature Role Comment

Read-Only Oracle Home

oraswdb_install

readonly_home: true and oracle_home_name in db_homes_config

Install from base release

oraswdb_install

Without imagename in db_homes_config

Patch the new ORACLE_HOME

oraswdb_manage_patches

opatch/opatchauto in db_homes_config, apply_patches: true in db_homes_installed. No stop_processes needed, because no database is running from the new ORACLE_HOME

Download Golden Image from Oracle (recommended)

autoupgrade (download_patches.yml)

gold_image in autoupgrade_download_patches or autoupgrade_download_gold_image

Create Golden Image (optional)

oraswdb_golden_image

Install from Golden Image (optional)

oraswdb_install

imagename or filesource in db_homes_config

Relink options

oraswdb_install

Enforced for every ORACLE_HOME, independent from the source

Network configuration

oradb_manage_db

Files are created in ORACLE_BASE_HOME of the ORACLE_HOME of the listener (listener_installed[].home)

datapatch

oradb_datapatch

Remove ORACLE_HOME

oraswdb_install

state: absent in db_homes_installed

Switch ORACLE_HOME of database

oradb_manage_db (switch_home.yml)

Change of oracle_databases[].home. Single Instance without and with Oracle Restart, not Grid Infrastructure (cluster). Stop from old ORACLE_HOME, copy of SPFile/PFile/password file (when the location differs), change of /etc/oratab, start from new ORACLE_HOME, datapatch. Oracle Restart: stop/start with srvctl, srvctl modify database -oraclehome (and -spfile/-pwfile, when copied), start with the start option of the resource, datapatch for the role PRIMARY only. Flag file $ORACLE_BASE/switch_home_<SID>.pending for an incomplete switch.

Switch ORACLE_HOME of listener

oradb_manage_db (loop_listener_details.yml)

Change of listener_installed[].home. Stop from old ORACLE_HOME, start from new ORACLE_HOME.

/etc/oratab without duplicate entries

oradb_manage_db (manage-db.yml)

lineinfile with regexp replaces the entry for the SID. With Oracle Restart the entry <db_unique_name>:<ORACLE_HOME>:N is written as well.

ORACLE_HOME without database

orasw_meta, oraswdb_install

An ORACLE_HOME in db_homes_installed without a database in oracle_databases is valid (e.g. new ORACLE_HOME before the switch, dedicated listener home).

ORACLE_HOME of listener, tnsnames and sqlnet must be installed

orasw_meta (assert_sqlnet.yml)

home in listener_installed, sqlnet_installed and tnsnames_installed must be in db_homes_installed without state: absent (unless state: absent or use_gi_home: true for the entry).

local_listener with EZConnect

orasw_meta (assert_oracle_databases.yml)

local_listener is mandatory in init_parameters of Single Instance databases with state: present.

Switch Grid Infrastructure home of Oracle Restart

oraswgi_install (switch_home.yml)

Change of oracle_home_gi. Oracle Restart 26ai only. See Switch of the Grid Infrastructure home (Oracle Restart and cluster).

6.2. Gaps

Table 6. Missing features for Out-of-Place Patching
Gap Location Description

Incomplete check of oracle_home_name

roles/oraswdb_install/tasks/roohctl.yml

Only the existence of oracle_home_name is checked for readonly_home: true, and only during the installation. Missing: check for allowed characters, unique value over all ORACLE_HOME`s in `db_homes_installed, check in orasw_meta before any change on the server.

6.3. ORACLE_HOME without database

An ORACLE_HOME in db_homes_installed without a database in oracle_databases is a valid configuration:

  • The new ORACLE_HOME has no database before the switch.

  • The old ORACLE_HOME has no database after the switch, until it is removed.

  • Listeners can run from a dedicated ORACLE_HOME without database.

The oracle_db_type must still be identical for all databases in the same ORACLE_HOME. All roles and asserts must support an ORACLE_HOME without database.

7. Switch of the Grid Infrastructure home (Oracle Restart and cluster)

The Grid Infrastructure home of Oracle Restart and of Grid Infrastructure (cluster) is switched by oraswgi_install, when oracle_home_gi differs from crs_home in /etc/oracle/olr.loc. The new home is created from oracle_install_image_gi, normally a gold image of the new Release Update.

Important
Supported for Oracle Restart 26ai and Grid Infrastructure (cluster) 26ai inside the same release (e.g. 23.26.3 ⇒ 23.26.4).
Not supported for 19c: -switchGridHome is not supported by Oracle for Oracle Restart 19c (INS-45101, MOS 2764906.1), 19c Grid Infrastructure (cluster) is not implemented.
Grid Infrastructure (cluster): tested with 2 nodes and a RAC database (molecule scenario rac, side_effect.yml).
Table 7. Steps of oraswgi_install (switch_home.yml)
Step User Description

Extract oracle_install_image_gi into oracle_home_gi

grid

Without downtime.
Grid Infrastructure (cluster): on every node.

gridSetup.sh -silent -setupHomeAs <old home>

grid

Software only installation with Oracle base and groups of the old home. Registers the new home in the central inventory.
Without it, -switchGridHome fails with INS-45127. Without downtime.
Grid Infrastructure (cluster): on every node, one after the other. -setupHomeAs works on the local node only, -switchGridHome needs the new home on all nodes (INS-42016 otherwise).

gridSetup.sh -silent -switchGridHome

grid

Prepares the switch. Without downtime.
Grid Infrastructure (cluster): on the primary node (1st host of orasw_meta_cluster_hostgroup).

root.sh of the new home

root

Oracle Restart: roothas.pl -prepatch/-postpatch -dstcrshome <new home>: stops the stack in the old home and starts it in the new home.
Downtime of ASM, the listeners in the Grid Infrastructure home and all databases managed by Oracle Restart.
The databases are started with their start option. oracle-ohasd.service and the +ASM entry in /etc/oratab point to the new home.
Grid Infrastructure (cluster): rootcrs.pl -prepatch/-postpatch -dstcrshome <new home>, rolling: primary node first, then the other nodes one after the other (throttle: 1). Only the instances of the node in progress are stopped, the RAC databases stay available. rootcrs.pl -prepatch runs cluvfy stage -pre patch: /etc/oraInst.loc must be owned by root (PRVG-2031/PRVG-2032), oraswgi_install sets the owner before root.sh.

runInstaller -updateNodeList

grid

CRS=TRUE for the new home, CRS=FALSE for the old home in the central inventory (not changed by root.sh).
Grid Infrastructure (cluster): in the central inventory of every node (-local).

gi_ignoreprereq: true adds -ignorePrereqFailure to gridSetup.sh and -ignoreSysPrereqs to runInstaller.

Listeners running from the Grid Infrastructure home (e.g. LISTENER) are moved with the stack, listener.ora is copied into the new home. Listeners and databases in database homes are not changed.

The old Grid Infrastructure home is not removed by the switch.

Oracle Restart: a switch back to the old home is done in the same way by setting oracle_home_gi to the old home.

Grid Infrastructure (cluster):

  • A switch back into a former home is not implemented: the former running home is locked by root.sh (e.g. bin/crsctl.bin owned by root), oraswgi_install stops with an assert. Use a new oracle_home_gi.

  • When root.sh fails before the switch (e.g. the CVU check of rootcrs.pl -prepatch), the next run resumes with root.sh: -switchGridHome is skipped, when rootconfig.sh of the new home is already owned by root.

7.1. Removal of obsolete Grid Infrastructure homes

oraswgi_install_remove_obsolete_homes: true removes obsolete Grid Infrastructure homes with deinstall. Without it, old homes stay for a switch back.

The source is the central inventory only. Obsolete is every home in the central inventory

  • with the component oracle.crs (1st component in <home>/inventory/ContentsXML/comps.xml), database homes are never removed,

  • which is not crs_home in /etc/oracle/olr.loc and not oracle_home_gi.

Homes or directories, which are not registered in the central inventory, are not touched and must be removed manually. The removal is only done, when Oracle Restart runs from oracle_home_gi (completed switch).

Table 8. Steps of oraswgi_install (remove_obsolete_home.yml) for each obsolete home
Step User Description

<old home>/deinstall/deinstall -silent -checkonly -o <dir>

grid

Creates the response file. The role stops with an error, when deinstall detects a configuration in the home:
CRS_HOME=false, ASM_IN_HOME=false, ASM_DROP_DISKGROUPS=false and ORA_CRS_HOME=<running home> are required, the output must contain The home being deconfigured is NOT a configured Grid Infrastructure home.

<old home>/crs/install/roothas.sh -unlock -dstcrshome <old home>

root

Only for a locked home (owner root), a former running home.

<old home>/deinstall/deinstall -silent -paramfile <response file>

grid

Removes the software and the home from the central inventory. No downtime.

rootdeinstall.sh

root

Only, when requested in the output of deinstall. Removes the files owned by root and the directory of the home.

Note
deinstall of an obsolete home prints Oracle Restart is stopped and de-configured successfully and requests rootdeinstall.sh "to remove the configuration of the Oracle Restart". This is the standard text of deinstall: the running Oracle Restart in crs_home is not changed (tested with 23.26.3 ⇒ 23.26.4: resources, /etc/oracle/olr.loc, oracle-ohasd.service, +ASM in /etc/oratab and listener.ora of the running home unchanged).
Note
Oracle documents deinstall of the home itself (same release) after an unlock for the previous release Grid home after an upgrade ("Unlocking and Deinstalling the Previous Release Grid Home"). The removal of an old home of the same release after -switchGridHome is not described explicitly.
Inventory example
oracle_install_option_gi: HA_CONFIG
oracle_install_version_gi: 26.1.0.0
# 1st installation: gi_goldimage_26_03.zip, /u01/app/oracle/product/26/restart_26.03
oracle_install_image_gi: gi_goldimage_26_04.zip
oracle_home_gi: /u01/app/oracle/product/26/restart_26.04

8. Inventory example

The ORACLE_HOME for each Release Update gets its own entry in db_homes_config with a unique oracle_home and oracle_home_name. Both ways to prepare the new ORACLE_HOME can be mixed, e.g. the old ORACLE_HOME from a Golden Image and the new one installed and patched on the server.

Example db_homes_config with Golden Image
db_homes_config:
  db19_2601:
    version: 19.3.0.0
    oracle_home: /u01/app/oracle/product/19/db19_2601
    oracle_home_name: db19_2601
    readonly_home: true
    edition: EE
    imagename: db_home_19.30.zip
Example db_homes_config installed and patched on the server
db_homes_config:
  db19_2604:
    version: 19.3.0.0
    oracle_home: /u01/app/oracle/product/19/db19_2604
    oracle_home_name: db19_2604
    readonly_home: true
    edition: EE
    opatch_minversion: 12.2.0.1.48
    opatch:
      - patchid: <RU patch id>
        # Database Release Update 19.31
        state: present
      - patchid: <OJVM patch id>
        # Oracle JavaVM Component Release Update 19.31
        state: present
Note
stop_processes is not needed for the patches of the new ORACLE_HOME.
Example: switch of database ORCL from db19_2601 to db19_2604
db_homes_installed:
  - home: db19_2601
    state: present
  - home: db19_2604
    state: present

oracle_databases:
  - home: db19_2604    # (1)
    oracle_db_name: ORCL
    state: present
  1. Changing home is the trigger for the switch. See Existing features.

9. Open questions

  • oraswdb_install derives the type of an ORACLE_HOME (SI or cluster installation) from the databases in oracle_databases. An ORACLE_HOME without database is installed as SI. A new RAC ORACLE_HOME needs the cluster installation before the switch ⇒ explicit type in db_homes_config (e.g. oracle_db_type)?

  • Handling of a listener shared by databases in different `ORACLE_HOME`s

  • Fallback to the old ORACLE_HOME (datapatch rollback)