Oracle Autonomous Health Framework (AHF)

Important
The roles ahf and ahf_fleet_insights are under development and experimental.
Breaking changes can happen at any time and are not always documented (e.g. in Breaking Changes - Upgrade Guide).

Oracle Autonomous Health Framework (AHF) bundles the diagnostic and health check tools of Oracle:

  • Trace File Analyzer (TFA): collection of diagnostic data for Service Requests, automatic collections after incidents

  • Compliance checks with orachk (exachk on Engineered Systems)

  • AHF Insights: offline HTML reports about the state of the server and the databases

  • OS Watcher, Cluster Health Monitor and Cluster Health Advisor (Grid Infrastructure)

AHF is not needed in every environment. This guide helps with the decisions for an installation and describes the usage in ansible-oracle:

  • Role ahf: installation, upgrade and configuration of AHF on the database servers

  • Role ahf_fleet_insights: AHF Fleet Insights as central web interface for many database servers

The variables are described in roles/ahf/README.md and roles/ahf_fleet_insights/README.md.

1. Architecture overview

Architecture

The role ahf installs AHF on every database server. The role ahf_fleet_insights is optional and installs the central web interface AHF Fleet Insights on a separate server. AHF works without Fleet Insights - the data is then evaluated locally on each database server.

Table 1. Data storage
Location Content Owner Remark

ahf_loc (/opt/oracle.ahf) on the database server

AHF software: TFA, orachk, ahfctl

root

Replaced by an upgrade. No data.

ahf_data_dir (/opt/oracle.ahf/data) on the database server

AHF repository: TFA collections, Compliance results, Insights reports, logs of AHF

root

Primary storage of the data. Local filesystem (no NFS), size limit with reposizeMB, purge with minfileagetopurge.

ahf_ansible_state_file (/opt/oracle.ahf.ansible.json) on the database server

Settings applied by the role ahf

root

Detection of changed settings, see Settings.

ahf_fi_install_dir ({{ oracle_base }}/ahffi) on the Fleet Insights server

Fleet Insights software, embedded Nginx, logs, processing of uploads

ahf_fi_install_user

Plan 4 GB per registered cluster.

Repository database (19c or newer)

Uploaded data of all clients: Compliance results and TFA collections

schema AHFFI

Central copy for the evaluation. Available when a database server is down.

The data flows in one direction only: from the database servers to Fleet Insights (push over HTTPS). The local AHF repository stays the primary storage - a database server keeps working when Fleet Insights is not reachable. Fleet Insights needs no access to the database servers, neither for the collection nor for the evaluation in the browser.

Note
The diagram is rendered by plantuml.com from the master branch on GitHub. Changes of doc/guides/ahf.puml are visible after the merge.

2. When is AHF useful?

Table 2. Use cases for AHF
Situation Recommendation

Grid Infrastructure (RAC, Oracle Restart)

AHF is recommended. Oracle Support requests TFA collections for most Service Requests. GI contains an older TFA - a current AHF replaces it.

Single instance on filesystem with Oracle Support contract

Useful for faster Service Requests (tfactl diagcollect) and regular Compliance checks.

Many database servers

AHF with AHF Fleet Insights gives a central view of the Compliance results and the collections.

Test/development VMs, short-lived environments

Normally not needed. The TFA daemon and the Compliance scheduler need CPU, memory and disk space.

3. Decisions before the installation

3.1. root or non-root installation

Table 3. Installation modes (ahf_install_mode)
Feature root (default) extract (non-root)

User for the installation

root

oracle_user

TFA daemon (systemd), automatic collections

yes

no

Compliance scheduler (ahfctl startahf -compliance)

yes

no

Access to OS logs (/var/log/messages)

yes

no

AHF Fleet Insights client

yes

no

Default ahf_loc / ahf_data_dir

/opt

oracle_base

Recommended by Oracle

yes

only when root is not possible

3.2. Installation directory and data directory

ahf_setup creates oracle.ahf below ahf_loc (software) and oracle.ahf/data below ahf_data_dir (repository with collections, Compliance results and Insights reports).

Table 4. Default directories
Variable Default Reason

ahf_loc

/opt ⇒ /opt/oracle.ahf

Default of Oracle. The binaries are executed by root.

ahf_data_dir

ahf_loc ⇒ /opt/oracle.ahf/data

Default of Oracle. At least 5 GB, 10 GB recommended.

Important
With ahf_install_mode: root every directory in the path of ahf_loc and ahf_data_dir must be owned by root. ahf_setup rejects a data directory in a path of another owner (AHF-00014: AHF Data Location …​ is not owned by root in directory hierarchy).
A directory below oracle_base or below a mount point of host_fs_layout (e.g. /u01, owned by oracle_user by default) is not possible: oracle_user could rename or replace a directory in the path and get the binaries executed by root. The role checks the owner of all directories of ahf_loc and ahf_data_dir before the installation.

The software and the data need about 10 GB in /opt/oracle.ahf. Use a logical volume of its own for /opt/oracle.ahf, owned by root, when the root filesystem has not enough space. A full AHF repository does not fill the root filesystem then.

Logical volume of its own for AHF (role orahost)
host_fs_layout:
  - vgname: vgora
    state: present
    filesystem:
      - { mntp: /u01, lvname: lvorabase, lvsize: 25G, fstype: xfs }
      - { mntp: /opt/oracle.ahf, lvname: lvahf, lvsize: 10G, fstype: xfs, owner: root, group: root, mode: "0755" }
    disk:
      - { device: /dev/sdb, pvname: /dev/sdb1 }
Note
The installer appends oracle.ahf to ahf_loc and oracle.ahf/data to ahf_data_dir. With the defaults, software and data are both in /opt/oracle.ahf and therefore on this logical volume.

3.3. Resource requirements

The values are from the documentation of Oracle and measured in the molecule scenario ahf (AHF 26.10.2, AHF Fleet Insights 25.2.1, freshly installed, idle, 8 CPUs).

Table 5. Database server (role ahf)
Resource Value Source

Disk: software /opt/oracle.ahf

1.2 GB

measured

Disk: data /opt/oracle.ahf/data

at least 5 GB, 10 GB recommended. Grows with collections and Compliance results, limit with reposizeMB

Oracle

Memory: TFA daemon

about 170 MB RSS (Java with -Xmx256m)

measured

CPU: TFA daemon

limit by default: the lower of 4 CPUs or 75% of the CPUs (ahfctl getresourcelimit: cpu 4.0)

Oracle, measured

Compliance checks (orachk)

only during the scheduled runs (default: daily 02:03, Sunday 03:03)

not measured

Table 6. AHF Fleet Insights server (role ahf_fleet_insights)
Resource Value Source

CPU / memory

4 cores, 16 GB RAM (minimum recommendation)

Oracle

Memory idle

about 2.5 GB RSS (34 processes)

measured

Disk: software ahf_fi_install_dir

0.8 GB

measured

Disk: data

4 GB per registered cluster

Oracle

Repository database

separate database (19c or newer), e.g. Oracle Database Free: about 2.2 GB memory

measured

The CPU and memory of TFA can be limited with ahfctl setresourcelimit (only on Linux with a root installation):

Limit TFA to 2 CPUs and 1 GB memory
ahfctl setresourcelimit -value 2
ahfctl setresourcelimit -resource kmem -value 1024
ahfctl getresourcelimit
Note
The documentation of Oracle has different ranges for the memory limit in different releases. Check the allowed values with the installed release. The role ahf does not manage the limits.

3.4. Which components should be active?

Table 7. Components and variables
Component Variable Default Recommendation

TFA daemon

ahf_tfa_enabled

true

Active for GI and production. Disable on small test systems - tfactl stays usable on-demand.

Automatic diagnostic collections

ahf_tfa_settings.autodiagcollect

Oracle: OFF

ON for production with Service Requests. Needs space in the repository.

Compliance scheduler (orachk)

ahf_compliance_autostart

false

Active, when the results are evaluated (mail or AHF Fleet Insights).

AHF Insights

ahf_tfa_settings.autoinsights

Oracle: ON

Insights are created with collections. Usable for the static web server (see below).

AHF Fleet Insights client

ahf_fleet_insights_client

false

Active, when an AHF Fleet Insights server exists.

Minimal installation: TFA on-demand only
ahf_tfa_enabled: false
ahf_compliance_autostart: false
Full installation for production
ahf_tfa_enabled: true
ahf_tfa_settings:
  autodiagcollect: "ON"
  reposizeMB: 10240
  ahfnotificationaddress: dba@example.com
ahf_compliance_autostart: true
ahf_compliance_settings:
  autostart_client:
    COLLECTION_RETENTION: 30
ahf_fleet_insights_client: true
ahf_fleet_insights_url: https://ahffi.example.com:5005
ahf_fleet_insights_user: ahfclient
ahf_fleet_insights_password: "{{ vault_ahf_fleet_insights_password }}"

3.5. systemd, cron or nothing?

The role creates no cron jobs or systemd units for AHF:

  • The TFA daemon is started by AHF itself (oracle-tfa.service / init.tfa).

  • The Compliance checks have their own scheduler (ahfctl startahf -compliance). Oracle recommends the scheduler instead of cron. The schedule is changed with AUTORUN_SCHEDULE of an autorun ID in ahf_compliance_settings.

  • AHF Fleet Insights clients upload their data automatically.

  • The AHF Fleet Insights server starts and monitors its processes with cron entries of ahf_fi_install_user - created by the installer.

A systemd timer is only needed for the static web server without AHF Fleet Insights (see below).

4. Installation media

AHF should be up to date. Oracle releases a new version every quarter.

Table 8. Sources of the installation media
Source Filename Remark

AutoUpgrade (role autoupgrade)

name from My Oracle Support

AHF in autoupgrade_download_patches - always the latest version

My Oracle Support Doc ID 2550798.1 / patch 30166242

AHF-LINUX_v<version>.zip or p30166242_*.zip

manual download into the software repository

Download AHF with AutoUpgrade into the software repository
autoupgrade_download_patches:
  - target_version: 26
    patch: AHF
    platform: LINUX.X64
ansible-playbook -i inventory/ opitzconsulting.ansible_oracle.autoupgrade_download_patches

The role ahf uses the newest file (modification time) matching ahf_installer_patterns in oracle_sw_source_local (Ansible controller) or oracle_stage_remote (oracle_sw_copy: false, e.g. NFS). A fixed file is set with ahf_installer_archive.

The version is taken from the filename AHF-LINUX_v<version>.zip. Patch 30166242 is extracted to find the version of the inner AHF-LINUX_v<version>.zip. Set ahf_installer_version, when the version is not part of the filename.

Note
AHF can upgrade itself (ahfctl setupgrade -autoupgrade on -swstage <dir>). This is not used by ansible-oracle. An upgrade with the role is reproducible, uses the software repository and the signature check, and is executed at a planned time.

5. Installation and configuration with role ahf

Requirements:

  • Perl 5.10 or newer with all core modules - the package perl, perl-interpreter alone is not enough (AHF-00074: Required Perl Modules not found). The role installs perl, openssl and unzip (ahf_packages, ahf_manage_packages).

  • Python module pexpect on the database server for the registration in AHF Fleet Insights

The role executes these steps:

  1. Check of variables, Perl and the owner of ahf_loc

  2. Installation with ahf_setup -silent -local when AHF is missing, or upgrade when the installation media is newer (ahf_upgrade)

  3. Check of the signature of ahf_setup (ahf_verify_signature)

  4. Start/stop of TFA and the Compliance scheduler, tfactl set and orachk -set for changed settings

  5. Registration or deregistration in AHF Fleet Insights

Install AHF on all database servers
ansible-playbook -i inventory/ opitzconsulting.ansible_oracle.ahf -e hostgroup=dbfs

5.1. Settings

ahf_tfa_settings is executed with tfactl set <key>=<value>, ahf_compliance_settings with orachk -set "<key>=<value>;…​" -id <autorun ID> - one call per autorun ID. orachk -set requires AUTORUN_SCHEDULE in every call: the role adds the current schedule, when it is not part of the settings.

The Compliance settings are stored per autorun ID by AHF. The autoruns exist only with a running Compliance scheduler: ahfctl stopahf -compliance removes them, ahfctl startahf -compliance does not recreate them. The role creates the default autoruns of Oracle with orachk -autostart, when they are missing, and applies ahf_compliance_settings again. ahf_compliance_settings requires ahf_compliance_autostart: true.

A new installation has two autorun IDs (orachk -get all -id all shows them with the prefix orachk.):

Table 9. Autoruns of a new installation (AHF 26.10)
Autorun ID AUTORUN_SCHEDULE Checks COLLECTION_RETENTION

autostart_client_oratier1

3 2 * * 1,2,3,4,5,6 (Monday - Saturday 02:03)

profile oratier1

7 days

autostart_client

3 3 * * 0 (Sunday 03:03)

all checks

14 days

Daily checks at 01:30 and a longer retention of the weekly results
ahf_compliance_settings:
  autostart_client_oratier1:
    AUTORUN_SCHEDULE: "30 1 * * 1-6"
  autostart_client:
    COLLECTION_RETENTION: 30

The role stores the applied settings in ahf_ansible_state_file (/opt/oracle.ahf.ansible.json). A setting is executed only, when it differs from the last run. Manual changes with tfactl are not detected. -e ahf_settings_force=true executes all settings again.

Note
Removing a key from ahf_tfa_settings does not reset the setting in AHF. Set the default value of Oracle instead.

5.2. Cluster

The role installs AHF with -local on each host of the play. Execute tfactl syncnodes once after the installation on all nodes of a cluster. An existing TFA of Grid Infrastructure is upgraded by ahf_setup.

ahf_setup uses the Perl of the Grid Infrastructure home on hosts with Oracle Restart or RAC. This Perl is owned by the GI owner and is rejected for a root installation (AHF-00026: Perl is not owned by root). The role passes the Perl of the OS with -perlhome /usr (ahf_perl_home) and checks its owner.

5.3. Uninstall

Uninstall AHF
ansible-playbook -i inventory/ opitzconsulting.ansible_oracle.ahf -e hostgroup=dbfs -e ahf_state=absent

The repository in ahf_data_dir is kept. ahf_uninstall_delete_repo: true removes it.

6. Central evaluation of many servers

6.1. Architecture

The data stays on the database servers and is pushed to a central server:

  • The AHF repository is a local directory on the database server. An NFS share for the repository is not needed - a hanging NFS server has no impact on the database server.

  • The database servers send their data to the central server (push). The central server needs no access to the database servers - neither for the collection nor for the evaluation.

  • The data is available on the central server, when the database server is down.

A pull from the central server (e.g. rsync over SSH from each database server) is possible as well, but requires access of the central server to all database servers.

6.2. AHF Fleet Insights

AHF Fleet Insights is the web interface of Oracle for many AHF clients (My Oracle Support Doc ID 3043060.1).

  • Registered clients upload Compliance results (orachk) and TFA collections automatically.

  • The data is stored in a repository database (Oracle Database 19c or newer). Oracle AI Database 26ai needs no additional license for the repository.

  • The web application (with an embedded Nginx) runs as non-root user on port ahf_fi_port (5005) with HTTPS.

  • Requirements: Oracle Linux / RHEL 8 or newer (x86_64), Java 22 or newer, Oracle Instant Client (Basic, SQL*Plus), 4 cores, 16 GB RAM and 4 GB disk space per registered cluster.

  • Clients need AHF 24.9 or newer.

Important
The Oracle documentation describes the registration for clusters ("AHF Client must be running on a cluster"), but describes the fleet as "database clusters or single-instance systems". Check the registration of single instance servers before planning AHF Fleet Insights for them.

6.2.1. Repository database

Fleet Insights needs a schema user in the repository database, before the role ahf_fleet_insights is executed. The role does not create it. The installation media contains the script create_db_user.sql (in ahf_fleet_setup_onprem.zip) with placeholders for the password and the tablespace:

create_db_user.sql of AHFFI-LINUX_v25.2.1.zip
CREATE USER AHFFI IDENTIFIED BY <password> DEFAULT TABLESPACE <tablespace>;
GRANT CREATE SESSION TO AHFFI;
GRANT CONNECT, RESOURCE TO AHFFI;
GRANT CREATE TABLE TO AHFFI;
GRANT INSERT ANY TABLE TO AHFFI;
alter user AHFFI quota unlimited on <tablespace>;

Optionally, the repository database and the user are managed with ansible-oracle, e.g. a PDB in oracle_pdbs with the user in users (role oradb_manage_users). The password is taken from dbpasswords like for every other user.

Schema user for Fleet Insights in a PDB of ansible-oracle
oracle_pdbs:
  - cdb: DB1
    home: db26-si-ee
    pdb_name: AHFFI
    pdbadmin_user: PDBADMIN
    listener_port: 1521
    state: present
    users:
      - schema: AHFFI
        state: present
        default_tablespace: USERS
        grants:
          - create session
          - connect
          - resource
          - create table
          - insert any table
        quotas:
          - tablespace: USERS
            size: unlimited

dbpasswords:
  DB1:
    AHFFI:
      AHFFI: "{{ vault_ahf_fi_db_password }}"

ahf_fi_db_service: AHFFI
ahf_fi_db_user: AHFFI
ahf_fi_db_password: "{{ vault_ahf_fi_db_password }}"
Note
Check the grants of create_db_user.sql of a new release of Fleet Insights. The molecule scenario ahf creates the user directly with SQL*Plus in an Oracle Database Free container.

6.2.2. Installation

Inventory for AHF Fleet Insights
ahf_fi_db_host: ahffi.example.com
ahf_fi_db_port: 1521
ahf_fi_db_service: ahffi.example.com
ahf_fi_db_user: AHFFI
ahf_fi_db_password: "{{ vault_ahf_fi_db_password }}"
ahf_fi_registration_user: ahfclient
ahf_fi_registration_password: "{{ vault_ahf_fleet_insights_password }}"
Install AHF Fleet Insights
ansible-playbook -i inventory/ opitzconsulting.ansible_oracle.ahf_fleet_insights -e hostgroup=ahffi

The role:

  1. installs cronie, iproute, hostname, the Oracle JDK and Oracle Instant Client (ahf_fi_packages)

  2. installs Fleet Insights with ahf_fleet_setup -quiet <properties> as ahf_fi_install_user and creates the registration user, or patches an existing installation when the installation media is newer

  3. opens ahf_fi_port in firewalld

  4. starts Fleet Insights (ahffi start), when ahffi status does not report it as running, and waits for the web application on ahf_fi_port (ahf_fi_start_timeout)

  5. provides the public certificate as fact ahf_fi_public_cert and optionally as file on the Ansible controller (ahf_fi_public_cert_fetch)

  6. shows ahffi status and the registration command from ahffi info

The properties file with the passwords is removed after the installation.

Fleet Insights has no systemd unit. Its processes are started by cron entries of ahf_fi_install_user, created by the installer - crond must be running.

The installer creates a self-signed certificate with the hostname as CN, but without Subject Alternative Name (SAN). AHF clients (AHF 26.x, Python 3.13) check only the SAN and reject it with CERTIFICATE_VERIFY_FAILED …​ Hostname mismatch, certificate is not valid for '<host>'. The role replaces it (ahffi importcert, Fleet Insights is restarted):

  • ahf_fi_cert_generate: true (default): self-signed certificate with the SAN ahf_fi_cert_names (default: hostname and FQDN). The clients must use one of these names in ahf_fleet_insights_url.

  • ahf_fi_cert_file / ahf_fi_cert_key_file: own certificate, e.g. from an internal CA.

6.2.3. Login to the web interface

The URL of the web interface is shown by ahffi info (Application URL), e.g. https://ahffi.example.com:5005/ahfservice.

Table 10. Default login of the installer
User Password Remark

fleet_admin

welcome1

UI administrator with access to the data of the whole fleet. The password must be changed at the first login.

Important
Change the password of fleet_admin immediately after the installation. The generic registration user (ahf_fi_registration_user) is only used by the AHF clients for the registration and the upload, not for the web interface.
Note
In the molecule scenario ahf the web interface is published on the host: https://localhost:5005/ahfservice.
6.2.3.1. Reset the password of fleet_admin

Oracle documents no reset of a forgotten password of fleet_admin. The UI users are stored in the table AHFFI_USERS of the repository schema (ahf_fi_db_user). The installer writes the hash of the default password welcome1 into this table (install_ahf_service.sh, variable ahffi_admin_default). Setting this hash again resets the password to welcome1, the new password is set at the next login.

Warning
This procedure is derived from the installer of AHF Fleet Insights 25.2.1 and is not documented by Oracle.
Reset the password of fleet_admin to welcome1 (as ahf_fi_install_user on the Fleet Insights server)
cd <ahf_fi_install_dir>
HASH=$(sed -n 's/^ahffi_admin_default="\(.*\)"/\1/p' install_ahf_service.sh)

sqlplus <ahf_fi_db_user>@//<ahf_fi_db_host>:<ahf_fi_db_port>/<ahf_fi_db_service> <<EOF
update ahffi_users
   set user_password = '${HASH}', status = 'ENABLED', update_date = sysdate
 where user_id = 'fleet_admin';
commit;
EOF
Note
The installer sets the default password, when the hash in AHFFI_USERS differs from it (register_admin_user). Check the password of fleet_admin after a patch of Fleet Insights.

6.2.4. Registration of the clients

Register the database servers
ahf_fleet_insights_client: true
ahf_fleet_insights_url: https://ahffi.example.com:5005
ahf_fleet_insights_user: ahfclient
ahf_fleet_insights_password: "{{ vault_ahf_fleet_insights_password }}"
# self-signed certificate of Fleet Insights (ahf_fi_public_cert_fetch of role ahf_fleet_insights)
ahf_fleet_insights_public_cert: "{{ lookup('ansible.builtin.file', inventory_dir + '/files/ahffi_cert.pem') }}"

The client checks the certificate of Fleet Insights. With the self-signed certificate of the installer the registration fails with CERTIFICATE_VERIFY_FAILED, unless the certificate is passed with ahf_fleet_insights_public_cert (ahf configuration set --public-cert). Fetch it with ahf_fi_public_cert_fetch while installing Fleet Insights:

Fetch the certificate to the Ansible controller
ahf_fi_public_cert_fetch: "{{ inventory_dir }}/files/ahffi_cert.pem"

The URL must use a name of the SAN of the certificate (ahf_fi_cert_names). The certificate file on the client must be readable by the owner only (AHF-07275), the role writes it with mode 0600.

The registration asks twice for the password of the registration user, the role answers it with ansible.builtin.expect (Python module pexpect on the database server). It is executed with --os-user {{ oracle_user }} (ahf_fleet_insights_os_user). Without an OS user for the REST access, the registration fails with AHF-02119: Fleet Insights configuration succeeded, but the REST Agent was not started.

ansible-playbook -i inventory/ opitzconsulting.ansible_oracle.ahf -e hostgroup=dbfs

The URL is the URL of ahffi info without /ahfservice. ahf_fleet_insights_force_register: true registers again, e.g. after a password change. ahf_fleet_insights_client: false deregisters a client registered by the role.

6.2.5. Data in AHF Fleet Insights

A registered client uploads its data automatically. The views of Fleet Insights are filled by different data:

Table 11. Data and views of Fleet Insights
Data View in Fleet Insights Automatic on the client Manual start (as root) Inventory (role ahf)

Topology: database servers, databases (CDB, PDB), ASM, clusters

Fleet Topology, Databases, Database Servers

AHF Insights collection at 05:00 and 17:00

tfactl diagcollect -insights -last 1h -silent

ahf_fleet_insights_client: true (no further setting)

Problems, events, OS issues, system changes, patches, RPMs, space

Insights (Detected Problems, Events, …​), Space Analysis

AHF Insights collection at 05:00 and 17:00

tfactl diagcollect -insights -last 1h -silent

ahf_fleet_insights_client: true (no further setting)

Compliance results (orachk)

Insights > Best Practice Issues

Compliance scheduler

orachk -usediscovery -profile ahf -showpass -tag manual_check -readenvconfig

ahf_compliance_autostart: true, schedule in ahf_compliance_settings

Diagnostic collections with log and trace files

collections of the clients

automatic collections after incidents

tfactl diagcollect -database <db> -last 1h -silent

ahf_properties: {ahf.fleet-insights.upload-only-minimal-files: "false"}, ahf_tfa_settings: {autodiagcollect: "ON"}

Capacity

Fleetwide Capacity Analysis, Cluster-Wise Usage Trends

capacity plugin of Fleet Insights and exachk (Exadata)

-

not supported

Table 12. Collector options on the CLI and in the inventory of the role ahf
Option CLI (as root) Inventory (role ahf)

Automatic diagnostic collections after incidents

tfactl set autodiagcollect=ON

ahf_tfa_settings: {autodiagcollect: "ON"}

AHF Insights in automatic collections

tfactl set autoinsights=ON

ahf_tfa_settings: {autoinsights: "ON"}

Size of the TFA repository

tfactl set reposizeMB=10240

ahf_tfa_settings: {reposizeMB: 10240}

Upload of complete collections (log and trace files)

ahf configuration set --property ahf.fleet-insights.upload-only-minimal-files --value false

ahf_properties: {ahf.fleet-insights.upload-only-minimal-files: "false"}

Alert History: timeline of ASM events (only ASM: dismount, dismount_force, offline, mount, last 30 days), view with ahf analysis create --type alert-history --component ASM

ahf configuration set --property ahf.collectors.alert-history --value on

ahf_properties: {ahf.collectors.alert-history: "on"}

Events of the alert logs of databases, ASM and Clusterware (base of the automatic collections), view with tfactl events

always active with a running TFA daemon

ahf_tfa_enabled: true

Enhanced OS metrics

ahf configuration set --property ahf.collectors.enhanced_os_metrics --value on

ahf_properties: {ahf.collectors.enhanced_os_metrics: "on"}

Checks excluded from the Compliance checks

ahf configuration set --property ahf.compliance.excluded-check-ids --value <id>

ahf_properties: {ahf.compliance.excluded-check-ids: "<id>"}

Compliance scheduler

ahfctl startahf -compliance / ahfctl stopahf -compliance

ahf_compliance_autostart: true / false

Schedule and retention of a Compliance autorun

orachk -set "AUTORUN_SCHEDULE=30 1 * * 1-6;COLLECTION_RETENTION=30" -id autostart_client_oratier1

ahf_compliance_settings: {autostart_client_oratier1: {AUTORUN_SCHEDULE: "30 1 * * 1-6", COLLECTION_RETENTION: 30}}

Additional directory for collections

tfactl directory add <dir> -public -noexclusions (asks for component, database, instance)

ahf_tfa_directories: [{path: <dir>, component: RDBMS, database: <db>, instance: <sid>, collection: noexclusions}]

TFA daemon

tfactl enable + ahfctl startahf -tfa / ahfctl stopahf -tfa + tfactl disable

ahf_tfa_enabled: true / false

Registration in Fleet Insights

ahf configuration set --type fleet-insights --user-name <user> --url <url> --public-cert <file> --os-user oracle

ahf_fleet_insights_client: true, ahf_fleet_insights_url, ahf_fleet_insights_user, ahf_fleet_insights_password, ahf_fleet_insights_public_cert

CPU and memory limit of TFA

ahfctl setresourcelimit -value 2

not supported, see Resource requirements

The role executes a setting only, when it differs from the last run (state file ahf_ansible_state_file). Manual changes on the CLI are not detected - -e ahf_settings_force=true executes all settings again.

The data is processed about 2 minutes after the upload. A new client shows its databases after the first Insights collection. The role ahf does not start a collection - start one manually, when the data is needed immediately.

6.2.5.1. Log files and diag_dest of the databases

TFA discovers the directories of Grid Infrastructure, ASM, listeners and the running databases automatically, including the ADR (diag_dest) of each database with the database and instance name:

Directories discovered by TFA (example from Oracle Restart, shortened)
tfactl print directories
| /u01/app/oracle/diag/rdbms/dbhas/DBHAS/trace | [RDBMS]{RDBMS|database=dbhas, RDBMS|instance=DBHAS} | public | root |
| /u01/app/oracle/diag/asm/+asm/+ASM/trace     | [ASM]{ASM|instance=+ASM}                            | public | root |

Directories, which are not discovered (e.g. audit files or log files of an application), are added with ahf_tfa_directories. The role answers the questions of tfactl directory add (component, database, instance) and removes directories, which it has added and which are no longer in the list.

By default the clients upload only minimal files into Fleet Insights: the Insights data and the Compliance results. Diagnostic collections with the log and trace files are uploaded only with ahf.fleet-insights.upload-only-minimal-files: false.

Note
The property ahf.fleet-insights.upload-only-minimal-files is not documented by Oracle. The behavior is observed with AHF 26.10: with true (default) a tfactl diagcollect -database is not uploaded, with false it is uploaded. Complete collections need more space in the repository database and on the Fleet Insights server.
Upload of log files and an additional directory
ahf_properties:
  ahf.fleet-insights.upload-only-minimal-files: "false"

ahf_tfa_directories:
  - path: /u01/app/oracle/admin/DB1/adump
    component: RDBMS
    database: db1
    instance: DB1
    collection: noexclusions

The properties of AHF are set with ahf configuration set --property <name> --value <value> (ahf_properties). ahf configuration get --help lists the supported properties, ahf configuration get --property <name> shows the value.

6.2.6. Check the upload on the client

ahf configuration check --type fleet-insights tests the upload of a client into Fleet Insights. The commands for the clients and the Fleet Insights server, the manual start of a collection with upload and the known errors are described in Troubleshooting.

6.3. Alternative: static web server with AHF Insights

This variant is not implemented in ansible-oracle. It is an option for environments without a repository database.

AHF Insights reports are offline HTML reports (zip file with index.html) - they need no server component.

  1. A systemd timer on the database server creates a report and copies it to the web server:

    # /etc/systemd/system/ahf-insights-push.service
    [Unit]
    Description=Create AHF Insights report and push it to the web server
    
    [Service]
    Type=oneshot
    ExecStart=/opt/oracle.ahf/bin/ahf analysis create --type insights --last 24h
    ExecStart=/usr/bin/rsync -a --include='*/' --include='*insights*.zip' --exclude='*' \
      /u01/app/oracle/ahf/oracle.ahf/data/repository/ ahfpush@web.example.com:
    # /etc/systemd/system/ahf-insights-push.timer
    [Timer]
    OnCalendar=daily
    RandomizedDelaySec=1h
    Persistent=true
    
    [Install]
    WantedBy=timers.target
  2. The SSH key of the database server is restricted to a directory on the web server with rrsync (command="rrsync -wo /srv/ahf/<host>" in authorized_keys).

  3. The web server (Nginx/httpd) extracts the reports (e.g. with a systemd path unit) and serves the directories as static files.

A systemd timer is preferred to cron: random delay (RandomizedDelaySec), catch-up after downtime (Persistent) and the output in the journal.

Table 13. Comparison of the variants
Criteria AHF Fleet Insights Static web server

Product

Oracle, supported

own solution

Data

Compliance results and TFA collections, aggregated

single Insights reports per host and day

Requirements

repository database, Java, Instant Client, 16 GB RAM

web server, SSH/rsync

Transport

HTTPS by the AHF client

rsync over SSH

Overview of the fleet

yes

no - one report per host

The Health Check Collection Manager (APEX application for orachk -setdbupload) is an older alternative for Compliance results only. It is not supported by ansible-oracle.

7. Troubleshooting

The commands are executed as root on the database server (/opt/oracle.ahf/bin with the default ahf_loc) and as ahf_fi_install_user in ahf_fi_install_dir on the Fleet Insights server.

7.1. Database server (AHF client)

7.1.1. Status

Table 14. Status of AHF on the database server
Command Description

ahfctl statusahf

Status of TFA (PID, port, version) and the Compliance scheduler

ahf software get-version --component all

Installed versions of AHF, TFA and Compliance

tfactl print config

Settings of TFA (ahf_tfa_settings)

tfactl print directories

Trace directories discovered by TFA, e.g. CRS, ASM, RDBMS with Oracle Restart

ahfctl getresourcelimit

CPU and memory limits of TFA

orachk -get all -id all

Compliance autoruns with AUTORUN_SCHEDULE and COLLECTION_RETENTION (ahf_compliance_settings)

cat /opt/oracle.ahf.ansible.json

Settings and registration applied by the role ahf (ahf_ansible_state_file)

7.1.2. Connection to Fleet Insights

Test the upload into Fleet Insights
ahf configuration check --type fleet-insights

The command uploads a test file (TestUploadFile.txt) into Fleet Insights and checks the complete path including the certificate:

  • Parameters for 'odbda' are configured correctly (rc 0): the upload works.

  • Parameters are not configured correctly to upload (rc 1): the upload fails. The message has no details, check the known errors below. After a new installation or a new certificate of Fleet Insights, run the role ahf again - it registers the client again, when the certificate has changed. ahf_fleet_insights_force_register: true registers again in any case.

The client has no log of its uploads. Failed uploads of Compliance results are kept and can be uploaded again:

Failed uploads of Compliance results
orachk -checkservicefaileduploads    # list failed uploads
orachk -uploadservicefailed          # upload them again

7.1.3. Collect data and upload it immediately

The registered client uploads Compliance results and TFA collections automatically. For a check in the web interface, a collection can be started manually:

Compliance check with upload (profile ahf: about 1 minute)
orachk -usediscovery -profile ahf -showpass -tag manual_check -readenvconfig

The output ends with <name> successfully uploaded to Autonomous Health Framework Service. Other profiles (orachk -profile -h), e.g. dba or clusterware, take longer. Without -profile, all checks are executed.

TFA collection of the last hour with AHF Insights and upload
tfactl diagcollect -insights -last 1h -silent

The dashboard of Fleet Insights is built from AHF Insights data. The clients collect it at 05:00 and 17:00, -insights uploads it immediately (about 2 minutes until the data is processed). -silent starts the collection in the background without questions. Without it, tfactl diagcollect asks for an event or an event time (e.g. on hosts with Grid Infrastructure). The status is shown by tfactl print actions and tfactl print collections.

7.1.4. Logs and directories

Table 15. Logs on the database server (default ahf_loc / ahf_data_dir)
Path Content

/opt/oracle.ahf/data/<host>/diag

Logs of AHF (requested by Oracle Support for AHF-00005)

/opt/oracle.ahf/data/<host>/orachk/user_root/output

Compliance results (orachk_<host>_<date>_<tag>.zip, HTML report)

/opt/oracle.ahf/data/repository

TFA collections and Insights reports

/opt/oracle.ahf/data/ahf/diag/ahf

Log of the installation (ahf_install_*.log)

7.1.5. Known errors

Table 16. Known errors on the database server
Message Cause Solution

AHF-00014: AHF Data Location …​ is not owned by root in directory hierarchy

ahf_data_dir below a directory of oracle_user, e.g. oracle_base or /u01

ahf_data_dir in a path owned by root (default /opt)

AHF-00026: Perl is not owned by root

ahf_setup uses the Perl of the GI-Home (Oracle Restart, RAC)

ahf_perl_home: /usr (default)

AHF-00074: Required Perl Modules not found

only perl-interpreter is installed

package perl (ahf_packages)

No scheduler for any ID

ahfctl stopahf -compliance removed the autoruns

ahf_compliance_autostart: true, the role recreates them with orachk -autostart

AUTORUN_SCHEDULE is a mandatory field to setup autorun entries

orachk -set without AUTORUN_SCHEDULE or with an unknown autorun ID

the role adds the current schedule, the ID is used without orachk.

CERTIFICATE_VERIFY_FAILED …​ self-signed certificate

certificate of Fleet Insights is unknown to the client

ahf_fleet_insights_public_cert

CERTIFICATE_VERIFY_FAILED …​ Hostname mismatch

certificate of Fleet Insights without SAN or the URL uses another name

ahf_fi_cert_generate / ahf_fi_cert_names on the server, a name of the SAN in ahf_fleet_insights_url

AHF-07275: Restrict the certificate file to the owner only

certificate file readable by others

mode 0600 (done by the role)

AHF-02119: …​ the REST Agent was not started

registration without OS user for the REST access

ahf_fleet_insights_os_user (default oracle_user)

7.2. AHF Fleet Insights server

7.2.1. Status

Table 17. Status of Fleet Insights (as ahf_fi_install_user in ahf_fi_install_dir)
Command Description

./ahffi status

Status of the web application and the collection processing engine

./ahffi info

URL of the web interface and registration command for the clients

./ahffi version

Installed version

./ahffi diagnose

Diagnostics of Fleet Insights

./ahffi start / ./ahffi stop / ./ahffi restart

Start and stop of all processes

crontab -l

Cron entries, which start and monitor the processes (no systemd unit)

ss -ltnp | grep 5005

Nginx listens on ahf_fi_port

Certificate of Fleet Insights (CN and SAN)
echo | openssl s_client -connect <host>:5005 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName -dates

7.2.2. Uploads of the clients

The access log of Nginx shows the registrations and uploads of all clients:

Registrations and uploads in the access log
LOG=<ahf_fi_install_dir>/ahf_service/third_party/nginx/logs/access.log
grep '"POST /ahfservice/register' $LOG                    # registrations
grep '"PUT /ahfservice/v1/collectionstorage' $LOG         # uploads

The uploads contain the hostname of the client in the path, e.g. …​/collections/tfa/<host>/orachk_<host>_<date>_<tag>.zip (Compliance results), …​/PERIODICPERFDATA_srdc_insights_collection_…​zip (Insights data) and checkserviceupload_*.txt / TestUploadFile.txt (tests).

7.2.3. Logs and directories

Table 18. Logs on the Fleet Insights server
Path Content

<ahf_fi_install_dir>/ahf_service/log/manage_app.log

Start, stop, status and importcert

<ahf_fi_install_dir>/ahf_service/third_party/nginx/logs/access.log

Requests of the browsers and the clients

<ahf_fi_install_dir>/ahf_service/third_party/nginx/logs/error.log

Errors of Nginx

<ahf_fi_install_dir>/ahf_service/certs/cert.pem

Certificate of the web interface

<ahf_fi_stage_dir>/tmp/ahf_fleet_insights_log

Log of the installation (removed with the stage directory after the installation)

7.2.4. Known errors

Table 19. Known errors on the Fleet Insights server
Message Cause Solution

Path to Java, 'JAVA_HOME' must be a valid directory

ahf_fi_java_home does not exist. The Oracle JDK RPM installs into a directory with the full version

ahf_fi_java_home: "" (default): the role uses the newest /usr/lib/jvm/jdk-*-oracle-x64

The installer will fail if run with root privileges

installation as root

ahf_fi_install_user (default oracle_user)

web interface not reachable after a reboot

processes are started by cron, crond is not running

systemctl enable --now crond, ./ahffi start (the role starts Fleet Insights, when it is not running)

Collection Processing Engine | Not Running after a reboot, uploads are not processed

the cron jobs of Fleet Insights restart the server and the collection processing engine at the same time. The restart of the engine removes its own cron entry and is interrupted (observed with Fleet Insights 25.2.1)

./ahffi start - starts all processes and adds the cron entries again. The role ahf_fleet_insights does it, when not all processes are running. Check with ./ahffi status and crontab -l (4 entries) after a reboot

clients fail with Hostname mismatch

certificate of the installer has no SAN

ahf_fi_cert_generate: true (default), ahf_fi_cert_names

login with fleet_admin not possible

password forgotten

see Reset the password of fleet_admin