It’s finally here! ODA patch 19.6 with EL7 and Oracle DB 19c (19.6.0.0.200114). In this blogpost, I will describe what steps I did and what issues I encountered as it is a big change from EL6 → EL7 and minor from 18c to 19c.

1. Resources used

• Oracle Database Appliance X7-2S, X7-2M, and X7-2-HA Patches (patches for other hardware models can be found on the same page)
  • ODA patch 19.6 (31010832)
  • DB clone 19.6 (30403662)
• Patching Instruction Oracle Database Appliance
  • OS patch (mandatory first step), note here is that /opt/oracle/dcs/bin/odacli update-server -c os –local should be /opt/oracle/dcs/bin/odacli update-server -v 19.6.0.0.0 -c os –local
  • server, storage, grid, db patch
• Known Issues with Oracle Database Appliance in This Release
  • Read through the known issues to anticipate changes I needed to do, lucky for me we just migrated to ODAX7-2 HA 18.7 this year. And applied the 18.8 patch recently as this is a pre-requisite
• ODA (Oracle Database Appliance): ODABR a System Backup/Restore Utility (Doc ID 2466177.1)
  • Which is mandatory for the EL6 → EL7 upgrade but doesn’t seem to be installed by default? And only seems to be a requirement for BM
• Analyzing the Pre-Checks Report for Operating System Upgrades
  • Saves your life (time) to read through it before actually doing the OS pre-patch
• Errors when running ORAchk or the odacli create-prepatchreport command
  • They added Orachk in all the pre-patch reports and needless to say there were a fair amount of issues but as described in the link above can be ignored.
• /opt/oracle/dcs/log/dcs-agent.log (verify useful as odacli describe-job is pretty picky with the information it shows)
• How to Download and Run Oracle’s Database Pre-Upgrade Utility (Doc ID 884522.1)

2. Cleanup

First step I can recommend is to make sure you have enough space on /opt (location where patches and clones will end up) and / (/tmp on second node will be used to place the zip files before placing them in the repository). I used the following commands to do this:

/opt

/ (/tmp)

3. Start of upgrade

3.1 OS Upgrade

💡 Tip: Before running /opt/oracle/dcs/bin/odacli create-prepatchreport -v 19.6.0.0.0 -os I would really advise to go through Analyzing the Pre-Checks Report for Operating System Upgrades and ODA (Oracle Database Appliance): ODABR a System Backup/Restore Utility (Doc ID 2466177.1) . This will save you a failed run, for 1 I didn’t know about de scheduled jobs on ODA so it kinda threw me off. But it’s easy to disable as mentioned in the document. And as for the requirement with odabr, I would’ve suspected it being installed if it was mandatory but this was not the case so I had to manually install it.

3.2 ODA patch (server, storage, grid, db)

08-MAY-2020: I discovered that the listener parameters were wrong on my database, I had the same thing happen when I moved to 18c. So I would advise to do show parameter listener and show spparameter listener to validate everything. Both after grid and db patch.
🐛12-MAY-2020: asr doesn’t startup automatically due to wrong java home configured. Details can be found at the bottom of this page.

3.3 Install 19.6 clone and upgrade database

💡 Tip: run Pre-Upgrade Information Tool (preupgrade.jar) Command before executing the upgrade-database command as it will save you another failed run 😉, this can be run from the new 19c home as it comes with preupgrade.jar (/u01/app/oracle/product/19.0.0.0/dbhome_1/rdbms/admin/preupgrade.jar). But the best approach would be to download the latest version from How to Download and Run Oracle’s Database Pre-Upgrade Utility (Doc ID 884522.1).
List of all the checks run by the preupgrade.jar can be found here Database Preupgrade tool check list. (Doc ID 2380601.1)

🐛 8-May-2020, read bottom of this page regarding a bug with DB 19.6

3.4 Remove 18c grid home

Output:

3.5 bugs and patches

Updated on 16-June-2020

1. 1. ODA TYPE CAN’T BE READ FROM SGA BECAUSE ERRORS INSIDE OAK CALLS
      After upgrading my database to 19.6 I saw a lot of tracefiles being generated with the above message. (edit: fixed in ODA 19.7)
      The full log is as follows
      
      Trace file /u01/app/oracle/diag/rdbms/DEV/DEV2/trace/DEV2_j002_20879.trc Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.6.0.0.0 Build label: RDBMS_19.3.0.0.0DBRU_LINUX.X64_190417 ORACLE_HOME: /u01/app/oracle/product/19.0.0.0/dbhome_1 System name: Linux Node name: by3aab Release: 4.14.35-1902.11.3.1.el7uek.x86_64 Version: #2 SMP Sat Mar 14 20:57:52 PDT 2020 Machine: x86_64 Instance name: DEV2 Redo thread mounted by this instance: 2 Oracle process number: 0 Unix process pid: 20879, image: *** 2020-05-07T22:00:02.486098+02:00 kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls SCHED 05-07 22:14:24.411 2 00 20879 J002 0(kkjspub):Slave was not marked as idle

      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

      Trace file /u01/app/oracle/diag/rdbms/DEV/DEV2/trace/DEV2_j002_20879.trc Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.6.0.0.0 Build label:    RDBMS_19.3.0.0.0DBRU_LINUX.X64_190417 ORACLE_HOME:    /u01/app/oracle/product/19.0.0.0/dbhome_1 System name:    Linux Node name:      by3aab Release:        4.14.35-1902.11.3.1.el7uek.x86_64 Version:        #2 SMP Sat Mar 14 20:57:52 PDT 2020 Machine:        x86_64 Instance name: DEV2 Redo thread mounted by this instance: 2 Oracle process number: 0 Unix process pid: 20879, image:     *** 2020-05-07T22:00:02.486098+02:00 kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls kgfmGetOdaType: ODA type can't be read from SGA because errors inside OAK calls SCHED 05-07 22:14:24.411 2 00 20879 J002 0(kkjspub):Slave was not marked as idle

      At this moment the bug is still not available for public but the patch to resolve this issue is -> https://updates.oracle.com/download/31009680.html
      * make sure grid users can do passwordless authentication
   2. asr doesn’t start due to wrong java home (cfr Auto Service Request (ASR) Manager Health Check (Doc ID 2205946.1))
      As a test you can execute systemctl status asra
      
      Shell

      [root@by3aaa ~]# systemctl status asra ● asra.service - LSB: Auto Service Request Manager Loaded: loaded (/etc/rc.d/init.d/asra; bad; vendor preset: disabled) Active: active (exited) since Sat 2020-05-02 05:16:15 CEST; 1 weeks 3 days ago Docs: man:systemd-sysv-generator(8) Tasks: 0 May 02 05:16:15 by3aaa systemd[1]: Starting LSB: Auto Service Request Manager... May 02 05:16:15 by3aaa asra[5818]: **************************************************************** May 02 05:16:15 by3aaa asra[5818]: JAVA is not found. Please set 'java.exec' property in file May 02 05:16:15 by3aaa asra[5818]: /var/opt/asrmanager/configuration/asr.conf May 02 05:16:15 by3aaa asra[5818]: to point to JAVA 1.8 or later and try again May 02 05:16:15 by3aaa asra[5818]: **************************************************************** May 02 05:16:15 by3aaa systemd[1]: Started LSB: Auto Service Request Manager. [root@by3aaa ~]# vi /var/opt/asrmanager/configuration/asr.conf [root@by3aaa ~]# cat /var/opt/asrmanager/configuration/asr.conf java.exec=/usr/java/jdk1.8.0_231-amd64/jre/bin/java #java.exec=/usr/java/jdk1.8.0_181-amd64/jre/bin/java [root@by3aaa log]# systemctl status asra ● asra.service - LSB: Auto Service Request Manager Loaded: loaded (/etc/rc.d/init.d/asra; bad; vendor preset: disabled) Active: active (exited) since Sat 2020-05-02 05:16:15 CEST; 1 weeks 3 days ago Docs: man:systemd-sysv-generator(8) Tasks: 0 May 02 05:16:15 by3aaa systemd[1]: Starting LSB: Auto Service Request Manager... May 02 05:16:15 by3aaa asra[5818]: **************************************************************** May 02 05:16:15 by3aaa asra[5818]: JAVA is not found. Please set 'java.exec' property in file May 02 05:16:15 by3aaa asra[5818]: /var/opt/asrmanager/configuration/asr.conf May 02 05:16:15 by3aaa asra[5818]: to point to JAVA 1.8 or later and try again May 02 05:16:15 by3aaa asra[5818]: **************************************************************** May 02 05:16:15 by3aaa systemd[1]: Started LSB: Auto Service Request Manager. [root@by3aaa ~]# systemctl stop asra [root@by3aaa ~]# systemctl start asra [root@by3aaa ~]# systemctl status asra ● asra.service - LSB: Auto Service Request Manager Loaded: loaded (/etc/rc.d/init.d/asra; bad; vendor preset: disabled) Active: active (running) since Tue 2020-05-12 13:42:13 CEST; 1s ago Docs: man:systemd-sysv-generator(8) Process: 34502 ExecStop=/etc/rc.d/init.d/asra stop (code=exited, status=0/SUCCESS) Process: 34664 ExecStart=/etc/rc.d/init.d/asra start (code=exited, status=0/SUCCESS) Tasks: 2 CGroup: /system.slice/asra.service ├─34716 /bin/sh /opt/asrmanager/bin/asrautoupdate ASRAuto └─34981 sleep 10 May 12 13:42:13 by3aaa systemd[1]: Starting LSB: Auto Service Request Manager... May 12 13:42:13 by3aaa systemd[1]: Started LSB: Auto Service Request Manager.

      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

      [root@by3aaa ~]# systemctl status asra ● asra.service - LSB: Auto Service Request Manager    Loaded: loaded (/etc/rc.d/init.d/asra; bad; vendor preset: disabled)    Active: active (exited) since Sat 2020-05-02 05:16:15 CEST; 1 weeks 3 days ago      Docs: man:systemd-sysv-generator(8)     Tasks: 0 May 02 05:16:15 by3aaa systemd[1]: Starting LSB: Auto Service Request Manager... May 02 05:16:15 by3aaa asra[5818]: **************************************************************** May 02 05:16:15 by3aaa asra[5818]: JAVA is not found. Please set 'java.exec' property in file May 02 05:16:15 by3aaa asra[5818]: /var/opt/asrmanager/configuration/asr.conf May 02 05:16:15 by3aaa asra[5818]: to point to JAVA 1.8 or later and try again May 02 05:16:15 by3aaa asra[5818]: **************************************************************** May 02 05:16:15 by3aaa systemd[1]: Started LSB: Auto Service Request Manager. [root@by3aaa ~]# vi /var/opt/asrmanager/configuration/asr.conf [root@by3aaa ~]# cat /var/opt/asrmanager/configuration/asr.conf java.exec=/usr/java/jdk1.8.0_231-amd64/jre/bin/java #java.exec=/usr/java/jdk1.8.0_181-amd64/jre/bin/java [root@by3aaa log]# systemctl status asra ● asra.service - LSB: Auto Service Request Manager    Loaded: loaded (/etc/rc.d/init.d/asra; bad; vendor preset: disabled)    Active: active (exited) since Sat 2020-05-02 05:16:15 CEST; 1 weeks 3 days ago      Docs: man:systemd-sysv-generator(8)     Tasks: 0 May 02 05:16:15 by3aaa systemd[1]: Starting LSB: Auto Service Request Manager... May 02 05:16:15 by3aaa asra[5818]: **************************************************************** May 02 05:16:15 by3aaa asra[5818]: JAVA is not found. Please set 'java.exec' property in file May 02 05:16:15 by3aaa asra[5818]: /var/opt/asrmanager/configuration/asr.conf May 02 05:16:15 by3aaa asra[5818]: to point to JAVA 1.8 or later and try again May 02 05:16:15 by3aaa asra[5818]: **************************************************************** May 02 05:16:15 by3aaa systemd[1]: Started LSB: Auto Service Request Manager. [root@by3aaa ~]# systemctl stop asra [root@by3aaa ~]# systemctl start asra [root@by3aaa ~]# systemctl status asra ● asra.service - LSB: Auto Service Request Manager    Loaded: loaded (/etc/rc.d/init.d/asra; bad; vendor preset: disabled)    Active: active (running) since Tue 2020-05-12 13:42:13 CEST; 1s ago      Docs: man:systemd-sysv-generator(8)   Process: 34502 ExecStop=/etc/rc.d/init.d/asra stop (code=exited, status=0/SUCCESS)   Process: 34664 ExecStart=/etc/rc.d/init.d/asra start (code=exited, status=0/SUCCESS)     Tasks: 2    CGroup: /system.slice/asra.service            ├─34716 /bin/sh /opt/asrmanager/bin/asrautoupdate ASRAuto            └─34981 sleep 10 May 12 13:42:13 by3aaa systemd[1]: Starting LSB: Auto Service Request Manager... May 12 13:42:13 by3aaa systemd[1]: Started LSB: Auto Service Request Manager.

      systemctl status asrm
      
      Shell

      [root@by3aaa ~]# systemctl status asrm ● asrm.service - LSB: Auto Service Request Manager Loaded: loaded (/etc/rc.d/init.d/asrm; bad; vendor preset: disabled) Active: inactive (dead) since Tue 2020-05-12 13:24:11 CEST; 1h 15min ago Docs: man:systemd-sysv-generator(8) Process: 82583 ExecStop=/etc/rc.d/init.d/asrm stop (code=exited, status=0/SUCCESS) May 02 05:16:15 by3aaa su[5874]: (to root) root on none May 02 05:16:15 by3aaa asrm[5819]: **************************************************************** May 02 05:16:15 by3aaa asrm[5819]: JAVA is not found. Please set 'java.exec' property in file May 02 05:16:15 by3aaa asrm[5819]: /var/opt/asrmanager/configuration/asr.conf May 02 05:16:15 by3aaa asrm[5819]: to point to JAVA 1.8 or later and try again May 02 05:16:15 by3aaa asrm[5819]: **************************************************************** May 02 05:16:15 by3aaa systemd[1]: Started LSB: Auto Service Request Manager. May 12 13:24:11 by3aaa systemd[1]: Stopping LSB: Auto Service Request Manager... May 12 13:24:11 by3aaa asrm[82583]: ASR Manager is stopped. May 12 13:24:11 by3aaa systemd[1]: Stopped LSB: Auto Service Request Manager. [root@by3aaa ~]# systemctl stop asrm [root@by3aaa ~]# systemctl start asrm [root@by3aaa ~]# systemctl status asrm ● asrm.service - LSB: Auto Service Request Manager Loaded: loaded (/etc/rc.d/init.d/asrm; bad; vendor preset: disabled) Active: active (exited) since Tue 2020-05-12 14:39:28 CEST; 4s ago Docs: man:systemd-sysv-generator(8) Process: 82583 ExecStop=/etc/rc.d/init.d/asrm stop (code=exited, status=0/SUCCESS) Process: 39601 ExecStart=/etc/rc.d/init.d/asrm start (code=exited, status=0/SUCCESS) May 12 14:39:27 by3aaa systemd[1]: Starting LSB: Auto Service Request Manager... May 12 14:39:27 by3aaa su[39602]: (to root) root on none May 12 14:39:27 by3aaa asrm[39601]: ASR Manager (pid 82968) is already RUNNING. May 12 14:39:28 by3aaa systemd[1]: Started LSB: Auto Service Request Manager.

      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

      [root@by3aaa ~]# systemctl status asrm ● asrm.service - LSB: Auto Service Request Manager    Loaded: loaded (/etc/rc.d/init.d/asrm; bad; vendor preset: disabled)    Active: inactive (dead) since Tue 2020-05-12 13:24:11 CEST; 1h 15min ago      Docs: man:systemd-sysv-generator(8)   Process: 82583 ExecStop=/etc/rc.d/init.d/asrm stop (code=exited, status=0/SUCCESS)   May 02 05:16:15 by3aaa su[5874]: (to root) root on none May 02 05:16:15 by3aaa asrm[5819]: **************************************************************** May 02 05:16:15 by3aaa asrm[5819]: JAVA is not found. Please set 'java.exec' property in file May 02 05:16:15 by3aaa asrm[5819]: /var/opt/asrmanager/configuration/asr.conf May 02 05:16:15 by3aaa asrm[5819]: to point to JAVA 1.8 or later and try again May 02 05:16:15 by3aaa asrm[5819]: **************************************************************** May 02 05:16:15 by3aaa systemd[1]: Started LSB: Auto Service Request Manager. May 12 13:24:11 by3aaa systemd[1]: Stopping LSB: Auto Service Request Manager... May 12 13:24:11 by3aaa asrm[82583]: ASR Manager is stopped. May 12 13:24:11 by3aaa systemd[1]: Stopped LSB: Auto Service Request Manager. [root@by3aaa ~]# systemctl stop asrm [root@by3aaa ~]# systemctl start asrm [root@by3aaa ~]# systemctl status asrm ● asrm.service - LSB: Auto Service Request Manager    Loaded: loaded (/etc/rc.d/init.d/asrm; bad; vendor preset: disabled)    Active: active (exited) since Tue 2020-05-12 14:39:28 CEST; 4s ago      Docs: man:systemd-sysv-generator(8)   Process: 82583 ExecStop=/etc/rc.d/init.d/asrm stop (code=exited, status=0/SUCCESS)   Process: 39601 ExecStart=/etc/rc.d/init.d/asrm start (code=exited, status=0/SUCCESS)   May 12 14:39:27 by3aaa systemd[1]: Starting LSB: Auto Service Request Manager... May 12 14:39:27 by3aaa su[39602]: (to root) root on none May 12 14:39:27 by3aaa asrm[39601]: ASR Manager (pid 82968) is already RUNNING. May 12 14:39:28 by3aaa systemd[1]: Started LSB: Auto Service Request Manager.

       

Interested in hearing more about the Oracle Database Appliance?
Contact us here or download the whitepaper below to get more info.

Download the ODA whitepaper