2 weeks ago, with the release of the April 2015 QFSP, it was time again to start patching our exadata.
As we apply the QFSP 2 times a year, this was nothing new for us and at first everything seemed to go smoothly.
Patching the storage servers completed in about 2 hours (we don’t use rolling patching) and updating the compute nodes with dbnodeupdate took 1 hour per node. No issues encountered and everything was pretty well documented.
But then, the problems started with the PSU patching of the GI (which is 18.104.22.168)
First conflict check reported a conflict with an existing patch:
Composite Patch : 20594149
Conflict with 19392604
Bug Superset of 19189240
Hmm, ok. Only bundle patches are installed on this home, so where is the conflict coming from.
[aaaa@xxxxx ~]$ $ORACLE_HOME/OPatch/opatch lspatches
19392604;OCW PATCH SET UPDATE : 22.214.171.124.1 (19392604)
19392590;ACFS Patch Set Update : 126.96.36.199.1 (19392590)
19189240;DATABASE BUNDLE PATCH : 188.8.131.52.1 (19189240)
So the conflict is for the OCW Patch. Weird.
When checking with Oracle support, we got confirmation that we indeed should rollback the 19392604 patch.
Unfortunately that turned out to be easier said than done when opatchauto threw a nice java exception:
Exception in thread "main" java.lang.NumberFormatException: For input string: ""
Patch version not found
After some searching this appeared to be bug 20370694 – OPATCHAUTO FAILS FOR COMPOSITE PATCH (see MOS doc 1998440.1) and after changing the system line in the bundle.xml file to include the unique_patch_id
<system_patch_bundle_xml type_version="2.0" bundle_type="ENGSYSTEM" patch_abstract="sample EsysPatch description" patch_id="20698050" unique_patch_id="18314356">
We were able to run the opatchauto command.
Which unfortunately failed to rollback the patch….
At that point we decided to perform a manual rollback as detailed in the supplemental readme (Doc ID 1591616.1), which did the trick and rolled back the OCW patch.
Finally we were able to continue with the patching.
No conflicts anymore, space check succeeded,…
But then, the patch failed because it could not find the $GRID_HOME/bin/odig file…
Which turned out to be Bug 20831754 – 184.108.40.206.7 BUNDLE PATCH INSTALLATION FAILS DUE TO MISSING ODIG (which is unfortunately not public)
And to solve it we simple copied it over from another node.
After all this, we were finally able to apply the PSU patch without further issues.
So, to recap, for the April 2015 PSU I had to:
- Manually rollback the OCW patch from the GI home (not necessary for the DB Home)
- Modify the bundle.xml file
- Copy over the odig file, which was wrongly removed by the rollback
But except for that, the patching went just fine.
NOTE: All the notes referred to can be found on Oracle Support