Fail to Apply WebLogic July 2023 CPU Patch using Opatch 13.9.4.2.13

 Oracle just released WebLogic CPU Patch on July 18th. More info below. 

Critical Patch Update (CPU) Patch Advisor for Oracle Fusion Middleware - Updated for July 2023 (Doc ID 2806740.2)

https://blogs.oracle.com/blogbypuneeth/post/oracle-weblogic-server-critical-patch-update-july-2023

The Patch requires newer version of Opatch 13.9.4.2.13. When trying to apply the 35601596 - WLS STACK PATCH BUNDLE 14.1.1.0.230713 (SPB)

or

35560771 - WLS PATCH SET UPDATE 14.1.1.0.230703

using 28186730 - OPatch 13.9.4.2.13

it fails with below error

[Jul 20, 2023 1:37:28 PM] [SEVERE]  Nextget could not process patch 35560771

[Jul 20, 2023 1:37:28 PM] [SEVERE]  oracle.glcm.opatch.common.api.install.HomeOperationsException: A failure occurred while processing patch: 35560771

                                    at com.oracle.cie.gdr.patch.HomeOperationsImpl.processPatch(HomeOperationsImpl.java:790)

                                    Caused by: com.oracle.cie.gdr.utils.LocalizedGdrException: GDR-70005: Failed to apply inventory patching to home.

                                    GDR-70005: A failure occurred while attempting to apply inventory patches to the home /opt/fedex/cdas/cporapp/wls

                                    GDR-70005: Contact customer support.

                                    at com.oracle.cie.gdr.patch.PatchHandlerImpl.processUpdatePatch(PatchHandlerImpl.java:1440)

                                    at com.oracle.cie.gdr.patch.PatchHandlerImpl.processUpdatePatch(PatchHandlerImpl.java:1345)

                                    at com.oracle.cie.gdr.patch.PatchHandlerImpl.processPatchUsingPaths(PatchHandlerImpl.java:771)

                                    at com.oracle.cie.gdr.patch.HomeOperationsImpl.processPatch(HomeOperationsImpl.java:786)

                                    ... 19 more

                                    Caused by: java.lang.NullPointerException

                                    at com.oracle.cie.gdr.patch.ComponentMetaDataPatchHandler.isPatchMetadataSuperseded(ComponentMetaDataPatchHandler.java:696)

                                    at com.oracle.cie.gdr.patch.ComponentMetaDataPatchHandler.processComponentUpdatePatches(ComponentMetaDataPatchHandler.java:581)

                                    at com.oracle.cie.gdr.patch.PatchHandlerImpl.processUpdatePatch(PatchHandlerImpl.java:1430)

Below MOS document mentions this issue.

July 2023 CPU: Oracle WebLogic Server 12.2.1.4 and 14.1.1 PSU/SPB Fail to Apply with "GDR-70005: Failed to apply inventory patching to home."(Doc ID 2962593.1)

The cause is currently under investigation. Bug 35619659 was filed with deveolopment.

Last updated: July 20, 2023 1:30pm CDT

At this time there is no confirmed solution. Please check back for further information.

Please do not apply this patch, till Oracle publishes the fix. 

Error executing pagelet.

The below error message appears when you try to enable an existing pagelet for a new Role or Permission List.

Error executing pagelet.

Detailed error description:

The current user has insufficient privileges to complete this operation.

This happens, when you are modifying an existing pagelet that is generated using Pagelet Wizard in Portal Interaction hub (aka Enterprise Portal 9.1) and added a new permission List or Role using Structure and Content.

The issue is Pagelet Wizard has its own security that is stored in the below table.

select * from PS_PTPPB_SECURITY where ptppb_pagelet_id = 'YOUR_PAGELET_ID';

You can add the new permission List, by Going thru Pagelet Wizard steps again.

Menu Navigation : Portal Administration -> Pagelet -> Pagelet Wizard

Once you add the permission List or Role using the Pagelet wizard, this error will go away.

I encountered this issue for a Pagelet based on Navigation Collection. This is not properly documented and is difficult to troubleshoot. 

Base64 encoding in PeopleSoft

 There is a sample peoplecode delivered in App engine program used by Search engine indexing. It is PTSF_GENFEED -> Setup -> Step04 – PeopleCode

   

Local JavaObject &header = CreateJavaObject("java.lang.String", &esuser | ":" | Decrypt("", &espass));

   Local JavaObject &oEncoder = CreateJavaObject("com.peoplesoft.tools.util.Base64");

   &auth = &oEncoder.encode(&header.getBytes());

   &auth = "Authorization: Basic " | &auth;

You can also find references of delivered peoplecode using base64 using below SQL

select * from pspcmtxt where upper(PCTEXT) like '%BASE64%';

Calling SSL or https Web Service using WebLogic : javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

When trying to call an external web service Using PeopleSoft Integration broker, it is giving the following message

<Jul 23, 2019 6:47:30 PM EDT> <Notice> <Stdout> <BEA-000000> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', RECV TLSv1.2 ALERT:  fatal, bad_certificate>

in Integration Broker Gateway Server, You will see

HttpTargetConnector:ExternalSystemContactException Received fatal alert: bad_certificate
Message Set : 158, Message ID : 10721 IOException: The host couldn't be resolved.
To further debug, use the following options in setEnv,sh

JAVA_OPTIONS_LINUX="-server -Xms1024m -Xmx1024m -Dtuxedo.jolt.LLEDeprecationWarnLevel=NONE -Djavax.net.debug=all -XX:MaxPermSize=256m -Dtoplink.xml.platform=oracle.toplink.platform.xml.jaxp.JAXPPlatform -Dcom.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0"

or
-Djavax.net.debug=ssl:handshake:verbose

Review this document

[Java SE] Training - How to Process javax.net.debug Diagnostic Output (Doc ID 2170565.1)

egrep -i "\* ClientHello|\* ServerHello|\* Certificate chain|\* CertificateRequest|\* ServerHelloDone|\* ClientKeyExchange|\* ServerKeyExchange|\* CertificateVerify|Change Cipher Spec|\* Finished|fatal|exception"  outputfile
egrep -v "^[0-9A-Z][0-9A-Z][0-9A-Z][0-9A-Z]:" outputfile
Resolution
This issue is caused by an expired SSL Certificate (PrivateKeyEntry) entry in pskey keystore located at

$PS_CFG_HOME/webserv/peoplesoft/piaconfig/keystore

To list the entries

cd $PS_CFG_HOME/webserv/peoplesoft/piaconfig/keystore
ls -lrt
keytool -list -keystore pskey -storepass xxx -alias xxx -v
or to list every thing
keytool -list -keystore pskey -storepass xxx -v > output.txt

Review the output for expired entry.

This can also happen, if you have a PrivateKeyEntry that is signed by a Root CA which is not present in the target Web Service Provider keystore.

Solution: Delete the expired or invalid PrivateKeyentry and restart the webserver

keytool -delete -keystore pskey -storepass xxx -alias xxx

This has resolved the issue.

Other issue is that if you do not have RootCA of the target Web Service provider SSL Certificate in your keystore then you will need to import it in your keystore.

errorLog

javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
 at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
 at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1972)
 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1089)
 at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1325)
 at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1352)
 at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1336)
 at psft.pt8.pshttp.https.HttpsClient.doConnect(HttpsClient.java:246)
 at sun.net.NetworkClient.openServer(NetworkClient.java:136)
 at psft.pt8.pshttp.https.HttpClient.openServer(HttpClient.java:543)
 at psft.pt8.pshttp.https.HttpClient.<init>(HttpClient.java:364)
 at psft.pt8.pshttp.https.HttpsClient.<init>(HttpsClient.java:76)
 at psft.pt8.pshttp.https.HttpsClient.newClient(HttpsClient.java:132)
 at psft.pt8.pshttp.https.HttpsClient.newClient(HttpsClient.java:100)
 at psft.pt8.pshttp.https.HttpsURLConnection.connect(HttpsURLConnection.java:468)
 at psft.pt8.pshttp.PSHttp.getResponseCode(PSHttp.java:423)
 at com.peoplesoft.pt.integrationgateway.targetconnector.HttpTargetConnector.send(HttpTargetConnector.java:858)
 at com.peoplesoft.pt.integrationgateway.service.BasicConnectorInvocator.execute(BasicConnectorInvocator.java:131)
 at com.peoplesoft.pt.integrationgateway.framework.GatewayManager.invokeService(GatewayManager.java:148)
 at com.peoplesoft.pt.integrationgateway.framework.GatewayManager.connect(GatewayManager.java:192)
 at com.peoplesoft.pt.integrationgateway.listeningconnector.PeopleSoftListeningConnector.doPost(PeopleSoftListeningConnector.java:186)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:751)
 at com.peoplesoft.pt.integrationgateway.listeningconnector.PeopleSoftListeningConnector.service(PeopleSoftListeningConnector.java:87)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
 at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
 at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
 at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)
 at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346)
 at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
 at com.peoplesoft.pt.integrationgateway.common.IBFilter.doFilter(IBFilter.java:84)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
 at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3456)
 at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3422)
 at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:323)
 at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
 at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
 at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2280)
 at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2196)
 at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
 at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1632)
 at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:256)
 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)


PIA_stderr

com.peoplesoft.pt.integrationgateway.common.ExternalSystemContactException: HttpTargetConnector:ExternalSystemContactException Received fatal alert: bad_certificate
 at com.peoplesoft.pt.integrationgateway.targetconnector.HttpTargetConnector.send(HttpTargetConnector.java:1296)
 at com.peoplesoft.pt.integrationgateway.service.BasicConnectorInvocator.execute(BasicConnectorInvocator.java:131)
 at com.peoplesoft.pt.integrationgateway.framework.GatewayManager.invokeService(GatewayManager.java:148)
 at com.peoplesoft.pt.integrationgateway.framework.GatewayManager.connect(GatewayManager.java:192)
 at com.peoplesoft.pt.integrationgateway.listeningconnector.PeopleSoftListeningConnector.doPost(PeopleSoftListeningConnector.java:186)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:751)
 at com.peoplesoft.pt.integrationgateway.listeningconnector.PeopleSoftListeningConnector.service(PeopleSoftListeningConnector.java:87)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
 at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
 at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
 at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)
 at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346)
 at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
 at com.peoplesoft.pt.integrationgateway.common.IBFilter.doFilter(IBFilter.java:84)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
 at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3456)
 at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3422)
 at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:323)
 at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
 at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
 at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2280)
 at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2196)
 at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
 at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1632)
 at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:256)
 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)

Automate download of patches from Oracle Support

If you need to download the patches or software from My Oracle Support, you can use the below utility available at Github

getMOSPatch

The Github site has instructions on how to use this utility.

I have found some additional challenges when using this utility if you are behind a proxy.

• The way patch download process works is that you connect to MOS (support.oracle.com) using HTTPS for authentication and use http for actual file download. This requires you use the below command line argument to support for both http and https. The Github site readme only specifies using https, which works for authentication but fails to download the file.

java -Dhttp.proxyHost=aaa.com -Dhttp.proxyPort=80 -Dhttps.proxyHost=aaa.com -Dhttps.proxyPort=80 -jar getMOSPatch.jar MOSUser=aaa@aaa.com MOSPass=aaa platform=226P download=all patch=26557153

• I have also found that the code uses (Line 144) conn.setReadTimeout(60000); which is 1 minute. I have found that this may not be sufficient for the busy sites and it fails with the following message.  To avoid this issue, I simply changed it to conn.setReadTimeout(6000000); – 100 minutes and used the below to recompile the jar file.

Unzip the current jar to extract META-INF\MANIFEST.MF file to the folder where you store the java file. Also change the path to java. The below path are for Windows 8 and JDK 8.

"C:\Program Files\Java\jdk8\bin\javac.exe" -bootclasspath "C:\Program Files\Java\jdk8\jre\lib\rt.jar" -source 1.6 -target 1.6 getMOSPatch.java

"C:\Program Files\Java\jdk8\bin\jar.exe" cvmf META-INF/MANIFEST.MF getMOSPatch.jar getMOSPatch*.class

del *.class

Error Message if we do not change the setReadTimeout

Processing patch 26557153 for Linux x86-64 and applying regexp .* to the filenames:
  1 - p26557153_170151_Linux-x86-64.zip
  Enter Comma separated files to download: all
  All files will be downloadad because download=all was specified.

Downloading all selected files:
  java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.io.BufferedInputStream.fill(Unknown Source)
        at java.io.BufferedInputStream.read1(Unknown Source)
        at java.io.BufferedInputStream.read(Unknown Source)
        at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
        at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
        at java.net.HttpURLConnection.getResponseCode(Unknown Source)
        at getMOSPatch.getHttpInputStream(getMOSPatch.java:146)
        at getMOSPatch.DownloadFile(getMOSPatch.java:171)
        at getMOSPatch.DownloadAllFIles(getMOSPatch.java:498)
        at getMOSPatch.main(getMOSPatch.java:540)

Applying latest Oracle Patch 22502456 Database Patch Set Update : 11.2.0.4.160419 for Oracle Run Time Client errors out

When trying to apply patch 22502456 to Oracle Runtime client using latest OPatch version 11.2.0.3.12
it fails with the following error. This patch is also included in combo patch : 22738777 - COMBO OF OJVM COMPONENT 11.2.0.4.160419 DB PSU + DB PSU 11.2.0.4.1604 (APR2016)


Verifying environment and performing prerequisite checks...
Skip patch 20760982 from list of patches to apply: This patch is not needed.
UtilSession failed: null
Log file location:
OPatch failed with error code 73
Log file error shows this.
[May 9, 2016 3:46:33 PM]     OUI-67073:UtilSession failed: null
[May 9, 2016 3:46:33 PM]     Finishing UtilSession at Mon May 09 15:46:33 EDT 2016
[May 9, 2016 3:46:33 PM]     Log file location: /opt/oracle/product/11.2.0.4/cfgtoollogs/opatch/opatch2016-05-09_15-46-25PM_1.log
[May 9, 2016 3:46:33 PM]     Stack Description: java.lang.NullPointerException
[May 9, 2016 3:46:33 PM]     StackTrace: java.lang.StringBuffer.<init>(StringBuffer.java:104)
[May 9, 2016 3:46:33 PM]     StackTrace: oracle.opatch.UtilSession.process(UtilSession.java:337)
[May 9, 2016 3:46:33 PM]     StackTrace: oracle.opatch.OPatchSession.main(OPatchSession.java:2580)
[May 9, 2016 3:46:33 PM]     StackTrace: oracle.opatch.OPatch.main(OPatch.java:634)


Oracle support is able to reproduce this issue however currently no workaround is provided.
They have internally 2 bugs opened to address this issue.


Bug 22932939 -- OPATCH FAILS TO PATCH RUNTIME CLIENT ENVIRONMENTS
Bug 22740194 - INSTALLATION OF 11.2.0.4.7 AND LATER PSU FAILS IN RUNTIME CLIENT ENVIRONMENT

Portal Home Page Displays Error : Error getting content for remote pagelet.

When you are logging on to a Portal Home Page which displays Remote pagelets, You get this error randomly.

Error Message: Error getting content

When you click on "Detailed Error Description", You will get this

Unable to get document

Error occurred while accessing target page.

To see the further details, Please log in to your webserver and open the following log files. If you have more than 1 web server you may have to go to each web server to see the content.


Log File Location : $PS_CFG_HOME/webserv/peoplesoft/servers/PIA/logs (This is for PeopleTools 8.53 and higher).
Log File Names are below
PIA_servlets0.log.0
PIA_servlets1.log.0
PIA_servlets2.log.0


If you open any one of these log file, You will see the following


2000-01-07T11:47:21.185         9259 163517125      unknown SEVERE psft.pt8.psp logError Error for User: abc Error Message: Unable to get document:  Requested URL: https://abc.com Target URL:  Error reading from server


Resolution:
If you do not have security issue or single signon issue, Please check the default HTTP Header size allowed by your Load balancer that you use for multiple webservers.  In Case of Cisco ACE Loadbalancer default is : 4096 bytes. Increasing it to 32768 helps in resolving this issue.


For Cisco Ace this parameter is :

set header-maxparse-length 32768

If you use a different Load balancer for e.g. F5 LTM then I believe the default is already 32768.  You can contact your Load balancer vendor to determine the default HTTP Header size.