When trying to upload about 7000 rows using ExcelToCI for a custom component, users are getting the following error.
Error occurred in routine sendSOAPRequest_SubmitToDB:
Error:
Number: -2147012894
Description: The operation timed out
After doing my investigation, checking the web server and appserver access log I found the following issues, and here is what i did to resolve the issue. This is not documented anywhere. Please create a resolution if possible to help others.
1. Found that ExcelToCI uses SERVERXMLHTTP MSXML 6.0 object to send the SOAP request to the webserver,
Set xHTTP = CreateObject("MSXML2.SERVERXMLHTTP.6.0")
This request has a default 30 second timeout for receiving a packet of response data from the target server,
See link : http://msdn.microsoft.com/en-us/library/ms760403 for more details on timeouts.
2. After checking the webserver access log (PIA_Access.log), determined that some of the POST request are taking about 59 seconds to complete and therefore we were receiving the error.
To resolve this issue, change the VB Macro code in ExcelToCI spreadsheet as follows
=======================================================
Add this line before xHTTP.send xDoc.xml in StagingAndSubmission Module under function sendSOAPRequest_SubmitToDB
Dim lResolve, lConnect, lSend, lReceive As Long
lResolve = 60 * CLng(1000)
lConnect = 90 * CLng(1000)
lSend = 90 * CLng(1000)
lReceive = 120 * CLng(1000)
xHTTP.setTimeouts lResolve, lConnect, lSend, lReceive
=======================================================
After changing this value, the ExcelToCI was still failing. Once again I checked the web Server and App server log and found that after loading about 50 to 60 rows, request is taking more than 5 minutes to respond and appserver is killing the appserv thread and therefore error was received saying that The Operation time out.
I had two options
1. Increase the XMLHTTP timeout and also increase the appserver timeout or find out why a request is taking more than 5 minutes to complete. Used Precise i3 performance monitoring tool and also the live Oracle
session and determined the following issue.
1. CI was firing sql to fetch location code description using a view that was not correctly joined with other large table using location code.
2. A Save Edit Peoplecode was written at Level 2 to fetch some data from the Oracle table using SQL Exec. This was causing this sql to fire 160,000 times as it was firing for every row in the scroll as new rows are being added.
Following improved the performance.
1. Remove the related display field from the page. Alternative option is to tune the sql for related display field.
2. Moved the save edit code to field edit, so that it fires only for the newly inserted rows. Other option is to write conditional logic to see if row is changed.
After these, 2 changes, the process finished inserting 7100 rows in 23 minutes. Only two request exceeded default timeout. One took 32 and other 35 seconds to respond.
I have also noticed, that if Chunking factor is reduced to 1 to have smaller number of rows processed, PSAPPSRV is restarting due to recycle count of 5000 being reached, and this results in a login Error and HTML (invalid XML) data is being sent to the Excel. Is this the expected behavior? Only solution to this issue is to increase the recycle count temporarily and change it back to original. Fortunately this parameter is dynamic and does not require restart of the appserver.
