Application Engine program remains in processing status forever.

Process Scheduler Server Agent PSUNX is below the Log Space Threshold
The Log/Output Directory /xyz/psft/pt/8.50/appserv/prcs/xyz/log_output for the Process Scheduler Server Agent PSUNX
in server xyz.com for database xyz has 8 MB of free space.
This is below the disk threshold value of 10 MB in the Log Space Threshold
found in the Process Scheduler Configuration file. The system is suspending
the server agent until more disk becomes available.  Until then, no queued
process requests will be processed in this Process Scheduler Server Agent.

 

This happened because, one of the user modified the Application engine Process definition, override options as follows and ran the process using process scheduler.

-debug Y –trace 3

 

This resulted in a trace file that keeps growing very fast and consumes the entire available disk space on the appserver.

Removing the above parameters (-debug Y) and deleting the huge log file resolved the issue.

Following sql can be used to determine if there are any app engines that have this defined.

SELECT lastupdoprid,
       lastupddttm,
       prcsname,
       parmlist,
       descrlong
FROM   ps_prcsdefn
WHERE  parmlist <> ' '
       AND prcstype = 'Application Engine'
       AND prcsname <> 'PSCONQRS'; 

 

-debug Y should only be used from psae command line for interactive debugging. It should not be used in Process definition.

Scheduled Crystal Process remains in initiated status when the account is locked

If you have implemented password controls that locks the users account after x no of invalid attempts and the user has a crystal report scheduled, it makes crystal report process to remain in initiated status forever. It also generates an ever increasing log file, which has a potential of consuming entire available disk space and disrupting the other batch processes.

Following Trace files are generated.
CRW_XRFWIN_12345.log
pssqltrace[1].trc

It appears that Crystal Report  repeatedly calls the database sql statements and never comes out of it.

Steps to reproduce the issue.
1. Create a testid testps and assign roles PeopleTools and PeopleSoft User.
2. Schedule XRFWIN Crystal Report to run within next 5 minutes.
3. Update the testps user profile and lock the account.
4. Go back to process monitor and observe that process remains in initiated status.
Also if you go back to server and check the log_output folder for xrfwin you will see that log file and trace file size keeps on increasing.

Only workaround is to Run a SQL to detect this situation and cancel the process. Unlocking the user account also fixes the issue. The fix is targeted in next Tools release 8.51. We observed this behavior in only PT 8.50 and Crystal Report 2008 SP1. We are currently using patch 8.50.10.

SQL to determine the Processes in initiated status.

SELECT 'Processes in Initiated Status ',
       prcsinstance,
       oprid,
       prcsname,
       servernamerqst,
       servernamerun,
       rundttm,
       lastupddttm
FROM   psprcsrqst
WHERE  runstatus = 6
       AND ( SYSDATE - Cast(lastupddttm AS DATE) ) * 24 * 60 > 15 

SQL to determine Scheduled Processes and user account is locked.

SELECT b.prcsinstance,
       b.prcstype,
       b.prcsname,
       (SELECT e.descr
        FROM   ps_prcsdefn e
        WHERE  e.prcstype = b.prcstype
               AND e.prcsname = b.prcsname)     descr,
       b.oprid,
       (SELECT a.oprdefndesc
        FROM   psoprdefn a
        WHERE  a.oprid = b.oprid)               NAME,
       runcntlid,
       (SELECT f.qryname
               || ', '
               || f.descr
        FROM   ps_query_run_cntrl f
        WHERE  f.oprid = b.oprid
               AND f.run_cntl_id = b.runcntlid) qryname,
       recurname,
       runstatus,
       (SELECT xlatshortname
        FROM   psxlatitem c
        WHERE  c.fieldname = 'RUNSTATUS'
               AND c.fieldvalue = b.runstatus)  rundescr,
       diststatus,
       (SELECT d.xlatshortname
        FROM   psxlatitem d
        WHERE  d.fieldname = 'DISTSTATUS'
               AND d.fieldvalue = b.diststatus) distdescr,
       pt_retentiondays,
       rundttm,
       rqstdttm,
       b.lastupddttm,
       servernamerqst
FROM   psprcsrqst b,
       psoprdefn z
WHERE  b.runstatus IN ( 5 )
       AND b.oprid = z.oprid
       AND z.acctlock = 1
ORDER  BY rundttm DESC 

You are not authorized to run process type XRFWIN and process name Crystal. (65,8)

This happens if the user’s Primary Permission List or Process Profile Permission list is blank and user is trying to submit a Process request using Process Scheduler.

To fix this, make sure that user profile has valid primary permission list and Process profile permission list defined.

You can use the following sql to find out which users does not have primary permission list or process profile permission list.

Primary Permission List SQL

select * from psoprdefn where oprclass = ' ';

Process Profile Permission List SQL
select * from psoprdefn where prcsprflcls = ' ';

You get the following Message if Primary permission list is blank.

---------------------------

Windows Internet Explorer

---------------------------

You are not authorized to run process type XRFWIN and process name Crystal. (65,8) PRCSRQSTDLG_WRK.LOADPRCSRQSTDLGPB.FieldFormula Name:LaunchAndRunProcessRequest PCPC:97211 Statement:1113

Called from:PRCSRQSTDLG_WRK.LOADPRCSRQSTDLGPB.FieldFormula Name:LaunchProcessRequestDlg Statement:1133

Called from:PRCSRQSTDLG_WRK.LOADPRCSRQSTDLGPB.FieldChange Statement:1

You must be specifically authorized to run this process. Authorization is granted using the Process Definition table and Maintain Security

---------------------------

You get the following message, if the Process Profile Permission list is blank.

Windows Internet Explorer

---------------------------

Process Profile defined for user ID %2 is invalid (65,111)

The process profile assigned to the user ID is either not valid or blank. Update the User Profile for the user ID in the Maintain Security component with a valid process profile.

---------------------------

You also get this message in addition to above.

E-PRCS: PeopleTools 8.50 After DST change processes run one hour ahead/behind scheduled time

After the recent DST time change in USA on November 7th 2010, Scheduled processes are running one hour behind in our case. Our base time zone is EST. This happens due to a bug in the current Tools version less than 8.50.13.  This issue is fixed in 8.50.13 and 8.51.

Workaround

To resolve this issue, restart all your batch server domains in DEV/QA/PROD. Both NT and unix server domains are affected by this issue.

 

For more detail see  : E-PRCS: 8.50 After DST change processes run one hour ahead/behind scheduled time (Doc ID 1265111.1)

References NOTE:1163113.1 - E-PRCS Processes Are in "Queued" Status when Process Scheduler Resides in a Different Time Zone

Root Cause

This happens due to inability of process scheduler to detect change in sessiontimezone after the DST change and it’s use of the following SQL in 8.50

select ..... from PSPRCSQUE........where RUNDTTM <= SYSTIMESTAMP

By understanding the Oracle DBMS concept of SYSTIMESTAMP, the session time zone also takes effect when a TIMESTAMP value is converted to the TIMESTAMP WITH TIME ZONE or TIMESTAMP WITH LOCAL TIME ZONE datatype.

Therefore, for this conversion to take place : RUNDTTM <= SYSTIMESTAMP, the user's session timezone is returned and is not the same timezone as the database server.

Note: This only impacts those customers using Oracle and PT 8.50.12 or less. It does not impact non Oracle or lower tools release customers.