(NET.334): Tuxedo cannot find the service RunAeAsync2. Make sure the application server advertising this service is booted.

If your users are calling and telling you that App engine processes are remaining in initiated status for longer time ( > 5 minute) and you observe the following message. (Assuming PSAESRV is configured)

in the $PS_CFG_HOME/appserv/prcs/<domain>/LOGS folder for SCHDLR_MMYY.LOG you will see the message

(NET.334): Tuxedo cannot find the service RunAeAsync2.  Make sure the application server advertising this service is booted.

To see all files that has this message

grep NET.334 SCHDLR*.LOG | cut -d : -f 1 | sort -u

Please note the number at the end of RunAeAsync. It could be 1 thru n depending on number of PSAESRV configured. What this message means that the PSAESRV and number at the RunAeSync is not running for e.g. in this case PSAESRV that provides service RunAeSync2 is not running.

You will also see in the same folder for TUXLOG.MMDDYY the following message

010135.abc.com!restartsrv.16887.414156544.-2: server AESRV/2: CMDTUX_CAT:1684: ERROR: Application initialization failure

010135.abc.com!restartsrv.16887.414156544.-2: server AESRV/2: CMDTUX_CAT:587: INFO: Cannot restart server, scheduling for cleanup

grep CMDTUX_CAT:1684 TUXLOG.*

Resolution

What this means is that for some reason PSAESRV has crashed and Tuxedo restart process is not able to restart PSAESRV process.

You have 2 option 2 fix the issue.

Option 1 : Shutdown and Restart the server

Option 2: Manually start the PSAERV process using command line without shutting down and restarting the server.

Option 2 is preferred, if you have other PSAESRV processes running in process monitor that will be killed if you try to stop the process scheduler.

Start PSADMIN (cd $PS_HOME/appserv; ./psadmin)

Select 2 for process Scheduler

Select 1 for Administer a domain

Select 1for appropriate Process Scheduler Domain

Select 5 for TUXEDO command line (tmadmin)

Type

boot -g AESRV -i 1

(Change -I 1 to appropriate number for PSAESRV process that crashed Determined from the number at the end of RunAeSync).

Observed this issue in PeopleTools : 8.53.12 and Linux : Red Hat 6.3 64 bit.

Which Run Status values are considered "Active" by the Process Scheduler

Question:  Which Run Status values are considered "Active" by the Process Scheduler ?
Answer: The Process Scheduler considers: Initiated, Blocked, and Processing as active slots.

Application Run Controls versus Process Run Controls

Please see this on Oracle support Site : (http://support.oracle.com or http://supporthtml.oracle.com for non flash based browsers.

E-PRCS: Master Note: Process Scheduler [ID 1266607.1]

Application Run Controls versus Process Run Controls 
There are two types of "Run Control" stored in different database tables.

Process Run Control (created by the system)
Application Run Control ( created by the Application)

Process Run Control
Process Run Controls are created automatically by the system when a user creates a process request. The process run controls are stored in multiple tables. User ID and Run Control ID are keys to each tables:
PS_PRCSRUNCNTL - Stores user's language code and language option
PSPRCSRUNCNTLS - Stores server name and time zone
PS_PRCSRUNCNTLEOPT - Stores parameters for process request runs to email such as email address, subject, body text…etc.
PS_PRCSRUNCNTLDTL - Stores run control details such as output type, output format, output destination, folder name, email output option for per process type and process name
PS_PRCSRUNCNTLDIST - Stores distribution list

Application Run Control
Application Run Controls are created by user when he/she submits a process to run. It holds parameters required and specific to the process/program. When the Application Run Control is created, it triggers the system to create the Process Run Controls as well using the same Run Control ID as the key.

From the end-user perspective, there's no distinction between Process and Application Run Controls. The Process Run Control is assigned the exact same name/id as the Application Run Control.
However, it's important for application developers to distinguish between the two when creating new application process/program.
Process run control and application run control are stored in different tables.
PeopleTools delivers the table to store only the process run control.
Application developer is responsible for creating the application run control table to store the application run control information.
If a user creates a new application run control to run a report, then the corresponding process run control will be given the same name.

Run Control information is stored in two PeopleTools tables (at least):
PSPRCSRUNCNTLS (stores information from the Process Scheduler request panel) and
PS_PRCSRUNCNTL (stores the language code),
and at least one Application Run Control Table.

Note: From 8.4x PSPRCSRUNCNTLS is introduced which is used in place of PSPRCSRUNCNTL.