SORT processing order for record handling, exits, statements, and options

The below diagram illustrates the processing order for record handling, exits, statements, and options.

Use this diagram with the text following it to understand the order DFSORT uses to run your job.

 

1. DFSORT first checks whether you supplied a SORTIN data set for SORT and COPY jobs or SORTINnn data sets for MERGE jobs. If so, DFSORT reads the input records from them.
   • If no SORTIN data set is present for a SORT or COPY job, you must use an E15 user exit to insert all the records. (This is also true if you invoke DFSORT from a program with the address of an E15 user exit in the parameter list, because SORTIN will be ignored.) DFSORT can use a COBOL E15 routine if you specified the E15 user exit in the MODS statement.
   • If no SORTINnn data sets are present for a MERGE job, you must use an E32 user exit to insert all the records.
2. If input records for SORT or COPY jobs are read from a SORTIN data set, DFSORT performs processing specified with the SKIPREC option. DFSORT deletes records until the SKIPREC count is satisfied. Eliminating records before a SORT or COPY gives better performance.
3. If the input records for a SORT or COPY job are read from a SORTIN data set, DFSORT checks whether you specified an E15 user exit. If so, DFSORT transfers control to the user exit routine. You can use a COBOL E15 routine if the E15 user exit is specified in the MODS statement. The E15 routine can insert, delete, or reformat records.
4. DFSORT performs processing specified on an INCLUDE or OMIT statement. If you used an E15 user exit routine to reformat the records, the INCLUDE/OMIT fields you specify must apply to the current format rather than to the original format. If you use the INCLUDE or OMIT statements to delete unnecessary records before SORT, MERGE, or COPY processing, your jobs run more efficiently.
5. For SORT or COPY jobs, DFSORT performs processing specified with the STOPAFT option. Record input stops after the maximum number of records (n) you specify have been accepted. DFSORT accepts records for processing if they are:
   • Read from SORTIN or inserted by E15
   • Not deleted by SKIPREC
   • Not deleted by E15
   • Not deleted by an INCLUDE or OMIT statement.
6. DFSORT performs processing specified in an INREC statement. Data records are parsed, edited and reformatted according to the options specified. If you reformatted the records before this step, the INREC fields you specify must apply to the current format rather than to the original format.
7. DFSORT performs processing specified in the SORT, MERGE, or OPTION COPY statement.
   • For SORT, all input records are processed before any output record is processed.
   • For COPY or MERGE, an output record is processed after an input record is processed.
   • For SORT or MERGE, if a SUM statement is present, DFSORT processes it during the SORT or MERGE processing. DFSORT summarizes the records and deletes duplicates. If you reformatted the records before this step, the SORT or MERGE and SUM fields you specify must apply to the current format rather than to the original format.
8. DFSORT performs processing specified in an OUTREC statement. Data records are parsed, edited and reformatted according to the options specified. If you reformatted the records before this step, the OUTREC fields you specify must apply to the current format rather than to the original format.
9. If an E35 user exit is present, DFSORT transfers control to your user exit routine after all statement processing (except OUTFIL) is completed. If you reformatted the records, the E35 user exit receives the records in the current format rather than in the original format. You can use a COBOL E35 routine if you specify the E35 user exit in the MODS statement. You can use the E35 exit routine to add, delete, or reformat records.

   If SORTOUT and OUTFIL data sets are not present, the E35 exit must dispose of all the records because DFSORT treats these records as deleted. (This is also true if you do not specify OUTFIL data sets and DFSORT is invoked with the address of an E35 user exit in the parameter list, because SORTOUT will be ignored.)

10. DFSORT writes your records to the SORTOUT data set, if present.
11. DFSORT performs processing specified in one or more OUTFIL statements, if present:
    • DFSORT performs processing specified with the STARTREC, SAMPLE, and ENDREC options. Record input for the OUTFIL data sets starts with the record indicated by STARTREC, ends with the record indicated by ENDREC, and is sampled according to the records indicated by SAMPLE.
    • DFSORT performs processing specified with the INCLUDE, OMIT, or SAVE option. Records are included or omitted from the OUTFIL data sets according to the criteria specified.
    • DFSORT performs processing specified with the ACCEPT option. Record processing ends when the ACCEPT limit is reached.
    • DFSORT performs processing specified with the PARSE, OUTREC (or BUILD), OVERLAY, FINDREP, IFTHEN, FTOV, VTOF (or CONVERT), VLFILL, VLTRIM, VLTRAIL and REPEAT options. Data records are parsed, edited, reformatted, converted and repeated according to the options specified.
    • DFSORT performs processing specified with the LINES, HEADER1, TRAILER1, HEADER2, TRAILER2, SECTIONS, NODETAIL, BLKCCH1, BLKCCH2, BLKCCT1 and REMOVECC options. Data records are reformatted and report records are generated for the OUTFIL data sets.
    • DFSORT performs SPLIT, SPLITBY, or SPLIT1R processing. Records are distributed among the OUTFIL data sets as evenly as possible.
12. DFSORT writes your OUTFIL records to the appropriate OUTFIL data sets.

RMF post processor report to print CPU busy, MVS busy, 4HRA

The below JCL prints CPU busy, MVS busy and 4HRA

//RMFSORT EXEC PGM=SORT
//SORTIN DD DISP=SHR,DSN=SMF file...
//SORTOUT DD DSN=TEMP.DTA,
// DISP=(,PASS,DELETE),UNIT=SYSDA,SPACE=(CYL,(25,25),RLSE)
//SORTWK01 DD DISP=(NEW,DELETE),UNIT=SYSDA,SPACE=(CYL,(10))
//SORTWK02 DD DISP=(NEW,DELETE),UNIT=SYSDA,SPACE=(CYL,(10))
//SORTWK03 DD DISP=(NEW,DELETE),UNIT=SYSDA,SPACE=(CYL,(10))
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSIN DD *
SORT FIELDS=(11,4,CH,A,7,4,CH,A,15,4,CH,A),EQUALS
MODS E15=(ERBPPE15,36000,,N),E35=(ERBPPE35,3000,,N)
//*
//RMFPOST EXEC PGM=ERBRMFPP,REGION=5M
//SYSABEND DD SYSOUT=*
//MFPMSGDS DD SYSOUT=*
//MFPINPUT DD DSN=TEMP.DTA,
// DISP=(OLD,DELETE)
//PPSUM001 DD SYSOUT=*
//PPPLT001 DD SYSOUT=*
//SYSIN DD *
SYSOUT(X)
OVERVIEW(REPORT)
OVW(4HRA(LACS))
OVW(CPUBSY(CPUBSY))
OVW(MVSBSY(MVSBSY))
/*

CA Detector Batch Reporting

The batch reporting facility lets you create reports in batch from a VSAM collection data store. These reports can then be reviewed online or in print.

Below is sample JCL to generate report for Plan Activity Report

//STEP01 EXEC PGM=PTLDRIVM,PARM='EP=PDTBATCC'
//STEPLIB DD DISP=SHR,DSN=high level qualifier.CDBALOAD
// DD DISP=SHR,DSN=high level qualifier.SDSNLOAD
// DD DISP=SHR,DSN=high level qualifier.SDSNEXIT
//PTILIB DD DISP=SHR,DSN=high level qualifier.CDBALOAD
// DD DISP=SHR,DSN=high level qualifier.SDSNLOAD
// DD DISP=SHR,DSN=high level qualifier.SDSNEXIT
//PTIPARM DD DISP=SHR,DSN=high level qualifier.CDBAPARM
//SYSPRINT DD DSN=output dataset
// DISP=(NEW,CATLG,DELETE),
// DCB=(MODDSCB,RECFM=FB,BLKSIZE=0,LRECL=133,DSORG=PS),
// DATACLAS=DCTLARGE
//SYSOUT DD SYSOUT=*
//SRTMSG DD SYSOUT=*
//SYSREC DD DUMMY
//SYSIN DD *
SSID = (ssid)
DATASTORE = (datastor)
VCAT = (datastore dataset name high level qualifier)
STARTDATE = (09/13/2020)
STARTTIME = (21:00)
ENDDATE = (09/13/2020)
ENDTIME = (23:59)
PRINT = (Y)
NORMALIZE = (Y)
CONTENT = (TABLE)
FORMAT = (SUMMARY)
REPORT = (PLAN)
/*

Below is the control card for Key Plan Activity Report—Summary Format

//SYSIN DD *
SSID = (ssid)
DATASTORE = (datastor)
VCAT = (datastore dataset name high level qualifier)
STARTDATE = (03/30/2021)
STARTTIME = (00:00)
ENDDATE = (03/31/2021)
ENDTIME = (00:00)
PRINT = (Y)
NORMALIZE = (Y)
CONTENT = (TABLE)
FORMAT = (SUMMARY)
REPORT = (KEYPLAN)
KEY = (USER)
PLAN = (plan name)
//*

Below is the control card for SQL Error Report—Detail Format

//SYSIN DD *
SSID = (ssid)
DATASTORE = (datastor)
VCAT = (datastore dataset name high level qualifier)
STARTDATE = (07/01/2021)
STARTTIME = (00:00)
ENDDATE = (07/01/2021)
ENDTIME = (23:59)
PRINT = (Y)
NORMALIZE = (Y)
CONTENT = (TEXT,TABLE,INDEX)
FORMAT = (DETAIL)
REPORT = (SQLE)
SQLCODE = (551)
/*

When generating a batch report, use the following parameters to filter the report or unloaded data to a specific set of data, according to its source:

COLLID=collectionID
CONNID=threadconnID
CORRID=threadcorrID
CNAME=connectionname
CTYPE=connectiontype
EUID=userID
EUTX=transactionID
EUWN=workstationID
PACKAGE=name
PLAN=name
PROGRAM=name
SQLCODE=codes
USER=userid

CICS File Openstatus Enablestatus

 In CICS, the open status of a VSAM file can either be OPEN or CLOSED, and this is indicated by the OPENSTATUS attribute in the file entry in the CICS file control table. If CICS closes a file, it is also de-allocated so that it can be processed by batch jobs. A file also has an enabled status, indicated by the ENABLESTATUS attribute, which can be either ENABLED, DISABLED, or UNENABLED. The enabled status determines whether or not a CICS application program request to a VSAM file can be attempted. If a file is disabled or unenabled, CICS fails the application program request without even attempting to send the request to VSAM. If the file is enabled, CICS issues the VSAM request on behalf of the application program.

The following are the most common combinations of these two states:

OPEN, ENABLED

CLOSED, ENABLED

OPEN, DISABLED

CLOSED, DISABLED

CLOSED, UNENABLED

In the first case (OPEN, ENABLED), the file is open and available to CICS application programs and if a batch job tries to access the file, it generally fails.

In the second case (CLOSED, ENABLED), the file is closed and not allocated, but it is available to CICS programs. In this state, if your application program issues a file access request, CICS attempts to allocate then open the file. If CICS cannot allocate and open the file, it is most probably because it is allocated to a batch job, and the application program request fails. Also in this case, as long as the file is closed, it is available for access by batch jobs.

In the last three cases in the above list (OPEN, DISABLED; CLOSED, DISABLED; and CLOSED, UNENABLED), CICS fails all file control requests without even attempting them because the file is not enabled. The key difference in these three cases is that in one the file is open and in the other two it is closed. In the open case, it is allocated to CICS so that batch jobs that attempt to access the file will probably fail. In the closed cases , it is not allocated to CICS, and is available for access by batch jobs.

Note that when a file is closed to CICS, and therefore de-allocated, the file can even be deleted by a batch job.

For the most part, the DISABLED and UNENABLED states are the same. Both states prevent an application program from accessing the file. However, there is one important difference that you must take into consideration when changing the state of a file.

The only way that a file becomes unenabled is when you close the file without altering its enabled status. You can do this using the CICS BAC batch request utility, or by using the EXEC CICS SET FILE command or the CEMT SET FILE transaction. For example, if FILEA is currently open and enabled, and you issue the CEMT command SET FILE(FILEA) CLOSED, the resulting file attributes are set to CLOSED and UNENABLED. However, if the file is currently open and disabled, the same command results in the file state being set to CLOSED and DISABLED.

The difference between a disabled and an unenabled file is in how the file state is set when a closed file is subsequently opened. If a file is currently closed and disabled, and a request is made through the CICS BAC batch request utility, CICS system programming interface (SPI), or CICS CEMT command to open the file without altering its enable status attribute, the file is opened (assuming there is nothing to keep it from being opened), but its enable status remains DISABLED. On the other hand, if a file is unenabled, an open request that does not also specify an enabled status results in the file state being set to OPEN and ENABLED.

If you issue a request that alters the enable status of a file explicitly, the file ENABLESTATUS attribute is changed regardless of whether the current state is ENABLED, DISABLED, or UNENABLED.

4HRA resets to 1 MSU after IPL

4HRA value resets to 1 MSU after IPL.  This can be seen in RMF Monitor III.

                         RMF V2R4   CPC Capacity                   Line 1 of 16 

 Command ===>                                                  Scroll ===> CSR  

                                                                                

 Samples: 100     System: C101  Date: 04/29/21  Time: 04.33.20  Range: 100   Sec

                                                                                

 Partition:   C101       3907 Model W02                         Boost: N        

 CPC Capacity:     280   Weight % of Max: 89.6   4h Avg:    1                   

 Image Capacity:   280   WLM Capping %:    0.0   4h Max:   44   Group:   N/A    

 MT Mode IIP:      N/A   C101 % IIP:       N/A   AbsMSUCap: N   Limit:   N/A    

                                                                                

 Partition  --- MSU ---  Cap     Proc    Logical Util %   - Physical Util % -   

              Def   Act  Def      Num    Effect   Total   LPAR  Effect  Total   

                                                                                

 *CP                              8.0                      0.5    14.0   14.5   

 C101           0    36  N N N    2.0      13.1    13.2    0.1    13.1   13.2   

 C102           0     1  N N N    2.0       0.3     0.3    0.0     0.3    0.3   

 C103           0     2  N N N    2.0       0.7     0.7    0.0     0.7    0.7   

 C104           0     0  N N N    2.0       0.0     0.0    0.0     0.0    0.0   

 PHYSICAL                                                  0.3            0.3   

                                                                                

 *IFL                             0.0                      0.0     0.0    0.0   

 PHYSICAL                                                  0.0            0.0   

                                                                                

 *IIP                             8.0                      0.0     0.7    0.8 

 SYSVIEW 16.0 C101 - CPCLPAR, CPC LPAR Capacity Information  2021/04/29 05:02:40

 Command ====>                                                 Scroll *===> PAGE

 GSVX005I Beginning of data rows ---------------- Lvl 4 Row 1-14/49 Col 1-79/118

 Partition       C101  Avg Weight     895  Group n/a       4h Avg         6   2%

 CPC Capacity     280  WLM Capping%   n/a  Limit n/a       4h Max        73  27%

 Image Capacity   280                                      4h Unused    266  98%

 -------------------------------------------------------------------------------

 Cmd  Intv Time    SrvUnits SrvUTime Unused 1hMSU Pct% ...25...50...75..100     

 ____    * TOTAL   28109385 03:58:10    265     7   3%                          

 ____    1 05:02    2535136 00:03:10    224    48  18%                          

 ____    2 04:59    6095872 00:05:00    199    73  27%                          

 ____    3 04:54    4664704 00:05:00    216    56  21%                          

 ____    4 04:49    3440832 00:05:00    231    41  15%                          

 ____    5 04:44    4042528 00:05:00    223    49  18%                          

 ____    6 04:39    3595040 00:05:00    229    43  16%                          

 ____    7 04:34    3735232 00:05:00    227    45  17%                          

 ____    8 04:29          1 00:04:59    280                                     

 ____    9 04:24          1 00:04:59    280                                     

 ____   10 04:19          1 00:04:59    280                                     

 ____   11 04:14          1 00:04:59    280                                     

 ____   12 04:09          1 00:04:59    280                                     

 ____   13 04:04          1 00:04:59    280                                     

 ____   14 03:59          1 00:04:59    280                                     

 ____   15 03:54          1 00:04:59    280                                     

 ____   16 03:49          1 00:04:59    280                                     

 ____   17 03:44          1 00:04:59    280                                     

 ____   18 03:39          1 00:04:59    280                                     

 ____   19 03:34          1 00:04:59    280                                     

 ____   20 03:29          1 00:04:59    280                                     

 ____   21 03:24          1 00:04:59    280                                     

 ____   22 03:19          1 00:04:59    280                                     

 ____   23 03:14          1 00:04:59    280                                     

 ____   24 03:09          1 00:04:59    280                                     

 ____   25 03:04          1 00:04:59    280                                     

 ____   26 02:59          1 00:04:59    280                                     

 ____   27 02:54          1 00:04:59    280                                     

 ____   28 02:49          1 00:04:59    280                                     

 ____   29 02:44          1 00:04:59    280                                     

 ____   30 02:39          1 00:04:59    280                                     

 ____   31 02:34          1 00:04:59    280                                     

 ____   32 02:29          1 00:04:59    280                                     

 ____   33 02:24          1 00:04:59    280                                     

 ____   34 02:19          1 00:04:59    280                                     

 ____   35 02:14          1 00:04:59    280                                     

 ____   36 02:09          1 00:04:59    280                                     

 ____   37 02:04          1 00:04:59    280                                     

 ____   38 01:59          1 00:04:59    280                                     

 ____   39 01:54          1 00:04:59    280                                     

 ____   40 01:49          1 00:04:59    280                                     

 ____   41 01:44          1 00:04:59    280                                     

 ____   42 01:39          1 00:04:59    280                                     

 ____   43 01:34          1 00:04:59    280                                     

 ____   44 01:29          1 00:04:59    280                                     

 ____   45 01:24          1 00:04:59    280                                     

 ____   46 01:19          1 00:04:59    280                                     

 ____   47 01:14          1 00:04:59    280                                     

 ____   48 01:09          1 00:04:59    280                                     

 ********************************* End of Data *********************************

How to extract CICS transaction log from CA Sysview

The Below JCL would pull the latest 500 transactions from CA sysview for CICS and put in MYID.PRN(M1). You need to allocate the MYID.PRN PDS before running this job 

//STEP1 EXEC PGM=GSVXBAT,PARM='LINECNT=66,SHOWINP=NO'          

//SYSPRINT DD SYSOUT=*,DCB=(LRECL=81,BLKSIZE=8100,RECFM=FBA)   

//SYSOUT   DD SYSOUT=*,DCB=(LRECL=81,BLKSIZE=8100,RECFM=FBA)   

//SYSIN DD *                                                   

COMMAND=(SET COPYDSNAME 'MYID.PRN')                         

COMMAND=(PRINT OPEN M1)                                        

COMMAND=(CTRANLOG LINES  500 LIMIT  500)                       

COMMAND=(PRINT ALL)                                            

COMMAND=(PRINT CLOSE)                                          

//*                                                            

How PR/SM calculates the number of Veritical High and Veritical Medium processors in HiperDispatch mode

Assume that we have 32 shared physical CPs and LPAR weight is 28% and HiperDispatch is turned on.

If the decimal number of the processor guarantee is ≥ 0.5 = the vertical high processors will be the integer number and there will be 1 vertical medium

𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐨𝐫 𝐆𝐮𝐚𝐫𝐚𝐧𝐭𝐞𝐞 = 𝟑𝟐 𝐱 𝟎.𝟐𝟖 = 𝟖.𝟗𝟔 = 𝟖 𝐕𝐇 + 𝟏𝐕M

If the decimal number of the processor guarantee is < 0.5 = the vertical high processors will be the integer number minus one and there will be 2 vertical mediums:

𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐨𝐫 𝐆𝐮𝐚𝐫𝐚𝐧𝐭𝐞𝐞 = 𝟑𝟐 𝐱 𝟎.𝟐𝟔 = 𝟖.𝟑𝟐 = 𝟕 𝐕𝐇 + 𝟐𝐕M

According to the IBM Best Practices recommendation, the number of logical processors should be the quantity required to satisfy the LPAR share/processor guarantee, i.e., the Vertical High and Vertical Medium, and 1 to 2 additional processors (Vertical Low).

The LPAR time slice is sensitive to the number of logical CPs and having more logicals may drive your time slice to a smaller interval for your vertical medium and vertical low logical processors.”

Based on IBM manual: the sum of all logical shared processors should not be more than triple the number of physical processors. Otherwise, the LPAR management time to reassign the PUs to the logical CPUs can increase to an unacceptable level.

Work will run most efficiently if you run within your defined weight, using vertical highs and vertical mediums to support the workload and avoid use of vertical lows except for occasional workload spikes. If the workload in the LPAR relies upon vertical lows for throughput you may want to change the weight to match actual usage.