𝗧𝗵𝗲 𝗗𝗮𝘆 𝗮 𝗠𝗮𝗶𝗻𝗳𝗿𝗮𝗺𝗲 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗟𝗣𝗔𝗥 𝗔𝗹𝗺𝗼𝘀𝘁 𝗖𝗮𝗺𝗲 𝘁𝗼 𝗮 𝗛𝗮𝗹𝘁

It was one of those bad days in production.

 

A mainframe production LPAR had almost ground to a halt.

 

Complaints started coming from every direction. “Nobody can log in to the mainframe.”

 

And then our hero stepped in. There was just one problem. He couldn’t log in either.

 

Fortunately, someone was already logged on. Through a screen-sharing session, he gained visibility into the struggling LPAR and immediately opened the z/OS monitoring tool.

 

He started checking the jobs and started tasks running on the system.

 

One address space stood out.

 

DB2 DIST was consuming an enormous amount of CPU.

 

Something coming through DDF was putting tremendous pressure on the system.

 

Emergency Brake:  

 

The immediate priority was simple: stop DB2 DIST from overwhelming the LPAR.

 

DB2 DIST was temporarily moved to a service class called HOLDON, configured with a Discretionary goal and associated with a resource group having a maximum Service Unit limit of 1.

 

DB2 DIST was effectively frozen from taking additional workload.

 

Finding the Real Culprit

 

He moved to the DB2 monitor. There, the picture became clearer.

 

Several DDF threads were consuming large amounts of CPU, and all of them were associated with the same user.

 

He terminated those threads.

 

And almost immediately... the mainframe came back to life.

 

Workloads began running normally again.

 

But killing the threads only treated the symptom. The next question was: Why had so many expensive DDF threads appeared in the first place?

 

The Root Cause

 

The user was running SQL queries from a distributed application. Unfortunately, those queries had an inefficient access path.

 

Because the expected results were not coming back quickly, the user kept triggering the same query again and again.

 

One slow query became many.  On the mainframe side, this resulted in multiple DDF threads consuming significant CPU.

 

And there was another contributing factor: the WLM goal was not configured appropriately for this workload.

 

As a result, these resource-intensive DDF threads were able to run at a high priority.

 

The combination was disastrous:

 

Inefficient SQL → repeated requests → multiple DDF threads → high CPU consumption → high WLM priority → LPAR under severe pressure

 

Eventually, even normal users struggled to log in.

 

Restoring Normal Operations

 

The user was contacted and asked to stop running the problematic queries.

 

Once the offending threads had been removed and the immediate problem was under control, the DB2 DIST address space was moved back to its original service class.

 

Normal DB2 distributed processing resumed.

 

And the production LPAR was back in business.

 

The Bigger Lesson

 

The interesting part of this incident wasn't simply that an inefficient SQL query consumed too much CPU.

 

It was the combination of several conditions:

 

• An inefficient SQL access path
• A user repeatedly submitting the same query
• Multiple expensive DDF threads accumulating
• An inappropriate WLM goal allowing those threads to run at high priority

 

Individually, each problem might have been manageable.

 

Together, they were enough to bring a production mainframe LPAR close to a standstill.

 

Sometimes a mainframe performance problem isn't caused by one catastrophic event.

 

It's several small problems lining up perfectly.

𝗧𝗵𝗲 𝗖𝘂𝗿𝗶𝗼𝘂𝘀 𝗖𝗮𝘀𝗲 𝗼𝗳 𝗜𝗻𝘁𝗲𝗿𝗺𝗶𝘁𝘁𝗲𝗻𝘁 𝗦𝗙𝗧𝗣 𝗧𝗶𝗺𝗲𝗼𝘂𝘁𝘀 𝗶𝗻 𝗠𝗮𝗶𝗻𝗳𝗿𝗮𝗺𝗲 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗝𝗼𝗯𝘀

One of our customers decided to improve security by replacing plain FTP with SFTP across their production batch jobs.

 

As part of the change, all existing FTP steps were replaced with Co:Z SFTP.

 

The implementation looked straightforward. However, once the changes reached production, we started seeing an unexpected problem.

 

Some SFTP steps were intermittently timing out.

 

What made the issue more interesting was that the failures were not consistent. The same jobs could run successfully at one time and fail at another.

 

After looking at the pattern more closely, we noticed one important clue: The SFTP timeouts were occurring primarily during periods of high CPU utilization.

 

That shifted our investigation away from simply treating this as an SFTP or connectivity problem.

 

SMF 30 Provided the Clue

 

Initially, we did not have a clear explanation for why the Co:Z SFTP steps were failing.

 

We eventually decided to investigate the workload using SMF Type 30 records, and that pointed us in the right direction.

 

We found that Co:Z SFTP file transfers were running under an OMVS address space. However, there was no specific WLM classification rule covering this workload.

 

As a result, the SFTP workload was falling into the default OMVS service class, which had a Discretionary goal.

 

And that turned out to be the real problem.

 

Why the Discretionary Goal Mattered

 

A workload running with a Discretionary WLM goal receives CPU resources only when higher-priority workloads do not need them.

 

Under normal CPU conditions, this was not noticeable. There was enough CPU available, and the SFTP transfers completed successfully.

 

During peak CPU utilization, however, higher-priority workloads took precedence. The OMVS workload running the Co:Z SFTP process did not receive enough CPU in a timely manner.

 

The result?  The SFTP operations started hitting timeouts.

 

So, what initially appeared to be an intermittent SFTP problem was actually a WLM workload-classification problem exposed by CPU contention.

 

The WLM Change

 

Our production batch workloads already had three service classes based on priority:

• PRDBATHI - High-priority batch
• PRDBATMD - Medium-priority batch
• PRDBATLO - Low-priority batch

 

We decided that the OMVS workload responsible for the production SFTP transfers should receive the PRDBATHI service class.

 

We therefore added a WLM classification rule similar to the following:

 

Qualifier  Qualifier      Starting       Service   Report

type       name           position       Class     Class

---------  -------------  ---------      --------  --------

TN         PRDJOBQUAL                    PRDBATHI  SFTP

 

After installing the updated WLM definition, the behavior changed immediately.

 

The Co:Z SFTP steps began completing successfully, including during periods of high CPU utilization, and the intermittent timeout problem disappeared.

 

This incident was a good reminder that an SFTP timeout does not necessarily mean there is a network or SFTP problem.

𝗪𝗵𝘆 𝗔𝗿𝗲 𝗦𝗼𝗺𝗲 𝗝𝗼𝗯𝘀 𝗣𝘂𝗿𝗴𝗲𝗱 𝗳𝗿𝗼𝗺 𝗝𝗛𝗦 𝗘𝗮𝗿𝗹𝗶𝗲𝗿 𝗧𝗵𝗮𝗻 𝗢𝘁𝗵𝗲𝗿𝘀?

In JHS (Job History System), we noticed that some jobs are purged from JHS sooner than other jobs.

We dug through the installation libraries below and found that JHS groups jobs by a Job Group number, ranging from 0 to 15.

HLQ.JHS.PARMLIB(EXITS)

HLQ.JHS.JHSASM(JHSGRPM1)

HLQ.JHS.PARMLIB(JHSPARMS)

JHS uses the Job Group number to determine the job retention period. These groups can be used to categorize jobs based on their retention requirements, such as daily, weekly, monthly, or annual jobs. Up to 16 Job Groups (0–15) can be specified.

 
The name of the JHS exit that assigns the Job Group number to jobs can be found in HLQ.JHS.PARMLIB(EXITS).
 
HLQ.JHS.PARMLIB(EXITS)
 
<EXIT>
  <TYPE>JOBGROUP</TYPE>
  <MODULE>JHSGRPM1</MODULE>
</EXIT>
 
The sample code for the JHS exit JHSGRPM1 can be found in:

HLQ.JHS.JHSASM(JHSGRPM1)
 
IF NAME=-------D
  GROUP 0
ENDIF
IF NAME=-------W
  GROUP 0
ENDIF
IF NAME=BXXXXXXD
  GROUP 4
ENDIF
IF NAME=MXXXXXXW
  GROUP 14
ENDIF
 
In PARMLIB(JHSPARMS), each Job Group is associated with a retention period specified in number of days.
 
HLQ.JHS.PARMLIB(JHSPARMS)
* NUMBER OF DAYS RETENTION OF JOB SUMMARY DATA
RETAIN              010    JOB GROUP 0
RETAIN1             005    JOB GROUP 1
RETAIN2             007    JOB GROUP 2
RETAIN3             012    JOB GROUP 3
RETAIN4             014    JOB GROUP 4
RETAIN5             018    JOB GROUP 5
RETAIN6             020    JOB GROUP 6
RETAIN7             022    JOB GROUP 7
RETAIN8             024    JOB GROUP 8
RETAIN9             030    JOB GROUP 9
RETAIN10            060    JOB GROUP 10
RETAIN11            091    JOB GROUP 11
RETAIN12            120    JOB GROUP 12
RETAIN13            150    JOB GROUP 13
RETAIN14            160    JOB GROUP 14
RETAIN15            170    JOB GROUP 15
 
Therefore, jobs assigned to different Job Groups can have different retention periods in JHS, which explains why some jobs are purged sooner than others.

𝗕𝗟𝗞𝗦𝗜𝗭𝗘 𝗠𝘆𝘀𝘁𝗲𝗿𝘆: 𝗦𝗮𝗺𝗲 𝗖𝗢𝗕𝗢𝗟 𝗮𝗻𝗱 𝗝𝗖𝗟, 𝗯𝘂𝘁 𝗗𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝗩𝗮𝗹𝘂𝗲𝘀 𝗳𝗼𝗿 𝗗𝗜𝗦𝗞 𝗮𝗻𝗱 𝗧𝗔𝗣𝗘

While testing the same COBOL program, I observed different BLKSIZE behavior for tape and disk output datasets.

 

The COBOL program had the following settings:

 

• NOBLOCK0 compiler option was in effect.
• The FD definition did not contain BLOCK CONTAINS 0.

Tape output dataset

 

For a tape output dataset, the system assigned a BLKSIZE equal to the LRECL, regardless of whether BLKSIZE=0 was specified in the JCL.

 

In other words:

• With BLKSIZE=0 in JCL → BLKSIZE was assigned the same value as LRECL.
• Without BLKSIZE=0 in JCL → BLKSIZE was still assigned the same value as LRECL.

Disk output dataset

 

The behavior was different for a disk output dataset.

Regardless of whether BLKSIZE=0 was specified in the JCL, the system assigned an appropriate BLKSIZE as expected.

 

Fix for the tape output dataset

 

To address the tape dataset issue, I added the following clause to the COBOL FD:

BLOCK CONTAINS 0

 

After making this change, the system properly assigned the BLKSIZE for the tape output dataset as well, irrespective of whether BLKSIZE=0 was specified in the JCL. 

𝗖𝗢𝗕𝗢𝗟 𝗣𝗿𝗼𝗴𝗿𝗮𝗺 𝗔𝗯𝗲𝗻𝗱𝘀 𝘄𝗶𝘁𝗵 𝗦𝗕𝟯𝟳 𝘄𝗵𝗶𝗹𝗲 𝗖𝗟𝗢𝗦𝗶𝗻𝗴 𝘁𝗵𝗲 𝗳𝗶𝗹𝗲

A COBOL program successfully executes all its WRITE statements and reaches the CLOSE statement. Then, unexpectedly, the program issues an SB37 space abend.
 
Why would an out-of-space condition become visible during CLOSE rather than during WRITE?
 
Let's investigate.
 
The Test Program
 
Consider this simple COBOL program:
 
        IDENTIFICATION DIVISION.                                    
        PROGRAM-ID. TESTPGM.                                        
        ENVIRONMENT DIVISION.                                       
        CONFIGURATION SECTION.                                      
        SPECIAL-NAMES.                                              
        INPUT-OUTPUT SECTION.                                       
        FILE-CONTROL.                                               
            SELECT OUTPUT-FILE       ASSIGN TO OUTFILE.             
        DATA DIVISION.                                              
        FILE SECTION.                                               
        FD  OUTPUT-FILE.                                            
        01  OUTPUT-REC    PIC X(30).                                
        WORKING-STORAGE SECTION.                                    
        01 WS-GRP.                                                  
           05 FILLER         PIC X(05) VALUE 'WS-I:'.               
           05 WS-I           PIC S9(8) COMP.                        
        01 WS-VAR            PIC  9(8).                             
        PROCEDURE DIVISION.                                         
           ACCEPT WS-VAR                                            
           DISPLAY WS-VAR                                           
           OPEN OUTPUT OUTPUT-FILE                                  
           PERFORM VARYING WS-I FROM 1 BY 1 UNTIL WS-I > WS-VAR     
             WRITE OUTPUT-REC    
           END-PERFORM           
           DISPLAY 'WS-I ' WS-I  
           CLOSE OUTPUT-FILE     
           GOBACK.               
 
The program simply accepts a record count and writes that many 30-byte records.
 
The corresponding JCL is:
 
//GO1    EXEC  PGM=TESTPGM
//SYSOUT   DD  SYSOUT=*                                          
//OUTFILE  DD  DSN=USERID.TESTFIL,
//          LRECL=30,RECFM=FB,VOL=(,,,1),                       
//          DISP=(,CATLG,CATLG),
//          SPACE=(TRK,(1,1),RLSE),BUFNO=5   
//*                                                              
//SYSIN    DD *                                                  
00034520                                                         
//*                                                              
 
So the program attempts to write: 34,520 records
 
The important file allocation characteristics are:
 
LRECL = 30
BLKSIZE = 27,990
SPACE = TRK,(1,1) = 1 Primary + 15 secondary tracks = total 16 tracks
BUFNO = 5
 
First, Let's Look at the Blocks
 
With fixed-block records:
 
Records per block = BLKSIZE / LRECL = 27,990 / 30 = 933 records
 
Therefore, one physical block can contain 933 logical COBOL records.
 
For 34,520 records:  34,520 / 933 = 36 remainder 932
 
So the output consists of:

• 36 complete blocks
• 1 partial block containing 932 records

 
Therefore the output dataset needs space for 37 blocks.
 
A track can accommodate two blocks, and each block holds 933 records.
 
With 16 tracks available(based on SPACE parameter), the total number of records that can be successfully written to output file is:
 
16 tracks × 2 blocks per track × 933 records per block = 29,856 records
 
So, the output data set can accommodate exactly 29,856 records before the space-related failure occurs.
 
If we translate 29,856 records to number of blocks, 29856/933 = 32 blocks.
 
But program is trying to write 37 blocks of records.
 
Now Look at What the Program Reports
 
The SYSOUT contains:
 
00034520   

WS-I 00034521                            
IGZ0034W The file with system-name OUTFILE could not be extended.  Secondary extents were not specified or were not available.  The last WRITE was at offset X'E23156CC' in program TESTPGM.                                  
 
The interesting part is:  “WS-I 00034521”
 
This tells us that the PERFORM loop finished.  In other words, COBOL executed all 34,520 WRITE statements.
 
Yet, after the job abended, the output data set contained only 29,856 records.
 
This means that the remaining records were in BUFFER.
 
CLOSE Is More Than "Close the File"
 
Although all the WRITE statements have completed, some records are still held in the output buffers.
 
During CLOSE, these buffered records are written to disk.
 
Since the data set cannot be extended to accommodate them, the program encounters an SB37 abend while closing the file.

Don't Trust Connect Time in SMF Type 30 for FICON Environments

I received the following tip from Enterprise Performance Strategies. 

If you're using SMF Type 30 connect time to gauge I/O performance, be careful - in FICON environments, that number is essentially meaningless.
 
Per IBM's documentation on the SMF 30 record's IO_CONNECT_SEC field: "The value of RqsvAIC for the FICON® channel utilization cannot be calculated. In this case, the system adjusts the connect time for FICON DASD to be 1 millisecond per request." (source https://www.ibm.com/docs/en/z-logdata-analytics/5.1.0?topic=data-smf-30-v2-record-type)

Here's why that matters: connect time has historically represented actual data transfer time. But when FICON introduced concurrent active I/Os per channel, that measurement got too complicated to capture accurately - so the system just hard-codes it to a flat 1ms per request instead.
 
Years ago, when average I/O response times were around 3ms, that 1ms floor was a reasonable stand-in. Today, with response times often well under 1ms (some experts suggest 0.1–0.2ms is closer to reality) that artificial 1ms number is now larger than many actual total response times. It throws off any comparison against more granular records like Type 42 (datasets) or Type 101 (DB2).
 
Bottom line: Don't rely on Type 30 connect time in FICON environments to judge batch I/O performance - use Type 42, Type 101, or other record types that capture true connect time instead.

𝗤𝘂𝗶𝗰𝗸𝗹𝘆 𝗘𝘀𝘁𝗶𝗺𝗮𝘁𝗶𝗻𝗴 𝘁𝗵𝗲 𝗦𝗶𝘇𝗲 𝗼𝗳 𝗬𝗼𝘂𝗿 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗗𝗕𝟮 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀

If all your production DB2 table spaces and index spaces are allocated within a dedicated SMS storage group, determining the approximate size of your production databases becomes a straightforward task.

 

A quick way to estimate the size of your production DB2 databases is by using ISMF (Interactive Storage Management Facility). Navigate to the Storage Group option and examine the storage group dedicated to your production DB2 environment. Pay particular attention to the following metrics:

 

• Total Space
• Free Space 

By subtracting the Free Space from the Total Space, you can calculate the used space within the storage group.

Used Space = Total Space − Free Space

 

The resulting value provides a good estimate of the storage currently consumed by your production DB2 databases.