𝗧𝗵𝗲 𝗖𝘂𝗿𝗶𝗼𝘂𝘀 𝗖𝗮𝘀𝗲 𝗼𝗳 𝗜𝗻𝘁𝗲𝗿𝗺𝗶𝘁𝘁𝗲𝗻𝘁 𝗦𝗙𝗧𝗣 𝗧𝗶𝗺𝗲𝗼𝘂𝘁𝘀 𝗶𝗻 𝗠𝗮𝗶𝗻𝗳𝗿𝗮𝗺𝗲 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗝𝗼𝗯𝘀

One of our customers decided to improve security by replacing plain FTP with SFTP across their production batch jobs.

 

As part of the change, all existing FTP steps were replaced with Co:Z SFTP.

 

The implementation looked straightforward. However, once the changes reached production, we started seeing an unexpected problem.

 

Some SFTP steps were intermittently timing out.

 

What made the issue more interesting was that the failures were not consistent. The same jobs could run successfully at one time and fail at another.

 

After looking at the pattern more closely, we noticed one important clue: The SFTP timeouts were occurring primarily during periods of high CPU utilization.

 

That shifted our investigation away from simply treating this as an SFTP or connectivity problem.

 

SMF 30 Provided the Clue

 

Initially, we did not have a clear explanation for why the Co:Z SFTP steps were failing.

 

We eventually decided to investigate the workload using SMF Type 30 records, and that pointed us in the right direction.

 

We found that Co:Z SFTP file transfers were running under an OMVS address space. However, there was no specific WLM classification rule covering this workload.

 

As a result, the SFTP workload was falling into the default OMVS service class, which had a Discretionary goal.

 

And that turned out to be the real problem.

 

Why the Discretionary Goal Mattered

 

A workload running with a Discretionary WLM goal receives CPU resources only when higher-priority workloads do not need them.

 

Under normal CPU conditions, this was not noticeable. There was enough CPU available, and the SFTP transfers completed successfully.

 

During peak CPU utilization, however, higher-priority workloads took precedence. The OMVS workload running the Co:Z SFTP process did not receive enough CPU in a timely manner.

 

The result?  The SFTP operations started hitting timeouts.

 

So, what initially appeared to be an intermittent SFTP problem was actually a WLM workload-classification problem exposed by CPU contention.

 

The WLM Change

 

Our production batch workloads already had three service classes based on priority:

• PRDBATHI - High-priority batch
• PRDBATMD - Medium-priority batch
• PRDBATLO - Low-priority batch

 

We decided that the OMVS workload responsible for the production SFTP transfers should receive the PRDBATHI service class.

 

We therefore added a WLM classification rule similar to the following:

 

Qualifier  Qualifier      Starting       Service   Report

type       name           position       Class     Class

---------  -------------  ---------      --------  --------

TN         PRDJOBQUAL                    PRDBATHI  SFTP

 

After installing the updated WLM definition, the behavior changed immediately.

 

The Co:Z SFTP steps began completing successfully, including during periods of high CPU utilization, and the intermittent timeout problem disappeared.

 

This incident was a good reminder that an SFTP timeout does not necessarily mean there is a network or SFTP problem.

𝗪𝗵𝘆 𝗔𝗿𝗲 𝗦𝗼𝗺𝗲 𝗝𝗼𝗯𝘀 𝗣𝘂𝗿𝗴𝗲𝗱 𝗳𝗿𝗼𝗺 𝗝𝗛𝗦 𝗘𝗮𝗿𝗹𝗶𝗲𝗿 𝗧𝗵𝗮𝗻 𝗢𝘁𝗵𝗲𝗿𝘀?

In JHS (Job History System), we noticed that some jobs are purged from JHS sooner than other jobs.

We dug through the installation libraries below and found that JHS groups jobs by a Job Group number, ranging from 0 to 15.

HLQ.JHS.PARMLIB(EXITS)

HLQ.JHS.JHSASM(JHSGRPM1)

HLQ.JHS.PARMLIB(JHSPARMS)

JHS uses the Job Group number to determine the job retention period. These groups can be used to categorize jobs based on their retention requirements, such as daily, weekly, monthly, or annual jobs. Up to 16 Job Groups (0–15) can be specified.

 
The name of the JHS exit that assigns the Job Group number to jobs can be found in HLQ.JHS.PARMLIB(EXITS).
 
HLQ.JHS.PARMLIB(EXITS)
 
<EXIT>
  <TYPE>JOBGROUP</TYPE>
  <MODULE>JHSGRPM1</MODULE>
</EXIT>
 
The sample code for the JHS exit JHSGRPM1 can be found in:

HLQ.JHS.JHSASM(JHSGRPM1)
 
IF NAME=-------D
  GROUP 0
ENDIF
IF NAME=-------W
  GROUP 0
ENDIF
IF NAME=BXXXXXXD
  GROUP 4
ENDIF
IF NAME=MXXXXXXW
  GROUP 14
ENDIF
 
In PARMLIB(JHSPARMS), each Job Group is associated with a retention period specified in number of days.
 
HLQ.JHS.PARMLIB(JHSPARMS)
* NUMBER OF DAYS RETENTION OF JOB SUMMARY DATA
RETAIN              010    JOB GROUP 0
RETAIN1             005    JOB GROUP 1
RETAIN2             007    JOB GROUP 2
RETAIN3             012    JOB GROUP 3
RETAIN4             014    JOB GROUP 4
RETAIN5             018    JOB GROUP 5
RETAIN6             020    JOB GROUP 6
RETAIN7             022    JOB GROUP 7
RETAIN8             024    JOB GROUP 8
RETAIN9             030    JOB GROUP 9
RETAIN10            060    JOB GROUP 10
RETAIN11            091    JOB GROUP 11
RETAIN12            120    JOB GROUP 12
RETAIN13            150    JOB GROUP 13
RETAIN14            160    JOB GROUP 14
RETAIN15            170    JOB GROUP 15
 
Therefore, jobs assigned to different Job Groups can have different retention periods in JHS, which explains why some jobs are purged sooner than others.

𝗕𝗟𝗞𝗦𝗜𝗭𝗘 𝗠𝘆𝘀𝘁𝗲𝗿𝘆: 𝗦𝗮𝗺𝗲 𝗖𝗢𝗕𝗢𝗟 𝗮𝗻𝗱 𝗝𝗖𝗟, 𝗯𝘂𝘁 𝗗𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁 𝗩𝗮𝗹𝘂𝗲𝘀 𝗳𝗼𝗿 𝗗𝗜𝗦𝗞 𝗮𝗻𝗱 𝗧𝗔𝗣𝗘

While testing the same COBOL program, I observed different BLKSIZE behavior for tape and disk output datasets.

 

The COBOL program had the following settings:

 

• NOBLOCK0 compiler option was in effect.
• The FD definition did not contain BLOCK CONTAINS 0.

Tape output dataset

 

For a tape output dataset, the system assigned a BLKSIZE equal to the LRECL, regardless of whether BLKSIZE=0 was specified in the JCL.

 

In other words:

• With BLKSIZE=0 in JCL → BLKSIZE was assigned the same value as LRECL.
• Without BLKSIZE=0 in JCL → BLKSIZE was still assigned the same value as LRECL.

Disk output dataset

 

The behavior was different for a disk output dataset.

Regardless of whether BLKSIZE=0 was specified in the JCL, the system assigned an appropriate BLKSIZE as expected.

 

Fix for the tape output dataset

 

To address the tape dataset issue, I added the following clause to the COBOL FD:

BLOCK CONTAINS 0

 

After making this change, the system properly assigned the BLKSIZE for the tape output dataset as well, irrespective of whether BLKSIZE=0 was specified in the JCL. 

𝗖𝗢𝗕𝗢𝗟 𝗣𝗿𝗼𝗴𝗿𝗮𝗺 𝗔𝗯𝗲𝗻𝗱𝘀 𝘄𝗶𝘁𝗵 𝗦𝗕𝟯𝟳 𝘄𝗵𝗶𝗹𝗲 𝗖𝗟𝗢𝗦𝗶𝗻𝗴 𝘁𝗵𝗲 𝗳𝗶𝗹𝗲

A COBOL program successfully executes all its WRITE statements and reaches the CLOSE statement. Then, unexpectedly, the program issues an SB37 space abend.
 
Why would an out-of-space condition become visible during CLOSE rather than during WRITE?
 
Let's investigate.
 
The Test Program
 
Consider this simple COBOL program:
 
        IDENTIFICATION DIVISION.                                    
        PROGRAM-ID. TESTPGM.                                        
        ENVIRONMENT DIVISION.                                       
        CONFIGURATION SECTION.                                      
        SPECIAL-NAMES.                                              
        INPUT-OUTPUT SECTION.                                       
        FILE-CONTROL.                                               
            SELECT OUTPUT-FILE       ASSIGN TO OUTFILE.             
        DATA DIVISION.                                              
        FILE SECTION.                                               
        FD  OUTPUT-FILE.                                            
        01  OUTPUT-REC    PIC X(30).                                
        WORKING-STORAGE SECTION.                                    
        01 WS-GRP.                                                  
           05 FILLER         PIC X(05) VALUE 'WS-I:'.               
           05 WS-I           PIC S9(8) COMP.                        
        01 WS-VAR            PIC  9(8).                             
        PROCEDURE DIVISION.                                         
           ACCEPT WS-VAR                                            
           DISPLAY WS-VAR                                           
           OPEN OUTPUT OUTPUT-FILE                                  
           PERFORM VARYING WS-I FROM 1 BY 1 UNTIL WS-I > WS-VAR     
             WRITE OUTPUT-REC    
           END-PERFORM           
           DISPLAY 'WS-I ' WS-I  
           CLOSE OUTPUT-FILE     
           GOBACK.               
 
The program simply accepts a record count and writes that many 30-byte records.
 
The corresponding JCL is:
 
//GO1    EXEC  PGM=TESTPGM
//SYSOUT   DD  SYSOUT=*                                          
//OUTFILE  DD  DSN=USERID.TESTFIL,
//          LRECL=30,RECFM=FB,VOL=(,,,1),                       
//          DISP=(,CATLG,CATLG),
//          SPACE=(TRK,(1,1),RLSE),BUFNO=5   
//*                                                              
//SYSIN    DD *                                                  
00034520                                                         
//*                                                              
 
So the program attempts to write: 34,520 records
 
The important file allocation characteristics are:
 
LRECL = 30
BLKSIZE = 27,990
SPACE = TRK,(1,1) = 1 Primary + 15 secondary tracks = total 16 tracks
BUFNO = 5
 
First, Let's Look at the Blocks
 
With fixed-block records:
 
Records per block = BLKSIZE / LRECL = 27,990 / 30 = 933 records
 
Therefore, one physical block can contain 933 logical COBOL records.
 
For 34,520 records:  34,520 / 933 = 36 remainder 932
 
So the output consists of:

• 36 complete blocks
• 1 partial block containing 932 records

 
Therefore the output dataset needs space for 37 blocks.
 
A track can accommodate two blocks, and each block holds 933 records.
 
With 16 tracks available(based on SPACE parameter), the total number of records that can be successfully written to output file is:
 
16 tracks × 2 blocks per track × 933 records per block = 29,856 records
 
So, the output data set can accommodate exactly 29,856 records before the space-related failure occurs.
 
If we translate 29,856 records to number of blocks, 29856/933 = 32 blocks.
 
But program is trying to write 37 blocks of records.
 
Now Look at What the Program Reports
 
The SYSOUT contains:
 
00034520   

WS-I 00034521                            
IGZ0034W The file with system-name OUTFILE could not be extended.  Secondary extents were not specified or were not available.  The last WRITE was at offset X'E23156CC' in program TESTPGM.                                  
 
The interesting part is:  “WS-I 00034521”
 
This tells us that the PERFORM loop finished.  In other words, COBOL executed all 34,520 WRITE statements.
 
Yet, after the job abended, the output data set contained only 29,856 records.
 
This means that the remaining records were in BUFFER.
 
CLOSE Is More Than "Close the File"
 
Although all the WRITE statements have completed, some records are still held in the output buffers.
 
During CLOSE, these buffered records are written to disk.
 
Since the data set cannot be extended to accommodate them, the program encounters an SB37 abend while closing the file.

Don't Trust Connect Time in SMF Type 30 for FICON Environments

I received the following tip from Enterprise Performance Strategies. 

If you're using SMF Type 30 connect time to gauge I/O performance, be careful - in FICON environments, that number is essentially meaningless.
 
Per IBM's documentation on the SMF 30 record's IO_CONNECT_SEC field: "The value of RqsvAIC for the FICON® channel utilization cannot be calculated. In this case, the system adjusts the connect time for FICON DASD to be 1 millisecond per request." (source https://www.ibm.com/docs/en/z-logdata-analytics/5.1.0?topic=data-smf-30-v2-record-type)

Here's why that matters: connect time has historically represented actual data transfer time. But when FICON introduced concurrent active I/Os per channel, that measurement got too complicated to capture accurately - so the system just hard-codes it to a flat 1ms per request instead.
 
Years ago, when average I/O response times were around 3ms, that 1ms floor was a reasonable stand-in. Today, with response times often well under 1ms (some experts suggest 0.1–0.2ms is closer to reality) that artificial 1ms number is now larger than many actual total response times. It throws off any comparison against more granular records like Type 42 (datasets) or Type 101 (DB2).
 
Bottom line: Don't rely on Type 30 connect time in FICON environments to judge batch I/O performance - use Type 42, Type 101, or other record types that capture true connect time instead.

𝗤𝘂𝗶𝗰𝗸𝗹𝘆 𝗘𝘀𝘁𝗶𝗺𝗮𝘁𝗶𝗻𝗴 𝘁𝗵𝗲 𝗦𝗶𝘇𝗲 𝗼𝗳 𝗬𝗼𝘂𝗿 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗗𝗕𝟮 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀

If all your production DB2 table spaces and index spaces are allocated within a dedicated SMS storage group, determining the approximate size of your production databases becomes a straightforward task.

 

A quick way to estimate the size of your production DB2 databases is by using ISMF (Interactive Storage Management Facility). Navigate to the Storage Group option and examine the storage group dedicated to your production DB2 environment. Pay particular attention to the following metrics:

 

• Total Space
• Free Space 

By subtracting the Free Space from the Total Space, you can calculate the used space within the storage group.

Used Space = Total Space − Free Space

 

The resulting value provides a good estimate of the storage currently consumed by your production DB2 databases.

𝗧𝗵𝗲 𝗠𝘆𝘀𝘁𝗲𝗿𝘆 𝗼𝗳 𝘁𝗵𝗲 𝟵𝟵% 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 𝗚𝗿𝗼𝘂𝗽 𝗙𝘂𝗹𝗹 𝗔𝗹𝗲𝗿𝘁: 𝗔 𝗜𝗻𝘃𝗲𝘀𝘁𝗶𝗴𝗮𝘁𝗶𝗼𝗻 𝗦𝘁𝗼𝗿𝘆

One early morning, an alert appeared that immediately caught the attention of the infrastructure team: A production database storage group had reached 99% utilization.

For any infrastructure professional, utilization at that level is cause for concern. When storage approaches full capacity, new DB2 dataset allocations can fail, utilities may abend, and application stability can be put at risk.

 

What made this incident particularly intriguing was the speed at which it unfolded. Within a very short period, storage consumption had surged dramatically, triggering automated alerts warning of insufficient space.

 

The obvious question was:  What Consumed So Much Storage So Quickly?

 

Following the Trail

 

A closer examination of the system logs revealed the first clue. The storage management subsystem was reporting that free capacity was rapidly diminishing, and soon afterward, DB2 began issuing messages that certain datasets could not be allocated because the storage group had run out of available space.

 

At first glance, it appeared that the production environment was genuinely running out of space.

 

Then something unexpected happened.

 

Just minutes later, storage utilization dropped significantly without any administrative intervention. The apparent storage crisis vanished as quickly as it had appeared.

 

Now there were two mysteries to solve:

1. What caused the sudden spike in storage consumption?
2. Why did utilization return to normal so quickly?

Correlating the Timeline

 

The breakthrough came when the storage alerts were correlated with active database maintenance activity.

 

At the exact time the utilization spike occurred, a DB2 utility job was performing an online table reorganization (REORG) on one of the largest tables in the environment.

 

Online REORG is designed to maintain application availability while reorganizing data. To achieve this, DB2 creates temporary shadow datasets that exist alongside the original table data throughout the reorganization process.

 

These shadow datasets consumes nearly as much storage as the original table itself.

 

As the utility progressed, storage consumption rose rapidly until the storage group was effectively exhausted. Eventually, the REORG utility failed because additional space could no longer be allocated.

 

The final piece of the puzzle appeared moments later.

 

When the failed utility terminated, DB2 automatically cleaned up the temporary shadow datasets it had created. As those datasets were deleted, storage utilization immediately dropped back to normal levels.

 

Case Closed

 

What initially appeared to be a critical and unexplained storage shortage was actually the side effect of an online REORG operating on a very large table.

 

The temporary shadow datasets created by the utility drove storage utilization to 99%, triggering alerts and allocation failures. Once the REORG failed and cleaned up its shadow files, the consumed space was released, causing utilization to fall back to normal almost instantly.