𝗧𝗵𝗲 𝗠𝘆𝘀𝘁𝗲𝗿𝘆 𝗼𝗳 𝘁𝗵𝗲 𝟵𝟵% 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 𝗔𝗹𝗲𝗿𝘁: 𝗔 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗗𝗲𝘁𝗲𝗰𝘁𝗶𝘃𝗲 𝗦𝘁𝗼𝗿𝘆

One early morning, an alert appeared that immediately caught the attention of the infrastructure team: A production database storage group had reached 99% utilization.

For any infrastructure professional, utilization at that level is cause for concern. When storage approaches full capacity, new DB2 dataset allocations can fail, utilities may abend, and application stability can be put at risk.

 

What made this incident particularly intriguing was the speed at which it unfolded. Within a very short period, storage consumption had surged dramatically, triggering automated alerts warning of insufficient space.

 

The obvious question was:  What Consumed So Much Storage So Quickly?

 

Following the Trail

 

A closer examination of the system logs revealed the first clue. The storage management subsystem was reporting that free capacity was rapidly diminishing, and soon afterward, DB2 began issuing messages that certain datasets could not be allocated because the storage group had run out of available space.

 

At first glance, it appeared that the production environment was genuinely running out of space.

 

Then something unexpected happened.

 

Just minutes later, storage utilization dropped significantly without any administrative intervention. The apparent storage crisis vanished as quickly as it had appeared.

 

Now there were two mysteries to solve:

1. What caused the sudden spike in storage consumption?
2. Why did utilization return to normal so quickly?

Correlating the Timeline

 

The breakthrough came when the storage alerts were correlated with active database maintenance activity.

 

At the exact time the utilization spike occurred, a DB2 utility job was performing an online table reorganization (REORG) on one of the largest tables in the environment.

 

Online REORG is designed to maintain application availability while reorganizing data. To achieve this, DB2 creates temporary shadow datasets that exist alongside the original table data throughout the reorganization process.

 

These shadow datasets consumes nearly as much storage as the original table itself.

 

As the utility progressed, storage consumption rose rapidly until the storage group was effectively exhausted. Eventually, the REORG utility failed because additional space could no longer be allocated.

 

The final piece of the puzzle appeared moments later.

 

When the failed utility terminated, DB2 automatically cleaned up the temporary shadow datasets it had created. As those datasets were deleted, storage utilization immediately dropped back to normal levels.

 

Case Closed

 

What initially appeared to be a critical and unexplained storage shortage was actually the side effect of an online REORG operating on a very large table.

 

The temporary shadow datasets created by the utility drove storage utilization to 99%, triggering alerts and allocation failures. Once the REORG failed and cleaned up its shadow files, the consumed space was released, causing utilization to fall back to normal almost instantly. 

𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗶𝗻𝗴 𝗮 𝗖𝗢𝗕𝗢𝗟 𝗔𝗿𝗿𝗮𝘆 𝗶𝗻𝘁𝗼 𝗮 𝗧𝗲𝗺𝗽𝗼𝗿𝗮𝗿𝘆 𝗧𝗮𝗯𝗹𝗲 𝗮𝗻𝗱 𝗝𝗼𝗶𝗻𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗮 𝗗𝗕𝟮 𝘁𝗮𝗯𝗹𝗲

It is not uncommon for COBOL programs to maintain a list of values in a WORKING STORAGE table and use those values to search for matching rows in a DB2 table. Consider the following working-storage structure:

 

01 WS-DIAG-CODE-TABLE.

   05 WS-DIAG-CODE-COUNT                   PIC S9(04) COMP.

   05 WS-DIAG-CODE-LIST.

      10 WS-DIAG-CODE OCCURS 1 TO 100 TIMES

          DEPENDING ON WS-DIAG-CODE-COUNT    PIC X(05).

 

Traditional Approach

 

The typical implementation is to perform a loop through each occurrence of WS-DIAG-CODE and execute a DB2 query to fetch matching rows.

 

PERFORM VARYING IDX FROM 1 BY 1

   UNTIL IDX > WS-DIAG-CODE-COUNT

   EXEC SQL

     SELECT ...

       FROM TEST_TABLE

     WHERE DIAG_CODE = :WS-DIAG-CODE(IDX)

   END-EXEC

END-PERFORM

 

While this approach is straightforward and easy to understand, it can become a significant performance bottleneck. Each iteration results in a separate database access, increasing CPU consumption and elapsed time.

 

A More Efficient Alternative

 

A better approach is to convert the COBOL table into a temporary result set within SQL and allow DB2 to process all diagnosis codes in a single query.

 

This can be achieved using a recursive Common Table Expression (CTE). The recursive CTE transforms the contents of the COBOL array into a relational structure that can be joined directly with the target DB2 table.

 

WITH TEMP (IDX, DIAG_CODE) AS

(SELECT 1, LEFT(:WS-DIAG-CODE-LIST, 5)

   FROM SYSIBM.SYSDUMMY1

UNION ALL

SELECT IDX + 1, SUBSTR(:WS-DIAG-CODE-LIST, (IDX * 5) + 1, 5)

  FROM TEMP

WHERE IDX < :WS-DIAG-CODE-COUNT),

SELECT ...

FROM TEST_TABLE A, TEMP T

WHERE A.DIAG_CODE = T.DIAG_CODE;

 

Why This Approach Performs Better

 

By leveraging a recursive CTE:

• Only one SQL statement is executed.
• DB2 can optimize the join operation internally.
• Database round trips are eliminated.
• CPU and elapsed time are significantly reduced, especially when the list contains many values.
• The solution scales better as the number of diagnosis codes grows.

 

Key Takeaway: Whenever you find yourself executing the same SQL repeatedly for values stored in a COBOL OCCURS table, consider transforming the data into a temporary table using a recursive CTE and let DB2 perform the matching in one pass. This is a classic example of replacing row-by-row processing with high-performance set-based SQL.

 

𝘇/𝗢𝗦 𝟯.𝟮 𝗗𝗲𝗳𝗲𝗰𝘁 : 𝟭𝟯 𝗦𝘁𝗲𝗽𝘀 𝗞𝗲𝗽𝘁 𝗔𝗽𝗽𝗲𝗻𝗱𝗶𝗻𝗴 𝗥𝗲𝗰𝗼𝗿𝗱𝘀 𝘁𝗼 𝘁𝗵𝗲 𝗗𝗮𝘁𝗮𝘀𝗲𝘁, 𝗯𝘂𝘁 𝗢𝗻𝗹𝘆 𝟳𝟬𝗠 𝗼𝗳 𝘁𝗵𝗲 𝗘𝘅𝗽𝗲𝗰𝘁𝗲𝗱 𝟱𝟵𝟰𝗠 𝗥𝗲𝗰𝗼𝗿𝗱𝘀 𝗔𝗿𝗲 𝗣𝗿𝗲𝘀𝗲𝗻𝘁

A batch job was executed to consolidate data from the most recent 13 generations of a GDG, representing approximately 13 months of historical information.

The process consists of 13 sequential steps, with each step appending its output records to a single output dataset. The dataset was allocated on a single DASD volume.

Dataset attributes:

• RECFM: FB
• LRECL: 21
• BLKSIZE: 27,993

The number of records appended by each processing step is shown below. STEP100 created the dataset, and all subsequent steps appended records to the same dataset using DISP=MOD.

Step	Records Written
STEP100	37,923,095
STEP110	44,967,733
STEP120	57,202,964
STEP130	44,714,138
STEP140	43,599,256
STEP150	43,838,386
STEP160	45,313,346
STEP170	46,349,138
STEP180	45,525,353
STEP190	46,162,534
STEP200	46,518,402
STEP210	45,979,524
STEP220	46,381,066

Total records written from STEP100 to STEP220 : 594,474,935

But the Final dataset had only 70,325,745 records.

 

Dataset Track Limitation

The output dataset was allocated as a basic-format sequential dataset, which is limited to a maximum of 65,535 tracks per volume.

But z/OS 3.2 allowed dataset to grow beyond 65,535 tracks per volume. This is a defect.

 

When the dataset grows beyond this limit:

 

1. The system can no longer correctly address the entire dataset.
2. It effectively treats the dataset as if it contains only the records corresponding to the overflowed tracks at the beginning of the file.
3. Existing records are overwritten as additional data is appended.
4. Consequently, part of the previously accumulated data is lost.

First Overflow Event (After STEP130)

The number of tracks is calculated as: (Number of Records × LRECL) / (2 × Block Size)

Step	Records Written	Running Record Count	Calculated Tracks
STEP100	37,923,095	37,923,095	14,225
STEP110	44,967,733	82,890,828	31,092
STEP120	57,202,964	140,093,792	52,549
STEP130	44,714,138	184,807,930	69,321

At the completion of STEP130, the dataset reaches 69,321 tracks, exceeding the maximum addressable limit of 65,535 tracks.

Overflow tracks: 69,321 - 65,535 = 3,785 tracks

System now thinks the dataset has 3785 tracks instead of 69,331 tracks.

3785 tracks equates to first 10,090,810 records written by STEP100. 

At this point, the system begins reusing existing track addresses, causing subsequent writes to overwrite data already present in the dataset

The process then continues appending additional data:

Step	Records Written	Running Record Count	Calculated Tracks
STEP100	10,090,810		3,785
STEP140	43,599,256	53,690,066	20,139
STEP150	43,838,386	97,528,452	36,582
STEP160	45,313,346	142,841,798	53,579
STEP170	46,349,138	189,190,936	70,965

 

At the completion of STEP170, the dataset again exceeds the track limit.

 

Overflow: 70,965 − 65,535 = 5,430 tracks

 

5,430 tracks corresponds to approximately 14,475,047 records, consisting of:

• First 10,090,810 records written from STEP100  
• First 4,384,237 records written from STEP140

Subsequent write operations reuse track addresses that have already been used, causing newly written records to overwrite previously stored data.

Processing then continues with the next set of steps:

Step	Records Written	Running Record Count	Calculated Tracks
STEP100	10,090,810		3,785
STEP140	4,384,237		1,645
STEP180	45,525,353	60,000,400	22,506
STEP190	46,162,534	106,162,934	39,821
STEP200	46,518,402	152,681,336	57,270
STEP210	45,979,524	198,660,860	74,517

At the completion of STEP210, the dataset exceeds the addressable limit for a third time.

 

Overflow tracks: 74,517 - 65,535 = 8,982 tracks

8,982 tracks corresponds to approximately 23,944,679 records, consisting of:

• First 10,090,810 records written from STEP100  
• First 4,384,237 records written from STEP140
• First 9,469,632 records written from STEP180

The final processing step STEP220 then appends an additional 46,381,066 records.

 

Final Dataset Contents

 

After all overflow events and the completion of the final processing step, the dataset contained records originating from only a subset of the processing steps.

 

Step Contribution Retained in Final Dataset	Record Count
STEP100	10,090,810
STEP140	4,384,237
STEP180	9,469,632
STEP220	46,381,066
Total Records in Final Dataset	70,325,745

 

Conclusion

 

The multiple overflow events caused entire sections of data written by several intermediate processing steps to be lost. As a result, the final dataset did not contain a continuous history of all records written during the job execution.  

The detailed issue outlined above was successfully resolved after implementing the z/OS 3.2 maintenance update (RSU2604/PUT2601).

𝗣𝗼𝘁𝗲𝗻𝘁𝗶𝗮𝗹 𝘇/𝗢𝗦 𝟯.𝟮 𝗗𝗲𝗳𝗲𝗰𝘁: 𝗕𝗮𝘀𝗶𝗰 𝗙𝗼𝗿𝗺𝗮𝘁 𝗦𝗲𝗾𝘂𝗲𝗻𝘁𝗶𝗮𝗹 𝗗𝗮𝘁𝗮𝘀𝗲𝘁𝘀 𝗘𝘅𝗰𝗲𝗲𝗱𝗶𝗻𝗴 𝘁𝗵𝗲 𝟲𝟱𝗞 𝗧𝗿𝗮𝗰𝗸 𝗟𝗶𝗺𝗶𝘁

A Mainframe Storage Group reached 99% utilization, resulting in unexpected abends for production jobs utilizing that storage group. As an immediate workaround, additional volumes were added to the storage group to restore capacity and prevent further job failures.

Following the temporary fix, we investigated which datasets were consuming the most space within the storage group. We identified the top storage-consuming datasets and analyzed the jobs responsible for creating them. The investigation revealed that a DB2 Image Copy job was generating these datasets.

The DB2 Image Copy job created backup datasets as GDG (Generation Data Group) generations on a daily basis. Since a new generation was created every day, the storage group's utilization continued to grow. To manage disk space, the job included mechanism to migrate older image copy generations to tape while retaining only the latest generation on disk.

Our analysis showed that most image copy datasets were being successfully migrated to tape. However, a small number of datasets remained on disk and were not being migrated. We further analyzed these datasets and compared the storage group growth trend. The investigation showed that the abnormal growth started after the z/OS 3.2 upgrade.

We then focused on understanding why these specific image copy datasets were not migrating to tape after the z/OS 3.2 upgrade. The problematic datasets were found to be multi-volume datasets, and occupying more than 65,535 tracks in one or more volumes. These datasets were Basic Format Sequential Datasets.

According to IBM documentation, Basic Format Sequential Datasets are limited to 65,535 tracks per volume. However, these datasets exceeded that limit. As a result, SMS was unable to migrate them to tape, causing the datasets to remain on disk and contributing significantly to the continued growth of the storage group.

SMS migration was failing with following messages.

ARC1001I DB2XXX.AAAAA.BBBBBBBB.CCCCCCCC.G5123V00 MIGRATE FAILED, RC=0099,

ARC1001I (CONT.) REAS=0040 

ARC1299I UNSUPPORTED DATA SET FOR MIGRATION

Reason code 40 says:

The data set occupies more than 65535 tracks and it is not a VSAM, PDSE, or extended format data set.

Based on the findings, it appears that the z/OS 3.2 was allowing Basic Format Sequential Datasets to exceed the documented 65K track limit per volume. This behavior is likely a defect.

𝗪𝗵𝘆 𝗗𝗦𝗟𝗜𝗦𝗧(𝟯.𝟰) 𝗥𝗲𝗽𝗼𝗿𝘁𝗲𝗱 𝟲% 𝗮𝗻𝗱 𝟯% 𝗨𝘁𝗶𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗼𝗻 𝗙𝘂𝗹𝗹𝘆 𝗢𝗰𝗰𝘂𝗽𝗶𝗲𝗱 𝗠𝘂𝗹𝘁𝗶-𝗩𝗼𝗹𝘂𝗺𝗲 𝗗𝗮𝘁𝗮𝘀𝗲𝘁𝘀: 𝗔 𝗗𝗙𝗦𝗢𝗥𝗧 𝗜𝗻𝘃𝗲𝘀𝘁𝗶𝗴𝗮𝘁𝗶𝗼𝗻

The below dataset spans four volumes, yet DSLIST reports utilization values of only 6% and 3% for the first two volumes. Since sequential datasets normally fill earlier volumes before extending to additional volumes, these figures appeared anomalous and warranted further investigation.

The following DSLIST 3.4 output shows the dataset allocation across four volumes.

Command - Enter "/" to select action                  Message           Volume
      Tracks  %     XT Device  Dsorg Recfm Lrecl Blksz  Created    Referred   
-------------------------------------------------------------------------------
         XY67890.PRODHIST.ORDRARCH.BACK.DATA                             XXX710
       69745   6    16 3390     PS   FB      228 27816 2026/08/07 2026/08/07  
-------------------------------------------------------------------------------
         XY67890.PRODHIST.ORDRARCH.BACK.DATA                             XXX674
       67860   3    11 3390     PS   FB      228 27816 2026/08/07 2026/08/07  
-------------------------------------------------------------------------------
         XY67890.PRODHIST.ORDRARCH.BACK.DATA                             XXX612
       60320 100    14 3390     PS   FB      228 27816 2026/08/07 2026/08/07  
-------------------------------------------------------------------------------
         XY67890.PRODHIST.ORDRARCH.BACK.DATA                             XXX678
       60320  93    11 3390     PS   FB      228 27816 2026/08/07 2026/08/07  
 
                             Data Set Information                            
                                     
 Data Set Name . . . . : XY67890.PRODHIST.ORDRARCH.BACK.DATA            
                                                                        
 General Data                          Current Allocation              
  Management class . . : XXXACTX        Allocated megabytes : 13,701   
  Storage class  . . . : XXXX           Allocated extents . : 52       
   Volume serial . . . : XXX710 +                                      
   Device type . . . . : 3390                                          
  Data class . . . . . : DCXXXXX                                       
   Organization  . . . : PS            Current Utilization             
   Record format . . . : FB             Used megabytes  . . : 6,532    
   Record length . . . : 228            Used extents  . . . : 2        
   Block size  . . . . : 27816                                         
   1st extent megabytes: 100                                           
   Secondary megabytes : 400           Dates                           
   Data set name type  :                Creation date . . . : 2026/08/07
   Data set encryption : NO             Referenced date . . : 2026/08/07
                                        Expiration date . . : ***None***
 
The following DFSORT job was used to determine how records are distributed across the dataset's four volumes.
 
The first DFSORT step reads the entire dataset and provides the total record count. Subsequent DFSORT steps specify VOL=(,,2), VOL=(,,3), and VOL=(,,4), causing processing to begin from the second, third, and fourth volume respectively. By subtracting the resulting record counts, the number of records residing on each individual volume can be derived.
 
 
//STEP010  EXEC PGM=SORT                                         
//SORTIN   DD DSN=XY67890.PRODHIST.ORDRARCH.BACK.DATA,DISP=SHR    
//SORTOUT  DD DUMMY                                              
//SYSOUT   DD SYSOUT=*                                           
//SYSIN    DD *                                                  
  SORT FIELDS=COPY                                               
//*                                                              
//STEP020  EXEC PGM=SORT                                         
//SORTIN   DD DSN=XY67890.PRODHIST.ORDRARCH.BACK.DATA,DISP=SHR,   
//            VOL=(,,2)                                          
//SORTOUT  DD DSN=XY67890.PRODHIST.ORDRARCH.BACK.DATA.SNGLVOL2,   
//            DISP=(,CATLG),                                     
//            SPACE=(1,(200,200),RLSE),AVGREC=M                  
//SYSOUT   DD SYSOUT=*                                           
//SYSIN    DD *                                                  
  SORT FIELDS=COPY                                               
//*                                                              
//STEP030  EXEC PGM=SORT                                         
//SORTIN   DD DSN=XY67890.PRODHIST.ORDRARCH.BACK.DATA,DISP=SHR,   
//            VOL=(,,3)                                          
//SORTOUT  DD DSN=XY67890.PRODHIST.ORDRARCH.BACK.DATA.SNGLVOL3,   
//            DISP=(,CATLG),                                     
//            SPACE=(1,(200,200),RLSE),AVGREC=M                  
//SYSOUT   DD SYSOUT=*                                         
//SYSIN    DD *                                                
  SORT FIELDS=COPY                                             
//*                                                            
//STEP040  EXEC PGM=SORT                                       
//SORTIN   DD DSN=XY67890.PRODHIST.ORDRARCH.BACK.DATA,DISP=SHR, 
//            VOL=(,,4)                                        
//SORTOUT  DD DSN=XY67890.PRODHIST.ORDRARCH.BACK.DATA.SNGLVOL4, 
//            DISP=(,CATLG),                                   
//            SPACE=(1,(200,200),RLSE),AVGREC=M                
//SYSOUT   DD SYSOUT=*                                         
//SYSIN    DD *                                                
  SORT FIELDS=COPY                                             
//*                                                            
 
Below is the SYSOUT of each step
 
SORTIN   : RECFM=FB   ; LRECL=   228; BLKSIZE= 27816  
SORTIN   : DSNAME=XY67890.PRODHIST.ORDRARCH.BACK.DATA    
SORTOUT  : DSNAME=NULLFILE                            
RCD IN   62025525, OUT   62025525 
 
SORTIN   : DSNAME=XY67890.PRODHIST.ORDRARCH.BACK.DATA        
SORTOUT  : DSNAME=XY67890.PRODHIST.ORDRARCH.BACK.DATA.SNGLVOL2
RCD IN   45007745, OUT   45007745 
 
SORTIN   : DSNAME=XY67890.PRODHIST.ORDRARCH.BACK.DATA           
SORTOUT  : DSNAME=XY67890.PRODHIST.ORDRARCH.BACK.DATA.SNGLVOL3  
RCD IN   28449905, OUT   28449905 
 
SORTIN   : DSNAME=XY67890.PRODHIST.ORDRARCH.BACK.DATA         
SORTOUT  : DSNAME=XY67890.PRODHIST.ORDRARCH.BACK.DATA.SNGLVOL4
RCD IN   13731825, OUT   13731825 
 
Record Distribution Across Volumes
 

Volume	Record Count Calculation	Records in Volume
XXX710 (Vol 1)	62,025,525 − 45,007,745	17,017,780
XXX674 (Vol 2)	45,007,745 − 28,449,905	16,557,840
XXX612 (Vol 3)	28,449,905 − 13,731,825	14,718,080
XXX678 (Vol 4)	Remaining Records	13,731,825

 
Dataset Size and Track Usage Calculation
 

Volume	Dataset Size (Bytes)	Track Calculation	Tracks Used
XXX710(Vol 1)	3,880,053,840	3,880,053,840 ÷ (2 × 27,816)	69,745
XXX674(Vol 2)	3,775,187,520	3,775,187,520 ÷ (2 × 27,816)	67,860
XXX612(Vol 3)	3,355,722,240	3,355,722,240 ÷ (2 × 27,816)	60,320
XXX678(Vol 4)	3,130,856,100	3,130,856,100 ÷ (2 × 27,816)	56,278

 
Formula Used
Dataset Size (Bytes) = Records × LRECL
Tracks Used = Dataset Size ÷ (2 × BLKSIZE)
 
where:
LRECL = 228
BLKSIZE = 27816
 
DSLIST vs. DFSORT Comparison of Volume Utilization
 

Volume	DSLIST Tracks	DSLIST Used %	Calculated Tracks Used (DFSORT)	Comparison
XXX710	69,745	6%	69,745	Exact match
XXX674	67,860	3%	67,860	Exact match
XXX612	60,320	100%	60,320	Exact match
XXX678	60,320	93%	56,278	Close match. 60,320 × 93% = 56,098 tracks, which is very close to the calculated value of 56,278 tracks

 
 
Conclusion
 
Although DSLIST reports utilization values of only 6% and 3% for volumes XXX710 and XXX674, DFSORT analysis confirms that both volumes are effectively 100% utilized by dataset data. The calculated track usage matches the DSLIST track counts exactly, demonstrating that the reported percentages are misleading and do not reflect the actual occupancy of these volumes. 

For basic format sequential datasets, the maximum number of tracks that can be addressed on a single volume is 65,535. When DSLIST encounters an allocation that exceeds this limit, it appears to calculate the utilization percentage using only the tracks beyond 65,535.

• Volume XXX710: 69,745 tracks allocated
• Excess tracks = 69,745 − 65,535 = 4,210
• DSLIST reported utilization = 4,210 / 69,745 ≈ 6%
• Volume XXX674: 67,860 tracks allocated
• Excess tracks = 67,860 − 65,535 = 2,325
• DSLIST reported utilization = 2,325 / 67,860 ≈ 3%

As a result, DSLIST displays 6% and 3% utilization for the first two volumes even though those volumes are effectively full. The percentages reflect only the tracks beyond the 65,535-track threshold rather than the actual space utilization of the volumes.

The dataset XY67890.PRODHIST.ORDRARCH.BACK.DATA is a basic-format sequential dataset. According to IBM documentation,https://www.ibm.com/docs/en/zos/2.1.0?topic=set-types-dasd-sequential-data-sets, a basic-format sequential dataset on a single volume is limited to a maximum of 65,535 tracks.

However, the system appears to allow allocations that exceed this documented limit. This behavior may indicate a z/OS defect.