𝗧𝗵𝗲 𝗠𝘆𝘀𝘁𝗲𝗿𝘆 𝗼𝗳 𝘁𝗵𝗲 𝟵𝟵% 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 𝗔𝗹𝗲𝗿𝘁: 𝗔 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗗𝗲𝘁𝗲𝗰𝘁𝗶𝘃𝗲 𝗦𝘁𝗼𝗿𝘆

One early morning, an alert appeared that immediately caught the attention of the infrastructure team: A production database storage group had reached 99% utilization.

For any infrastructure professional, utilization at that level is cause for concern. When storage approaches full capacity, new DB2 dataset allocations can fail, utilities may abend, and application stability can be put at risk.

 

What made this incident particularly intriguing was the speed at which it unfolded. Within a very short period, storage consumption had surged dramatically, triggering automated alerts warning of insufficient space.

 

The obvious question was:  What Consumed So Much Storage So Quickly?

 

Following the Trail

 

A closer examination of the system logs revealed the first clue. The storage management subsystem was reporting that free capacity was rapidly diminishing, and soon afterward, DB2 began issuing messages that certain datasets could not be allocated because the storage group had run out of available space.

 

At first glance, it appeared that the production environment was genuinely running out of space.

 

Then something unexpected happened.

 

Just minutes later, storage utilization dropped significantly without any administrative intervention. The apparent storage crisis vanished as quickly as it had appeared.

 

Now there were two mysteries to solve:

1. What caused the sudden spike in storage consumption?
2. Why did utilization return to normal so quickly?

Correlating the Timeline

 

The breakthrough came when the storage alerts were correlated with active database maintenance activity.

 

At the exact time the utilization spike occurred, a DB2 utility job was performing an online table reorganization (REORG) on one of the largest tables in the environment.

 

Online REORG is designed to maintain application availability while reorganizing data. To achieve this, DB2 creates temporary shadow datasets that exist alongside the original table data throughout the reorganization process.

 

These shadow datasets consumes nearly as much storage as the original table itself.

 

As the utility progressed, storage consumption rose rapidly until the storage group was effectively exhausted. Eventually, the REORG utility failed because additional space could no longer be allocated.

 

The final piece of the puzzle appeared moments later.

 

When the failed utility terminated, DB2 automatically cleaned up the temporary shadow datasets it had created. As those datasets were deleted, storage utilization immediately dropped back to normal levels.

 

Case Closed

 

What initially appeared to be a critical and unexplained storage shortage was actually the side effect of an online REORG operating on a very large table.

 

The temporary shadow datasets created by the utility drove storage utilization to 99%, triggering alerts and allocation failures. Once the REORG failed and cleaned up its shadow files, the consumed space was released, causing utilization to fall back to normal almost instantly.