It was one of
those bad days in production.
A mainframe
production LPAR had almost ground to a halt.
Complaints
started coming from every direction. “Nobody can log in to the mainframe.”
And then our
hero stepped in. There was just one problem. He couldn’t log in either.
Fortunately,
someone was already logged on. Through a screen-sharing session, he gained
visibility into the struggling LPAR and immediately opened the z/OS monitoring
tool.
He started
checking the jobs and started tasks running on the system.
One address
space stood out.
DB2 DIST was
consuming an enormous amount of CPU.
Something
coming through DDF was putting tremendous pressure on the system.
Emergency
Brake:
The immediate
priority was simple: stop DB2 DIST from overwhelming the LPAR.
DB2 DIST was
temporarily moved to a service class called HOLDON, configured with a Discretionary
goal and associated with a resource group having a maximum Service Unit
limit of 1.
DB2 DIST was
effectively frozen from taking additional workload.
Finding the
Real Culprit
He moved to the
DB2 monitor. There, the picture became clearer.
Several DDF
threads were consuming large amounts of CPU, and all of them were
associated with the same user.
He terminated
those threads.
And almost
immediately... the mainframe came back to life.
Workloads began
running normally again.
But killing the
threads only treated the symptom. The next question was: Why had so many
expensive DDF threads appeared in the first place?
The Root
Cause
The user was
running SQL queries from a distributed application. Unfortunately, those
queries had an inefficient access path.
Because the
expected results were not coming back quickly, the user kept triggering the
same query again and again.
One slow query
became many. On the mainframe side, this
resulted in multiple DDF threads consuming significant CPU.
And there was
another contributing factor: the WLM goal was not configured appropriately
for this workload.
As a result,
these resource-intensive DDF threads were able to run at a high priority.
The combination
was disastrous:
Inefficient
SQL → repeated requests → multiple DDF threads → high CPU consumption → high
WLM priority → LPAR under severe pressure
Eventually,
even normal users struggled to log in.
Restoring
Normal Operations
The user was
contacted and asked to stop running the problematic queries.
Once the
offending threads had been removed and the immediate problem was under control,
the DB2 DIST address space was moved back to its original service class.
Normal DB2
distributed processing resumed.
And the
production LPAR was back in business.
The Bigger
Lesson
The interesting
part of this incident wasn't simply that an inefficient SQL query consumed too
much CPU.
It was the combination
of several conditions:
- An inefficient SQL access path
- A user repeatedly submitting the same query
- Multiple expensive DDF threads accumulating
- An inappropriate WLM goal allowing those threads to
run at high priority
Individually,
each problem might have been manageable.
Together, they
were enough to bring a production mainframe LPAR close to a standstill.
Sometimes a
mainframe performance problem isn't caused by one catastrophic event.
It's several
small problems lining up perfectly.
