Getting a log of the locks in order, can it be done?

[ConvertFromOldNGs]

by Carl Ranson >> Sun, 30 Dec 2007 6:26:14 GMT

Hey all,

I've got some work to do on tracking down some hideous deadlock exceptions and I was thinking about the approach I'd use. I've already identified the pairs of methods that deadlock each other.

What I'd ideally like is a listing of all locks that a process has requested, in the order that it requested them.

I think one method that would work is to write a script that logs the current locks and single step through the code logging the locks at each step (using System.getLocks), but it seems like it would be slow and error prone.

Anyone know of a more elegant method?
cheers
CR

[ConvertFromOldNGs]

by Scott >> Mon, 31 Dec 2007 4:38:09 GMT

It's been a while since I last used it, but I seem to remember that Node Sampling will give you the information you're wanting about what has been locked and in what order.

[ConvertFromOldNGs]

by Carl Ranson >> Wed, 2 Jan 2008 2:49:04 GMT

I hand't twigged to that fact from the few times ive flipped through that part of the documentation. It looks like you are right, though, there is info in there about each lock request. I recon the name "node sampling" is a bit misleading in this case.

Anyway, I'll be checking it out, thanks for the tip.
CR

p.s. Don't suppose anyone's got a script that can process the logs and produce meaningful oid numbers and method names out of it?

[ConvertFromOldNGs]

by Dean Cooper >> Wed, 2 Jan 2008 10:29:07 GMT

If you go ahead with Node Sampling, suggest you email JADE Support and ask for the notes and app we have that may make it easier for you to use.

Alternatively, if you are investigating the locks/deadlocks in a test environment (and your production environment isn't on JADE 6.2), you could download JADE 6.2 and upgrade your test environment, and then use the new JADE 6.2 Monitor to investigate things. The JADE 6.2 Monitor provides significantly more information and you can target individual nodes and processes if required.

It isn't possible (even with sampling) to see the method from which a lock was requested.

However, if you're specifically chasing deadlocks (as opposed to general lock contention/queuing performance), then it's often easiest to start with the DoubleDeadlockException ini file setting. By default this is disabled. You can enable it in the JadeServer ini file section as follows:

[JadeServer]
DoubleDeadlockException=true

By default, when a deadlock occurs, the process requesting the lock that caused the deadlock receives the exception. This gives you one side of the picture. What you also want to see is the other process that's holding the conflicting lock(s). That's what the DoubleDeadlockException setting does. When it's set to true, both processes receive a deadlock exception. So you can then see a stack dump for each process, and you can use a global deadlock exception handler to write out all locks for each process (using System::getLocks) at the point of the deadlock, if required. Unless your application is using session locks and/or is holding transaction duration locks across idle state (eg: app.doWindowEvents - not recommended), then the conflicting locks will most likely have been requested by methods in one or both of the call stacks. That's generally sufficient to start chasing them down.

Dean.

[ConvertFromOldNGs]

by Carl Ranson >> Sat, 12 Jan 2008 23:41:07 GMT

Node sampling worked a treat. As easy as starting and stopping it in code around the transaction you're looking at.

I just did a script to parse out the info i wanted from the file (the objects being locked and the methods involved). It doens't quite let you drill down to the call stack in place at each one, but you can tell the method used to trigger the lock and distinguish manual from automatic locks.
Cheers
CR