Very basic lock exception avoidance

[concord]

I've been developing a fairly large ERP suite in Jade for almost 15 years and never had to bother with a locking strategy. I know most of you are thinking... "he's crazy". We run a singleUser app server (no Jade RAP) and funnel our core (contentious) transacations through a single process. With hardware and Jade improvements this strategy has just continued to work really really well.

Anyway, I now have a situation where I'm going to encounter lock exceptions and need to handle it. I feel like a bright eyed newbie again.

I'm not sure how this situation gave me a lock in the first place, and I I'm not sure which simple strategy I should to implement to avoid it.

Here's a ficticious scenario:

Library.allBooksByISBN
Book.myLibrary
All book objects already exist.

At a key time in the night there is a 2 hour window when one process is performing quite a few:
if library.allBooksByISBN.getAtKey (987654) then

At the same time another process is updating book isbn properties.
beginTransaction
book.setISBN(123456);
commitTransaction

About every 10 minutes I might get 40-50 book isbn look-ups, with approx 5 occurring per second.
The book isbn update transaction is occurring no more frequently than once every 5 seconds.

Given the simplicity of these 2 systems I would have thought I'd NEVER get a lock exception. I always assumed Jade performed a very basic lock acquire, timeout, retry and that under these very lite conditions the locks would always work them selves out.

Not the case, my book isbn update process got a lock exception last night and the culprit process was the book isbn look-up.

So is the best approach a simple lock exception handler right before my book.setISBN(123456), that simply performs a series of tryLock(...) attempts for a couple of seconds and hopefully (surely almost guaranteed?) acquires the lock, otherwise Ex_Pass_Back?

[torrie]

We have lock exception handlers as you have described. Basically, they back off and retry after a delay. The process may try three times before doing a pass back. If you are doing nightly processing, it may be better to put the object in a collection and then go back to it in a second pass. For example your processing loop could be something like:

Code: Select all

// Put All objects to process in a collection (set or dictionary) while not collection.isEmpty do foreach object in collection do if tryLock( object ) then // Do all processing here. collection.remove( object ); endif; endforeach; // break out if you have tried too many times. endwhile;

However we've also done a number of things up stream of acquiring a lock to try to prevent these sorts of things happening:

• Nightly process etc have an initial step that identifies all items to update and puts them into a collection to process as a second step. A transient class can be used to store references to all the relevant items for later processing.
• Perform all calculations / searches etc prior to entering transaction state
• Keep the transaction state as short as possible, ours is (almost!) just property updates, all processing is done before we enter transaction state.
• Set key properties prior to setting references that put objects into collections. (if you add an object to a collection before setting the keys, then it's moved each time as you set a new key locking the collection for a longer time.)
• Update processes so that they acquire locks in a specified order (we put all objects into a set so they are in their OID order and also have a mechanism for including exclusive collections in this list)
• Deal with collections, especially global or/ contentious collections at the end of the process so they are locked for a shorter time.
• Get exclusive locks on the objects that we are going to update prior to starting the transaction.
• Designing the object structure so that frequently updated fields (e.g. account balances etc) are on a separate object. This includes counters for the next transaction ID etc.
• Avoiding large global collections. We're moving global collections do to sites or departments and then using merge iterators etc when processing or performing lookups.

We've found that locking issues are generally to do with collections rather than objects themselves. I would recommend that you use the Monitor to view the locks while the background processing is occuring. It will give you a better idea of what is actually occuring.

[ghosttie]

It sounds like you should be fine - if this is the first LockException you've had in 15 years then I'd suspect something unusual happened to slow down that one transaction, for example a slow disk access caused by the hardware or another process.

If you're worried about this happening again then you could add a simple automatic retry to your global exception handler:

Code: Select all

if pEx.isKindOf(LockException) then le := pEx.LockException; while le.retryCount < 10 do // retry the lock 10 times if le.retryLock then return Ex_Continue; // we got the lock, continue endif; endwhile; endif;

[concord]

Hi Torrie,
It's Andrew here, it's been a while. Way back I used to tap you on the shoulder every 15 minutes with questions a bit like this.

I've covered off most of your points: temp processing collection, nothing inside tran state unless it has to be, key's updated before refs collections last. More than likely picked up these tips from you 10+ years back.

I believe the only contention I'm going to strike is on the master collection (library.allBooksByISBN). I'm reluctant to manually lock this collection myself.

In fact I'm not manually acquiring any locks myself, I've always just left that up to Jade. Any time I've profiled one of our serious transactions I've been amazed at how many locks and updates actually took place, 100's often 1000's! This is why I have avoided managing locks manually.

I'm intrigued by your split collection and merge iterator, will look into that. Thanks for the advice.

[concord]

Hi ghosttie,
Certainly not the first lock, but 99% of our locks are occasional and U related, the user barely notices them. Our key data crunching is done by one process and almost never fails. I think I'm going to implement a simple re-try as you've suggested. And perhaps in our global handler rather than local to the batch process. Cheers.

[BeeJay]

Others have already provided advice on how to handle the lock exception, but I thought it worth adding my $0.02 for your following question/statement:

<snip>

I'm not sure how this situation gave me a lock in the first place...

<snip>

At a key time in the night there is a 2 hour window when one process is performing quite a few:
if library.allBooksByISBN.getAtKey (987654) then

<snip>

The library.allBooksByISBN.getAtKey will implicitly place a transaction duration shared lock on the library.allBooksByISBN collection, assuming the library.allBooksByISBN collection is not already locked.

If the code that does this getAtKey is "inside" Transaction or Load or Lock state, then that implicit sharedLock will be retained until the next commitTransaction/abortTransaction or endLoad or endLock statement.

Cheers,
BeeJay.

[concord]

BeeJay,
You're quite right, I was still intrigued as to how these 2 process could bang heads like this and so started digging a little further...
I was absolutely horrified to find that the collection look-up was in fact nested inside a parent call that had unnecessarily opened a transaction state way too early. Absolute shocker, if the original coder had still been with us I would have forced him to write out Torrie's golden rule bullet points 100 times!

I'm pretty sure that once I correct this I'll never see a a lock like this again.

[suzuki1100]

You also may get some performance gains and reduced locking
by using includeKey(..) instead of getAtKey(..) determining if a object exists in a collection.
re your code sample.

Code: Select all

if library.allBooksByISBN.getAtKey (987654) then

should be

Code: Select all

if library.allBooksByISBN.includesKey (987654) then

[concord]

I need the object so it has to be a getAtKey.
The scenario was totally fictitious to get the point across. Thanks for your input.