Notifications slowing system

[John.Dalmau]

I have a list box control on a form, that I am using to show a list of products (that need to have their price updated because the cost has increased or whatever).
I am using notifications to update the list box as events happen, if the user has the form open.
In my development system everything works perfectly, but on the clients live database (about 100Gb system, 30 users running thin-client) these events are causing the users system to 'hang' until it's finished the events notifications.
The 'hanging' is not really noticeable when doing tests at night time after the users have logged out.

I thought notifications occurred outside the node that code is running in, so the user should be able to continue using the system while the notifications happen in the background.
Any ideas on how to remove the hanging?

Thanks,

John


My setup:
I have used beginNotification on the control, specifying the currently logged in User object and an event number.
eg. beginNotification(app.myCurrentUser, 6004, Response_Continuous, 0);

The control has a userNotify method that calls a method to rebuild the list on the control.

When a price event occurs I call a causeEvent method.
eg. oUser.causeEvent(6004, false, oUser);

Since a single price change can cause multiple events I cause each event as 'non-immediate', then run a begin/commit transaction after they are all banked up.
eg. foreach oUser in oGroup.allMembers discreteLock do
oUser.causeEvent(6004, false, oUser);
oUser.causeEvent(6012, false, oUser);
endforeach;
if not process.isInTransactionState() then
beginTransaction;
commitTransaction;
endif;

[ghosttie]

If the listbox only needs to be refreshed once regardless how many price events occur then why not just send one notification in total instead of one for each change?

[John.Dalmau]

Each price event can occur separate from other events, although they can process a bunch of them at once.
I was encapsulating the notifications against each event, maybe I could reduce the hanging time by reducing the number of notifications that are sent.
But that would only help with the hanging issue if it was the number of notifications that was causing it, I would have thought Jade could handle a lot of notifications without a noticable speed issue.

eg. I can process 20 prices in 3 seconds on my development system running single user, including the notifications.
Processing 20 prices on the clients system takes about 5 minutes. Using write statements I could see that my code took less than a second to process, the rest of the 5 minutes is waiting for the notifications to finish.

[ghosttie]

I think you need to narrow down where the time is being spent - having a lot of notifications shouldn't cause a problem, but if you're doing work for each one then it can add up.

How long does it take to run the code in userNotify and how many times is it being called?

[JohnP]

JADE Monitor's Method Analysis can help show where the time is going. Go to the Method Analysis sheet in the Navigator pane. Select the Process to Profile and click the Start button. After sending through the workload - maybe a price or two - click the Stop button. The methods at the top of the list are the ones where most of the time is being spent. Try it on the development system first, then the production. Also, you can check the counts on the userNotify method to confirm the number of notifications is what you expect.

[John.Dalmau]

Thanks for your help guys, I managed to resolve the problem.
There were 2 issues causing the slowness.
1) It was changing the colour of the items in the list box based on their status, the status was calling a method that used prices and costs, the price logic was taking a couple of seconds per item which added up to minutes once the number of items increased.
2) It was rebuilding the full list of items on the tile when the notification occured which meant the new items were in the correct sort order, but also took more time. I changed it to just add a new item to the list box, which means it shows at the bottom of the list and is not in its sorted position, but it saves heaps of time. The user can always press the re-sort button if they want the new items sorted. Also, I remove items individually from the list box on a tile.

[BeeJay]

Thanks for your help guys, I managed to resolve the problem.
There were 2 issues causing the slowness.
1) It was changing the colour of the items in the list box based on their status, the status was calling a method that used prices and costs, the price logic was taking a couple of seconds per item which added up to minutes once the number of items increased.
2) It was rebuilding the full list of items on the tile when the notification occured which meant the new items were in the correct sort order, but also took more time. I changed it to just add a new item to the list box, which means it shows at the bottom of the list and is not in its sorted position, but it saves heaps of time. The user can always press the re-sort button if they want the new items sorted. Also, I remove items individually from the list box on a tile.

As others stated, investigating what the userNotify code does when receiving the notification is normally the best place to start when you're noticing performance issues around notifications. However, you do still need to keep in mind that a very large flood of notifications could still produce an unresponsive application whilst it is processing those notifications, particularly if each notification results in changes to the UI. To that end, even though you've solved the immediate problem I will make the following comment about the example code as well:

I have used beginNotification on the control, specifying the currently logged in User object and an event number.
eg. beginNotification(app.myCurrentUser, 6004, Response_Continuous, 0);

The control has a userNotify method that calls a method to rebuild the list on the control.

When a price event occurs I call a causeEvent method.
eg. oUser.causeEvent(6004, false, oUser);

Since a single price change can cause multiple events I cause each event as 'non-immediate', then run a begin/commit transaction after they are all banked up.
eg. foreach oUser in oGroup.allMembers discreteLock do
oUser.causeEvent(6004, false, oUser);
oUser.causeEvent(6012, false, oUser);
endforeach;
if not process.isInTransactionState() then
beginTransaction;
commitTransaction;
endif;

Doing 2 causeEvents for each user in oGroup.allMembers won't necessarily scale very well as you add more and more users to the group as each user added to the group means more notifications to be distributed. If the system is only ever going to have a very small number of users in a group, then that's probably not of concern but if you could potentially have thousands of users in a group that's thousands of causeEvents being individually generated. A better approach would be to register for the notifications and do the causeEvents against a much smaller number of objects. For example, a common approach is to do the notifications against the "system" object, as then you're only doing causeEvents for a single object. In that case, there would be only two causeEvents issued per price change irrespective of the number of members in the group, rather than 2 x number of members in the group that the current code issues.

Further to this, you may want to consider creating a notification manager that takes care of grouping, dropping duplicates, and eventually sending the causeEvents. This allows you to handle the situation of an update that affects only a single price, versus an update that affects say 30 prices, still only issuing 2 causeEvents even when you update 30 prices in one go rather than 60 causeEvents, or 60 * number of members as happens with the current code. For a group with say 2,000 members, that would mean you go from issuing 2 * 2000 * 30 = 120,000 causeEvents to issuing only 2 causeEvents, thereby significantly reducing the load on the system. You would also be able to remove that horrible kludgy code that is checking process.isInTransactionState...

Cheers,
BeeJay.

[ghosttie]

the price logic was taking a couple of seconds per item

Wow, that's a long time to spend doing anything - I'd think you could improve that considerably, perhaps by pre-calculating some things and storing the results or having collections sorted on the right keys rather than searching through all of the items in the collection. It's hard to be specific without knowing more about what it's doing.

[John.Dalmau]

Doing 2 causeEvents for each user in oGroup.allMembers won't necessarily scale very well as you add more and more users to the group as each user added to the group means more notifications to be distributed. If the system is only ever going to have a very small number of users in a group, then that's probably not of concern but if you could potentially have thousands of users in a group that's thousands of causeEvents being individually generated. A better approach would be to register for the notifications and do the causeEvents against a much smaller number of objects. For example, a common approach is to do the notifications against the "system" object, as then you're only doing causeEvents for a single object. In that case, there would be only two causeEvents issued per price change irrespective of the number of members in the group, rather than 2 x number of members in the group that the current code issues.

When the status of a price updated is changed it has to be removed from one list box and added to another, so I created a 'remove from list x' event and an 'add to list y' event. The events do not always go together, for example cancelling a price would only call the remove event. Also, each price update might be submitted by one user (A) but have to be approved by a different user (B), so I'd cause an event for user A to remove the price update from his 'unsubmitted' list, and also cause an event for user B to add the price update to his 'awaiting approval' list. I did this at the user level because the beginNotifications were listening for the user object, and thus would only respond to anything affecting that user. I thought I'd have more notifications but only one thing responding per notification. If I did the causeEvents at the system level, I thought it would mean a lot less notifications being caused, but a lot more things responding per notification, in this case all users.

Further to this, you may want to consider creating a notification manager that takes care of grouping, dropping duplicates, and eventually sending the causeEvents. This allows you to handle the situation of an update that affects only a single price, versus an update that affects say 30 prices, still only issuing 2 causeEvents even when you update 30 prices in one go rather than 60 causeEvents, or 60 * number of members as happens with the current code. For a group with say 2,000 members, that would mean you go from issuing 2 * 2000 * 30 = 120,000 causeEvents to issuing only 2 causeEvents, thereby significantly reducing the load on the system. You would also be able to remove that horrible kludgy code that is checking process.isInTransactionState...

This is an excellent idea, I'll look at implementing this in the next version. The users have options for submitting or approving in bulk, and it would make a lot more sense to do a single list rebuild at the end of the database work via a single notification, instead of one notification per item that's processed.
Thanks to the speed increases I've also been able to remove the need for the 'process.isInTransactionState' by making the causeEvents immediate.

On a separate note, there was another problem that popped up thanks to using notifications.
When cancelling a price update my code would dispose/delete the price update and send a notification to remove that price update from the list it was in. This would cause an error (4: invalid object) because the notification was being processed after the price update had been deleted. Moving the causeEvent in my code to be before the delete didn't help, the delete still occurred first. Adding an 'if global.isValidObject' call on the removal didn't help because the object wasn't valid and thus the removal would never happen.
In the end, I changed the removal userNotify method to rebuild the list instead of trying to remove the object from the list.

[John.Dalmau]

Wow, that's a long time to spend doing anything - I'd think you could improve that considerably, perhaps by pre-calculating some things and storing the results or having collections sorted on the right keys rather than searching through all of the items in the collection. It's hard to be specific without knowing more about what it's doing.

Unfortunately my code is a customisation to another product, so I have to use the other products methods for determining the cost/price.
But, as you suggest, I did change the calculation of the cost/price/margin to occur only when they changed the price update. This meant it had no effect on the building of the list box. It does open up a slight chance that the calculated margins may be wrong if the underlying cost/price changes after the user has created the price update, but in the end it shouldn't matter because if they approve an out of date price update the system will automatically create a new price update due to the margins being too low.