Integrity checking

[allistar]

Hi all,
In most database applications we want to prevent an object from being deleted if another object is referencing it. An easy way to do this is to add a collection maintained by an inverse, and then do an "isEmpty" check on the collection. If the collection is not empty, then the object cannot be deleted. A major problem with this approach is that the collection becomes a high point of contention, killing performance in high volume systems. In many cases we cannot live with this contention issue and so the collection gets removed. This leaves the question: how do we prevent the user from deleting the object if it's being used? One solution it to not allow them to delete it, and instead we hide it from them in user interfaces (by having a parallel collection that has a condition "not isHidden" against it). Another solution is we run around all objects that could reference this object and check - but in many cases this is an extremely expensive check, and it has the timing hole of allowing someone to refer to the object before the delete check has passed.

Have you had this problem in the design of your JADE applications and what strategy have you used to solve it?

Thanks,
Allistar.

[torrie]

We use similar approaches, having a "Visible" flag with conditional collections.

In order to minimise the contention, our transaction layer recognises certian collections as contention points. The manual end of these inverses are the very last properties updated as part of the transaction cycle, minimising the time that the collections are locked. We also strive to keep the transaction time as short as possible with most validation etc done prior to any updates.

We've also used approaches where the child object is updated and put into a staging collection on another object (often the object representing the user who made the change.) A back ground process, then updates that object, removing it from the staging collection and adding it to the global collection. This way only one process is updating that collection.

[JohnP]

Sometimes, in updating transactions, a significant amount of time is spent reading from the database. For example, if an object is to be added into a collection, the collection blocks containing the appropriate key values are read in. This is frequently done in transaction state, after locking the collection. If the blocks are already in the node's cache, this goes very quickly, but if they have to be read in from disk, this can add significantly to the length of time spent in transaction state, with collections locked. This is usually worse if the collections are large, or there are a large number of collections involved, and the updates happen at random points spread throughout the collections.

You can help reduce transaction times and the length of time collections are locked by warming the node cache prior to updating. Quoting from page 16 of the JADE Perforfmance Analysis whitepaper (included with JADE 6.3 documentation):

"If an object is to be added, the collections to which the object will be added can be warmed. This can be done with Collection::includes or Dictionary::includesKey. Similarly, if objects are to be deleted, the collections can be warmed with Collection::includes."

These functions will help ensure that the collections blocks required for the update will be in cache when required, and can significantly reduce the time collections are locked. It does increase the overall amount of processing, but that can be a small price to pay for reduced contention.

There is more discussion of this topic in the whitepaper, although the scenario is not exactly the same.