Manual versus Automatic collections

[ConvertFromOldNGs]

by Eric Peachey >> Sun, 22 Aug 1999 22:54:42 GMT

Hello,

Some advice we've received from more experienced JADE consultants recommend that you avoid manually maintained collections and stick to automatic collections. Using only automatic collections seems to present problems in some areas and you seem to fight JADE all the time. I understand the problems associated with maintaining manual collections but how hard can it be to build classes such that the objects make sure they're in the right collections?

I'd be interested in what approach you others take to this issue.

Eric Peachey
Dunedin

[ConvertFromOldNGs]

by Craig Shearer >> Mon, 23 Aug 1999 0:50:50 GMT

Hi Eric

This is certainly good advice. I try to avoid manually maintained collections "like the plague".

The major issue with manually maintaining collections is when you have Dictionary-type collections where the keys must be maintained. For example, if you add an object to a MemberKeyDictionary collection, the object is accessable only via the values of the keys that existed on the object when it was added to the collection. If you subsequently change the values of the keys then, unless the collection is automatically maintained, you won't be able to access the object in the collection any longer (for example, when trying to remove the object from the collection, you'll get an exception saying that the object's not in the collection). JADE's automatic collection feature is very powerful, and usually transparent to the developer which is why it's so easy to use.

Perhaps you can give us some examples where this presents problems and you are fighting JADE.

Building classes to automatically maintain manual collections is actually quite difficult. Remember that to do this you would have to trap changes to all properties on an object that participated in a key or key path and remove them from their associated dictionary collections, change the key, then re-add them to the collection. While this can be done using mapping methods or accessor methods, it becomes a maintenance nightmare. If the keys of the member key dictionary class are changed then you need to also maintain the methods that trap changes to key values in parallel. Also, how do you identify all the collections that the object belongs to if you don't have an inverse? Handling this stuff with key paths would be even more difficult!

The main area where manually maintained collections might be useful would be having objects conditionally belonging to a collection. JADE once was going to have a conditional collection membership feature where presumeably you'd specify some rule that determined whether or not an object belonged to a collection.

To take an example, say you wanted to have a collection of all customers with outstanding invoices, ordered by customer name. You could build a manually maintained collection on your (say) company object and then trap changes to the invoice status to add or remove customers from the collection. But then, you'd also have to remember to maintain the collection if the customer's name changed.

A better (if somewhat artificial) solution to the above problem would be to introduce an additional reference on the Customer class to its Company, eg. myOutstandingInvoiceCompany. Then, arrange for this reference to point to the customer's company when it has outstanding invoices, and have a null value when it doesn't. Then, you have a collection on Company (say) allMyCustomersWithOSInvoices that has an automatically maintained inverse to myOutstandingInvoiceCompany.

What do others think?

Craig.
Auckland!

[ConvertFromOldNGs]

by Eric Peachey >> Mon, 23 Aug 1999 2:11:11 GMT

Thank you Craig for your summary of the issues involved and an interesting idea.

Perhaps you can give us some examples where this presents problems and you are fighting JADE.

Okay, let's try with this example. Imagine we have three classes: Employer, Insurer and Policy. An employer has a collection of policies keyed by date - the latest one being the current policy, and old policies are there for policy history. A policy has an employer (myEmployer) and an insurer (myInsurer). An insurer object has a collection of current policies (i.e. the insurer is not interested in historical policies) called allPolicies. The inverse of Policy::myInsurer is Insurer.allPolicies. Problem is the Insurer only wants the current policies, but the historical policies also point to an insurer's collection of policies. We still need the reference on policy to insurer so that we can tell who the insurer is. Unless we add some other property into the keys of Insurer::allPolicies so that we can group the current and historical policies to avoid trawling through all of them I'm not sure what you'd do. Craig's point about 'additional references' (see below) may be an interesting work round.

The main area where manually maintained collections might be useful would be having objects conditionally belonging to a collection. JADE once was going to have a conditional collection membership feature where presumeably you'd specify some rule that determined whether or not an object belonged to a collection.

This would be handy. The LINC systems that go through the JET sausage machine to produce JADE code/schema use a lot of conditional collections (all manually maintained). I imagine we'll get JADE conditional collections in a future release. Even the idea of null keys so that an object with null values in all its keys (in the case of a composite keyed index) doesn't appear in a collection (okay you can a reference to null so that the object won't appear in the reference's inverse collection). From memory Btrieve and Microfocus COBOL had this kind of feature - so you could easily have an index/collection of all invoices that only included unpaid ones.

A better (if somewhat artificial) solution to the above problem would be to introduce an additional reference on the Customer class to its Company, eg.

I suppose having 'artificial' references seems to be less than pure but if it helps use the functionality provided then we should use it. I think using this kind of mechanism might be an answer...

Craig.
Auckland!

You poor b*****d

Eric in Dunedin where 3 scuba divers are stealing the crayfish under our building!

[ConvertFromOldNGs]

by Craig Shearer >> Mon, 23 Aug 1999 2:52:08 GMT

Okay, let's try with this example. Imagine we have three classes: Employer, Insurer and Policy. An employer has a collection of policies keyed by date - the latest one being the current policy, and old policies are there for policy history. A policy has an employer (myEmployer) and an insurer (myInsurer). An insurer object has a collection of current policies (i.e. the insurer is not interested in historical policies) called allPolicies. The inverse of Policy::myInsurer is Insurer.allPolicies. Problem is the Insurer only wants the current policies, but the historical policies also point to an insurer's collection of policies. We still need the reference on policy to insurer so that we can tell who the insurer is. Unless we add some other property into the keys of Insurer::allPolicies so that we can group the current and historical policies to avoid trawling through all of them I'm not sure what you'd do. Craig's point about 'additional references' (see below) may be an interesting work round.

Well, as you've guessed, what I would do in this situation would be as follows:

Policy Class:

myInsurer - no inverse
myActiveInsurer - automatic inverse to:

Insurer Class:

allMyPolicies - inverse to Policy::myActiveInsurer

thus myInsurer is always populated and is used to find out who the insurer is for the policy. The myActiveInsurer is only populated when the policy is active. Thus could could write methods such as:

isActive(): Boolean;
begin

return myActiveInsurer <> null;
end;

activate() updating;
begin

myActiveInsurer := myInsurer;
end;

deactivate() updating;
begin

myActiveInsurer := null;
end;

You poor b*****d

It's not so bad! At least we don't suffer from snow!

[ConvertFromOldNGs]

by Allistar Melville >> Thu, 26 Aug 1999 9:48:38 GMT

[snip]

A better (if somewhat artificial) solution to the above problem would be to introduce an additional reference on the Customer class to its Company, eg. myOutstandingInvoiceCompany. Then, arrange for this reference to point to the customer's company when it has outstanding invoices, and have a null value when it doesn't. Then, you have a collection on Company (say) allMyCustomersWithOSInvoices that has an automatically maintained inverse to myOutstandingInvoiceCompany.

What do others think?

We have the need for conditional membership all opver the place in our product and the above solution is exactly the way we do it. It is so
much easier to maintain than manually adding/removing the objects from the appropriate collections.

Allistar.

------------------------------------------------------------------
Allistar Melville (BSc) Home: allistar@ihug.co.nz \_
Software Developer Work: allistar@focussoft.co.nz </'
Auckland, NEW ZEALAND /)
(/`
"Science built the Academy, superstition the inquisition."
[Robert G. Ingersoll] ------------------------------------------------------------------

[BeeJay]

by Allistar Melville >> Thu, 26 Aug 1999 9:48:38 GMT

[snip]

A better (if somewhat artificial) solution to the above problem would be to introduce an additional reference on the Customer class to its Company, eg. myOutstandingInvoiceCompany. Then, arrange for this reference to point to the customer's company when it has outstanding invoices, and have a null value when it doesn't. Then, you have a collection on Company (say) allMyCustomersWithOSInvoices that has an automatically maintained inverse to myOutstandingInvoiceCompany.

What do others think?

We have the need for conditional membership all opver the place in our product and the above solution is exactly the way we do it. It is so
much easier to maintain than manually adding/removing the objects from the appropriate collections.

Allistar.

Out of interest, are you still doing it this way or have you now moved to using condition methods as constraints on your inverses and removed the then redundant myNonInversed reference(s)?

Cheers,
BeeJay.

[allistar]

by Allistar Melville >> Thu, 26 Aug 1999 9:48:38 GMT

[snip]

A better (if somewhat artificial) solution to the above problem would be to introduce an additional reference on the Customer class to its Company, eg. myOutstandingInvoiceCompany. Then, arrange for this reference to point to the customer's company when it has outstanding invoices, and have a null value when it doesn't. Then, you have a collection on Company (say) allMyCustomersWithOSInvoices that has an automatically maintained inverse to myOutstandingInvoiceCompany.

What do others think?

We have the need for conditional membership all opver the place in our product and the above solution is exactly the way we do it. It is so
much easier to maintain than manually adding/removing the objects from the appropriate collections.

Allistar.

Out of interest, are you still doing it this way or have you now moved to using condition methods as constraints on your inverses and removed the then redundant myNonInversed reference(s)?

Cheers,
BeeJay.

For the most part this has not changed - primarily because the approach described above "just works". We have used conditions for new structures that would benefit from them. I have noted that if you set the property that the condition checks after you set the inverse, performance is impacted a lot more than if you were changing a key property for a dictionary. Of course, it's wise to set the condition properties and keys first - it's just interesting that the performance hit with maintaining the condition is a lot more than with maintaining a key.

We are going through a process of trimming down the number of collections in the database with the intention of reducing contention. We have a traditional rule that "every persistent reference or collection must have an inverse". We insist on this to maintain data integrity - dealing with collections that have holes in them is not nice. In many cases the collection on the "other" side of the inverse is used only for deletion checking. I.e. I can't delete this object if any of it's parent collections are not empty. We've realised that this policy isn't very good for performance - those collections can become a point of contention. The goal now is to remove the collections (and hence the inverse) and we don't allow the user to delete the object - instead they make it "inactive" and we hide it from them in most parts of the user interface.

It's a shame in a way to trade off elegance in design for performance, but at the end of the day it's the end user that matters, and so performance wins.

Cheers,
Allistar.

[Dr Danyo]

Hi Beejay,

On a similar vein I would be curious to know what people think of the man/auto feature, I personally advocate avoiding it where possible as I think forcing the developer to think about which side is auto and which is manual results in a better design. I'm sure there are plenty of exceptions to this, but I struggle to come up with real world examples.

- Dr Danyo.

[allistar]

Hi Beejay,

On a similar vein I would be curious to know what people think of the man/auto feature, I personally advocate avoiding it where possible as I think forcing the developer to think about which side is auto and which is manual results in a better design. I'm sure there are plenty of exceptions to this, but I struggle to come up with real world examples.

- Dr Danyo.

I agree - man/auto is nasty unless it's really needed. One example is where you want the ability to easily clear a peer-peer collection without iterating through instances in the collection and setting the "my" side to null. Doing a .clear is much easier, and so man/auto is justified. This is also true of peer-peer. I find it's a common mistake of "junior" developers to use peer-peer and man/auto everywhere. Parent/child is a key part of the inverse relationship and it infers which object "owns" the other object. Using this information in code is handy for things like "if I disable an object, then automatically disable all child objects".

One of the great powers of the Jade language is the ability to look at metadata at runtime. Without that there are so many nifty features that would be much more difficult to implement.

Allistar.

[BeeJay]

We are going through a process of trimming down the number of collections in the database with the intention of reducing contention. We have a traditional rule that "every persistent reference or collection must have an inverse". We insist on this to maintain data integrity - dealing with collections that have holes in them is not nice. In many cases the collection on the "other" side of the inverse is used only for deletion checking. I.e. I can't delete this object if any of it's parent collections are not empty. We've realised that this policy isn't very good for performance - those collections can become a point of contention. The goal now is to remove the collections (and hence the inverse) and we don't allow the user to delete the object - instead they make it "inactive" and we hide it from them in most parts of the user interface.

It's a shame in a way to trade off elegance in design for performance, but at the end of the day it's the end user that matters, and so performance wins.

With the exception of "global" collections of "all instances" of the related class, in most cases the collections can be targetted in such a way as to reduce contention between users. In the 6.3 release, the option of using merged iterators means that "global" collections, which often were created mainly for reporting purposes, can now be removed in favour of a merged iterator when you need to report over "all instances" of a given class.

Another option if you "must" have the global collections, is to consider using a single background process in "lazy update" mode to avoid the contention between multiple processes. Such lazy updaters will normally keep up with a significant number of interactive users with very little lag time before the newly created instances are in the collection. This doesn't help when you're updating the key value on a MKD inverse, but for the most part this is a far less frequent activity than the adding of new entries. Care does need to be taken to handle situations where two people simultaneously create an entry whereby the "lazy update" would result in duplicate key exceptions.

If the contention is between readers and an updater, rather than multiple updaters, then consideration could also be given to using the new Update Lock option to allow other non-updating processes to continue to read the affected collection(s) for as long as possible until such time as the transaction has to be committed.

ie: I'd look to find other ways of reducing contention before I started risking referential integrity with non-inversed references, especially if that means going to manually maintained collections over objects where the key value can and does change, and would only ever consider this as an absolute last resort option.

On a similar vein I would be curious to know what people think of the man/auto feature, I personally advocate avoiding it where possible as I think forcing the developer to think about which side is auto and which is manual results in a better design. I'm sure there are plenty of exceptions to this, but I struggle to come up with real world examples.

I'm with you and Allistar on this one. There should be very few occasions at all where a man/auto inverse is ever justifiable. The lack of thought in this area rates right up there with code such as the following:

Code: Select all

vars didBeginTrans : Boolean; begin if not process.isInTransactionState then beginTransaction ; didBeginTrans := true ; endif; ... if didBeginTrans then commitTransaction ; endif; end;

Cheers,
BeeJay.