Maintaining collections that are frequently updated

Discussions about design and architecture principles, including native JADE systems and JADE interoperating with other technologies
ConvertFromOldNGs
Posts: 5321
Joined: Wed Aug 05, 2009 5:19 pm

Re: Maintaining collections that are frequently updated

Postby ConvertFromOldNGs » Fri Aug 07, 2009 11:27 am

by cdshearer >> Tue, 26 Mar 2002 21:48:17 GMT

OK, perhaps I should have given some context to what I'm using this for...

I'm using the cache for many small-ish (a few hundred objects at most) collections of lookup objects that are used to populate ComboBox and ListBox controls. The collections are very rarely updated but likely to be frequently read. Caching the collections on the client, particularly in an Application Server environment will eliminate the need to go to the database to lock the collection when it's used to populate the controls. Additionally, the controls register for notifications on the cached collection rather than the persistent one so the notifications are transient, and don't involve the database server.

We also have the concept of effective date ranges for these lookups, so loading the collection into the control involves more that just attaching the collection to the control as it needs to be filtered based upon the current date. For this reason, we effectively do the filtering once per day for all clients. We expect that this will make the solution more scalable, not less!!!

I agree though that the solution wouldn't be appropriate for every collection - there are some that are too large or update too frequently for the technique to be of any benefit.

Craig

ConvertFromOldNGs
Posts: 5321
Joined: Wed Aug 05, 2009 5:19 pm

Re: Maintaining collections that are frequently updated

Postby ConvertFromOldNGs » Fri Aug 07, 2009 11:27 am

by LHUfvoCGqMV >> Mon, 11 Jun 2007 2:08:28 GMT

dfgfdhgsd

ConvertFromOldNGs
Posts: 5321
Joined: Wed Aug 05, 2009 5:19 pm

Re: Maintaining collections that are frequently updated

Postby ConvertFromOldNGs » Fri Aug 07, 2009 11:27 am

by YivSfCitvVceIeh >> Mon, 11 Jun 2007 20:58:36 GMT

sdgdssddsg

ConvertFromOldNGs
Posts: 5321
Joined: Wed Aug 05, 2009 5:19 pm

Re: Maintaining collections that are frequently updated

Postby ConvertFromOldNGs » Fri Aug 07, 2009 11:27 am

by FbxuSdSdfHS >> Sat, 16 Jun 2007 6:40:27 GMT

<a href="http://yeahsearch.info/blog/2007/06/09/ ... som-hotel/"
Mixing of sphere phentermine</a>

ConvertFromOldNGs
Posts: 5321
Joined: Wed Aug 05, 2009 5:19 pm

Re: Maintaining collections that are frequently updated

Postby ConvertFromOldNGs » Fri Aug 07, 2009 11:27 am

by FbxuSdSdfHS >> Sat, 16 Jun 2007 6:40:37 GMT

<a href="http://yeahsearch.info/blog/2007/06/09/ ... som-hotel/"
Mixing of sphere phentermine</a>

ConvertFromOldNGs
Posts: 5321
Joined: Wed Aug 05, 2009 5:19 pm

Re: Maintaining collections that are frequently updated

Postby ConvertFromOldNGs » Fri Aug 07, 2009 11:27 am

by FbxuSdSdfHS >> Sat, 16 Jun 2007 6:40:49 GMT

<a href="http://yeahsearch.info/blog/2007/06/09/ ... som-hotel/"
Mixing of sphere phentermine</a>

ConvertFromOldNGs
Posts: 5321
Joined: Wed Aug 05, 2009 5:19 pm

Re: Maintaining collections that are frequently updated

Postby ConvertFromOldNGs » Fri Aug 07, 2009 11:27 am

by paulMathews >> Wed, 9 Apr 2003 15:03:52 GMT

Has anyone noticed a reduction in lock exceptions in 5.2 because of this change.

ConvertFromOldNGs
Posts: 5321
Joined: Wed Aug 05, 2009 5:19 pm

Re: Maintaining collections that are frequently updated

Postby ConvertFromOldNGs » Fri Aug 07, 2009 11:27 am

by Patwos >> Sat, 12 Apr 2003 9:48:20 GMT

I never really had many lock contentions occurring in my applications anyway, but I have certainly noticed an improvement in performance on the 5.2 release.

ConvertFromOldNGs
Posts: 5321
Joined: Wed Aug 05, 2009 5:19 pm

Re: Maintaining collections that are frequently updated

Postby ConvertFromOldNGs » Fri Aug 07, 2009 11:27 am

by allistar >> Thu, 21 Mar 2002 1:34:49 GMT

Hi Geoff,
A way that we have gotten around this very problem is to have 10
order collections on the root object, not just one, like this:

allOrders0
allOrders1
allOrders2
allOrders3
.....
allOrders9

Each of these collections will need a corresponding inverse on the
Order class, meaning there will be 10 "myRoot1", "myRoot2" etc
inverses.

If the key in the allOrders collection is an integer, then great, as
you can take the last digit of the order and use that to determine
which collection the order should go in (which determines which
"myRoot" inverse you should set).

When doing a .getAtKey to find a particular order, you first need to
work out which collection it will be in, then do a .getAtKey on that collection.

The end result is that you have broken down the collection into
multiple collection,s, and if orders get added with sequential
numbers, then each collection will only be updated in 1 out of every
10 orders.

If the key is a string, then what we have done is used a hashing
method to convert that string to a number (between 0 and 10). The
trick with this is to design the hashing method so it return an even distribution of numbers for the keys you will put into it. (I.e. there
is no point if the hashing method wants to stick every order into the
"0" collection).

We have used this approach, and it greatly alleviated locking problems
on the main collection. You can have as many or as few collections as
you would like. And the great thing is that in the future you could
add more collections into the mix (although that would require a
change to the hashing method and also a large reorg/script to move the objects around).

Hope this idea helps,
Allistar. ------------------------------------------------------------------
Allistar Melville
Software Developer
Auckland, NEW ZEALAND

Greentree International,
Developer of Greentree Financial Software. ------------------------------------------------------------------

ConvertFromOldNGs
Posts: 5321
Joined: Wed Aug 05, 2009 5:19 pm

Re: Maintaining collections that are frequently updated

Postby ConvertFromOldNGs » Fri Aug 07, 2009 11:27 am

by cdshearer >> Thu, 21 Mar 2002 2:26:20 GMT

Hi Allistar

I must say I admire your determination at solving performance problems in this way, but you'd think that this sort of optimisation (which really is dictating the physical database design) would best be left up to the DBMS rather than the application. It seems to me you're buying yourself a maintenance nightmare with this sort of design!

What sort of level of hardware and volume of transactions required this sort of approach?

Just interested...

Craig.


Return to “Design and Architecture”

Who is online

Users browsing this forum: No registered users and 12 guests