quickest/most efficient way to get a subset of a keyed colle

[ConvertFromOldNGs]

by johnmunro >> Wed, 22 Sep 2004 11:08:51 GMT

You have a collection keyed on, for example, date. You want a collection containing only the objects between two dates. Is this the quickest/most efficient way of doing it?

getMKDSubset(pMKD : MemberKeyDictionary; pAny_StartKey, pAny_EndKey : Any) : Collection final;

vars
collResults : Collection;
iter : Iterator;
obj, objStop : Object;
begin

create collResults transient;

iter := pMKD.createIterator;

pMKD.startKeyGeq(pAny_StartKey, iter);

objStop := pMKD.getAtKeyLeq(pAny_EndKey);

while iter.next(obj) and (obj <> objStop) do
coll.add(obj);
endwhile;

return collResults;

epilog
delete iter;

end;

It would be cool if there was a built in (and therefore by definition quicker) method for doing this sort of thing...

--

John Munro

FileVision UK Ltd.
The Bioscience Innovation Centre
Cowley Road, Cambridge, UK
CB4 0DS

Telephone: +44 (0) 1223 478200
Fax: +44 (0) 1223 477969
Email: john.munro@filevision.com
Web: http://www.filevision.com

The contents of this communication are confidential and are only intended to be read by the addressee. We apologize if you receive this communication in error and ask that you contact FileVision UK Ltd. immediately to arrange for its return. The use of any information contained in this communication by an unauthorized person is strictly prohibited. FileVision UK Ltd. cannot accept responsibility for the accuracy or completeness of this communication as it is being transmitted over a public network. If you suspect this message may have been intercepted or amended, please inform FileVision UK Ltd.

[ConvertFromOldNGs]

by allistar >> Thu, 23 Sep 2004 1:24:44 GMT

You can use the "where" clause or a "foreach" statement as JADE optimises the search if the conditions in the where clause are based on keys of the dictionary. In the latest releases of JADE 6 you don't get the collection locking issues with foreaches that you did in JADE 5.

Otherwise your while loop looks like a good solution. You may want to do a few sanity checks to make sure that pAny_StartKey is actually less that pAny_EndKey and that the dictionay is not empty. Also I would pass through the "collResults" collection into the method as an input parameter rather than creating it and returning it (will the caller always know to delete the collection that is returned?) Making the caller responsible for both creating and deleting the transient collection will lead to less transirnt leaks - especially if other people will be using your code as well.

Putting the "obj <> objStop" is a good idea, better than "obj.keyProperty <= pAny_EndKey" as the former will fetch the object from the database to compare the properties. Your approach will not need to fetch the object from the database as only the oid is being compared.(Someone correct me if I am wrong on this).

Regards,
Allistar.
--
------------------------------------------------------------------
Allistar Melville
Software Developer, Analyst allistar@silvermoon.co.nz
Auckland, NEW ZEALAND

Silvermoon Software
Specialising in JADE development and consulting
Visit us at: http://www.silvermoon.co.nz
*NEW* Simple web access to Jade at: www.silvermoon.co.nz/jhp.html ------------------------------------------------------------------

[ConvertFromOldNGs]

by johnmunro >> Thu, 23 Sep 2004 14:47:46 GMT

I wanted to use foreach rather than an iterator because it seems faster if you use lots of them (possibly time saved by not creating the Iterator instance) but using a foreach would require referencing the keyed property on the object in the collection, and in this case I needed it to be generic. The only assumption this method makes is that the MKD only has one key.

I would have actually preferred to use Dictionary.getIteratorKeys rather than objStop in case this resulted in less loads from the db, but Any doesn't support less than or greater than to compare the current iterator key against pAny_EndKey.

This method is part of a query agent which allows searching any collection, so creating and returning a collection isn't a problem because this method will only be called internally by the query agent, which is responsible for tracking transients it creates. In fact I simplified the method before posting it - the original version didn't create the return collection explicitly, it called a method on the query agent which created the collection and kept a reference to it in a collection which is purged in the destuctor as a failsafe in case a delete was missing from an epilog.

Sanity checks are all done at the point of initialising the query agent, so this method doesn't have to worry about it.

--

John Munro

FileVision UK Ltd.
The Bioscience Innovation Centre
Cowley Road, Cambridge, UK
CB4 0DS

Telephone: +44 (0) 1223 478200
Fax: +44 (0) 1223 477969
Email: john.munro@filevision.com
Web: http://www.filevision.com

The contents of this communication are confidential and are only intended to be read by the addressee. We apologize if you receive this communication in error and ask that you contact FileVision UK Ltd. immediately to arrange for its return. The use of any information contained in this communication by an unauthorized person is strictly prohibited. FileVision UK Ltd. cannot accept responsibility for the accuracy or completeness of this communication as it is being transmitted over a public network. If you suspect this message may have been intercepted or amended, please inform FileVision UK Ltd.

[ConvertFromOldNGs]

by allistar >> Fri, 24 Sep 2004 6:22:23 GMT

Hi John,
If ultra-efficient searches is what you're after then a good solution I have used in the past is to generate optimised JADE code (in a String) that then gets compiled and executed at runtime. You do have the overhead of generating the code (which is very fast) and compiling it (which is noticable for small queries) but the advantage of searching on big collections is quite noticable. The query code you generate can be tailored to your specific query, and in my mind gives the best, fastest generic search tool.

Regards,
Allistar.
--
------------------------------------------------------------------
Allistar Melville
Software Developer, Analyst allistar@silvermoon.co.nz
Auckland, NEW ZEALAND

Silvermoon Software
Specialising in JADE development and consulting
Visit us at: http://www.silvermoon.co.nz
*NEW* Simple web access to Jade at: www.silvermoon.co.nz/jhp.html ------------------------------------------------------------------

[ConvertFromOldNGs]

by johnmunro >> Fri, 24 Sep 2004 9:26:49 GMT

Yeah, one of the searches I'm replacing with the query agent uses a transient method, the problem I found with it is that it's impossible to debug when something goes wrong


--

John Munro

FileVision UK Ltd.
The Bioscience Innovation Centre
Cowley Road, Cambridge, UK
CB4 0DS

Telephone: +44 (0) 1223 478200
Fax: +44 (0) 1223 477969
Email: john.munro@filevision.com
Web: http://www.filevision.com

The contents of this communication are confidential and are only intended to be read by the addressee. We apologize if you receive this communication in error and ask that you contact FileVision UK Ltd. immediately to arrange for its return. The use of any information contained in this communication by an unauthorized person is strictly prohibited. FileVision UK Ltd. cannot accept responsibility for the accuracy or completeness of this communication as it is being transmitted over a public network. If you suspect this message may have been intercepted or amended, please inform FileVision UK Ltd.

[ConvertFromOldNGs]

by allistar >> Sun, 26 Sep 2004 6:43:06 GMT

Impossible using the debugger, yes. I have always favoured write statements for debugging over using the debugger. Especially for debuggin a GUI, as the precense of the debugger causes erroneous events, like lostFocus, to be triggered which can stuff up the test.

Regards,
Allistar.
--
------------------------------------------------------------------
Allistar Melville
Software Developer, Analyst allistar@silvermoon.co.nz
Auckland, NEW ZEALAND

Silvermoon Software
Specialising in JADE development and consulting
Visit us at: http://www.silvermoon.co.nz
*NEW* Simple web access to Jade at: www.silvermoon.co.nz/jhp.html ------------------------------------------------------------------

[ConvertFromOldNGs]

by johnmunro >> Mon, 27 Sep 2004 8:49:24 GMT

well yeah, but what i meant was you end up with two layers of code - the code that generates the search method and the search method itself, so when something isn't working it makes it that much more difficult to work out where the problem is

--

John Munro

FileVision UK Ltd.
The Bioscience Innovation Centre
Cowley Road, Cambridge, UK
CB4 0DS

Telephone: +44 (0) 1223 478200
Fax: +44 (0) 1223 477969
Email: john.munro@filevision.com
Web: http://www.filevision.com

The contents of this communication are confidential and are only intended to be read by the addressee. We apologize if you receive this communication in error and ask that you contact FileVision UK Ltd. immediately to arrange for its return. The use of any information contained in this communication by an unauthorized person is strictly prohibited. FileVision UK Ltd. cannot accept responsibility for the accuracy or completeness of this communication as it is being transmitted over a public network. If you suspect this message may have been intercepted or amended, please inform FileVision UK Ltd.

[ConvertFromOldNGs]

by JadeWatcher >> Mon, 27 Sep 2004 22:13:57 GMT

One could also argue that developers shouldn't have to resort to dirty techniques like code generation on the fly just to get performance. Surely such "query optimisation" is what Jade should be doing by iteslf?

[ConvertFromOldNGs]

by allistar >> Tue, 28 Sep 2004 1:18:38 GMT

I agree, but you have to work with what you've got. The JADE database doesn't have any query tools built into it, other than using the JADE langauge itself.

Regards,
Allistar.
--
------------------------------------------------------------------
Allistar Melville
Software Developer, Analyst allistar@silvermoon.co.nz
Auckland, NEW ZEALAND

Silvermoon Software
Specialising in JADE development and consulting
Visit us at: http://www.silvermoon.co.nz
*NEW* Simple web access to Jade at: www.silvermoon.co.nz/jhp.html ------------------------------------------------------------------

[ConvertFromOldNGs]

by cnwjhp1 >> Wed, 29 Sep 2004 4:27:11 GMT

That is a big ask though, and even then you would have to know what optimisations it was doing so you could drive it properly. A certain prominent database vendor has been doing query optimisations for years, but last I knew if you tried this type of query, you might even have difficulty convincing it to use the collection you thoughtfully keyed by date. It might decide to read up all the objects and sort them.