Transient cleanup and DictIterator

[ConvertFromOldNGs]

by JensRichnow >> Sun, 22 Dec 2002 20:27:23 GMT

The tight coupling with the Jade DB could be used to have a solid transient garbage collection. In Java the garbage collection is not 'fool proof' as there might arise circumstances in which transient leaks occur.

I just come across a nice pattern dealing with the management of objects: "Object Lifetime Manager" pattern which can be used to manage the entire life time of objects. Although it concentrates mainly on the issues of object creation and deletion in respect to application initialisation and termination, the pattern can be extended to the management (recycling) of transients in the course of the application.

One reference to this pattern would be: "The Object Lifetime Manager Pattern", D.L. Levine, C.D. Gill & D.C. Schmidt In "Design Patterns in Communication Software" ed. by Linda Rising, 2001, SIGS Reference Library.

A Google search gave many online documents for this pattern as well.

[ConvertFromOldNGs]

by JensRichnow >> Thu, 7 Nov 2002 3:55:10 GMT

I was going to reply but maybe I hit the wrong button - was back at home page....

Any way. An approach I'm using is more task-based. Whenever it is (i)hard to keep track of created/cloned/copied transients, and (ii) I have a well-defined task I put all these transients in a 'garbage bin' (ObjectArray) either on the application level or the task manager level. After accomplishing the task the bin gets purged. This works fine for repeated tasks where re-use of these transients is not feasible.

I agree, that implementing the garbage collection at the kernel level would free us programmers from accomplishing this task.

[ConvertFromOldNGs]

by JensRichnow >> Thu, 7 Nov 2002 1:33:53 GMT

Hi Craig,

thanks for the lengthy reply. Wasn't in the office yesterday. I do use a modified version of your example in the app::finalise(), e.g.,

vars
coll :ObjectArray;
o :Object;
count :Integer;
s :Schema;
c :Class;
classColl :ClassColl;

begin
s := currentSchema;
classColl := s.getAllClasses(true);
create coll transient;

foreach c in classColl do
c.allProcessTransientInstances(coll, 0, false); // this returns all the transients
// for this process // to obtain all the transients for a Node use
// c.allNodeTransientInstances(coll, 0, false);
foreach o in coll do
write ' Transient not yet deleted -' & o.String;
count := count + 1;
endforeach;

coll.clear;
endforeach;

delete coll;
delete classColl;

write "total transients not deleted = " & count.String;
end;

Note, that I DO INCLUDE the system classes. And I WAS SURPISED to find many transient instances of system classes (see example out put below). This includes default system classes I'm actively not using in the schema, such as CurrencyFormat, Printer and others. Most concerning however was the exceeding listing of the DictIterator which sometimes runs in the hundreds of instances. And that for only a few executions some methods within the app.

Transient not yet deleted -CurrencyFormat/332.13
Transient not yet deleted -DateFormat/329.13
Transient not yet deleted -DictIterator/97.1324
Transient not yet deleted -DictIterator/97.1325
Transient not yet deleted -DictIterator/97.1343
Transient not yet deleted -LocaleFullInfo/199.13
Transient not yet deleted -NumberFormat/331.13
Transient not yet deleted -Printer/190.13
Transient not yet deleted -TimeFormat/330.13

I hope that this clarifies my confusing question.

Do you have an idea where the transient instances of the DictIterator are generated? Does any dictionary lookup, such as ::getAtKey() implicitely creates such a guy without me haviing a handle to deal with it?

[ConvertFromOldNGs]

by allistar >> Thu, 7 Nov 2002 3:05:51 GMT

On Thu, 7 Nov 2002 1:33:53 GMT, jrichnow@maketxt.com (JensRichnow)
wrote:

[snip]

Note, that I DO INCLUDE the system classes. And I WAS SURPISED to find many transient instances of system classes (see example out put below). This includes default system classes I'm actively not using in the schema, such as CurrencyFormat, Printer and others. Most concerning however was the exceeding listing of the DictIterator which sometimes runs in the hundreds of instances. And that for only a few executions some methods within the app.

Transient not yet deleted -CurrencyFormat/332.13
Transient not yet deleted -DateFormat/329.13
Transient not yet deleted -DictIterator/97.1324
Transient not yet deleted -DictIterator/97.1325
Transient not yet deleted -DictIterator/97.1343
Transient not yet deleted -LocaleFullInfo/199.13
Transient not yet deleted -NumberFormat/331.13
Transient not yet deleted -Printer/190.13
Transient not yet deleted -TimeFormat/330.13

I hope that this clarifies my confusing question.

Do you have an idea where the transient instances of the DictIterator are generated? Does any dictionary lookup, such as ::getAtKey() implicitely creates such a guy without me haviing a handle to deal with it?

You can find out the collection the iterator is for by using the "getCollection" method (or something like that). That should tell you whether it is Jade code or your code that is leaking iterators.

An interesting thing is that leaking iterators can cause major performance issues (more than just using up memory). We have noticed
that if you leak a lot of iterators things start to perform quite
badly. There must be somethnig special about the Jade treats them.

Regards,
Allistar.

------------------------------------------------------------------
Allistar Melville
Software Developer
Auckland, NEW ZEALAND

Greentree International,
Developer of Greentree Financial Software. ------------------------------------------------------------------

[ConvertFromOldNGs]

by CarlRanson >> Thu, 7 Nov 2002 3:33:59 GMT

If you really get desperate you may be able to implement a create method for dictIterator class and cause it to write out the oid and the next method up the stack chain.

You could then reconcile that with your output at application shutdown.

But i would first try to work out which dictionaries are involved. Its probably a dead giveaway once you find that out.
CR

[ConvertFromOldNGs]

by JensRichnow >> Thu, 7 Nov 2002 5:46:21 GMT

Thanks all! I did find the culprit! Myself

Looking at the dictionaries it turned out to be only one external dictionary I use for the meta-data implementation which get frequetly queried application wide. On oversight on may part. But I learned something ... and was surprised by the feedback. Thanks to all again.

Hope that jade is looking at the other side-branch of the thread and the idea to implement gc at kernel level -->

[ConvertFromOldNGs]

by cdshearer >> Thu, 7 Nov 2002 3:46:34 GMT

Here's the actual code I'm using to check...

This has various refinements to it including the ability to recognise those that JADE leaves lying around (Printer, CurrencyInfo etc.) It also has the ability to turn the checking off, which is useful during development as it can take some time to execute the code.

Note, I've only ever seen DictIterator appear once, then I couldn't make it happen again. So maybe it is JADE leaking these... some comment from the plant would be useful!



vars
obj : Object;
realUndeleted,
insts : ObjectArray;
knownObjs : ObjectSet;begin

if checkUndeletedTransients then
app.log(LogLevel_Debug, "Checking for undeleted transients...");
create insts transient;
Object.allProcessTransientInstances(insts, 0, true);

create knownObjs transient;
knownObjs.add(insts);
knownObjs.add(app);
knownObjs.add(app.printer);
knownObjs.add(app.currentLocaleInfo);
knownObjs.add(app.currentLocaleInfo.currencyInfo);
knownObjs.add(app.currentLocaleInfo.timeInfo);
knownObjs.add(app.currentLocaleInfo.dateInfo);
knownObjs.add(app.currentLocaleInfo.numericInfo);

create realUndeleted transient;
foreach obj in insts do
// Check whether the object is one of the "known" objects
if not knownObjs.includes(obj) then
realUndeleted.add(obj);
endif;
endforeach;

if realUndeleted.size > 0 then
app.log(LogLevel_Debug, "Found " & realUndeleted.size.String &
" undeleted transients:");

foreach obj in realUndeleted do
app.log(LogLevel_Debug, obj.String);
endforeach;
else
app.log(LogLevel_Debug, "No undeleted transients found ");
endif;
else
app.log(LogLevel_Debug, "No check for undeleted transients performed."); endif;
epilog
delete insts;
delete realUndeleted;
delete knownObjs;
end;

[ConvertFromOldNGs]

by JensRichnow >> Thu, 7 Nov 2002 6:54:13 GMT

Thanks, just a brief comment re the filtering of the 'known objects'. If we want to achieve a filtering by system vs. user object why just don't use the Object.isSystemObject() method. Unless we know exactly all the system objects Jade is creating by default this seems to be a more generic approach. Jade might decide at some stage to add system objects (of new classes) which will not be caught by the above object comparison.

[ConvertFromOldNGs]

by hml >> Thu, 7 Nov 2002 8:49:44 GMT

Thanks, just a brief comment re the filtering of the 'known objects'. If we want to achieve a filtering by system vs. user object why just don't use the Object.isSystemObject() method. Unless we know exactly all
the system objects Jade is creating by default this seems to be a more generic approach. Jade might decide at some stage to add system
objects (of new classes) which will not be caught by the above object comparison.

The problem with the isSystemObject method is that is returns true for a DictIterator as this class is a system class, but the instance may have been created by a developer.

Torrie Moore

[ConvertFromOldNGs]

by JensRichnow >> Fri, 8 Nov 2002 7:43:17 GMT

As I read your answer I agreed, but something was in the back in my mind all day long. A cycle ride and beer got me.

I just checked the Object.isSystemObject() behaviour for the DictIterator class, and yes it returns true for those iterators created by the developer for user dictionaries.

I would have expected that the method checks the run-time nature of the object queried. If the DictIterator.getCollectionClass() returns the user dictionary class (or collection for that matter) the ::isSystemObject() should reflect the run-time nature of the class instance.