Forums

by Rich >> Wed, 26 Mar 2008 16:18:40 GMT

Hello!

I have a very strange issue. I program for a payroll bureau, and the largest data structure for our software has an instance of every payment item, for every employee, for ever period, for every company in that database for a standard of 2 years! So, lots and lots of instances!!

We have two memberKeyDictionaries that control the viewing of the instances, they both use the same keys (employee, item, date) but return the data in two different orders. The first returns every item for an employee for a given date, starting with the item in the parameter. The other returns a history of an employee for a single item, starting with the given date and working backwards for a given number of periods.

A couple of weeks ago, all of the weekly paid companies in one database noticed that they couldn't see any history for any item before that week whilst using the program that uses the second dictionary type; but when they used the program running the first dictionary type, they could see all the instances with no problems. Strangely, the monthly paid companies can all use both with no issues. Since then, the structure is working as expected, but they can only see the history back until that point. We did have a space issue at the end of that week, but that shouldn't have affected this at this point... it also doesn't explain why it has started working again!

It seems what has happened is that the records in the data structure for all weekly paid employees have been removed from the dictionary, no code nor errors could possibly have caused this. Our first thought was that due to the week in question being the week containing Feb 29th, it may be an issue with the date field, but testing has proved this not to be the case. We have tried doing reorgs on the data structure and the dictionary to no avail. Today, i have also created a copy of the dictionary and made them refer to the newly populated one, but that doesn't work either.

We are baffled as to why this has happened, any information or advice will be very appreciated. All other companies on all other databases work with no problems, its just this database with the weekly companies (there should be no reason it treats these differently).

All inverses are correctly set up, the instances still exist and are formed ok. It is just this reading issue.

Thank you in advance

Rich

by Dean Cooper >> Thu, 27 Mar 2008 0:54:43 GMT

When you say "It seems what has happened is that the records in the data structure for all weekly paid employees have been removed from the dictionary", have you verified this by doing an iteration through the entire collection and checking the member objects, or are you checking this by doing getAtKey, startKeyGeq, startKeyLeq, etc lookups? And if you expect both collections to contain exactly the same member objects, does coll1.size = coll2.size? Does a physical certify (use jdbutil.exe) of the database report any errors?

Dean.

by Rich >> Thu, 27 Mar 2008 11:56:05 GMT

Rather than checking through the iteration, somebody has done blackbox testing on the database and they can only access the monthly data. They did enough testing to be sure of this. I am checking for records by using getAtKey and it simply finds the item that would come after that company's records - as though none exist. There are no apparant errors as it is now happily creating them again. The size of the collections are different, as i would expect. The collection that has 'lost' the entries is about 2/3 the size of the other collection - which would be about right. Something obviously happened during that week but getting it working again is more important at the moment than avoiding it in the future - that can be looked at later.

Thanks again,

Rich

by BeeJay >> Thu, 27 Mar 2008 23:38:17 GMT

Have you tried a fixup script that iterates all the instances and when it identifies one that has this "problem" state save off the value of the myXYZ reference that is the inverse of the two collections, set the myXYZ reference to null and then set it back to the saved off value. Repeat for each of the instances with the problem, with appropriate commitTransaction frequency to prevent it locking the collections for too long or creating too large a transaction.

Alternately, you could run the Jade Logical Certifier utility against last night's backup of the system to identify issues such as this one. If your database is a reasonable size you may want to consider limiting the classes to just the ones in question to speed up the certify process. Then, you can use the generated fix file to correct the live production system at a convenient time.

Cheers,
BeeJay.

by Dean Cooper >> Sat, 29 Mar 2008 3:01:49 GMT

Are you able to check the dictionary by using a foreach to iterate it completely and reference every member object? If you can traverse the affected dictionary without encountering an error, then that's a good indicator that there's no physical corruption. However, you should still perform a physical certification on a backup to make sure (if you haven't already). Assuming that's error free, the next step would be to perform a logical certification as BeeJay has suggested.

Also, have you performed any file rebuilds on the database recently and if so, why please?

Dean.

by Rich >> Wed, 2 Apr 2008 10:13:15 GMT

Hi, sorry for the delayed response.

I have already run a complete database certify and it has returned errors from the dictionaries in question, along with a couple in the actual data structure. I can iterate the entire dictionary without error, but only around half of the instances are in there - that is the issue. If i attempt a foreach iteration on the entire datastructure it does fall over part way through, i think the error was 3107 (may be wrong), but eitherway, the documentation recommended i contacted Jade.

No, i haven't performed any file rebuilds on the database - if i am honest, i wouldn't have a clue how or why i would do this!

I have opened up a Parsys call and attached the Jommsg.log, certify.log and the 4 .dat files in question. I shall first see what reply i get from the call, but if nothing comes of it then i will try BeeJay's suggestion of a logical certification and fix.

Thanks guys for your help in this,

Rich

by BeeJay >> Thu, 3 Apr 2008 3:39:40 GMT

If you're getting 3107 errors from the database, that is definitely suggestive of physical corruption to the database files. You need to get that resolved before attempting any logical certification or scripted fixes along the lines of those I've suggested as they're unlikely to help much while physical corruption like this is present.

I would definitely not consider running logical certify or fixup scripts until you get appropriate advice from Jade Support on restoring the physical integrity of your database.

by Dean Cooper >> Thu, 3 Apr 2008 21:40:21 GMT

A contact has been opened with Jade Support and is being investigated by the support team and the Plant.

Dean.

by Rich >> Mon, 14 Apr 2008 14:34:04 GMT

Hi

Just to update you all for interest purposes. This situation was fixed with the help of Jade Support and it turned out that it was a physical corruption.

Running a Rebuild, Logical Certification and Logical Repair on the files replaced the instances that were missing. We are investigating exactly how this corruption occured but there were several issues around that time that could easily have been culprits (for example, at one point an infinite loop started in code that is barely ran which eventually filled up the HDD causing the server to crash!

Anyway, thank you all for your help and advice in the matter.

Regards,

Rich