Upgrading from JADE 4 to JADE 5

[ConvertFromOldNGs]

by Craig Shearer >> Mon, 17 Jan 2000 19:40:38 GMT

Hello All

Just a cautionary tale to relate in upgrading a deployed system from JADE 4.2.2 to JADE 5.

You need to be very careful upgrading. We have a deployed JADE 4 system that has been in production for some time now. As usual, it has been patched many times, adding new functionality, fixing bugs etc. and as with most deployed systems, the deployed and development systems get "out of sync" in terms of class numbering - class numbers in the deployed system are different from those in the development system for one reason or another.

When upgrading a deployed database from JADE 4 to JADE 5, you need to have the development system extract that EXACTLY matches the deployed system in terms of its class numbers and also the class structure (ie all the class's properties).

We had a sticky situation where we used the JADE 5 upgrade to upgrade our deployed JADE 4 system to version 5 and it appeared to work, even though the structure was different. At least the message that came back was that the upgrade was successful. Not until the users started using the system did we find some rather puzzling errors, such as Exception 4 - Object Not Found, and Exception 1010 - The requested method is not defined for the type of the receiver. It would seem that the upgrade does not detect changes in the class structure, and consequently won't mark classes for reorg. This causes 3039 exceptions.

The solution was as follows:

1. Get the development environment in sync with the deployment environment. For us, this meant finding all the class numbers for classes in our deployed database. We wrote a simple method to do this:

getClasses();

vars
cls : Class;
f : File;begin

create f transient;
f.openOutput("c:\classes.txt");
foreach cls in currentSchema.getAllClasses(false) where cls.schema = currentSchema do
f.writeLine(cls.name & Tab & cls.number.String);
endforeach;

epilog
delete f;
end;

This produced a file which I then loaded into an Excel Spreadsheet. I ran the same code on the development version, and loaded its output into the spreadsheet, side by side so they could be compared.

2. Identify all the classes where the class numbers are different in the development environment.

3. Do a full schema extract from your development environment.

4. Load up your .scm file in a text editor, and under the typeHeaders section, fix up all the class numbers so that they reflect the deployed environment. Save your .scm file.

5. Now, delete all your schemas from your development copy (make sure you've got a backup first!). Unfortunately, you're going to lose all your test data!

6. Reload the edited schemas. Note no changes are necessary to your .ddb file unless you happen to have a control class with a different class number. Luckily we didn't, but if you did - you'd need to fix up the .ddb file too (scary!)

7. You should now have the development and deployed environments in synch. You can upgrade your development environment to JADE 5 then use the multiple schema extract to upgrade the deployment database too.

Finally, just another point, once this is completed, you could then load in a new version of the schema, even if it has differing class numbers as the normal schema load handles this. This is what we had in our situation where we'd already upgraded our development environment to JADE 5 and had started making changes.

Hope this helps...

Craig.

[ConvertFromOldNGs]

by John Porter >> Mon, 17 Jan 2000 23:04:25 GMT

Just a hint if you've got lots of classes - you can compare two text files in Word using Compare Versions (on the Tools|Revisions or Tools|Track Changes menu). It automatically finds deleted / inserted lines, changes text, etc and flags them for you. You can then search for the flagged items.

Cheers,
John P

[ConvertFromOldNGs]

by Craig Shearer >> Wed, 19 Jan 2000 19:45:12 GMT

Actually, there is a further requirement which we have since discovered. As well as all classes having the same class numbers, all properties must have identical subIds. Properties that have subIds are collections, and strings.

After upgrading, we discovered that a couple of our persistent collections were empty - and this occurred during the upgrade process. The system appeared to be fine, except for these collections. So, we ran a script to extract the subIds for all properties of all the classes, and discovered that on one class, two of the subIds had been swapped. It would seem that the result of this is a JADE effectively clears the collections during the upgrade. Fixing the subIds (by deleting the schema, then reloading a modified .scm file) fixed the problem.

Here's the script that extracts the subIds as well as the class numbers:

getClasses();

vars
subclasses : ClassColl;
cls : Class;
p : Property;
f : File;
begin

create subclasses transient;
create f transient;
f.openOutput("c:\classes.txt");

currentSchema.getClass("Object").allSubclasses(subclasses);
foreach cls in subclasses where cls.schema = currentSchema do
f.writeLine(cls.name & Tab & cls.number.String);
foreach p in cls.allProperties where p.name[1] <> "_" and p.subId.Integer <> 0 do
f.writeLine(Tab & p.name & Tab & p.subId.Integer.String);
endforeach;
endforeach;

epilog
delete f;
delete subclasses;
end;

[ConvertFromOldNGs]

by Dean Cooper >> Thu, 20 Jan 2000 1:51:44 GMT

Properties that have subIds are collections, and strings.

Shared collections don't have sub ids; only exclusive collections do. Binaries can have sub ids too. When a string or a binary is not embedded in its owner object (ie: it is a slob or a blob), it will have a sub id.

It would seem that the result of this is a JADE effectively clears the collections during the upgrade.

The sub id forms part of an exclusive oid. The original collection hasn't actually been deleted, but your app is going after a different object (which is an empty collection). If you were to fix the sub id in-place after the upgrade, you'd be able to access the original collection.

These problems can occur when your development schemas and deployed schemas get out of sync, and you're replacing/rebuilding the _user*.dat files while wanting to preserve existing user data. Normally, when you load a schema into a database where that schema already exists, the compiler always uses the class numbers and sub ids defined in the target database (ie: any class number or sub id differences in the schema file are ignored), thus preserving user data. But in cases where you are copying in new _user*.dat files (ie: file-based deployment rather than loading schemas), or during the 5.0 upgrade (when the _user*.dat files are rebuilt), there is no load over an existing schema hence the requirement that class numbers and sub ids be exactly the same.

If you suspect that you have mismatched class numbers and/or sub ids (the scripts Craig has posted are useful for checking this), please contact Jade Support. They have a RootSchema patch you can load into your deployed system that will extract the structural definition (ie: the typeHeaders section) of a schema from your deployed database. You can then replace the typeHeaders section in your schema file with this generated typeHeaders section to ensure that class numbers and sub ids match.

We accept that preserving the same class numbers and sub ids across systems can be difficult to manage, and we are currently looking at automating the upgrade process with the goal of being able to perform in-place system upgrades (ie: without the need to reload schemas at all). Achieving this should avoid situations of class and feature number mismatches.

However, upgrade issues aside, can I suggest that it is still important to keep schemas (either as extracts or full development environment backups) that match your deployed/production systems. We have seen several cases where sites have been unable to obtain schema definitions that match their system; with the closest match often having method changes, property/inverse changes, even the addition or deletion of entire classes. This is like writing an app in C++ or Delphi or something and chucking away the source code after each release. Maintenance, debugging and upgrades can be much easier (and safer) if you retain snapshots of your schemas each time you ship a release.

Dean.

[ConvertFromOldNGs]

by Craig Shearer >> Thu, 20 Jan 2000 19:54:02 GMT

Hi Dean

Your comments are very useful, and explain clearly what is going on. I think this is a deficiency of the upgrade process from JADE 4 to 5. In the past, upgrades have been quite manual, and the developer has been aware of what is going on. Now, with the upgrade wizard, the developer is much less aware of what is going on underneath in the process of upgrading. (I know, people complained about not having an automatic upgrade process - I guess you can't satisfy everybody!)

Had it been clear that the upgrade process actually deleted existing schemas and reloaded them while preserving existing data, perhaps the need to have schemas perfectly in sync would have been more obvious. It would have perhaps been useful for the upgrade process to read the supplied .scm files and compare them to the structural definition in the database and stop the user if they were incompatible. While this would add extra time to the upgrade process, it would at least stop users (ie. us) from making mistakes.

Looking back at the instructions for upgrading from JADE 4 to 5, nowhere (that I can find!) does it mention that development and deployment versions MUST be in sync. I guess in hindsight that this is obvious now, but it certainly wasn't at the time. It is all too easy for development and deployment versions to get out of sync, so this really should have been made clear from the start.

Here's hoping that the upgrade to JADE 6 is less painful!

Craig.