String Pool Theory

[Hayden]

Hi All.

Here in Sydney, we're having a little bit of trouble with an error 4024 - Out of Resources for Operation. Jade Support & the Jade Plant have been very helpful in reading our dumps and pointing out that it is our String Pool which is causing the problem.

That's nice, but we're unsure of the values to set it to, and how we go about calculating what is best for the system.
We can't just use our production system as a test environment, as the 4024 error causes the system to crash, and 350 users aren't too happy.

What we know is this:
Adjusting the StringPoolLimit and MethodCacheLimit along with the Min & MaxServerThreads all play a part.
(MethodCacheLimit + StringPoolLimit) x 2 = Interpreter cache size.
This is the amount of memory used by each thread on the server that has it's own interpreter cache. (MaxServerThreads)

So our current calculation is
(2,000,000 + 2,000,000) x2 = 8,000,000 x 100 = 800,000,000 - approx 800Mb.

But when we are compressing & uncompressing Binary files (usually Word docs of 250+Kb), during periods of high activity, the 4024 still occurs.

Jade Support have made 2 suggestions.
1. Reducing the number of ServerThreads
2. Going to a Single Pool, instead of a value for each Thread

But what are the pro's & con's of each?
Has anyone else had experience of this? Your advice or thoughts would be most welcome.

And does anyone know how to measure the StringPool that a method could/does use? The Profiler gives some info, but not enough.

We're kinda lost when it comes to memory management, especially as the machine is looked after by a different group of people, who also do our Jade Admin, but they generally don't know much and only do what we tell them to do.
Also RAM and CPU on the machine isn't a problem - we have plenty of each.

Thanks in advance.
- Hayden

[Hayden]

Just one more thing to add.
Does anyone know the best way for testing this?
If I can replicate the problem in a test system, then I can prove how changing the settings eliminates, or reduces this occuring.
Basically, I need to show some proof behind the theory.

Thanks

[torrie]

Hi Hayden

There is a post in the Design and Archiecture group called Method Cache which discusses the multiple vs single cache issue. Dean pointed me to the Performance Analysiswhite paper which has an example about multiple method caches. Hwoever this is from the MethodCacheLimit view point, not the string pool. We never did any testing on the settings though.

Out of interest, are you using Blobs or JadeBytes for the documents?

[BeeJay]

I'll not try to show my ignorance of String Pool Theory, but rather I'll take a different tack on the problem at hand by asking a pointed question: Is there a particular reason why you're compressing and uncompressing all these files on the DbServer Node rather than spreading this work-load across multiple AppServer Nodes?

If you're compressing them for storage saving purposes, and uncompressing them to display them to the users, would it not make more sense to do the compress/uncompress on the AppServer side.

1. You'll be spreading the workload across multiple AppServer Nodes - I'm assuming with 350 concurrent(?) users you have quite a number of AppServer Nodes.

2. By using multiple AppServer Nodes you're less likely to have one AppServer exceed the 2 gig Windows heap limit, 4 gig for 32bit Jade on 64bit Windows, than you are with all the compress/decompress operations running in the DbServer Node.

3. If you do encounter a resource issue in your AppServer Node this is likely to have less impact on users connected to the other AppServer Nodes than a resource issue on the DbServer Node.

4. You'll be sending compressed data between the DbServer and the AppServer, thereby reducing the amount of data being sent to the AppServer Node over the RPC connection between the DbServer and the AppServer.

Of course if there is some other reason why you're doing the compress/uncompress on the DbServer Node these become moot points.

Cheers,
BeeJay.

[Dino]

Jade Support have had further communication with Hayden, which I'll post here for completeness so that other readers can see what needs to be considered in these situations. This assumes that the compression/uncompression remains on the database server (ie: RAP) as opposed to BeeJay's suggestions of moving it to the app servers.

The issue is address space fragmentation. Being JADE 6.2, this pre-dates JADE's native 64-bit implementation, which doesn't suffer from this particular issue. If you run 32-bit JADE on 32-bit Windows you have a limit of a 2GB address space, and if you run 32-bit JADE on 64-bit Windows the address space limit goes up to 4GB.

Ultimately, the solution to this problem is to upgrade to a 64-bit version of JADE 6.3 on a 64-bit OS platform. However, the JADE Plant suggests that the following steps may mitigate or address the crashes that you are experiencing.

1. Change the RAP's ini file (and ONLY the RAP) to specify the use of a single method cache manager, and

2. Increase the following Interpreter limits (again, in the RAP ini file ONLY):

[JadeInterpreter]
MethodCache=Single
MethodCacheLimit=350M
WriteEnabled=false
StringPoolLimit=150M

Note that these are what we consider to be useful starting points for your system, and are chosen primarily to mitigate the crashes you have been experiencing. However, because the behaviour of each application is different, some experimentation and tuning will most likely be necessary in order to find the optimal numbers from a performance standpoint.

Again, because of application differences, we cannot be certain whether you will experience an improvement or degradation in performance by moving to a single method cache manager. On the one hand, having a single, much larger method cache should reduce the need to load methods into the cache. The use of single method cache means that only one copy of a given method will be in cache, and results in the most efficient use of memory. It will also eliminate the 'sloshing' of method code from one cache to another when processes call methods on a server thread whose cache doesn't contain the required method code tree.

On the other hand, there may be some degradation in performance because server processes may compete for access to the single cache. We can’t predict the net effect of this possible degradation along with any performance benefits arising from the use of the single method cache, as these effects are quite application-dependent.

Dean.

[Hayden]

Thanks All.
Torrie, we haven't looked at JadeBytes - we're just using blobs. Those Whitepapers have been helpful.
BeeJay, good thinking. We've got 16 App Servers - removing the server execution from that method might help. I think it's been coded like that from a "default thinking" of server = faster/more efficient - which may not be the case. The docs are compressed for storage on our company-wide SAN, and it gives some basic security too. (ie must be accessed from the Jade App)
Dean, thanks for posting that update. It made some good points, but the question remains: How do we test this?

And, unfortunately, at the end of the day, that is the question, because duplicate hardware & all users logged in to test just isn't reality. (no offense Jade Support!)

As it stands, the production issue has gone away.

I'm pretty sure I understand what we need to do to manage this, I just need to be able to sell it to the management team.

And that's not an issue that can be resolved via the Jade Forums!
But I do hope that this thread might help others out there who come across this issue.
Short Answer: Upgrade to 64Bit.