Forums

by JADE Kid - Ray Hidayat >> Wed, 12 Jul 2000 2:41:18 GMT

I just worked out why I could not load large files into my database. But still, I don't have a way to fix the problem.
So until now, I have chunked the file into bite-size pieces because I thought that if I added little pieces to the file at a time, it wouldn't bring up the error "No memory for buffers"

But the problem wasn't that. Here is the code with the problem (in the FileTrans transient class, the place where files are stored to send to other people):

while rec <> null do
rec := _file.readBinary (65536);begin
TransientTransaction;
file := file & rec;
commitTransientTransaction;
endwhile;

That may look harmless. But look closer.
begin
TransientTransaction;
file := file & rec;
commitTransientTransaction;

The problem here is right here

file := file & rec;
^
When this file variable gets to a significantly large size, it won't fit in the transaction buffer.
That causes the error "No memory for buffers"
But I don't need a buffer. The transaction is not going to be aborted.

And as far as I know, there is no way to fix this at the current moment. Anyone have a solution to do an "Instantaneous Transaction" which enters the information straight away into the database without a buffer?

Now that I think of it, does anyone actually use the 'abortTransaction' method?

--
Ray Hidayat
JADE Kid - 2000
www.jadekids.com

by Craig Shearer >> Wed, 12 Jul 2000 3:11:06 GMT

Hi Ray

Firstly, why are you using a transient transaction? Unless you are using shared transients, there is no need for this. However, you'd probably have the same issues with a persistent database transaction anyway.

What is the issue with your cache size? Is there are reason why you can't have a large cache, or do you want to be able to handle an arbitrarily large file? If you're worried about having an exception occur, why not just arm an exception handler and tell the user that the file is too big.

Basically, your cache needs to be bigger than the largest object you're going to have in it, otherwise you'll get the 1018 - No memory for buffers exception. If you want to load large files into a binary field, the cache HAS to be big enough to hold that size.

There is a way around this - and that would be to store the file as you read it - as chunks, in a BinaryArray. I'm not sure how efficient this would be, but you can do it. Basically, read a chunk, add the chunk to the array, read the next chunk, etc. until you're done. Then, you can use the same process for writing the file out again.

Hope this helps,
Craig.

Oh, and yes, abortTransaction is a very useful instruction!!!

by Darrell Duniam >> Wed, 12 Jul 2000 3:28:35 GMT

There is a way around this, by setting the following keys in the JadeClient and JadeServer sections of your initialization file (set the Key that corresponds to the type of object you're reading the file into):

ObjectCacheSizeLimit=5000000
TransientCacheSizeLimit=5000000
RemoteTransientCacheSizeLimit=5000000

.....these settings need to be of a size at least as large as the file you're trying to read into JADE. Be careful here however, since if they're set too large, you'll degrade JADE's performance.

darrell.

by JADE Kid - Ray Hidayat >> Thu, 13 Jul 2000 6:07:44 GMT

Yes I am using shared transients.
The reason is that the message has to get from one app to another.
I think I'll have a go at the binaryArray. It sounds like a useful idea, because I'm sure that people might find the need to send huge files across the network.

--
Ray Hidayat
JADE Kid - 2000
www.jadekids.com

by Krull >> Wed, 12 Jul 2000 7:02:15 GMT

Ray,

Your "No memory for buffers" problem is due to the cache sizing issue pointed out by Craig and others on this thread. However, I do have a couple of other comments that may (or may not) be of use to you:

In your code fragment:

while rec <> null do
rec := _file.readBinary (65536);
beginTransientTransaction;
file := file & rec;
commitTransientTransaction;
endwhile;

It looks like file is a property of self (shared transient), otherwise you would need to bound the update in a transient transaction. It is often better (performance-wise) to do this type of assembly of string or binary fragments in a local variable and then assign this to the property once. Try timing the above loop and compare it to something like:

while rec <> null do
rec := _file.readBinary (65536);
tempBinary := tempBinary & rec;
endwhile;
begin
TransientTransaction;
file := tempBinary;
commitTransientTransaction;

You should see a performance improvement. You would also get better performance if you increase your cache sizes as suggested by Craig and Darrell and go back to a single read. However, these approaches won't scale up to handling large files.

Do you really need to store the entire file in an object in the first place? Given that you are storing a copy of the file in a transient object implies that you don't need to share a copy between nodes so you could consider reading it each time you need to process the file. If you need to read the file more than once, you could make file access relatively transparent using a *virtual* mapping method similar to the following:

fileImage(set:Boolean; value:Binary io) mapping;begin

if not set then
value := myFile.readBinary (myFile.fileLength);
endif;
end;

or a just a method of the form:

fileImage() : Binary;begin

return myFile.readBinary (myFile.fileLength);
end;

Neither of these approaches involves updating a (blob) binary large object property, so cache size is no longer an issue (i.e. you won't get the 1018, regardless of cache size); the size of file you can handle this way is now limited by the JADE interpreter string pool size, which is mainly limited by the amount of virtual memory available on your machine. Doing it this way will give you better performance for files with a size approaching the size of your cache, even if you have to read the file more than once, since whenever a large binary object overflows cache it has to be written out to disk and gets read back in when reaccessed. A possible optimisation (using the virtual mapping method) would be to store files below a certain size threshold in an internal binary property; only useful if you process the file more than once (but I wouldn't bother with this).

All of the above approaches are only good for "moderate size" files, were moderate depends on the average amount of available *physical* (not virtual) memory on your machine. To get an idea what I mean, try opening a file that has a size close to your machines physical memory size with wordpad (wordpad also takes the approach of loding the entire file into memory).

Handling very large files efficiently is beyond the scope of this thread (that's why we have databases).

by Craig Shearer >> Wed, 12 Jul 2000 10:29:57 GMT

Excellent response Mr McSckull... anybody would think you worked in the plant or something

The best thing is that even the most basic of questions can result in somebody posting something that we can all learn from.

Craig.

by JADE Kid - Ray Hidayat >> Thu, 13 Jul 2000 6:18:28 GMT

I agree. This database optimisation could be used throughout my program. Throughout everyone's programs!

--
Ray Hidayat
JADE Kid - 2000
www.jadekids.com

by JADE Kid - Ray Hidayat >> Thu, 13 Jul 2000 6:16:59 GMT

The FileTrans class is a shared transient because it has to get from one app to another, I have to share it.
So yes, I guess I have to store the whole file, unless there is another way.

The virtual thing won't work because there is one app storing it, and one retrieving it, and one would have a few problems reading another file.

But now that I think of it, is it possible that I can create a reference to the file on the first machine, and send it to the other one? Or would that just point to the same path on a different node?

If all I have to do to load the file was to collect it into a binary variable and then load it into the database, then I'm sure I could do it all in one gulp. The reason I chunked it was because I had the error. So it still loads a huge amount of data either way.

It still works though. For a certain size.

--
Ray Hidayat
JADE Kid - 2000
www.jadekids.com

by Krull >> Thu, 13 Jul 2000 7:21:28 GMT

The FileTrans class is a shared transient because it has to get from one app to another, I have to share it. So yes, I guess I have to store the whole file, unless there is another way.

Passing the contents of the file from one app to another in the same node is on its own not a good enough reason to store the file in a transient object; sharing the contents between multiple nodes is a better reason, but then you will need to switch to using a persistent object. The methods I suggested earlier still apply to persistent sharing; what differs is where the actual physical file resides.

The virtual thing won't work because there is one app storing it, and one retrieving it, and one would have a few problems reading another file.

Why does any app have to store the file? If you are storing the file in a shared transient you can only share the transient object between processes on the *same node*, not between processes in different nodes; you can of course pass transient objects between client and server execution methods executing within the same process. I haven't seen what you want to do with the file, so I don't know yet why one application has to store the file to pass it to another application executing on the same node. The intent of the 'virtual thing' was so that *no* application actually stored the file; the application that needs to process the file data simply accesses the fileImage virtual property (or method). The application that in your scenario, stored the contents of the file simply has to setup the filename on a shared object; you need to also encapsulate the create/delete of a File instance (and optionally open/close the file if you wish).

But now that I think of it, is it possible that I can create a reference to the file on the first machine, and send it to the other one? Or would that just point to the same path on a different node?

Ah, now you have changed the ballgame, the above implies you wish to share the file data between nodes on different machines, in this scenario you can no longer use shared transient objects to share the data. You could, however, still store a filename in a persistent object and use the virtual method approach to provide access to the contents of an external file. To provide network wide access you could consider using a UNC name or make the file accessible to the database server and use a server method, of the form already suggested, to return the contents of the file. If you really do want to share the file contents between multiple users, there are good reasons for storing the file in the persistent database as opposed to just a link to an external file; you need to consider such things as the importance of the data, ease of access versus performance to name a few.

Whatever you decide, have fun!

cheers
Krull

by JADE Kid - Ray Hidayat >> Thu, 13 Jul 2000 12:19:34 GMT

Well somehow I got this shared transient thing working accross multiple applications on multiple nodes. We tested that at the JADE Kids meeting. Gosh that was fun.

I think that shared transients can be shared accross multiple nodes. Can anyone confirm this?

--
Ray Hidayat
JADE Kid - 2000
www.jadekids.com