Moving large files between nodes

[allistar]

Hi all,
We have the requirement to move large files (database journals and backups, which could have individual files up to 80Gb in size) from the database server to another server on the local LAN. This is to be controlled by Jade (triggered based on notifications). We do not want to require any file shares to be mapped, so a simple file copy using the OS copy commands won't work. We have investigate using Jade with serverExecution and clientExecution methods but the overhead is too large.

At the moment we're leaning towards using SSH to do the transport of data. It's simple to setup, offers strong encryption and has built in compression.

Has anyone had a similar requirement with similar restrictions and if so how have you tackled this?

Thanks,
Allistar.

[ghosttie]

We only do smaller files (usually between 50KB and few MB), and we only do the following types of transfers:

• upload from the client to the server
• download from the server to the client
• copy within the server
• copy within the client

For uploads and downloads we create an instance of File with usePresentationFileSystem := false and one with usePresentationFileSystem := true, then readBinary from one and writeBinary on the other. We do it in 500KB chunks to keep from killing the string cache.

For copies within the server or client we use the CopyFileA external function in kernel32.

So in summary, we don't do anything similar to what you're doing, so can't help

We tend to keep things in JADE wherever possible to avoid having another communication method to set up, maintain, let through firewalls etc. Using only one port makes things a lot simpler.

[allistar]

Thanks. We have done tests on using just Jade and the results from a performance point of view are not that good for very large files. We can do streaming in Jade using TCP/IP but there's still the issue of impacting the caches. From that point of view, doing it using an external application makes more sense to us.

[stevek]

I have created a similar transfer method to that describe by ghostie (though I have started with 64k chunks) and note that the binary file transfer between server and client is about 8-10 times faster than Windows Explorer. I am happy with that!