Skip to main content

Batch file transfer

Download sample

Download source code for demo application: 05-batch-transfer.zip

In previous articles I explained how to use Rebex.SFTP to perform transfers between local and remote server file by file. Which is just fine in many cases, but in many cases you will prefer the batch transfer, where you can upload or download multiple files by single command.

Methods for batch file transfer

The Sftp class has the Upload and Download blocking methods and their async versions UploadAsync and DownloadAsync, with multiple overloads.

Use the following arguments to define what and how should be transferred:

  • localPath can be either single file name, single directory name or a wildcard specifying the items to be transferred.
    • If you specify a file, the file will be transferred.
    • If you specify directory name, ie. C:\Some\Folder, the directory itself would be transferred (directory called Folder will be created on remote server).
    • If you specify wildcard in typical file style, ie. C:\Some\Folder\*.*, all files matching the wildcard would be copied into the remote directory. Directories will be copied only if they contain a dot in their name (Some.Folder would be copied, SomeFolder would be ignored).
    • To process all directories and files without the containing directory itself, use C:\Some\Folder\*
  • remoteDirectoryPath specifies where the items are to be copied.
  • traversalMode controls how recursion in source works:
    • Recursive performs full recursion and will upload everything: files, directories with contents, including nested directories.
    • NonRecursive will process all items in the specified path:
      • If the localPath is file, the fire will be transferred.
      • If the localPath is folder, empty folder of given name will be created. Everything inside will be ignored.
      • If the localPath is wildcard, files matching the wildcard will be transferred and directories matching the wildcard will be created empty.
    • MatchFilesShallow is equivalent to NonRecursive, but only files are transferred. Directories are ignored.
    • MatchFilesDeep is equivalent to Recursive, but empty directories are not transferred.
  • transferMethod is either Copy or Move. In case of the later, source items are deleted after they are transferred.
  • existingFileMode sets how the existing files are handled:
    • SkipAll means that existing files are never overwritten and are skipped.
    • OverwriteAll means that existing files are always overwritten.
    • Rename will rename the new file by adding number in square brackets to the file name. If test.txt exists, the newly uploaded file will be named test[1].txt, test[2].txt etc. I'll explain later how you may customize this behaviour.
    • OverwriteOlder will overwrite older files with newer. Use of this mode is strongly discouraged, though. Depending on server, client, file system and options, the file dates may be unreliable, and the results may be faulty.
    • OverwriteDifferentSize will overwrite files if they differ in size. Which isn't usually what do you want, because equal size does not mean equal content.
    • ResumeIfPossible is a very tricky one, see note below.
      • If the new file is smaller or equal size, it's skipped.
      • If the new file is larger, the new content is appended to existing file.
    • OverwriteDifferentChecksum overwrites the file if it has different checksum. Which works nicely, if the server supports checksum. And in case of large files it might be slow.
warning

Using existingFileMode: ResumeIfPossible may result in interrupted transfers to be resumed correctly. But if the files have been modified, the results may be disastrous. If the file was modified in source, it depends on how and where exactly it was modified.

If the modification is after the size of the target file, is transferred correctly. But any modification in the first part of file are not, so the target file may be inconsistent. Similar situation may happen when file is modified on the target side.

Using this mode is not recommended, unless you can be sure that files on the target side are never modified and on the source file they are not modified or are just appended into (such as typical text log files).

Instead of giving localPath as string, you can use overload taking a set argument, which is of FileSet type. The FileSet class gives you greater control over selection of files. You may specify multiple wildcards for inclusion and exclusion and various file path manipulation options. See documentation for more info.

The ProblemDetected event

The Sftp class has event called ProblemDetected. This event is raised if there is some kind of problem during batch operations (Upload, Download, Delete, GetItems and their asynchronous variants). In this event you can handle the problem programmatically, based on your scenario.

You will get instance of a SftpProblemDetectedEventArgs class and you may use its properties and methods to find out and resolve the problem.

Crucial is the ProblemType property, which tells you what kind of problem occured. The value is an enum. The most common problems you would probably like to handle in custom fasion are:

  • FileExists - target file of the same name exist and you want to implement your custom strategy to handle this situation.
  • DirectoryNameIsInvalidOnTargetFileSystem - name of the directory is valid on source file system, but invalid on the target. This may happen when copying from Linux or similar systems to Windows, because in typical Linux file systems the paths can contain characters considered invalid in typical Windows file systems, such as NTFS. You have to implement some strategy how to deal with it, ie. rename the directory.
  • FileNameIsInvalidOnTargetFileSystem - the same situation, but for file names.

Implementing custom existing file handling strategy

To resolve a problem, call the appropriate method. Some of them are self-explanatory: Cancel, Fail, Retry and Skip. For custom strategy of handling existing files we have several more:

  • Overwrite without arguments will simply overwrite the target file. Optionally, you can specify condition under which it may happen, as explained above.
  • Rename will rename the file using the default algorithm (adding [number] to file name), or with custom name you provide.
  • Resume will append to the existing file, as explained above.

Let's imagine we want to rename existing files during transfer, but not by adding sequential numbers to their names, but the date of last modification (or precisely, date when the file was last written). It's not trivial, because we have to solve problems arising when the file with new name already exists, which will be fairly often for repeated transfers. Also, we have to implement bit of different strategy for uploading and downloading.

This is one of possible solutions:

using Rebex.Net;
using Rebex.IO;

const string ServerName = "localhost";
const string UserName = "tester";
const string Password = "password";
const string UploadDirectory = @"C:\Windows\Web\Wallpaper\ThemeA\*";

Rebex.Licensing.Key = "Get trial key at https://www.rebex.net/support/trial/";

using var ftp = new Sftp();

ftp.ProblemDetected += (sender, e) => {
// Handle only file exists problems, use default handling for all other problems
if (e.ProblemType != TransferProblemType.FileExists) return;

if (e.Action == TransferAction.Uploading) {
// Rename the file being uploaded by appending the last write time to the file name
var time = e.LocalItem.LastWriteTime ?? throw new Exception("Local item's last write time is null.");
var newFileName = $"{Path.GetFileNameWithoutExtension(e.LocalItem.Name)}_{time:yyyyMMddHHmmss}{Path.GetExtension(e.LocalItem.Name)}";

// Check if the file with the new name already exists on the server
var ftp = sender as Sftp ?? throw new NullReferenceException("Sftp instance is null in event handler.");
var remotePath = Path.GetDirectoryName(e.RemoteItem.Path) + '/' + newFileName;
var remoteExists = ftp.FileExists(remotePath);
if (remoteExists) {
// If the file with the new name already exists, skip the file being uploaded
e.Skip();
} else {
// Rename the file being uploaded
e.Rename(newFileName);
}
} else if (e.Action == TransferAction.Downloading) {
// Rename the file being downloaded by appending the last write time to the file name
var time = e.RemoteItem.LastWriteTime ?? throw new Exception("Remote item's last write time is null.");
var newFileName = $"{Path.GetFileNameWithoutExtension(e.RemoteItem.Name)}_{time:yyyyMMddHHmmss}{Path.GetExtension(e.RemoteItem.Name)}";

// Check if the file with the new name already exists on the local file system
var localPath = Path.Combine(Path.GetDirectoryName(e.LocalItem.FullPath) ?? string.Empty, newFileName);
if (File.Exists(localPath)) {
// If the file with the new name already exists, skip the file being downloaded
e.Skip();
} else {
// Rename the file being downloaded
e.Rename(newFileName);
}
} else {
throw new Exception($"Unexpected transfer action: {e.Action}");
}
};

Console.Write("Connecting to SFTP server...");
await ftp.ConnectAsync(ServerName);
Console.WriteLine("OK");

Console.Write($"Logging in user {UserName}...");
await ftp.LoginAsync(UserName, Password);
Console.WriteLine("OK");

Console.Write($"Uploading directory {UploadDirectory}...");
await ftp.UploadAsync(UploadDirectory, "/", TraversalMode.Recursive, TransferMethod.Copy, ActionOnExistingFiles.Overwrite);
Console.WriteLine("OK");

Console.Write("Disconnecting...");
await ftp.DisconnectAsync();
Console.WriteLine("OK");

Please note that although I specified ActionOnExistingFiles.Overwrite, the custom handling takes precedence.