Batch file transfer
Download source code for demo application: 05-batch-transfer.zip
In previous articles I explained how to use Rebex.SFTP to perform transfers between local and remote server file by file. Which is just fine in many cases, but in many cases you will prefer the batch transfer, where you can upload or download multiple files by single command.
Methods for batch file transfer
The Sftp class has the Upload and Download blocking methods and their async versions UploadAsync and DownloadAsync, with multiple overloads.
Use the following arguments to define what and how should be transferred:
localPathcan be either single file name, single directory name or a wildcard specifying the items to be transferred.- If you specify a file, the file will be transferred.
- If you specify directory name, ie.
C:\Some\Folder, the directory itself would be transferred (directory calledFolderwill be created on remote server). - If you specify wildcard in typical file style, ie.
C:\Some\Folder\*.*, all files matching the wildcard would be copied into the remote directory. Directories will be copied only if they contain a dot in their name (Some.Folderwould be copied,SomeFolderwould be ignored). - To process all directories and files without the containing directory itself, use
C:\Some\Folder\*
remoteDirectoryPathspecifies where the items are to be copied.traversalModecontrols how recursion in source works:Recursiveperforms full recursion and will upload everything: files, directories with contents, including nested directories.NonRecursivewill process all items in the specified path:- If the
localPathis file, the fire will be transferred. - If the
localPathis folder, empty folder of given name will be created. Everything inside will be ignored. - If the
localPathis wildcard, files matching the wildcard will be transferred and directories matching the wildcard will be created empty.
- If the
MatchFilesShallowis equivalent toNonRecursive, but only files are transferred. Directories are ignored.MatchFilesDeepis equivalent toRecursive, but empty directories are not transferred.
transferMethodis eitherCopyorMove. In case of the later, source items are deleted after they are transferred.existingFileModesets how the existing files are handled:SkipAllmeans that existing files are never overwritten and are skipped.OverwriteAllmeans that existing files are always overwritten.Renamewill rename the new file by adding number in square brackets to the file name. Iftest.txtexists, the newly uploaded file will be namedtest[1].txt,test[2].txtetc. I'll explain later how you may customize this behaviour.OverwriteOlderwill overwrite older files with newer. Use of this mode is strongly discouraged, though. Depending on server, client, file system and options, the file dates may be unreliable, and the results may be faulty.OverwriteDifferentSizewill overwrite files if they differ in size. Which isn't usually what do you want, because equal size does not mean equal content.ResumeIfPossibleis a very tricky one, see note below.- If the new file is smaller or equal size, it's skipped.
- If the new file is larger, the new content is appended to existing file.
OverwriteDifferentChecksumoverwrites the file if it has different checksum. Which works nicely, if the server supports checksum. And in case of large files it might be slow.
Using existingFileMode: ResumeIfPossible may result in interrupted transfers to be resumed correctly. But if the files have been modified, the results may be disastrous. If the file was modified in source, it depends on how and where exactly it was modified.
If the modification is after the size of the target file, is transferred correctly. But any modification in the first part of file are not, so the target file may be inconsistent. Similar situation may happen when file is modified on the target side.
Using this mode is not recommended, unless you can be sure that files on the target side are never modified and on the source file they are not modified or are just appended into (such as typical text log files).
Instead of giving localPath as string, you can use overload taking a set argument, which is of FileSet type. The FileSet class gives you greater control over selection of files. You may specify multiple wildcards for inclusion and exclusion and various file path manipulation options. See documentation for more info.
The ProblemDetected event
The Sftp class has event called ProblemDetected. This event is raised if there is some kind of problem during batch operations (Upload, Download, Delete, GetItems and their asynchronous variants). In this event you can handle the problem programmatically, based on your scenario.
You will get instance of a SftpProblemDetectedEventArgs class and you may use its properties and methods to find out and resolve the problem.
Crucial is the ProblemType property, which tells you what kind of problem occured. The value is an enum. The most common problems you would probably like to handle in custom fasion are:
FileExists- target file of the same name exist and you want to implement your custom strategy to handle this situation.DirectoryNameIsInvalidOnTargetFileSystem- name of the directory is valid on source file system, but invalid on the target. This may happen when copying from Linux or similar systems to Windows, because in typical Linux file systems the paths can contain characters considered invalid in typical Windows file systems, such as NTFS. You have to implement some strategy how to deal with it, ie. rename the directory.FileNameIsInvalidOnTargetFileSystem- the same situation, but for file names.
Implementing custom existing file handling strategy
To resolve a problem, call the appropriate method. Some of them are self-explanatory: Cancel, Fail, Retry and Skip. For custom strategy of handling existing files we have several more:
Overwritewithout arguments will simply overwrite the target file. Optionally, you can specify condition under which it may happen, as explained above.Renamewill rename the file using the default algorithm (adding[number]to file name), or with custom name you provide.Resumewill append to the existing file, as explained above.
Let's imagine we want to rename existing files during transfer, but not by adding sequential numbers to their names, but the date of last modification (or precisely, date when the file was last written). It's not trivial, because we have to solve problems arising when the file with new name already exists, which will be fairly often for repeated transfers. Also, we have to implement bit of different strategy for uploading and downloading.
This is one of possible solutions:
using Rebex.Net;
using Rebex.IO;
const string ServerName = "localhost";
const string UserName = "tester";
const string Password = "password";
const string UploadDirectory = @"C:\Windows\Web\Wallpaper\ThemeA\*";
Rebex.Licensing.Key = "Get trial key at https://www.rebex.net/support/trial/";
using var ftp = new Sftp();
ftp.ProblemDetected += (sender, e) => {
// Handle only file exists problems, use default handling for all other problems
if (e.ProblemType != TransferProblemType.FileExists) return;
if (e.Action == TransferAction.Uploading) {
// Rename the file being uploaded by appending the last write time to the file name
var time = e.LocalItem.LastWriteTime ?? throw new Exception("Local item's last write time is null.");
var newFileName = $"{Path.GetFileNameWithoutExtension(e.LocalItem.Name)}_{time:yyyyMMddHHmmss}{Path.GetExtension(e.LocalItem.Name)}";
// Check if the file with the new name already exists on the server
var ftp = sender as Sftp ?? throw new NullReferenceException("Sftp instance is null in event handler.");
var remotePath = Path.GetDirectoryName(e.RemoteItem.Path) + '/' + newFileName;
var remoteExists = ftp.FileExists(remotePath);
if (remoteExists) {
// If the file with the new name already exists, skip the file being uploaded
e.Skip();
} else {
// Rename the file being uploaded
e.Rename(newFileName);
}
} else if (e.Action == TransferAction.Downloading) {
// Rename the file being downloaded by appending the last write time to the file name
var time = e.RemoteItem.LastWriteTime ?? throw new Exception("Remote item's last write time is null.");
var newFileName = $"{Path.GetFileNameWithoutExtension(e.RemoteItem.Name)}_{time:yyyyMMddHHmmss}{Path.GetExtension(e.RemoteItem.Name)}";
// Check if the file with the new name already exists on the local file system
var localPath = Path.Combine(Path.GetDirectoryName(e.LocalItem.FullPath) ?? string.Empty, newFileName);
if (File.Exists(localPath)) {
// If the file with the new name already exists, skip the file being downloaded
e.Skip();
} else {
// Rename the file being downloaded
e.Rename(newFileName);
}
} else {
throw new Exception($"Unexpected transfer action: {e.Action}");
}
};
Console.Write("Connecting to SFTP server...");
await ftp.ConnectAsync(ServerName);
Console.WriteLine("OK");
Console.Write($"Logging in user {UserName}...");
await ftp.LoginAsync(UserName, Password);
Console.WriteLine("OK");
Console.Write($"Uploading directory {UploadDirectory}...");
await ftp.UploadAsync(UploadDirectory, "/", TraversalMode.Recursive, TransferMethod.Copy, ActionOnExistingFiles.Overwrite);
Console.WriteLine("OK");
Console.Write("Disconnecting...");
await ftp.DisconnectAsync();
Console.WriteLine("OK");
Please note that although I specified ActionOnExistingFiles.Overwrite, the custom handling takes precedence.