You want to fetch files with FTP from the source directory, another directory, as well as all their subdirectories at once. The ready-made FTP endpoint does not support this: it fetches the files from a single directory only. Therefore you need to create your own FTP endpoint that does this.
Note: This end point works only as a source end point, using it as a destination would not offer any additional functionality over the basic FTP-endpoint.
This example has been designed to work on an environment which has IIS 7 set up.
There are three ready-made endpoints you can extend:
These endpoints implement the IEndPoint interface.
NOTE: For the location of source files see extending cobalt.
Since the directory contains multiple values instead of just one, we need to split them for use later. The value of Dir is set to "", so later when we call base.Open, the initial directory is not changed.
We override the Open-method so we can store the initial directory we start in, this way relative paths from elsewhere than the root are supported.
private string startDirectory;
public CustomFtpEndpointForMultipleSourceDirs(TransferEndPointConfig endPointConfig) : base(endPointConfig)
{
if (base.Dir == null)
throw new ArgumentException("Directory cannot be left null");
//If the Dir is a empty string this will result in a single empty string in the collection
//For FTP this is valid and means the initial directory
_directories = base.Dir.Split(';');
//trim whitespaces, because they cause "cannot find path specified" problems
for (int i = 0; i < _directories.Length; i++)
_directories[i] = _directories[i].Trim();
// set the base dir to empty so we stay in the initial directory
base.Dir = "";
}
public override void Open()
{
base.Open();
//since the user may start elsewhere than the root of the ftp-server, eg. /home/user/ instead of /,
//we need to store the original directory we start in to be able to use relational paths multiple times.
//this is why whe set the empty initial directory in the constructor
startDirectory = FtpClient.GetCurrentDirectory();
}
Override ListFiles() to go through the directories recursively and return the list of all files with absolute paths instead of the default relative ones
- the Get() method will then use the path:
/// <summary>
/// The list of directories, initialized in constructor
/// </summary>
private readonly string[] _directories;
public override IList<FileItem> ListFiles()
{
List<FileItem> result = new List<FileItem>();
foreach (string directory in _directories)
{
result.AddRange(GetFilesRecursivelyFromDirectory(directory));
}
return result;
}
private IEnumerable<FileItem> GetFilesRecursivelyFromDirectory(string directoryPath)
{
//Always change to the start directory, this way both relative and absolute paths work
FtpClient.ChangeDirectory(startDirectory);
//Change to the path defined in the configuration, may be relative or absolute
FtpClient.ChangeDirectory(directoryPath);
//Get the Current directory with absolute path for Get()
var currentDirectory = FtpClient.GetCurrentDirectory();
List<FileItem> result = new List<FileItem>();
foreach (FtpItem ftpItem in FtpClient.GetList())
{
if (ftpItem.IsDirectory)
result.AddRange(GetFilesRecursivelyFromDirectory(Path.Combine(directoryPath, ftpItem.Name)));
else if (Util.FileMatchesMask(ftpItem.Name, this.FileName))
{
var item = new FileItem()
{
Modified = ftpItem.Modified,
Size = ftpItem.Size,
//Set the name to absolute path so Get() can fetch the files
Name = Path.Combine(currentDirectory, ftpItem.Name)
};
result.Add(item);
}
}
return result;
}
Note: When using this sample with recursion code do NOT use any Source endpoint directory, or any directory under it, as the
Destination endpoint directory. Such configuration will result in unwanted
behavior. The process will run successfully for the first time,
but later on it will always fail as there are already files existing in the destination directory. The process will fail because all the file rename and
copy actions done to the file(s) occur in the same directory and file cannot be copied onto itself.
Note that you can access the given directory string and file mask parameters via the inherited properties Dir and FileName properties
respectively, whilst in this sample only FileName property is used. The GetList() method can take a
file mask as
a parameter, but it would not find the sub-directories. Therefore the code uses the same Util.FileMatchesMask()
method that the actual FtpEndPoint uses.
You can use the FileItem constructor to initialize the file details. The absolute path needs to
be explicitly set (otherwise the name is just the file name) so the Get() method can locate the files.
A simple integration test for this method from Frends.Cobalt.Extensions.Tests.CustomFtpEndpointForMultipleSourceDirsTest (the directories are initialized at the test setup method)
[Test]
public void ShouldReturnOnlyTextFilesInAllDirectories()
{
var context = new TransferEndPointConfig
{
Directory = "/ftpTest/foo;/ftpTest/bar",
FileName = "*.txt",
ServerAddress = "localhost",
ServerPort = 21,
ServerUsername = "anonymous",
ServerPassword = "password@example.org",
EndPointType = "Custom",
FtpParameters = new FtpParameters
{
ConnectionMode = "Passive",
TransferType = "Binary"
}
};
IList<FileItem> files;
using (var endPoint = new CustomFtpEndpointForMultipleSourceDirs(context))
{
endPoint.Open();
files = endPoint.ListFiles();
endPoint.Close();
}
Assert.That(files.Count, Is.EqualTo(3));
var fileNames = files.Select(f => f.Name).ToArray();
Assert.That(fileNames, Has.Some.SamePath("/ftpTest/foo/txt.txt"));
Assert.That(fileNames, Has.Some.SamePath("/ftpTest/bar/txt.txt"));
Assert.That(fileNames, Has.None.SamePath("/ftpTest/bar/nontxt.non"));
Assert.That(fileNames, Has.Some.SamePath("/ftpTest/bar/baz/txt.txt"));
}
The setup for the test environment can be found from the beginning of the test file.
In order to use the code with Cobalt you first need to compile it to an assembly and deploy the assembly somewhere FRENDS Iron can find the code.
This means either the \Program Files\Frends Technology\FRENDS Iron -directory or the GAC
Once this is done, you need to configure Cobalt to use the new custom endpoint. To do this, just create a new Cobalt connection point as described in the manual and:
Source endpoint type to "Custom"Directory field separated by semicolonsCustom element:
AssemblyName to the assembly name where the custom class resides in, e.g. Frends.Cobalt.Extensions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.ClassName to class name of custom class, e.g.Frends.Cobalt.Extensions.CustomFtpEndpointForMultipleSourceDirs.Parameters empty, as our custom endpoint does not take any parameters.