Extension HowTo - Extending an existing endpoint

Scenario: Get files from multiple FTP source directories at once

You want to fetch files with FTP from the source directory, another directory, as well as all their subdirectories at once. The ready-made FTP endpoint does not support this: it fetches the files from a single directory only. Therefore you need to create your own FTP endpoint that does this.

Note: This end point works only as a source end point, using it as a destination would not offer any additional functionality over the basic FTP-endpoint.

This example has been designed to work on an environment which has IIS 7 set up.

The ready-made endpoints

There are three ready-made endpoints you can extend:

Frends.Cobalt.FileEndPoint
Frends.Cobalt.FtpEndPoint
Frends.Cobalt.SftpEndPoint

These endpoints implement the IEndPoint interface.

NOTE: For the location of source files see extending cobalt.

Implement the needed operations

Since the directory contains multiple values instead of just one, we need to split them for use later. The value of Dir is set to "", so later when we call base.Open, the initial directory is not changed.

We override the Open-method so we can store the initial directory we start in, this way relative paths from elsewhere than the root are supported.

01private string startDirectory;
02 
03public CustomFtpEndpointForMultipleSourceDirs(TransferEndPointConfig endPointConfig) : base(endPointConfig)
04{
05    if (base.Dir == null)
06        throw new ArgumentException("Directory cannot be left null");
07 
08    //If the Dir is a empty string this will result in a single empty string in the collection
09    //For FTP this is valid and means the initial directory
10    _directories = base.Dir.Split(';');
11 
12    //trim whitespaces, because they cause "cannot find path specified" problems
13    for (int i = 0; i < _directories.Length; i++)
14        _directories[i] = _directories[i].Trim();
15 
16    // set the base dir to empty so we stay in the initial directory
17    base.Dir = "";
18}
19 
20public override void Open()
21{
22    base.Open();
23 
24    //since the user may start elsewhere than the root of the ftp-server, eg. /home/user/ instead of /,
25    //we need to store the original directory we start in to be able to use relational paths multiple times.
26    //this is why whe set the empty initial directory in the constructor
27    startDirectory = FtpClient.GetCurrentDirectory();
28}

Override ListFiles() to go through the directories recursively and return the list of all files with absolute paths instead of the default relative ones - the Get() method will then use the path:

01/// <summary>
02/// The list of directories, initialized in constructor
03/// </summary>
04private readonly string[] _directories;
05 
06public override IList<FileItem> ListFiles()
07{
08    List<FileItem> result = new List<FileItem>();
09     
10    foreach (string directory in _directories)
11    {
12        result.AddRange(GetFilesRecursivelyFromDirectory(directory));
13    }
14    return result;
15}
16 
17private IEnumerable<FileItem> GetFilesRecursivelyFromDirectory(string directoryPath)
18{           
19    //Always change to the start directory, this way both relative and absolute paths work
20    FtpClient.ChangeDirectory(startDirectory);
21     
22    //Change to the path defined in the configuration, may be relative or absolute
23    FtpClient.ChangeDirectory(directoryPath);
24 
25    //Get the Current directory with absolute path for Get()
26    var currentDirectory = FtpClient.GetCurrentDirectory();
27 
28    List<FileItem> result = new List<FileItem>();
29    foreach (FtpItem ftpItem in FtpClient.GetList())
30    {
31        if (ftpItem.IsDirectory)
32            result.AddRange(GetFilesRecursivelyFromDirectory(Path.Combine(directoryPath, ftpItem.Name)));
33        else if (Util.FileMatchesMask(ftpItem.Name, this.FileName))
34        {
35            var item = new FileItem()
36                           {
37                               Modified = ftpItem.Modified,
38                               Size = ftpItem.Size,
39                               //Set the name to absolute path so Get() can fetch the files
40                               Name = Path.Combine(currentDirectory, ftpItem.Name)
41                           };
42            result.Add(item);
43        }
44 
45    }
46     
47    return result;
48}

Note: When using this sample with recursion code do NOT use any Source endpoint directory, or any directory under it, as the Destination endpoint directory. Such configuration will result in unwanted behavior. The process will run successfully for the first time, but later on it will always fail as there are already files existing in the destination directory. The process will fail because all the file rename and copy actions done to the file(s) occur in the same directory and file cannot be copied onto itself.

Note that you can access the given directory string and file mask parameters via the inherited properties Dir and FileName properties respectively, whilst in this sample only FileName property is used. The GetList() method can take a file mask as a parameter, but it would not find the sub-directories. Therefore the code uses the same Util.FileMatchesMask() method that the actual FtpEndPoint uses. You can use the FileItem constructor to initialize the file details. The absolute path needs to be explicitly set (otherwise the name is just the file name) so the Get() method can locate the files.

A simple integration test for this method from Frends.Cobalt.Extensions.Tests.CustomFtpEndpointForMultipleSourceDirsTest (the directories are initialized at the test setup method)

01[Test]
02public void ShouldReturnOnlyTextFilesInAllDirectories()
03{            
04    var context = new TransferEndPointConfig
05                                        {
06                                            Directory = "/ftpTest/foo;/ftpTest/bar",
07                                            FileName = "*.txt",
08                                            ServerAddress = "localhost",
09                                            ServerPort = 21,
10                                            ServerUsername = "anonymous",
11                                            ServerPassword = "password@example.org",
12                                            EndPointType = "Custom",
13                                            FtpParameters = new FtpParameters
14                                                                {
15                                                                    ConnectionMode = "Passive",
16                                                                    TransferType = "Binary"
17                                                                }
18 
19                                        };
20    IList<FileItem> files;
21    using (var endPoint = new CustomFtpEndpointForMultipleSourceDirs(context))
22    {
23        endPoint.Open();
24        files = endPoint.ListFiles();
25        endPoint.Close();
26    }
27    Assert.That(files.Count, Is.EqualTo(3));
28    var fileNames = files.Select(f => f.Name).ToArray();
29    Assert.That(fileNames, Has.Some.SamePath("/ftpTest/foo/txt.txt"));
30    Assert.That(fileNames, Has.Some.SamePath("/ftpTest/bar/txt.txt"));
31    Assert.That(fileNames, Has.None.SamePath("/ftpTest/bar/nontxt.non"));
32    Assert.That(fileNames, Has.Some.SamePath("/ftpTest/bar/baz/txt.txt"));
33}

The setup for the test environment can be found from the beginning of the test file.

Using the new module in FRENDS Cobalt

Step 1: Compile and deploy the assembly

In order to use the code with Cobalt you first need to compile it to an assembly and deploy the assembly somewhere FRENDS Iron can find the code. This means either the \Program Files\Frends Technology\FRENDS Iron -directory or the GAC

Step 2: Configure Cobalt

Once this is done, you need to configure Cobalt to use the new custom endpoint. To do this, just create a new Cobalt connection point as described in the manual and:

Change the Source endpoint type to "Custom"
Give many directories to the Directory field separated by semicolons
Under the Custom element:
- Set the AssemblyName to the assembly name where the custom class resides in, e.g. Frends.Cobalt.Extensions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
- Set the ClassName to class name of custom class, e.g.Frends.Cobalt.Extensions.CustomFtpEndpointForMultipleSourceDirs.
- Leave the Parameters empty, as our custom endpoint does not take any parameters.

Step 3: Create and execute the file transfer routine

As in the manual create and execute file transfer routine.