Endpoint from scratch HowTo - Creating a new endpoint

Scenario: Get and put files via HTTP

You want to fetch a file with HTTP from a specific URL or POST files to a web page.

This example has been designed to work on an environment which has IIS 7 set up. A few tests have been written for the HttpEndPoint. The example solution location can be found here.

The unit tests for the end point can be found in the Frends.Cobalt.Extensions.Tests project's HttpEndPointTests.cs file and the integration tests are located in IntegrationTests/HttpEndPointWorkflowIntegrationTest.cs file. The integration tests test the EndPoint with the Cobalt workflow. The requirements for the test environment can be found from the beginning of the test files.

Basics

All endpoints have to be derived from the abstract EndPointBase class. Which is defined as following:

001public interface IEndPointDetails
002{
003    /// <summary>
004    /// The name of the file (or file mask) that this endpoint is configured to use. Meant for reporting purposes.
005    /// </summary>
006    string FileName { get; }
007 
008    /// <summary>
009    /// The name of the directory this endpoint is configured to use. Meant for reporting purposes.
010    /// </summary>
011    string Dir { get; }
012 
013    /// <summary>
014    /// The possible server address this endpoint is configured to use. Meant for reporting purposes.
015    /// </summary>
016    string Address { get; }
017 
018    /// <summary>
019    /// The string representation of the type of transfer for the endpoint. Meant for reporting purposes.
020    /// </summary>
021    string TransferType { get; }
022}
023 
024///<summary>
025/// The common interface for all Cobalt endpoints.
026/// This interface defines the methods needed for both source and destination endpoints
027///</summary>
028public abstract class EndPointBase : IEndPointDetails, IDisposable
029{
030 
031    /// <summary>
032    /// Returns the name or description of EndPoint
033    /// </summary>
034    /// <returns></returns>
035    public virtual string GetEndPointName();
036    ///<summary>
037    /// Deletes the given remote file.        
038    ///</summary>
039    ///<param name="remoteFile">The name of the remote file to remove. 
040    /// If this is a relative path ("foo/file.txt") or just a file name ("file.txt"), it will be based on the current directory.
041    /// If you give an absolute path (that has a root, i.e. '/' or e.g. 'C:\'), it will be used as is              
042    /// </param>
043    public abstract void Delete(string remoteFile);
044 
045    ///<summary>
046    /// Renames the remote file.
047    /// <remarks>This method is called when: 
048    /// <list type="bullet">
049    /// <item>Starting the transfer of a file: renames the source file with a unique name in order to lock the file during transfer</item>
050    /// <item>Executing the source file operation: Nothing renames the source temp file back to the original file name, Rename renames the source temp file to the given name</item>
051    /// <item>Rolling back the transfer (due to an error): The temporary source file is renamed back to the original file name to release it.</item>
052    /// <item>Ending the transfer: renames the temporary destination file (again, done to lock the file during transfer) to the final file name</item>
053    /// <item></item>
054    /// </list>
055    /// </remarks>
056    ///</summary>
057    ///<param name="remoteFile">
058    /// The name of the remote file to rename.
059    /// If this is a relative path ("foo/file.txt") or just a file name ("file.txt"), it will be based on the current directory.
060    /// If you give an absolute path (that has a root, i.e. '/' or e.g. 'C:\'), it will be used as is. 
061    /// </param>
062    ///<param name="toFile">
063    /// The new name of the file.
064    /// As with <see cref="remoteFile"/>, this can be either a relative or absolute path.
065    /// </param>
066    ///<returns>The full path to the renamed file (so it can be used in when executing the SourceOperation, Get etc.) </returns>
067    public abstract string Rename(string remoteFile, string toFile);
068 
069    /// <summary>
070    /// Moves the file to another directory
071    /// <remarks>
072    /// Used when executing the source file operation: Move moves the file to the given directory
073    /// </remarks>
074    /// </summary>
075    /// <param name="remoteFile">Source file name</param>
076    /// <param name="toPath">Full path, with filename, where to move source file after transfer</param>
077    public abstract string Move(string remoteFile, string toPath);
078 
079    /// <summary>
080    /// Opens the endpoint connection. 
081    /// This method must be called before using the endpoint, and 
082    /// after this method has been called, the user should be able to call other methods of the endpoint
083    /// This can mean different things depending on the endpoint type, e.g.
084    /// opening a connection, logging in and setting the default directory
085    /// </summary>
086    public abstract void Open();
087 
088    /// <summary>
089    /// Closes the endpoint connection.
090    /// When this method has been called, all internal connections should be disconnected and resources released.        
091    /// </summary>
092    public abstract void Close();
093 
094    /// <summary>
095    /// Gets a file from the endpoint.
096    /// Called for each file returned by <see cref="ListFiles"/>
097    /// </summary>
098    /// <param name="remoteFile">
099    /// The name of the file to fetch from the source.
100    /// Relative path ("foo/file.txt") or just a file name ("file.txt") will be based on the current directory.
101    /// Absolute path (that has a root, i.e. '/' or e.g. 'C:\') will be used as is. 
102    /// </param>
103    ///  <param name="localFilePath">The full path to the local temporary file to store the file to</param>
104    public abstract void Get(string remoteFile, string localFilePath);
105 
106    /// <summary>
107    /// Creates and uploads a file to the endpoint         
108    /// </summary>
109    /// <param name="sourceFile">
110    /// The full path to the local temporary file to upload (the result of possible transforms etc.)
111    /// </param>
112    /// <param name="remoteFile">
113    /// The name of the file to create on the remote endpoint.
114    /// Relative path ("foo/file.txt") or just a file name ("file.txt") will be based on the current directory.
115    /// Absolute path (that has a root, i.e. '/' or e.g. 'C:\') will be used as is. 
116    /// </param>
117    public abstract void Put(string sourceFile, string remoteFile);
118 
119    /// <summary>
120    /// Appends the content of the local source file to the destination file
121    /// </summary>
122    /// <param name="sourceFile">The full path to the local temporary file</param>
123    /// <param name="remoteFile">The name of the file to append to on the remote endpoint.
124    /// Relative path ("foo/file.txt") or just a file name ("file.txt") will be based on the current directory.
125    /// Absolute path (that has a root, i.e. '/' or e.g. 'C:\') will be used as is. 
126    /// </param>
127    public abstract void Append(string sourceFile, string remoteFile);
128 
129    /// <summary>
130    /// Gets the list of files from the source endpoint that match the <see cref="FileName"/> mask.
131    /// The method skips all entries for the routine 
132    /// </summary>
133    /// <returns>The list of file details</returns>
134    public abstract IList<FileItem> ListFiles();
135 
136    /// <summary>
137    /// Checks if the given file exists on the remote endpoint
138    /// </summary>
139    /// <param name="remoteFilePath">Full path to the file to check</param>
140    /// <returns>True if the file exists</returns>
141    public abstract bool FileExists(string remoteFilePath);
142}

For this HTTP example, the most important methods to implement are:

ListFiles - When used as a source endpoint, ListFiles is called to return the list of files that match the file mask. The files will then be transferred one by one. Because HTTP cannot list directory contents, the method should just return the combined path for directory/file name
Get - When used as a source endpoint, Get is called to actually transfer the file from the source location to the local, temporary file on the server running the Cobalt transfer. For HTTP, this is a simple GET command.
Put - When used as a destination endpoint, Put is called to transfer the local temporary file (fetched with Get and possibly modified by message processing steps) to the destination location. For HTTP, this could be the PUT or POST command. In this example, we use POST.

If an endpoint can not support some of the methods defined in the base, it should throw an exception if those methods are called. In our HTTP example these unsupported methods are:

Delete
Rename
Move
Append

These are not supported by HTTP alone and would have to have specific pages or web services to handle their functionality. The Put functionality also needs help to function and we have provided a simple example ASP .NET-page which can receive a file through a POST-request. This page can be found in the Frends.Cobalt.Extensions project.

HTTP example walkthrough

WebClientWrapper

We wrap the .NET class WebClient in our own wrapper WebClientWrapper which implements the interface IWebClientWrapper to make it mockable.

01[assembly: InternalsVisibleTo("Frends.Cobalt.Extensions.Tests")]
02namespace Frends.Cobalt.Extensions
03{
04    public class WebClientWrapper : IWebClientWrapper
05    {
06        private WebClient _client;
07 
08 
09        public WebClientWrapper(WebClient client)
10        {
11            _client = client;
12        }
13 
14        public void Dispose()
15        {
16            if(_client != null)
17            {
18                _client.Dispose();
19                _client = null;
20            }
21             
22        }
23 
24        public ICredentials Credentials
25        {
26            get { return _client.Credentials; }
27            set { _client.Credentials = value; }
28        }
29 
30        public byte[] DownloadData(Uri uri)
31        {
32            return _client.DownloadData(uri);
33        }
34 
35        public byte[] UploadFile(Uri uri, string method, string file)
36        {
37            return _client.UploadFile(uri, method, file);
38        }
39    }
40 
41    public interface IWebClientWrapper
42    {
43        void Dispose();
44        ICredentials Credentials { get; set; }
45        byte[] DownloadData(Uri uri);
46        byte[] UploadFile(Uri uri, string method, string file);
47    }

Dispose method

Since the EndPointBase interface implements IDisposable we have to write a Dispose() method which frees the resources reserved by the class. This means we should dispose of all disposable classes we have initialized. In our case this means the WebClient.

01public override void Dispose()
02{
03    Dispose(true);
04}
05 
06private void Dispose(bool disposing)
07{
08    if (!disposing)
09        return;
10 
11    if (_webClient != null)
12    {
13        _webClient.Dispose();
14        _webClient = null;
15    }
16}

The properties

In addition to the properties inherited from EndPointBase class, HTTP endpoint has some other data stored in private fields as well. We get all of these from the TransferEndPointConfig, which is initialized from the config-xml, except for the WebClient.

1private readonly string _username;
2private readonly string _password;
3private readonly int _port;
4 
5private IWebClientWrapper _webClient;

The constructors

There are two constructors, one to enable testing with a mocked WebClientWrapper and another one to implement the EndPointBase, which initializes a new WebClient. TransferEndPointConfig contains the basic connection information for the enpoint. If there were some extra parameters that we needed, we would parse them from the parameters string in the constructor, which is defined in the xml-configuration.

01internal HttpEndPoint(TransferEndPointConfig endPointConfig, IWebClientWrapper webClient) : base(endPointConfig)
02{
03    _webClient = webClient;
04 
05    TransferType = "Http";
06    _port = endPointConfig.ServerPort <= 0 ? 80 : endPointConfig.ServerPort;
07    _username = endPointConfig.ServerUsername ?? "";
08    _password = endPointConfig.ServerPassword ?? "";
09 
10    //If password or username defined, add them to the request
11    SetPossibleCredentials();
12}
13 
14public HttpEndPoint(TransferEndPointConfig endPointConfig) : this(endPointConfig, new WebClientWrapper(new WebClient())) { }

GetEndPointName -method

This returns a basic description of the endpoint, in our case we return the URL for the endpoint. It could contain more information, such as the file size if we knew it.

1public string GetEndPointName()
2{
3    return GetUrl(FileName).ToString();
4}

Delete -method

This is the delete method used for deleting files at the endpoint, the workflow uses this with Source operation: Delete and Destination File Exists operation: Overwrite. The delete operation is not possible with the HTTP-endpoint, so an exception is thrown if it is called.

1public void Delete(string remoteFile)
2{
3    //deleting is not possible
4    throw new NotSupportedException("Deletion is not possible with HTTP Endpoint");
5}

Rename -method

The Rename -method is used while transferring files for locking purposes. Since renaming is not always possible with endpoints, it can be disabled in the configuration Paramaters section with the RenameSourceFileBeforeTransfer and RenameDestinationFileDuringTransfer -settings. Rename is not supported by HTTP, so an exception is thrown if it is called.

1public string Rename(string remoteFile, string toFile)
2{
3    //rename is not possible
4    throw new NotSupportedException("Renaming is not possible with HTTP Endpoint");
5}

Move -method

The Move -method is used with the Source Operation: Move to move the file that was transferred to another directory. This is not supported by the HTTP -endpoint so an exception is thrown when called.

1public string Move(string remoteFile, string toPath)
2{
3    //Move is not possible
4    throw new NotSupportedException("Moving files is not possible with HTTP Endpoint");
5}

Open/Close -methods

The open method is called when the workflow is started and close is called when the workflow has finished its transfer. They should contain the code for opening and closing the connection if needed. Our HTTP -endpoint does not need them because each transfer is handled in its own connection, so we do nothing in these methods.

1public void Open()
2{
3    //Transfers are done in their own connections
4}
5 
6public void Close()
7{
8    //Transfers are done in their own connections
9}

GetUrl -utility method

This method does not belong to the interface and is used to build the url for the webpage. This is used to create the URI for the WebClient and with the GetEndPointName-method.

01internal Uri GetUrl(string remoteFile)
02{
03    UriBuilder builder = new UriBuilder();
04    builder.Host = Address;
05    builder.Port = _port;
06    builder.Scheme = "http";
07    builder.Path = Path.Combine(Dir, remoteFile);
08 
09    return builder.Uri;
10}

Get -method

The Get -method contains the code used for getting the files from the source endpoint. In our HTTP -endpoint we use the .NET WebClient class to download the data from the URI.

1public void Get(string remoteFile, string localFilePath)
2{
3    Uri uri = GetUrl(remoteFile);
4 
5    var fileBytes = _webClient.DownloadData(uri);
6 
7    File.WriteAllBytes(localFilePath, fileBytes);
8 
9}

SetPossibleCredentials -utility method

This method does not belong to the interface and is used to setup the credentials to our WebClient if either username or password is provided.

1private void SetPossibleCredentials()
2{
3    if (!String.IsNullOrEmpty(_username) || !String.IsNullOrEmpty(_password))
4        _webClient.Credentials = new NetworkCredential(_username, _password);
5}

Put -method

The Put method contains the code used for sending the files to destination endpoint. The HTTP -endpoint simply uses the .NET WebClient class to POST the file the URI. The response is discarded, but it could be used to detect errors.

1public void Put(string sourceFile, string remoteFile)
2{
3    Uri uri = GetUrl(remoteFile);
4 
5    var response = _webClient.UploadFile(uri, "POST", sourceFile); //TODO: Handle response, errors such as 404 are thrown out by the _webClient as exceptions
6}

Append -method

Append method is called if the Destination Exists operation: Append is selected and the destination file exists(FileExists returns true). The method should contain the code to append the content of the sourcefile to the destination file. The sourcefile is the path to the temporary local file and remoteFile is the path to the file on the remote machine. The HTTP -endpoint does not support append operations so it throws an exception if called.

1public void Append(string sourceFile, string remoteFile)
2{
3    throw new NotSupportedException("Append is not supported by http");
4}

ListFiles -method

The ListFiles -method is used in the beginning of the workflow to retrieve the list of files to transfer from the source endpoint. It should handle matching the filenames to the Filemask and only return an IList of FileItems which will be transferred. The HTTP -endpoint as a source endpoint supports only the retrieval of a single file, so it returns the filename that was provided in the config.

1public IList<FileItem> ListFiles()
2{
3    //HttpEndpoint can only transfer single files so we return the original FileName, 
4    //which normally is used for reporting only. 
5    //This example only supports getting a single file because implementing a directory
6    //listing parser would be too tedious for the scope of this example.
7    //TODO: Check if the FileName is a filemask and throw an exception if this is the case
8    return new List<FileItem>() { new HttpFileItem(FileName) };
9}

FileExists -method

FileExists is called when transferring files to the destination endpoint before each transfer. If it returns true the workflow acts accordingly to the DestinationFileExists Operation setting in the config. The HTTP -endpoint always returns false because the destination page should always exist because it handles receiving the file data.

1public bool FileExists(string remoteFilePath)
2{
3    //Destination filename is the page which is receiving the data with the POST
4    //This method is used when checking if the remote file exists so we won't overwrite it
5    //but remoteFilePath is a page which should always exist, and we cannot overwrite it
6    //therefore we always return false so the Cobalt workflow will function correctly
7    return false; 
8}

HttpFileItem -utility class

We create a new HttpFileItem class that inherits the FileItem class. We don't use the base class because most of the data defined is unavailable for us with the HTTP -endpoint. This is why we instantiate the FileItem object with some default values.

01    public class HttpFileItem : FileItem
02    {
03        public HttpFileItem(string fileName)
04        {
05            //Using static values because we cannot get the file information without getting the whole file.            
06            this.Modified = DateTime.MaxValue;
07            this.Name = fileName;
08            this.Size = -1;
09        }
10    }
11}

Using the new module in FRENDS Cobalt

Step 1: Compile and deploy the assembly

In order to use the code with Cobalt you first need to compile it to an assembly and deploy the assembly somewhere FRENDS Iron can find the code. This means either the \Program Files\Frends Technology\FRENDS Iron -directory or the GAC

Step 2: Configure Cobalt

Once this is done, you need to configure Cobalt to use the new custom endpoint. To do this, just create a new Cobalt connection point as described in the manual and:

Change the Source endpoint type to "Custom"
The filename has to be the actual filename of the file to get, it cannot contain masks, e.g. index.htm, text.txt, order.xml, but not *.txt
Change the Destination endpoint type to "Custom"
Change the Directory to the location of the page that will receive the files
Under the Custom elements under Source and Destination:
- Set the AssemblyName to the name of the assembly containing the custom class, e.g. Frends.Cobalt.Extensions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null and ClassName to the name of the endpoint class, e.g. Frends.Cobalt.Extensions.HttpEndPoint
- Leave the Parameters empty, as our custom endpoint does not take any parameters.
Change the RenameSourceFileBeforeTransfer to false, because our source end point is a HttpEndPoint, which does not support renaming.
Change the RenameDestinationFileDuringTransfer to false, because our destination end point is a HttpEndPoint, which does not support renaming.

Sample configuration which gets the file from http://localhost:80/test/test.txt and POSTs it to http://localhost:80/test/HttpFileReceiver.aspx.

You could also put the whole address+query to Directory field (http://localhost:80/test/test.txt) and leave FileName empty (server address and port will be ignored).

Connection point with parameters

Step 3: Create and execute the file transfer routine

As in the manual create and execute file transfer routine.