Extension HowTo

Scenario: Get / put file from a server that only supports SCP

FRENDS Cobalt supports SFTP, which provides secure file transfers. We recommend using SFTP when possible, but if SCP is the only secure option for some reason, then this example may be of help. SCP has some limitations, for example it does not support file listings, appending, or checking if a file exists. For more information about SCP, see its Wikipedia page.

Configuration

The SCP endpoint utilizes the SFTP settings, since both SFTP and SCP are based on SSH connections. To use the SCP sample endpoint, you therefore need to configure the endpoint as you would an SFTP endpoint, i.e. using the Server and SFTP entries etc. See the main Cobalt documentation for more on the SFTP configuration.

NOTE: This SCP endpoint currently supports only getting one file at a time when used as the Source endpoint. This is due to Cobalt always transferring one file at a time, based on the directory listing: because SCP does not allow listing the directory contents, the ListFiles() method just returns the name of the file given as the FileName parameter. Therefore the FileName cannot contain any wildcard characters; the file name must be exactly the name of the file to get.

Therefore to configure the SCP endpoint, use the Server and SFTP settings, just make sure you use the custom endpoint instead of SFTP, i.e. the following settings:

Type: Custom

Directory: [The directory to get (Source) or put (Destination) the file(s)]

FileName: [When used as Source, must be the actual file name: no wildcards are supported! When used as Destination, macros are supported, though.]

Server

Address: [The server name]
Port: [The SSH port, by default 22]
Username: [The user name]
Password: [The user password, if you are using 'Username-Password' as the SFTP LoginType. Note that you should use an encrypted CP field reference so the password is not shown in plain text.]

Custom

AssemblyName: Frends.Cobalt.Extensions
ClassName: Frends.Cobalt.Extensions.ScpEndPoint
Parameters: [can be left empty because all the settings are defined in the Server, SFTP etc. sections]

Also note that because the SCP protocol is so limited, the SCP endpoint does not support most of the available settings in the Parameter section:

RenameSourceFileBeforeTransfer / RenameDestinationFileDuringTransfer when using must be false, because SCP cannot rename files.

DestinationFileExistsAction doesn't matter, because SCP cannot check if the file exists before transfer. If the file already exists, it will be overwritten.

NoSourceAction doesn't matter, because SCP cannot check if the file exists before transfer. If the file does not exist, an error will be thrown during transfer.

SourceOperation must be Nothing, because SCP cannot rename, move or delete files.

The code

001public class ScpEndPoint : EndPointBase
002{
003    private static readonly ILog Log = LogManager.GetLogger(typeof(ScpEndPoint));
004    protected Scp ScpClient;
005    protected TransferEndPointConfig EndPointConfig { get; set; }
006 
007    /// <inheritdoc />
008    public ScpEndPoint(TransferEndPointConfig endPointConfig)
009        : base(endPointConfig)
010    {
011        EndPointConfig = endPointConfig;
012 
013        ScpClient = new Scp();
014    }
015 
016    /// <inheritdoc />
017    public override void Dispose()
018    {
019        Dispose(true);
020    }
021 
022    /// <inheritdoc />
023    protected virtual void Dispose(bool disposing)
024    {
025        if (disposing)
026        {
027            if (ScpClient != null)
028            {
029                ScpClient.Dispose();
030                ScpClient = null;
031            }
032        }
033    }
034 
035    /// <summary>
036    /// Delete operation is not supported by SCP, throws NotSupportedException
037    /// </summary>
038    /// <param name="remoteFile"></param>
039    public override void Delete(string remoteFile)
040    {
041        throw new NotSupportedException("Delete action is not supported by SCP, this means that SourceOperation 'Delete' is not supported");
042    }
043 
044    /// <summary>
045    /// Rename operation is not supported by SCP, throws NotSupportedException
046    /// </summary>
047    /// <param name="remoteFile"></param>
048    /// <param name="toFile"></param>
049    /// <returns></returns>
050    public override string Rename(string remoteFile, string toFile)
051    {
052        throw new NotSupportedException("Rename action is not supported by SCP, this means that Parameters RenaeSourceFileBeforeTransfer and RenameDestinationFileDuringTransfer are not supported.");
053    }
054 
055    /// <summary>
056    /// Move operation is not supported by SCP, throws NotSupportedException
057    /// </summary>
058    /// <param name="remoteFile"></param>
059    /// <param name="toPath"></param>
060    /// <returns></returns>
061    public override string Move(string remoteFile, string toPath)
062    {
063        throw new NotSupportedException("Move action is not supported by SCP, this means other SourceOperations than Nothing are not supported.");
064    }
065 
066    /// Opens the connection to the remote server. Uses the SftpParameters from the 
067    /// endpoint configuration as authentication options. 
068    public override void Open()
069    {
070        if (Log.IsDebugEnabled)
071        {
072            // enable verbose logging
073            ScpClient.LogWriter = new Log4NetRebexLogWriter(Log, Rebex.LogLevel.Verbose);
074        }
075 
076        Connect(EndPointConfig.ServerAddress, EndPointConfig.ServerPort);
077 
078        var scpParameters = EndPointConfig.SftpParameters;
079        if (!String.IsNullOrEmpty(scpParameters.ServerFingerPrint))
080        {
081            string fp = ScpClient.Fingerprint;
082            string verify = scpParameters.ServerFingerPrint;
083            if (!verify.Equals(fp))
084            {
085                ScpClient.Disconnect();
086                throw new Exception(String.Format("Can't trust SCP server. The server fingerprint does not match. Expected fingerprint: '{0}', but was: '{1}'",
087                                                    verify, fp));
088            }
089        }
090 
091        var loginType = (SftpLoginType)Enum.Parse(typeof(SftpLoginType), scpParameters.LoginType);
092        if (loginType == SftpLoginType.UsernamePassword)
093        {
094            ScpClient.Login(EndPointConfig.ServerUsername, EndPointConfig.ServerPassword);
095        }
096        else if (loginType == SftpLoginType.UsernamePasswordPrivatekey)
097        {
098            ScpClient.Login(EndPointConfig.ServerUsername, EndPointConfig.ServerPassword, scpParameters.GetPrivateKey());
099        }
100        else if (loginType == SftpLoginType.UsernamePrivateKey)
101        {
102            ScpClient.Login(EndPointConfig.ServerUsername, scpParameters.GetPrivateKey());
103        }
104        else
105        {
106            ScpClient.Disconnect();
107            throw new Exception("Unknown SFTP login type.");
108        }
109 
110 
111    }
112 
113    private static readonly Mutex ConnectMutex = new Mutex();
114 
115    //Timeout for acquiring the Connect mutex
116    private const int MutexTimeout = 300000;
117 
118    /// <inheritdoc />
119    protected void Connect(string host, int port)
120    {
121        if (ConnectMutex.WaitOne(MutexTimeout, false))
122        {
123            try
124            {
125                ScpClient.Connect(host, port);
126            }
127            finally
128            {
129                ConnectMutex.ReleaseMutex();
130            }
131        }
132        else
133        {
134            throw new TimeoutException("Timeout while waiting for connection mutex.");
135        }
136    }
137 
138    /// <inheritdoc />
139    public override void Close()
140    {
141        ScpClient.Disconnect();
142    }
143 
144    /// <summary>
145    /// Helper to execute SCP actions with a reconnect and retry, if the actions fails again,
146    /// the exception from the retry is thrown out.
147    /// </summary>
148    /// <param name="action"></param>
149    private void ExecuteActionWithOneRetry(Action<Scp> action)
150    {
151        try
152        {
153            action.Invoke(ScpClient);
154        }
155        catch (InvalidOperationException ioe)
156        {
157            ReconnectAndRetry(action, ioe);
158        }
159        catch (ScpException e)
160        {
161            ReconnectAndRetry(action, e);
162        }
163    }
164 
165    /// <summary>
166    /// Reconnects to the server and then retries the action once, any exception that is thrown by the action is allowed to be thrown up
167    /// </summary>
168    /// <param name="action"></param>
169    /// <param name="ex"></param>
170    private void ReconnectAndRetry(Action<Scp> action, Exception ex)
171    {
172        Log.InfoFormat("Error when trying to execute action '{0}'. Error: '{1}'. Retrying once.", action.Method, ex.Message);
173 
174        // Rebex does not support checking the connection state for SCP so
175        // we reopen the connection just in case
176        Reopen();
177        action.Invoke(ScpClient);
178    }
179 
180    /// <summary>
181    /// Closes and reopens the connection.
182    /// </summary>
183    private void Reopen()
184    {
185        Close();
186        Open();
187    }
188 
189    /// <inheritdoc />
190    public override void Get(string remoteFile, string localFilePath)
191    {
192        ExecuteActionWithOneRetry(c => c.GetFile(remoteFile, localFilePath));
193    }
194 
195    /// <inheritdoc />
196    public override void Put(string sourceFile, string remoteFile)
197    {
198        ExecuteActionWithOneRetry(c => c.PutFile(sourceFile, remoteFile));
199    }
200 
201    /// <summary>
202    /// Append operation is not supported by SCP, throws NotSupportedException
203    /// </summary>
204    /// <param name="sourceFile"></param>
205    /// <param name="remoteFile"></param>
206    public override void Append(string sourceFile, string remoteFile)
207    {
208        throw new NotSupportedException("Append action is not supported by SCP, only supported value for DestinationFileExistAction is Overwrite.");
209    }
210 
211    /// <summary>
212    /// Listing files is not supported by SCP, instead is assumes the file always exists.
213    /// </summary>
214    /// <returns>List which contains one element, which is the source file name.</returns>
215    public override IList<FileItem> ListFiles()
216    {
217        //SCP does not support listing files or checking if a file exists,
218        //therefore we assume that a single file that is exactly the filemask exists
219        return new List<FileItem>()
220                   {
221                       new ScpFileItem(FileName)
222                   };
223    }
224 
225    /// <summary>
226    /// Checking for file existance is not directly supported by SCP, we would have to try
227    /// to copy the file to check for its existence. Since it would be ugly, and SCP always
228    /// overwrites an existing destination file, we can always assume that the file does not
229    /// exist.
230    /// </summary>
231    /// <param name="remoteFilePath"></param>
232    /// <returns>false</returns>
233    public override bool FileExists(string remoteFilePath)
234    {
235        //Checking for file existence is not supported by SCP and the existing files are always
236        //overwritten so we return false.
237 
238        return false;
239    }
240}
241 
242/// <summary>
243/// ScpFileItem, derived from FileItem and only has the Name property set as SCP does not
244/// provide any other information about files.
245/// </summary>
246public class ScpFileItem : FileItem
247{
248    public ScpFileItem(string fileName)
249    {
250        Name = fileName;
251        //Other properties are unattainable with the SCP protocol            
252    }
253}

Extension HowTo - SCP Endpoint

Scenario: Get / put file from a server that only supports SCP

Configuration

The code