FRENDS COBALT MANUAL

CONTENTS

• TERMS OF DELIVERY AND USE
• OVERVIEW
• MAJOR CHANGES BETWEEN VERSIONS
• INSTALLATION
  • 1. Install FRENDS Iron
  • 2. Configure FRENDS Iron
  • 3. (Optional) Configure logging file transfers with log4net
• UPGRADING
• CONFIGURING A SAMPLE TRANSFER ROUTINE
  • 1. Setup sample folders and files
  • 2. Create a connection point
  • 3. Create a new routine
  • 4. Run the routine
• TECHNICAL OVERVIEW
  • Transfer overview
  • Error handling
  • Transfer log database
  • Extending FRENDS Cobalt
  • Using Cobalt task Output in sequential tasks
• CONFIGURABLE FILE TRANSPORT PARAMETERS
  • Cobalt Task transfer parameters
  • Transfer parameters (CP Schema)
    • Source
    • Destination
    • General parameters
    • Message processing steps
    • Other parameters
    • Macro reference
• KNOWN ISSUES

TERMS OF DELIVERY AND USE

OVERVIEW

FRENDS Cobalt is a FRENDS Iron Simple task library built for scheduled transfer of files.

With FRENDS Cobalt you can easily automate recurring file transfers within the enterprise. The transfers are configured as a single XML structure from the FRENDS Iron management console. The same console can be used to run and monitor the transfers as well.

FRENDS Cobalt can read files from and deliver files to file system, FTP, FTPS and SFTP servers. Text file encoding can be changed and XML files can be converted using XSLT in message processing steps. Files can naturally be transferred without conversions as well.

Possible errors in FRENDS Cobalt executions are reported to the Windows EventLog and in FRENDS Iron as well, so operators can take action as required. There is also a sample transfer log database that can be used to log transfer details and show reports on failed and successful transfers. In addition, FRENDS Cobalt uses Log4Net for logging purposes that can be configured to e.g. send reports as needed.

MAJOR CHANGES BETWEEN VERSIONS

1.5

Initial public version

1.6

Minor bug fixes. Changed SourceOperation.Rename to also accept directories, so you can move and rename source files after they have been transferred. Now SourceOperation.Move only accepts directories: it can only be used for moving source files with the original names.

1.7

Added extension API.
Major changes to message processing steps: backups, conversions and custom processing are now done in the message processing steps. Note that this change requires you use Iron 2.4 SR1 or later.
Updated Rebex.Net assemblies to version 1.0.3723. See section "Upgrading" for details on what this means for upgrading from previous versions.

1.8

File endpoint now supports specifying username and password for remote file shares.
Message processing step extension API changed. More details added to step's transfer context, see extension API for details. Message processing steps and endpoints need to be rebuilt, see Known issues section for more details.
Cobalt workflow supports FRENDS Iron's new ActionSkipped state. See FRENDS Iron documentation for more information on ActionSkipped state.
Source and destination folders support using macros.
Destination folders can be configured to be created during transfer.

2.0

New version that utilizes the Simple task support in Iron 3.3 and uses .NET 4.0.
Main new features are:

• Support for logging transfer information via the new log service
• Better performance, especially with parallel transfers
• Bulk logging message processing step
• Transferred files are now provided as task Output

2.1

Version adds a new endpoint and message processing step:

• ServiceBus endpoint type which enables sending/retriving messages to/from Azure Service Bus or Service Bus for Windows Server
• FileSystemBackup message processing step which allows using renaming patterns for backup files

INSTALLATION

FRENDS Cobalt is installed and configured with FRENDS Iron.

1. Install FRENDS Iron

Unzip the FRENDS Iron package and run the Windows Installer installation package. Check the prerequisites and follow the instructions in readme.rtf

The installer installs the needed libraries and executables. After installing, you need to run the FRENDS Iron Configuration Tool to set up the Windows services, databases and other settings. The installer suggests you do this at the end of the installation.

2. Configure FRENDS Iron

FRENDS Cobalt is configured together with FRENDS Iron with the FRENDS Iron Configuration tool. For details on this, see the FRENDS Iron configuration tool manual and the FRENDS Iron readme.rtf.

In order to configure FRENDS Cobalt, you need to make sure the checkbox for "Import FRENDS Radon/Cobalt sample routines" is checked. This will import the sample routines, tasks and CPs, as well as copy the needed assemblies to the FRENDS Iron installation directory.

If previous versions of the assemblies are already installed in the FRENDS Iron directory, the configuration tool will ask for confirmation before copying the files. If you choose to continue, the existing assemblies will be overwritten. In this case, if the sample tasks are also installed, new copies of them will be imported using new names like "FRENDS Cobalt Sample Routine (2)"

FRENDS Cobalt can be configured to log all file transfers and details on transferred files to a SQL Server database via Service Bus. Transferred files can be stored into Windows Azure storage as well. More information on logging can be found from section Transfer log database. Configuration options for using Service Bus and Windows Azure Storage for logging can be found in configuration tool's documentation.

3. (Optional) Configure logging file transfers with log4net

log4net can be used to help with logging / notifications on file transfers. All succeeded file transfers are logged as INFO level events and all failures are logged as ERROR level events.

The name of the logger used is provided in the routines task parameter "TransferLoggerName" and the logger can be configured in the FrendsAccService.exe.config file.

Example logger configuration with SmtpAppender for log4net:

<appender name="CobaltSmtpAppender" type="log4net.Appender.SmtpAppender"> 
    <to value="to@domain.com" />
    <from value="from@domain.com" />
    <subject value="Cobalt transfer result" />
    <smtpHost value="SMTPServer.domain.com" />
    <bufferSize value="1" />
    <lossy value="true" />
    <evaluator type="log4net.Core.LevelEvaluator">
        <threshold value="ERROR"/>
    </evaluator>
    <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%newline%date [%thread] %-5level %logger [%property{NDC}] - %X{transfername} file: %X{sourcefile} ended with result %X{result} - %message%newline%newline%newline " />
    </layout>
</appender>

<logger name="FileTransferSmtpLog">
    <appender-ref ref="CobaltSmtpAppender" />
    <level value="ERROR" />
</logger>

The following variables can be accessed in the conversionPattern with the %X{variableName}:

• transfername
• transfergroupname
• batchid - The GUID of the Cobalt workflow process.
• sourcefile
• sourcepath
• destinationfile
• destinationpath
• transferstarted
• transferended
• filesize - Size of transferred file in bytes.
• errorinfo - Information about error (can be null).
• batchsize - Number of files transferred.
• sequenceinbatch - The file's sequence number in the transfer batch.
• result - The result of the individual file transfer, can be success or fail.
• sourceaddress - The address from which the file is transferred from (can be null).
• destinationaddress - The address to which the file is transferred to (can be null).
• sourcetransfertype - The type of the transfer from the source (FTP / SFTP / FTPS / FILE / etc...).
• destinationtransfertype - The type of the transfer to the destination (FTP / SFTP / FTPS / FILE / etc...).

Log4net is designed to fail silently if it fails while trying to write to the log appender. This should be taken into account when creating reporting with log4net.

UPGRADING

Upgrading from 1.5 or older versions

Upgrading from 1.5 or older versions requires you to recreate the transfer configuration CP's with the new schema versions as well as use the newer versions of the Cobalt routines.

Upgrading from 1.6

Routines and CPs from 1.6 are compatible with the latest 1.7 version: even though the schemas have been changed, the old schema is still accepted and e.g. the encoding operations are dynamically converted to corresponding message processing steps.

Please note that the Rebex.Net assemblies have been updated from 1.0.2896 to 1.0.3723, and you cannot use the old Rebex versions with the newer Cobalt versions or vice versa - unless you deploy the older Rebex.Net versions first to the Global Assembly Cache (GAC).

Also please note the changed CP schema requires you use Iron 2.4 SR1 or later. On previous Iron versions the existing schemas may not update correctly, and MessageProcessingStep nodes will get duplicated every time you edit the CP. This issue was fixed in 2.4 SP1.

Upgrading from 1.7 beta

The API references were modified since the 1.7 beta, so upgrading to the final 1.7 version requires you to update and recompile possible extensions made against the 1.7 beta. Also, since 1.7 beta used the same old Rebex.net assemblies as 1.6, upgrading to the final 1.7 version means you need to update all workflows using the Rebex assemblies at the same time.

Using multiple Cobalt versions side by side

You may not want to upgrade all existing Cobalt workflows to use the latest workflow, e.g. if you have made extensions that rely on a specific version of Cobalt. To do this, you need to deploy the different Cobalt versions to the Global Assembly Cache (GAC), where they can be loaded to different workflows as needed.

Upgrading Cobalt manually

Because the latest version of Cobalt requires many improvements and fixes to FRENDS Iron, the recommended way to upgrade Cobalt is to upgrade Iron (and Cobalt with it) to the latest version. This way, there should be no compatibility problems or issues with bugs already fixed.

However, if you absolutely cannot update Iron, and accept that you will likely encounter problems, you can update Cobalt manually as follows:

1. Get the internal Cobalt release zip file with the latest assemblies and schemas and extract the files to a local directory.
2. Copy the Frends.Cobalt.dll assembly to \Program Files\Frends Technology\FRENDS Iron
3. Copy also the latest Rebex.*.dll assemblies to \Program Files\Frends Technology\FRENDS. Normally these are updated with Iron, but you can find the latest assemblies also from the zip file, in folder Frends.Cobalt.Extensions\lib
4. Import the sample Cobalt routine - which also imports the latest schemas - to Iron by importing the routine from the FRENDSCobalt-Routine.xml file.

CONFIGURING A SAMPLE TRANSFER ROUTINE

1. Set up sample folders and files

1. Create folders for testing:
   • C:\Cobalt - Root folder for testing.
   • C:\Cobalt\In - Source folder for files to be transferred.
   • C:\Cobalt\Out - Destination folder for the files.
2. Give the FRENDS Iron service account read, write and modify access to the C:\Cobalt directory and its subdirectories.
3. Create a new text file to C:\Cobalt\In with the name Sample.txt. It can be left empty if you wish.

2. Create a connection point

1. In FRENDS Iron management console navigate to FRENDS / Routine elements / Connection points.
2. Right click and select "Add new Connection Point".
3. Name it SampleCP.
4. Click "Load Schema".
   
5. If there is no existing FRENDS Cobalt schema in the list, or you want to be sure to use the latest version of Cobalt do as follows, otherwise continue from step 6.
   1. Click on the browse (...) button.
   2. Select "Frends_Cobalt_CP.xsd" from the installation folder, default: C:\Program Files\Frends Technology\FRENDS Iron.
6. Click OK.
7. Configure source:
   
8. Configure destination:
   
9. Configure parameters:
   
10. Leave the message processing step to use PassThrough type, since we do not want to use any processing.
    

3. Create a new routine

1. In FRENDS Iron management console navigate to FRENDS / Routines.
2. Right click and select "Add new Routine".
3. Name the routine, Click Add button to add a new Task for Routine and select the FRENDS Cobalt 2.0 from the opening list.
4. Check the "Use Connection Points" checkbox, select the "Single" radio button and then select SampleCP from the Connection Point drop down menu.
5. Set up task arguments on the "Task Arguments" tab (see the Cobalt task transfer parameters section for more details on the options):
   
   
   Note: This routine is to be started manually for now, so no schedule is needed.
6. Click OK.

4. Run the routine

Now the created FRENDS Cobalt transfer is ready to be run from the routines view. Just right click and select "Run Once".

If so desired the routine can be scheduled to be run with the regular FRENDS Iron scheduling tools.

TECHNICAL OVERVIEW

FRENDS Cobalt is implemented .NET 4.0 class that is run in a FRENDS Iron routine. The next section describes the basic steps of a typical file transfer.

Transfer overview

The file transfer progress has the following steps:

1. Initialize
   

   Initialize the configuration based on the XML config and open source connection(s).

   Note: If you have set the MaxConcurrentConnections to more than 1, the given number of source connections are opened.

2. ListFiles
   

   Get a list of files from the source endpoint according to the filename/mask.

   If there are no files to transfer, the source connections are closed and the transfer ends with the ActionSkipped state

3. Transfer files
   

   If there are files to transfer, they are then transferred individually.

   If there are more than one source connection (MaxConcurrentConnections is set to > 1), the equal number of destination connections will be opened, and the files are transferred in parallel using these source-destination connection pairs.

   For every file in the list returned from the source endpoint, the following process is repeated:

   1. GetFile
      Get a file from the source endpoint to the local work directory.
      If the parameter "RenameSourceFileBeforeTransfer" is set to true (default), the file is first renamed with a temporary filename before transfer.
   2. ExecuteMessageProcessingSteps
      Execute every MessageProcessingStep defined in the configuration. The steps are executed in sequence, in the same order that they are defined in the configuration.
      The steps process local temporary files as follows: the path to the file originally fetched from the source endpoint is given to the first step. The first step returns the path to the file it created as a result of its processing (this may be the same path as given originally to the step if it did not need to change the file or create a new file). This path is then passed to the next step, which again returns a path to its result file, which is given to the next step, and so on.
      
   3. Execute SourceFileOperation
      Rename or move the source file.
      This is done before transferring the file to the destination, this means that possible errors in the renaming or moving that would cause the transfer to fail will happen as early as possible - before we actually try to transfer files onward.
   4. Put the file to destination endpoint
      If destination file already exists, depending on the parameter "DestinationFileExistsAction" either an exception is thrown, the destination file is overwritten or the source file is appended to the destination file.
      If the parameter "RenameDestinationFileDuringTransfer" is true, the file is first transferred with a temporary file name and afterwards renamed to intended filename, otherwise the file is transferred with the intended filename. The intended filename has its possible file masks expanded.
      Note: When "DestinationFileExistsAction" is set to Error, there is a small chance that the file with the same name will be overwritten. This will only happen if a file with the same name is written by another process after the Cobalt workflow checks for the file's existence.
   5. Delete source file
      Delete the source file if the parameter "SourceOperation" is set to Delete.
4. Finish
   Close the source and destination endpoint connections.

If the transfer is cancelled (e.g. by calling Terminate on the routine instance), the files that are currently being transferred will be processed until finished, but no new files will be transferred. The task instance will finish once all file transfers that were processing at the time of cancellation have finished. The cancelled transfer end result will be Failed.

Error handling

If the opening of the connection to the source endpoint fails, or the listing of the source files fails, the transfer is immediately aborted and an error is logged to the event log.
Note: This error will not be logged to the database.

If the source connection is opened correctly, but the transfer of a single file fails, the workflow will try to continue from the next file until all the files have been processed.
Note: This means there may be many redundant tries to move files if e.g. the user does not have rights to write to the destination. There will be a separate error for each attempt.
On connection errors FTP(S) and SFTP endpoints will try to reopen the connection and retry the failed operation once per operation.

Transfer log database

FRENDS Cobalt Log AddIn is an add-in for FRENDS Log service that logs transferred file batches (details of single FRENDS Cobalt execution) and details of each individual transferred file into FRENDSCobaltTransferLog database. File transfer data stored in FRENDSCobaltTransferLog is used for generating the transfer view and transfer list in FRENDS Iron's web user interface. Log message for executed transfers will be sent to service bus from where it will be processed by FRENDS Log service and log it into a database. File transfer data persisted in database will be used for creating the transfer view shown in FRENDS Iron web user interface.

In addition to logging simply transfers and their details, file contents can be logged and stored into a Windows Azure Table Storage and Blob Storage using the FRENDS Bulk Logger add-in. Files under the size of 32 kilobytes are stored in Azure Table Storage and larger files into Azure Blob Storage. If bulk logger add-in is installed and configured, enabling it for FRENDS Cobalt transfers requires only to select one of the transfer's MessageProcessingSteps to be set to BulkLog.

Extending FRENDS Cobalt

While in most cases the core functionality FRENDS Cobalt provides for file transfer is sufficient, you may have cases which require more functionality. If, for example, the file conversion options are not sufficient enough, you could create a custom MessageProcessingStep for handling the required conversion for transferred files. You may also want to use a completely different protocol for transferring files than what are available in FRENDS Cobalt by default. Support for different protocols and other custom endpoint behavior can be achieved by creating a custom endpoint. Information on how to extend FRENDS Cobalt and some samples can be found Extending FRENDS Cobalt.

Using Cobalt task Output

The FRENDS Iron service provides support for passing data to sequential tasks, Cobalt supports this by providing information about the task execution(e.g. transferred files and notification messages), which can be then utilized by the next tasks. For example a task which sends a list of transferred files via email is possible.

The data provided as the task output is in XML-format in the PreviousTaskResult structure which is described in the FRENDS Iron documentation under Sharing information between tasks. Please note that since the PreviousTaskResult is also XML and the output element is string type, the CobaltOutput is escaped. The CobaltOutput XML conforms to the following schema:

<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xs="http://www.w3.org/2001/XMLSchema" attributeFormDefault="unqualified" elementFormDefault="qualified">
  <xsd:simpleType name="notificationType">
    <xs:restriction base="xs:string">
      <xs:enumeration value="Info"/>
      <xs:enumeration value="Error"/>
    </xs:restriction>
  </xsd:simpleType>
  <xsd:element name="CobaltOutput">
    <xsd:complexType>
      <xsd:sequence>
        <xsd:element name="Notifications">
          <xsd:complexType>
            <xsd:sequence>
              <xsd:element maxOccurs="unbounded" name="Notification">
                <xsd:complexType>
                  <xsd:sequence>
                    <xsd:element name="Type" type="notificationType" />
                    <xsd:element name="BatchId" type="xsd:string"/>
                    <xsd:element name="Message" type="xsd:string"/>
                  </xsd:sequence>  
                </xsd:complexType>
              </xsd:element>
            </xsd:sequence>
          </xsd:complexType>
        </xsd:element>
        <xsd:element name="Batch">
          <xsd:complexType>
            <xsd:sequence>
              <xsd:element name="Id" type="xsd:string" />
              <xsd:element name="UserResultMessage" type="xsd:string" />
              <xsd:element name="Started" type="xsd:dateTime" />
              <xsd:element name="Ended" type="xsd:dateTime" />
              <xsd:element name="Success" type="xsd:boolean" />
              <xsd:element name="ActionSkipped" type="xsd:boolean" />
              <xsd:element name="SourceAddress" type="xsd:string" />
              <xsd:element name="SourceType" type="xsd:string" />
              <xsd:element name="DestinationAddress" type="xsd:string" />
              <xsd:element name="DestinationType" type="xsd:string" />
              <xsd:element name="SourcePort" type="xsd:unsignedByte" />
              <xsd:element name="SourcePath" type="xsd:string" />
              <xsd:element name="DestinationPath" type="xsd:string" />
              <xsd:element name="DestinationPort" type="xsd:unsignedByte" />
              <xsd:element name="RoutineId" type="xsd:unsignedByte" />
              <xsd:element name="ConnectionPointId" type="xsd:unsignedByte" />
              <xsd:element name="TransferName" type="xsd:string" />
              <xsd:element name="TransferGroupName" type="xsd:string" />
              <xsd:element name="FileTransfers">
                <xsd:complexType>
                  <xsd:sequence>
                    <xsd:element maxOccurs="unbounded" name="FileTransfer">
                      <xsd:complexType>
                        <xsd:sequence>
                          <xsd:element name="BatchId" type="xsd:string" />
                          <xsd:element name="SourceFile" type="xsd:string" />
                          <xsd:element name="TransferStarted" type="xsd:dateTime" />
                          <xsd:element name="TransferEnded" type="xsd:dateTime" />
                          <xsd:element name="DestinationFile" type="xsd:string" />
                          <xsd:element name="FileSize" type="xsd:unsignedInt" />
                          <xsd:element name="Result" type="xsd:string" />
                          <xsd:element name="SourcePath" type="xsd:string" />
                          <xsd:element name="DestinationPath" type="xsd:string" />
                        </xsd:sequence>
                      </xsd:complexType>
                    </xsd:element>
                  </xsd:sequence>
                </xsd:complexType>
              </xsd:element>
            </xsd:sequence>
          </xsd:complexType>
        </xsd:element>
      </xsd:sequence>
    </xsd:complexType>
  </xsd:element>
</xs:schema>

Example CobaltOutput XML:

<?xml version="1.0" encoding="utf-16"?> <CobaltOutput xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Notifications />
  <Batch>
    <Id>bf9b8098-5f0b-4a41-a6a5-5ae22a2496d0</Id>
    <UserResultMessage>2 files transferred: priceChanges.xml, invoices.xml</UserResultMessage>
    <Started>2013-09-25T11:35:04.7665631Z</Started>
    <Ended>2013-09-25T11:35:04.8516014Z</Ended>
    <Success>true</Success>
    <ActionSkipped>false</ActionSkipped>
    <SourceAddress>127.0.0.1</SourceAddress>
    <SourceType>File</SourceType>
    <DestinationAddress>127.0.0.1</DestinationAddress>
    <DestinationType>Ftp</DestinationType>
    <SourcePort>21</SourcePort>
    <SourcePath>C:\Transfers\Out</SourcePath>
    <DestinationPath>/Out</DestinationPath>
    <DestinationPort>21</DestinationPort>
    <RoutineId>12</RoutineId>
    <ConnectionPointId>10</ConnectionPointId>
    <TransferName>Invoices and price changes</TransferName>
    <TransferGroupName>Biz</TransferGroupName>
    <FileTransfers>
      <FileTransfer>
        <BatchId>bf9b8098-5f0b-4a41-a6a5-5ae22a2496d0</BatchId>
        <SourceFile>priceChanges.xml</SourceFile>
        <TransferStarted>2013-09-25T11:35:04.8265732Z</TransferStarted>
        <TransferEnded>2013-09-25T11:35:04.8476276Z</TransferEnded>
        <DestinationFile>priceChanges.xml</DestinationFile>
        <FileSize>4112</FileSize>
        <Result>Success</Result>
        <SourcePath>C:\Transfers\Out\</SourcePath>
        <DestinationPath>/Out</DestinationPath>
      </FileTransfer>
      <FileTransfer>
        <BatchId>bf9b8098-5f0b-4a41-a6a5-5ae22a2496d0</BatchId>
        <SourceFile>invoices.xml</SourceFile>
        <TransferStarted>2013-09-25T11:35:04.7875668Z</TransferStarted>
        <TransferEnded>2013-09-25T11:35:04.8245696Z</TransferEnded>
        <DestinationFile>invoices.xml</DestinationFile>
        <FileSize>20570</FileSize>
        <Result>Success</Result>
        <SourcePath>C:\Transfers\Out\</SourcePath>
        <DestinationPath>/Out</DestinationPath>
      </FileTransfer>
    </FileTransfers>
  </Batch>
</CobaltOutput>

TRANSFER PARAMETER REFERENCE

There are two places where transfer parameters can be set:

• Technical transfer parameters, like workdir location, given as routine parameters
• The transfer specific parameters, like source and destination details, given as connection point data

Cobalt task transfer parameters

The Cobalt task takes four parameters: the transfer configuration as XML, and a set of optional technical parameters for the transfer:

ConfigXml

The transfer configuration as XML string. It is recommended the configuration is given as a CP reference, i.e. {CP:/}

WorkDir (optional)

Folder that FRENDS Cobalt uses to store local working copies of files. If empty, the FRENDS Iron service user account's temporary directory is used. Note: If you define the WorkDir, make sure the FRENDS Iron Service user account has read, write and modify access to the directory.

TransferGroupName (optional)

Free-form name describing the transfer. It is shown in error and transfer log reports, and can be used to group many different transfers together in the reports

TransferLoggerName (optional)

Name for the log4net transfer logger, if one is used. For details on this, see section "Configure logging file transfers with log4net"

Transfer specific parameters (CP Schema)

Source

Type

[File | Ftp | Ftps | Sftp | ServiceBus | Custom]
The protocol type to use. The type specific configuration is defined in the sections below

Directory

Directory on server where the transferred files are located. You can also use relative paths, in which case the path will be resolved relative to the initial working directory. For FTP/SFTP, this directory is the user's home directory, and relative paths should work OK. However, for File endpoints, it will be \Windows\System32\ by default. As this almost certainly is not correct, you must always use absolute paths for File endpoints.

Note: When using FTP transfer the FTP user must have read, write and modify permissions to source/destination folder. If local directory is used, read, write and modify permissions to the source/destination directory must be assigned to FrendsAccService user.

Macros can be also used for defining the directory. See Macro reference for details on how to use macros.

FileName

Name of the file(s) to be transferred. File masks can also be used here for transferring several files. The file mask uses regular expressions, but for convenience, it has special handling for '*' and '?' wildcards. See the table below for examples.

Mask	Matches	Does not match
*t.txt	text.txt, t.txt	foo.txt, text.foo
?t.txt	2t.txt	t.txt, tt.foo
test.(txt|xml)	test.txt, test.xml	test.foo, foo.txt
test.[^t][^x][^t]	test.xml, test.foo	test.txt, test.tii
test\d{4,4}.txt	test1234.txt, test6789.txt	test123.txt
<regex>^(?!prof).*_test.txt	pro_test.txt, pref_test.txt, _test.txt	prof_test.txt, pro_tet.txt

Note: Because regular expressions are used, you need to escape special characters such as parentheses in your filenames with a backslash(\).
E.g. Filename: "customer\(123\)_Q1.xml" to only match the file: customer(123)_Q1.xml.

By default, the regular expressions are compiled as case-insensitive, which means that a regular expression: file\d{6,6}env.dta would match both filenames file123456env.dta and FILE445566ENV.DTA. If you want to use pure regular expressions for matching, like if you want your regular expression to be compiled as case sensitive or you want to avoid the special handlings for '*' and '?' characters you have to use <regex> keyword at the beginning of your regular expression. If, for example, you want to make the previous example to match only the uppercase filenames, you would use a regular expression like: <regex>(?-i)(FILE\d{6,6}ENV.DTA).

Server

Common server parameters for the transfer types that fetch files from a remote server.

Address

Address of the server.

Port

Port number for server.

Username

Username to logon to server. NOTE: File endpoint only supports username for remote shares and needs that the username must be in the format DOMAIN\Username.

Password

Password for user on the server.

Ftp

FTP specific settings

TransportType

[Ascii | Binary]

• Ascii mode sends files via FTP as 'text' and should *only* be used for transferring ASCII text files.
• Binary modes send files via FTP as raw data and should be used for transferring non-text files. (default)

Mode

[Passive | Active]
FTP connection mode.

Sftp

SFTP specific settings

LoginType

[Username-Password | Username-PrivateKey | Username-Password-PrivateKey]

• Username-Password = the user name and password given in the Server section is used for authentication
• Username-PrivateKey = the user name (given in the Server section) and the private key file (given in this section) are used for authentication.
• Username-Password-PrivateKey = the user name and password as well as the private key is used for authentication. This may not be very commonly supported. GlobalScape SFTP server for example does support it, but use it at your own risk.

PrivateKeyFileName

Location of private key file to be used for authentication if public key authentication is used on the server.

PrivateKeyFilePassword

Password for private key file.

ServerFingerPrint

Fingerprint for SFTP server. When using Username-Password authentication it is recommended to use server fingerprint in order to be sure of the server you are connecting.

Ftps

FTPS specific settings

TransportType

[Ascii | Binary]

• Ascii mode sends files via FTP as 'text' and should *only* be used for transferring ASCII text files.
• Binary modes send files via FTP as raw data and should be used for transferring non-text files. (default)

Mode

[Passive | Active]
FTP connection mode.

SecureDataChannel

[true | false]
Is protection of transferred data required.

SslMode

[Explicit | Implicit]

• Explicit = a separate request is made for establishing secure connection. Default FTP port is used for secure connection.
• Implicit = secure connection is established as soon as the connection to a server is made, but a different port (990) is used for implicit security.

EnableClientAuth

[true | false]
If client authentication is enabled both client and server are authenticated. The client certificate is searched from user's certificate store.

ServerName

Required if client authentication is enabled. The ServerName must be exactly the same as in server certificate.

Timeout for FTP operations is 60 seconds. This means that data transfer is timed out after 60 seconds of inactivity and not from the start of the transfer.

ServiceBus

Service Bus specific settings. Service bus endpoint can be used for sending and retrieving messages. Due to the limitations of service bus the maximum supported file size for sending is limited to 250KB (Service bus supports message sizes up to 256KB). Endpoint can be used to push many and to retrieve one message per execution. MaxConcurrentConnections should be set to 1 as the same connection is shared by all threads (increasing the value will just cause overhead). The endpoint sends and receives the message body serialized as a Stream.

ConnectionString

Connection string for Azure Service Bus or Service Bus for Windows Server instance to use for messaging.

Path

Path of the entity (queue, topic or subscription) to send or receive messages to or from. When sending messages, give the name of the queue or topic. When receiving messages, give the queue name, or if the entity is a subscription, the entire path to it, in the format [topic name]/subscriptions/[subscription name]

Things to note:

• The topics, queues or subscriptions will not be created automatically. If they do not exist, an error will be thrown and the transfer will fail.
• For source endpoints, only a single message will be received for a transfer, i.e. batch receive is not supported. The message will also be completed immediately after receive, even before it is written to disk to a temporary file. Therefore if you need guarantees that the messages are processed, you will need to create a custom endpoint that e.g. persists the messages being processed, or otherwise handles possible system crashes etc.

Custom

You can provide your own custom endpoint implementation as a compiled assembly, and pass initialization parameters to it via this section. For more details on how to do this, see the Extending FRENDS Cobalt guide.

NOTE: The implementation must extend the EndPointBase base class, and provide the necessary constructors in order to work.

AssemblyName

Assembly name where the custom class resides in, e.g. Frends.Cobalt.Extensions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null

ClassName

Class name of custom class, e.g. Frends.Cobalt.Extensions.FileLogger

Parameters

Custom parameter string to be passed as is to the custom implementation constructor. This parameter can be e.g. a connection string or any other custom parameters

Destination

Destination parameter section contains same parameters as in Source section, so same descriptions generally apply to them too with exception in FileName parameter.

FileName

File name or file mask for the remote file. '?' characters are not allowed in file mask. If no file name is given the original file name is used for destination file name.

Examples of usage:

• No value = use the source file name
• *.xml = rename files with xml extension
• *temp = append 'temp' to the source file name
• new_* renamed file will start with string new_

Macros can also be used for destination file names. See Macro reference for details on how to use macros.

Parameters

General parameters for the transfer

MaxConcurrentTransfers

The maximum number of concurrent transfers, by default 1.

If set to more than 1, the given number of source and destination connections are opened, and the files transferred using the opened source-destination connection pairs.

RenameSourceFileBeforeTransfer

[true | false]
Should the source file be renamed with a temporary file name before transferring the file. This is a way to prevent other users to access the file with its original name while transfer is in progress. Set this to false if the used source endpoint implementation doesn't support source file renaming. Default value is true.

The temporary file name is created using the following scheme: cobalt_[current time in ticks][random 8 character file name].8CO
For example: cobalt_634509139954167644lomy3uzg.8CO

RenameDestinationFileDuringTransfer

[true | false]
Should the file be transferred to destination endpoint with a temporary file name and rename it after the transfer is completed. This is a way to prevent other users to access the file at the destination endpoint with its actual name before the transfer is completed. Set this to false if the used destination endpoint implementation doesn't support file renaming. Default value is true.

The temporary file name is created using the same scheme as for the source file: cobalt_[current time in ticks][random 8 character file name].8CO

CreateDestinationDirectories

[true | false]
Should directories be created in transfer destination if they do not exist already. Default value is false.

DestinationFileExistsAction

[Append | Overwrite | Error]
What to do if destination file exists. By default throws an error.

• Error = Throw an error and fail the transfer.(default)
• Append = Append to an existing file.
• Overwrite = Overwrite existing file.

NoSourceAction

[Error | Info | Ignore]
What to do if the source file is not found. By default throws an error.

NOTE: This error is not logged by log4net transfer logger, neither is it written in the file transfer log database

• Error = Throw an error and fail the transfer. (default)
• Info = Alarm info and quit with success status.
• Ignore = Quit execution with success status.

SourceOperation

What to do to the source file after a successful transfer

Operation

[Delete | Rename | Move | Nothing]
Defines the operation to perform to source file(s) after successful transfer. By default deletes the file from the source directory.

• Delete = Delete the source file(s).(default)
• Rename = Renames the source file(s) to use the given name. The name (given in the To parameter) can use file macros (and file masks, although they are not recommended). From 1.6 onwards, you can also move the renamed files to another directory, by giving a directory path.
  Note: This may overwrite files.
• Move = Moves the source file(s) to different directory (given in the To parameter) using the original file names.
  Note: This may overwrite files.
• Nothing = Do nothing, leaves the file to the source directory.

To

The path to move or to rename the files to.

• For Rename, you can use file macros and also specify a directory where to move the files to, e.g. /subdir/%Date%file.txt. If you don't define a directory path, the source directory is used. When using rename, the To-parameter must always contain a file name.
  Note: The possible directory must exist before the transfer is executed, because it is not created automatically.
• For Move, you must give a directory name. You cannot use file macros, or define file names in the parameter.
  Note: The possible directory must exist before the transfer is executed, because it is not created automatically.

Message processing steps

You can define steps for processing messages after they have been fetched from the source endpoint but before they are sent to the destination endpoint.

The default Cobalt CP schema has three steps you can use. However, you can have as many steps you like, if you just create a custom CP schema.

Type

[PassThrough | LocalBackup | FileSystemBackup | Xslt | Encoding | Custom]
The message processing step type. By default PassThrough.

• PassThrough = No conversion is done.
• LocalBackup = Save a local copy of the transferred file and optionally clean up old backups.
• FileSystemBackup = Back up transferred file on a local file system and clean up old backups.
• Xslt = Convert the files with an XSL transformation e.g. to a text file, HTML file or another XML format.
• Encoding = Changes the encoding of the file.
• Custom = Use a custom message processing step implementation.

LocalBackup

Parameters for the backup step that takes a local backup of the transferred files.

BackupDirectory

Root directory for the backups. Must be an absolute local path or network share, like "E:\Backup\" or "\\store\backup".
For each transfer, a new backup directory named after the file transfer start time and the batch ID is created underneath the root directory. Copies of all files transferred in the single transfer will be put to this transfer specific directory.

Cleanup/DaysOlder

Number of days to keep the backups. Backup subdirectories older than this are deleted. Dates are compared by the last write time of the directory.

Cleanup/Enabled

[true|false]
Whether or not the automatic cleanup is enabled. If false (default), the files will not be cleaned up by the step. Note that if the automatic cleanup is disabled, you must clean up the files in some other way.

FileSystemBackup

Parameters for the new backup step that takes a local file system backup of the transferred files. Basically performs the same functionality as LocalBackup step, but with more control over the backup directory and backup file naming.

Destination

Destination must be an absolute local or network path ("E:\Backup\" or "\\store\backup\). Macros can be used to generate the backup directory or/and filename, e.g E:\Backup\%TransferId%\. This would generate a subfolder with transfer id (e.g. '8E9C5EC5-E838-4404-85BF-BC352BDA4462') under 'E:\Backup\' with all the transfered files inside. If a file already exists the existing file will be overwritten.

Cleanup/DaysOlder

Number of days to keep the backups. Backup files and subdirectories are deleted from the base path of the Destination (path without macros). The base path for cleanup for 'E:\Backup\%TransferId%\' would be 'E:\Backup\'. Dates are compared by the last write time for subdirectories. For files the dates are compared by the file creation time.

Cleanup/Enabled

[true|false]
Whether or not the automatic cleanup is enabled. If false (default), the files will not be cleaned up by the step. Note that if the automatic cleanup is disabled, you must clean up the files in some other way.

Xslt

XSLT processing step parameters

XslFileLocation

Absolute path to the XSL transformation file

Encoding

Encoding step parameters

SourceEncoding

Source file character encoding from which to convert.

DestinationEncoding

Destination file character encoding to which to convert.

Custom

You can provide your own custom message processing step implementation as a compiled assembly, and pass initialization parameters to it via this section. For more details on how to do this, see the Extending FRENDS Cobalt guide.

AssemblyName

Assembly name where the custom message processing step implementation class resides in, e.g. Frends.Cobalt.Extensions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null

ClassName

Class name of custom class, e.g. Frends.Cobalt.Extensions.FileLogger
NOTE: The implementation must extend the MessageProcessingStepBase base class, and provide the necessary constructors in order to work.

Parameters

Custom parameter string to be passed as is to the custom implementation constructor. This parameter can be e.g. a connection string or any other custom parameters

Other parameters

TransferName

Can be used to give a descriptive name to a single transfer.

Macro reference

Macros can be used to dynamically configure source directory, destination directory or destination file name for a file transfer.

Generally the following rules apply for macros:

• Macros are case insensitive.
• You can use any number of macros in all of the cases.
• Dates and times are formatted with leading zeros.

The following macros can be used with all of dynamically configurable locations for file transfer:

• %Ticks% = will be replace with the current time as Ticks.
• %DateTime% = will be replaced with date and time in format: "yyyy-MM-dd-HH-mm-ss".
• %DateTimeMs% = will be replace with date and time in format: "yyyy-MM-dd-HH-ss-fff".
• %Date% = will be replaced with date in format: "yyyy-MM-dd".
• %Time% = will be replaced with time in format: "HH-mm-ss".
• %Year% = will be replaced with current year.
• %Month% = will be replaced with current month.
• %Day% = will be replaced with current day.
• %Hour% = will be replaced with current hour.
• %Minute% = will be replaced with current minute.
• %Second% = will be replaced with current second.
• %Millisecond% = will be replaced with current millisecond.
• %WeekDay% = will be replaced with a number of weekday, ranging from 1 (monday) to 7 (sunday).
• %Guid% = will be replaced with a new unique identifier.
• %TransferId% = will be replaced with the transfer id.
• %TransferName% = will be replaced with TransferName parameter specified in Connection point schema.
• %TransferGroupName% = will be replaced with TransferGroupName parameter specified in routine's task arguments.

• %SourceFileName% = will be replaced with source file name without extension.
• %SourceFileExtension% = will be replaced with source file's extension, with the dot '.' included, i.e. if the source file is named 'foo.txt', the %SourceFileExtension% will be expanded as '.txt'. If the source file name does not have an extension, the macro result will be empty, i.e. for original file name "foo", "bar%SourceFileExtension%" will result in "bar"

Note:With destination file name it is recommended to only use file macros, since in future the usage of file masks for such purpose will be deprecated. Especially, if you want to add a time stamp to a transferred file name, you can only do this with file macros.

Below are some examples with imaginary sample.txt file:

Not recommended:

• *%Date% -> sample.txt2008-08-26
• %Date%* -> 2008-08-26sample.txt

Recommended:

• %SourceFileName%%Date%%SourceFileExtension% -> sample2008-08-26.txt

Known issues

PersistenceException blocks stopping FRENDS Iron service when Cobalt transfer is running

If the Cobalt workflow is running when the FRENDS Iron service is being stopped, the workflow runtime tries to pause and persist its state to the WorkflowStore database. Unfortunately, due to the way the Cobalt workflow is structured, it cannot be serialized and therefore persisted. This causes the System.Workflow.Runtime.Hosting.PersistenceException to be thrown, which also blocks stopping the workflow runtime and the FRENDS Iron service. On the other hand, the Cobalt transfer would not be able to resume the transfer anyways, so it is good to allow the transfer to finish.

Therefore, you should wait until all current Cobalt transfers have been completed before stopping the FRENDS Iron service.

Source file operation may overwrite existing files on FTP

When renaming or moving source the source files with SourceOperation with FTP, you should note that existing files may get overwritten. At least Microsoft's IIS FTP Server overwrites existing files renaming/moving files, this may happen on other servers as well.

If you want to store the original source files somewhere, and do not want to use the backup operation, it is recommended to append e.g. the datetime to the moved or renamed file name by using a file name like %DateTime%*

Failure in source file operation logs the transfer failed even if the files were transferred

SourceFileOperations Delete or Nothing are executed only after the actual file transfer has been completed. There is a possibility that they will fail (e.g. another file with the original source file name has been created in the directory, so renaming the temporary file back to the original name (option Nothing) will fail). This error will be logged as a failed file transfer, even though the file has been transferred to the destination. In this case, the logged error description will mention that only the source operation has failed and the file has been transferred.

If you get these kinds of errors, you should fix the original error, e.g. not writing a new file to the source directory with always the same name

Failures with fetching source files are not logged to transfer log

The transfer log only logs individual file transfers that have succeeded or failed. If fetching the source files fails, e.g. because the server cannot be accessed, or there are no files and the SourceFileNotFoundAction is set to Error, the transfer will fail, but nothing will be logged to the transfer log.

This issue will be fixed in future versions of FRENDS Cobalt.

Cobalt 1.7 custom message processing steps or custom endpoints don't work with Cobalt 1.8

While Cobalt 1.8 is technically backwards compatible with Cobalt 1.7, message processing steps or custom endpoints built against Cobalt 1.7 cannot be used by Cobalt 1.8, because the Cobalt reference is strong-named.

There are three ways to fix or work around this issue:

• It is recommended to rebuild any custom functionality assemblies updating the Cobalt reference with version 1.8.
• If rebuilding the custom functionality assemblies is not possible, you can force the runtime to use Cobalt 1.8 where Cobalt 1.7 is referenced. This can be done by modifying the FRENDSAccService.exe.config and adding the following configuration block into the file.

  <runtime>	
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
     <dependentAssembly>
        <assemblyIdentity name="Frends.Cobalt"
                          publicKeyToken="7d5d218da670e3aa"
                          culture="neutral" />
          <bindingRedirect oldVersion="1.7.0.0-1.7.9999.9999" newVersion="1.8.0.0" />
      </dependentAssembly>
    </assemblyBinding>
  <runtime>  
  	

  FRENDS Iron service needs to be restarted in order to configuration changes to take effect.
• You can still run Cobalt 1.7 and use the old custom functionality assemblies with it. In this case, you need to install the Cobalt 1.7 assembly to the GAC.