FTP Poll Directory activity

During run time, the FTP Poll Directory activity connects to FTP Server and polls for the existence of the specified file or files at the specified location. After reading the contents of the file or files and mapping the contents into variable or variables of the orchestration, the FTP Poll Directory activity may delete the file or files, move the file or files, or leave the file or files in the directory of the FTP Server.

The FTP Poll Directory activity is a starter activity. When an orchestration that starts with an FTP Poll Directory activity is deployed to the Integration Appliance, the Integration Appliance polls for file or files to be added to a specific directory on the FTP Server. When a file or files is shown on the FTP Server, an orchestration job is started to processes the file or files and the rest of the activities in the orchestration. For more information, see Polling Interval Behavior. For an example of an orchestration using the FTP Poll Directory activity, see the Getting Started Guide.

Depending on your application requirements, you can process a list of files in an FTP directory using one of the following methods:
  • For some cases, you can use only the FTP Poll Directory activity to process a set of files found in an FTP directory.
  • In other cases, you must use the FTP List Files activity in conjunction with the FTP Get File, FTP Put File, FTP Delete File, and FTP Rename File activities.
Use the FTP Poll Directory activity to process a set of files found in an FTP directory when the following conditions are true:
  • The FTP Poll Directory activity supports the format of the directory listing of your FTP Server. - The format of directory listings are different for the FTP Servers supported on different operating systems. The FTP Poll Directory activity supports many types of FTP Servers but if the directory listing format your FTP Server cannot be parsed by the FTP Poll Directory activity, use the FTP List Files activity with a Flat File Schema and the FTP Get File, FTP Put File, FTP Delete File, and FTP Rename File activities, to provide the same functionality as the FTP Poll Directory activity as shown in the Parse Directory Listing Example: Parse directory listing example.
  • After reading each file in the list, each file in the list can be post-processed with the same action. - For example, while processing a set of files from a FTP directory, if each file in the list can be moved to the same directory, you can use the FTP Poll Directory activity to process the set of files. If however, you need to process a different action based on the properties of a file, you must use the FTP List Files activity in conjunction with the FTP Get File, FTP Put File, FTP Delete File, and FTP Rename File activities. For example, if you must move files to different directories based on the filename, use the FTP List Files activity in conjunction with the FTP Get File, FTP Put File, FTP Delete File, and FTP Rename File activities as shown in the Parse Directory Listing Example: Parse directory listing example.

Symbolic links

The support for symbolic links by the FTP Poll Directory activity during run time is dependent on the following factors:  
  • The types of symbolic links supported by operating system that hosts the FTP Server.
  • The types of symbolic links supported by the FTP Server itself.
  • The types of symbolic links supported by the FTP Poll Directory activity, as listed in the following table:
    Table 1.
    Symbolic Links to... Hard Symbolic Links Soft Symbolic Links Notes®
    Files Not Supported Not Supported The FTP Poll Directory activity does not interpret symbolic links to files as files and therefore symbolic links to files are not processed by the FTP Poll Directory activity during run time.
    Directories Supported Supported  
To determine the specific symbolic link support for a specific FTP Server on a specific operating system, see your operating system and FTP Server documentation.

Persistence

You should enable persistence for the FTP Poll Directory activity:

  • If you have an orchestration job with a FTP Poll Directory activity that is processing large input files. Otherwise, you can experience problems like "Variable 'Lotdata' is too large, please turn on persistence for this orchestration".
  • If you disable persistence and have an orchestration job with a FTP Poll Directory activity with the Keep in place option selected, and during run time if a failure occurs that causes the job to stop, when the orchestration is restarted it processes the file again. The state of the job is lost and Integration Appliance does not know that it has already processed the file.

For more information, see Enabling persistence.

Processing subdirectories with the Microsoft FTP Endpoint

For the Microsoft IIS (Internet Information Services) FTP Endpoint, the run time behavior of the FTP Poll Directory activity when processing subdirectories and the files in subdirectories depends on the regular expression specified in the File Named field of the Configure task of the activity, as described in the following cases:
  • When the regular expression is equal to the string: *, the orchestration job processes the subdirectories in the specified directory.
  • When the regular expression is equal to the string: *.*, the orchestration job does not process the subdirectories in the specified directory.
  • When the regular expression is blank, the orchestration job does not process the subdirectories in the specified directory
.

Adding an FTP Poll Directory activity to the orchestration

  1. Create or open an orchestration. A graphical representation of the orchestration is displayed.
  2. Select the Activities tab and expand the FTP folder.
  3. Drag the Poll Directory activity onto the orchestration.
  4. Select the activity. The Checklist is displayed.

Creating, selecting, or editing an FTP endpoint

  1. Click Pick Endpoint task from the Checklist and choose from one of the following actions:
    • Select an existing endpoint:
      1. Select Browse. The Project Explorer is displayed.
        CAUTION:
        Specifying a new FTP endpoint with a different directory structure than the original FTP endpoint can cause the orchestration job to fail during run time. You must either specify a new FTP Server that matches the directory structure of the originally specified FTP Server or change the configured directory of the In Directory field by clicking Browse as described in the "Configure the activity" procedure.
      2. Select an existing endpoint and click OK. Skip to the "Configure the activity" procedure.
    • Select an existing endpoint and edit the endpoint:
      1. Select Browse. The Project Explorer is displayed.
      2. Select an existing endpoint and click OK.
        CAUTION:
        Specifying a new FTP endpoint with a different directory structure than the original FTP Endpoint can cause the orchestration job to fail during run time. You must either specify a new FTP Server that matches the directory structure of the originally specified FTP Server or change the configured directory of the In Directory field by clicking Browse as described in the "Configure the activity" procedure. Click Edit. The Edit Endpoint pane is displayed.
    • Create a new endpoint: Select New. The Create Endpoint pane is displayed.
  2. Create or edit an FTP endpoint.

Configuring the activity

  1. Select the Configure task from the Checklist. The Configure pane is displayed.
  2. Configure all the fields in this pane as described in the following table:
    Table 2.
    Field Description
    File Named Specifies a pattern or regular expression that defines the filename or filenames to get from the FTP Server. The processing of the regular expression is done by the FTP Server. For more information, see the FTP Server documentation on regular expressions . Typically, FTP Servers support the following wildcards in their regular expressions:
    • * - zero or more characters
    • ? - zero or one character
    Note: Because the processing of the regular expression is done on the FTP Server, the case sensitivity is determined by FTP Server. If the FTP Server is case sensitive, the file matching done by the FTP Poll Directory activity is also case sensitive.
    In Directory Specifies the directory where the Integration Appliance looks for files on the FTP Server. You can specify a directory using one of the following options:
    • Directly enter directory path by clicking in the field and typing in one of the following directory paths:
      • Relative directory path, for example: input.  (Assumes that FTP home has been configured for the user account that connects to the FTP Server. See "Note" for more information.)
      • Complete directory path, for example on a UNIX machine: /home/user/input.
    • Click Browse... to populate the drop-down list of available directories. Select the appropriate directory from the drop-down list.
    Note: UNIX: If you specify a relative directory path when connecting to an FTP Server on a UNIX machine, verify that the FTP home directory has been correctly specified in the /etc/password file for the user account connecting to the FTP Server.
    Test After a regular expression has been specified in the File Named field and the directory to look for files on the FTP Server has been specified in the In Directory field, click Test to test the regular expression against the set of files in the specified directory on the FTP Server. A dialog box will appear listing any matching files.
    Of Type Specifies how to read in the contents of the file - either as binary data or encoded text. If the Text option is selected, set the encoding type using one of the following options:
    • From the encoded with drop-down list, select one of the default encoding types.  
    • Enter your encoding type directly by clicking in the encoded with field and typing in your encoding type.
    Note: Some double-byte characters are not converted using the SHIFT-JIS encoding. For more information, see Using the Shift-JIS encoding.
     
    CAUTION:
    You must specify the Transfer as Text (ASCII) option for FTP activities when retrieving text files and placing text files on an FTP Server for Endpoints that manipulate text data while transferring data from/to the FTP endpoint in ASCII mode. For example, mainframe and DOS FTP Endpoints manipulate ASCII data when passing data in ASCII mode. For example, if your orchestration retrieves ASCII files using the FTP Poll Directory activity and places ASCII files using the FTP Put File activity to a mainframe FTP Server, you must specify the options in the following tasks in Studio:
    • In the Configure task of an FTP Poll Directory activity, for the Of Type field select the Text option.
    • In the Delivery Rules task of an FTP Poll Directory activity, for the File Transfer Type field select the Transfer as Text (ASCII) option.
    • In the Configure task of an FTP Put File activity, for the Of Type field select the Text option.
    • In the Delivery Rules task of an FTP Put File activity, for the File Transfer Type field select the Transfer as Text (ASCII) option.
    After reading the file

    Delete it - Delete the file after the file has been read.

    Move to directory - Move the file to the specified directory after the file has been read.

    Specify the directory that the file is moved to during run time, using one of the following options:
    • Directly enter a directory path by clicking in the field and typing in one of the following directory paths:
      • Relative directory path, for example: input/.  (Assumes that FTP home has been configured for the user account that connects to the FTP Server. See "Note" for more information.)
      • Complete directory path, for example on a UNIX machine: /home/user/input/.
    • Click Browse... to populate the drop-down list of available directories. Select the appropriate directory from the drop-down list.
     
    Attention: Specify the directory separator after the last listed directory name. For example, to move a file to the processed/acme directory in a relative path, specify the following directory in the Move to directory field: processed/acme/
    Optional - Specify the format of the file name after it has been renamed. From the Pattern drop-down list, select the desired pattern. To specify the character used to separate the sub-fields in the file name, enter a character in the Pattern Separator field. Specify the run time behavior when a file with the same file name exists:
    • Select the Overwrite check box to specify that the existing file should be overwritten if a file with the same name already exists.
    • Clear the Overwrite check box to specify that the existing file should not be overwritten if a file with the same name already exists.
    Note: The first time the FTP Poll Directory activity polls during run time, if the activity is unable to move the file because the specified move to directory does not exist, a warning is logged (not an error) to the Web Management Console (WMC) and the file stays in place. For all subsequent polls, the activity does not attempt to move the same file and an info level message is logged to the WMC stating that the file has already been processed.

    Keep in place - Leave the file in the input directory after the file has been read.

    Advanced Options Description
    Duplicate List Size Size of the internal list used by the Integration Appliance to keep track of which files on the FTP Server have been polled or not. The default value is 10000. You should only specify a size larger than the default.

Specifying the delivery rules

  1. Select the Delivery Rules task from the Checklist. The Delivery Rules pane is displayed.
  2. Specify the time interval to poll for the files on the FTP Server:
    1. Select a time unit from the drop-down menu: seconds, minutes, hours, or days.
    2. Specify the amount of time by clicking in the Polling Interval field and entering a value or using the up and down arrows.
    For more information, see Polling interval behavior.
  3. If security is not enabled for the FTP endpoint, the Transfer as Text (ASCII) and Transfer as Binary (BINARY) options are enabled in the Delivery Rules task. If security is enabled for the FTP Endpoint, the file transfer type is always binary.
    Select the type of file transfer between the Integration Appliance and the FTP Server from one of the following options:
    • Transfer as Text (ASCII) - Only seven bits of each character in the file is transferred from the FTP Server to the Integration Appliance. This option is used for transferring ASCII files such as HTML and text files. WARNING: Using this option with binary files can result in a loss of data.
    • Transfer as Binary (BINARY) - All eight bits of each byte in the file is transferred from the FTP Server to the Integration Appliance.  This option is used for transferring binary files.
      Note: Setting the transfer mode in the Delivery Rules task is equivalent to setting the transfer mode via a command line session with an FTP Server using the following commands:
      • ascii
      • binary
    Note: App Connect FTP connector SFTP endpoint supports only client side SFTP protocol version 3. Hence, the SFTP protocol does not support ASCII/text mode file transfer. Make sure to set the File Transfer Type to Transfer as Binary (BINARY) in the Checklist > Delivery Rules.

Specifying the retry options

  1. Select the Retry task from the Checklist. The Retry pane is displayed.
  2. Configure the retry options for connecting to an FTP Server. The following table defines the retry options for a connection to an FTP Server:
    Table 3.
    Field Description
    Retry  
    1) Wait __ second(s) between each retry. The number of seconds that the Integration Appliance waits before trying to establish a connection to the FTP Server.
    2) Try to connect __ times before failing. Specifies the retry count - the maximum number of times the Integration Appliance attempts to establish a connection to the FTP Server before issuing an error. If an orchestration that starts with an FTP Poll Directory activity is deployed and the Integration Appliance cannot connect to the specified FTP Server, the Integration Appliance logs the connection errors as warnings in the system log until the retry count value is reached. When the retry count is reached, the Integration Appliance logs an error in the system log, resets the connection error count to zero, and continues to attempt to establish a connection to the FTP Server.

    For example, you set the retry count to 3. The first, second, and third connection errors appear in the system log as warnings. The Integration Appliance logs the fourth connection error as an error and resets the connection error count to zero. Therefore, the fifth connection error generates a warning in the system log. The Integration Appliance continues to attempt to establish a connection to the FTP Server.

Mapping the output of the activity

  1. Select the Map Outputs task in the Checklist. The output parameters of the activity are displayed in the From Activity pane as listed in the following table:
    Table 4.
    Output Parameter Description
    filename Provides the name of the file found on the FTP Server.
    data Provides the contents of the file.
    timestamp Provides the date and time as a string in the XML Schema dataType format (YYYY-MM-DDhh:mm:ss, where YY is the year, MM is the month, DD is the day, hh is the timezone, mm is the minutes, and ss is the seconds.)
    Note: If the timestamp output parameter is mapped in the Map Outputs task but the MDTM (return the modification time of a file) functionality is not available or is not enabled for the FTP Server, during run time the orchestration will stop processing due to this error. A warning is generated and the Epoch date (1/1/1970) is returned in the timestamp output parameter.
  2. You are not required to map the output parameters of the activity. See Creating a map for general instructions on mapping.  



Feedback | Notices


Timestamp icon Last updated: Sunday, 09 September 2018


https://www.ibm.com/support/knowledgecenter/en/SS3LC4_7.5.3/com.ibm.wci.doc/ftp_poll_directory_activity.html