FTP List Files activity

During run time, the FTP List Files activity connects to an FTP Server and generates a list of files from the specified directory location using the specified regular expression. The FTP List Files activity returns the list of files to the orchestration using either the directoryListing or fileListing output parameter.

If the List Only Files (NLST) check box is cleared, a full directory listing including directory permissions, timestamps, and ownership is returned in the directoryListing output parameter as a single string. When a full directory listing is returned as a string, you must parse the directory listing using a Flat File Schema with the Read Flat File activity. If the List Only Files (NLST) check box is selected and the FTP List Files activity supports the format of the directory listing for your FTP Server, a repeating structure of just filenames (if supported by the FTP server) is returned in the fileListing output parameter. When a repeating structure of filename is returned, you do not need to parse the directory listing and can just iterate through the list using a For Each or While Loop activity. The FTP server you are connecting to must support the NLST command. The response provided by the FTP server is returned.

For some cases, you must use the FTP List Files activity in conjunction with the FTP Get File, FTP Put File, FTP Delete File, and FTP Rename File activities. to process a set of files found in an FTP directory, while in other cases you can just use the FTP Poll Directory activity.

Use the FTP List Files activity in conjunction with the FTP Get File, FTP Put File, FTP Delete File, and FTP Rename File activities, to process a set of files found in an FTP directory if any of the following conditions are true:
  • The FTP Poll Directory activity does not support the format of the directory listing of your FTP Server - The format of directory listings are different for the FTP Servers supported on different operating systems. The FTP Poll Directory activity supports many types of FTP Servers but if the directory listing format of your FTP Server cannot be parsed by the FTP Poll Directory activity, use the FTP List Files activity and the FTP Get File, FTP Put File, FTP Delete File, and FTP Rename File activities, to provide the same functionality as the FTP Poll Directory activity as shown in the Parse Directory Listing Example: Parse directory listing example.
  • Each file in the list must be post-processed with a different action. - For example, if you must move files to a different directories based on the filename, use the FTP List Files activity in conjunction with the FTP Get File, FTP Put File, FTP Delete File, and FTP Rename File activities as show in the Parse Directory Listing Example: Parse directory listing example. However, if each file can be post-processed using the same action, you can use the FTP Poll Directory activity to process the set of files.

Symbolic links

The support for symbolic links by the FTP List Files activity during run time is dependant on the following factors:  
  • The types of symbolic links supported by operating system that hosts the FTP Server.
  • The types of symbolic links supported by the FTP Server itself.
  • The types of symbolic links supported by the FTP List Files activity, as listed in the following table:
    Table 1.
    Symbolic Links to... Hard Symbolic Links Soft Symbolic Links Notes®
    Files Not Supported Not Supported The FTP List Files activity does not interpret symbolic links to files as files and therefore symbolic links to files are not processed by the FTP List Files activity during run time.
    Directories Supported Supported  
To determine the specific symbolic link support for a specific FTP Server on a specific operating system, see your operating system and FTP Server documentation.

Processing subdirectories with the Microsoft FTP Endpoint

For the Microsoft IIS (Internet Information Services) FTP Endpoint, the run time behavior of the FTP List Files activity when processing subdirectories and the files in subdirectories depends on the regular expression specified in the File Named field of the Configure task of the activity, as described in the following cases:
  • When the regular expression is equal to the string: *, the orchestration job processes the subdirectories in the specified directory.
  • When the regular expression is equal to the string: *.*, the orchestration job does not process the subdirectories in the specified directory.
  • When the regular expression is blank, the orchestration job does not process the subdirectories in the specified directory.

Adding an FTP List Files activity to the orchestration

  1. Create or open an orchestration. A graphical representation of the orchestration is displayed.
  2. Select the Activities tab and expand the FTP folder.
  3. Drag the List Files activity onto the orchestration.
  4. Select the activity. The Checklist is displayed.

Creating, selecting, or editing an FTP endpoint

  1. Click Pick Endpoint task from the Checklist and choose from one of the following actions:
    • Select an existing endpoint:
      1. Select Browse. The Project Explorer is displayed.
        Attention: Specifying a new FTP Endpoint with a different directory structure than the original FTP Endpoint can cause the orchestration job to fail during run time. You must either specify a new FTP Server that matches the directory structure of the originally specified FTP Server or change the configured directory of the In Directory field by clicking Browse as described below in the "Configuring the activity" procedure.
      2. Select an existing endpoint and click OK. Skip to the "Configuring the activity" procedure.
    • Select an existing endpoint and edit the endpoint:
      1. Select Browse. The Project Explorer is displayed.
      2. Select an existing endpoint and click OK.
        Attention: Specifying a new FTP Endpoint with a different directory structure than the original FTP Endpoint can cause the orchestration job to fail during run time. You must either specify a new FTP Server that matches the directory structure of the originally specified FTP Server or change the configured directory of the In Directory field by clicking Browse as described below in the "Configuring the activity" procedure.
      3. Click Edit. The Edit Endpoint pane is displayed.
    • Create a new endpoint: Select New. The Create Endpoint pane is displayed.
  2. Create or edit an FTP endpoint: Creating or Editing an FTP endpoint.

Configuring the activity

  1. Select the Configure task from the Checklist. The Configure pane is displayed.
  2. Configure all the fields in this pane as described in the following table:
    Table 2.
    Field Description
    In Directory Specifies the directory on the FTP server where the Integration Appliance generates a list of files in the directory. You can specify a directory using one of the following options:
    • Directly enter directory path by clicking in the field and typing in one of the following directory paths:
      • Relative directory path, for example: /process.  (Assumes that FTP home has been configured for the user account that connects to the FTP Server. See the note below for more information.)
      • Complete directory path, for example on a UNIX machine: /home/user/process.
    • Click Browse... to populate the drown-down list of available directories. Select the appropriate directory from the drop-down list.
    Note: UNIX: If you specify a relative directory path when connecting to an FTP Server on a UNIX machine, verify that the FTP home directory has been correctly specified in the /etc/password file for the user account connecting to the FTP Server.
    File Named Specifies a glob pattern that defines the file names you need to get from the FTP Server. The processing of the glob pattern is done by the FTP Server. For more information, see the FTP Server documentation on glob pattern.
    List Only Files (NLST) The List Only Files (NLST) check box determines the type of directory listing returned by the Test button and during run time. In addition, this check box defines if the fileListing or the directoryListing output parameter is displayed in the Map Outputs task and returned by the activity during run time.
      If the List Only Files (NLST) check box is selected, just the filenames (if supported by the FTP server) are returned when the Test button is clicked, as shown in Figure 1. In addition, when the List Only Files (NLST) check box is selected, the fileListing output parameter is listed in the Map Outputs task. During run time, the set of files are returned in the fileName repeating node of the fileListing output parameter. The FTP server you are connecting to must support the NLST command. The response provided by the FTP server is returned.
      If the List Only Files (NLST) check box is cleared, a full directory listing including directory permissions, timestamps, and ownership are returned when the Test button is clicked, as shown in Figure 2. In addition, when the List Only Files (NLST) check box is selected, the directoryListing output parameter is listed in the Map Outputs task. During run time, the full directory listing is provided as a single string and returned using the directoryListing output parameter.
    Figure 1. Example listing when List Only Files (NLST) is selected.
    bidAcmeInc.txt
    bidBudgetManufacture.txt
    bidCrankCo.txt
    bidSamAndSons.txt
    bidWidgetRUs.txt
    Figure 2. Example listing when List Only Files (NLST) is cleared.
    -rw-r--r--  1 551    551      65 May  3 11:39 bidAcmeInc.txt
    -rw-r--r--  1 551    551     43 May  3 11:39 bidBudgetManufacture.txt
    -rw-r--r--  1 551    551       0 May  3 11:39 bidCrankCo.txt
    -rw-r--r--  1 551    551      73 May  3 11:39 bidSamAndSons.txt
    -rw-r--r--  1 551    551      33 May  3 11:39 bidWidgetRUs.txt

Specifying the retry options

  1. Select the Retry task from the Checklist. The Retry pane is displayed.
  2. Configure the retry options for connecting to an FTP Server. The following table defines the retry options for a connection to an FTP Server:
    Table 3.
    Field Description
    Retry  
    1) Wait ___ second(s) between each retry. The number of seconds that the Integration Appliance waits before trying to establish a connection to the FTP Server.
    2) Try to connect ___ times before failing. Specifies the retry count - the maximum number of times the Integration Appliance attempts to establish a connection to the FTP Server before issuing an error.

    If an orchestration that starts with an FTP List Files activity is deployed and the Integration Appliance cannot connect to the specified FTP Server, the Integration Appliance logs the connections errors as warnings in the system log until the retry count value is reached. When the retry count is reached, the Integration Appliance logs an error in the system log, resets the connection error count to zero, and continues to attempt to establish a connection to the FTP Server.

    For example, you set the retry count to 3. The first, second, and third connection errors appear in the system log as warnings. The Integration Appliance logs the fourth connection error as an error and resets the connection error count to zero. Therefore, the fifth connection error generates a warning in the system log. The Integration Appliance continues to attempt to establish a connection to the FTP Server.

Mapping the input of the activity

  1. Select the Map Inputs task in the Checklist. The Retry pane is displayed. The input parameters of the activity are displayed in the To Activity pane.
  2. It is optional to map the input parameters of the FTP List Files activity. The FTP List Files activity has the following optional input parameters:    
    • directory
    • connection

    If the input parameter listed in the following table are not displayed in the To Activity pane of the Map Inputs pane, select Map > Show Optional Parameters from the toolbar menu or right-click in the To Activity pane and select the Show Optional Parameters option. If the Map > Show Optional Parameters option is not selectable in the toolbar menu, click on a node in the Map Inputs pane to make this option active.

    You can optionally override the settings you specified in the Configure and Pick Endpoint task of the Checklist dynamically during run time, using the input parameters defined in the following table.

    For example, the directory on the FTP Server where the Integration Appliance gets the list of files, could be set in the Configure task to the directory called input. During run time, the directory name: inputXML could be passed into the activity using the directory input parameter. The directory name specified dynamically during run time overrides the original setting specified in the Configure pane. In this example, the list of files is generated from the inputXML directory.

Table 4.
Input Parameter Name/Node Description Overrides the Setting in Task?
directory Specifies the directory on the FTP server where the Integration Appliance generates a list of files in this activity. The directory input parameter overrides the In Directory field of the Configure task.
connection/timeout Specifies the amount of time in seconds before a timeout occurs between the Integration Appliance attempting to establish a connection to the endpoint and the time it takes for the endpoint to respond to the connection request. For the FTP Delete File activity, the amount of time the Integration Appliance attempts to establish a connection to the FTP Server before timing out.   The timeout node of the connection input parameter overrides the Connection Timeout -Time out after ___ second(s) when establishing a connection to the Endpoint field of the FTP endpoint.  
connection/host Specifies the Host Name or IP address for the FTP server. The host node of the connection input parameter overrides the Host Name field of the FTP endpoint.
connection/port Specifies the port number for the specified host name. The standard port for FTP traffic is 21. The default port for SFTP traffic is 22. The port node of the connection input parameter overrides the Port field of the FTP endpoint.
connection/username Specify a user name for the account used to connect to the FTP Server. The username node of the connection input parameter overrides the User Name field of the FTP endpoint.
connection/password Specify a password for the account used to connect to the FTP Server. The password node of the connection input parameter overrides the Password field of the FTP endpoint.

Mapping the output of the activity

  1. Select the Map Outputs task in the Checklist. The directoryListing or fileListing output parameters is displayed in the From Activity pane.
  2. Map the directoryListing or fileListing output parameter to a variable. See Creating a map for general instructions on mapping.

During run time, the FTP List Files activity returns the list of files to the orchestration using the directoryListing or fileListing output parameter.

If the List Only Files (NLST) check box is cleared in the Configure task, a full directory listing including directory permissions, timestamps, and ownership is returned in the directoryListing output parameter as a single string. When a full directory listing is returned as a string, you must parse the directory listing using a Flat File Schema with the Read Flat File activity. You can create a Flat File Schema using the Flat File Schema Editor. For an example orchestration parsing a directory listing using a Flat File Schema, see the Parse Directory Listing Example: Parse directory listing example.

If the List Only Files (NLST) check box is selected and the FTP List Files activity supports the format of the directory listing for your FTP Server, a repeating structure of just filenames (if supported by the FTP server) is returned in the fileListing output parameter. When a repeating structure of filename is returned, you do not need to parse the directory listing and can just iterate through the list using a For Each or While Loop activity.




Feedback | Notices


Timestamp icon Last updated: Thursday, December 17, 2015


http://pic.dhe.ibm.com/infocenter/wci/v7r0m0/topic/com.ibm.wci.doc/ftp_list_files_activity.html