Create a mass ingestion task

Use a POST request to create a mass ingestion task.

POST request

To create a mass ingestion task through the API, use the following URI:

mftsaas/api/v1/mitasks

Include the following fields in the request:

Field

Type

Required

Description

name

String

Yes

Name of the task.

location

String

Location of the project.

projectId

String

ID number associated with the project.

projectName

String

Name of the project.

description

String

Description of the task.

sourceConnection

String

Directory from where files are transferred.

sourceType

String

Yes

Determines the type where files are transferred. Enter one of the following options:
CONNECTION. Use connection as a source.
FILELISTENER. Use file listener as a source.

includesubfolder

String

Determines whether to include the files in sub-folders in the transfer. Set the value to true to transfer files from all sub-folders under the defined source directory. Values are true or false.

checkDuplicate

String

Determines whether to check for duplicate files. Values are true or false. Set the value to true to check duplicate files and deny file transfer. If the value is set to false all files are transferred

filePatternType

String

Yes

File pattern type used to select files to transfer. Enter one of the following options:
wildcard
regex

filePattern

String

Yes

Enter file pattern types, depending on the file pattern that you have selected.
wildcard. You can use the following wildcard character filters:
An asterisk () matches any number of characters.
A question mark (?) matches a single character.
regex. Use regular expression to match the file pattern. Consider the following examples:
Use the following syntax to listen to all files except for files with a name that contains out, foo, and baz: ^(?!.
(?:out|baz|foo)).*$ all except
Use the following syntax to listen to all files with doc, docx, and pdf extensions: ([a-zA-Z0-9\s_\.-():])+(.doc|.docx|.pdf)$

postPickupAction

String

Determines what to do with source files after the transfer of files. The following options are available:
KEEP. Keep the files in the source directory.
DELETE. Delete the files from the source directory.
RENAME. Rename the files in the source directory. You must specify a file name suffix that Mass Ingestion adds to the file name when renaming the files.
ARCHIVE. Archive the files to a different location. You must specify an archive directory.

targetConnection

String

Yes

Directory details to which files are transferred.

taskActions

String

Actions to process files in the mass ingestion task. If you add multiple actions, mass ingestion processes files in a sequence.

actions

File processing action.
Enter the following file processing actions:
To compress files, enter Compression.
To decompress files, enter Decompression.
To encrypt files, enter Encryption.
To dencrypt files, enter Decryption.

action type

Enter the action type depending on the action that you add.
To compress files use one of the following methods.
Zip
Tar
Gzip
To decompress files use one of the following methods.
Unzip
Untar
Gunzip
To encrypt files add PGP. Enter the key ID in the properties.
The mass ingestion task uses the PGP method to encrypt files. Generate a key ring using the CLI. Enter the key ring in the Key ID. For more information about the keyring CLIs, refer to key ring command reference in Tasks.
To decrypt files, add PGP. Enter the key passphrase in the properties.
The mass ingestion task uses the PGP method to encrypt files. Generate the key passphrase using the CLI. Enter the key passphrase in the Key Passphrase. For more information about the keyring CLIs, refer to key ring command reference in Tasks.

POST request examplePOST request example

Uee this sample as a reference to create a mass ingestion task.

POST <serverURL>/public/core/v1/mitasks
Content-Type: application/json
Accept:application/json
Content-Type:application/json
IDS-SESSION-ID:{{IDS-SESSION-ID}}
{    
    "name": "Green Green v2",
    "location": {
        "projectId": "9JDNOBX9M31e2AD1dIUv6M",
        "projectName": "Default"
    },
    "description": "",
    "sourceType": "CONNECTION",
    "sourceConnection": {
        "id": "",
        "name": "",
        "type": "local"
},
        "sourceParameters": {
            "filePattern": "*.txt",
            "filePatternType": "WILDCARD",
            "includeSubfolder": "false",
            "sourceDirectory": "C:\\Monitor",
            "checkDuplicate": "false",
            "postPickupAction": "KEEP"
        
    },
    "targetConnection": {
        "id": "0100000B000000000002",
        "name": "ADLS",
        "type": "Azure Data Lake"
},
        "targetParameters": {
            "adlsTargetLocation": "/satyen/green"
        
    },
    "agentGroupId": "01000025000000000002"
                    taskActions":[
                    { 
                                "action": "Compression", 
                                "actionType": "Zip", 
                                "properties": {}
                     } 
                    ]
}

POST response example

If the request is successful, you might receive a response similar to the following example:

{
    "id": "cEMWKpibm44bNf5aMjbJ4U",
    "name": "Green Green v2",
    "location": {
        "projectId": "9JDNOBX9M31e2AD1dIUv6M",
        "projectName": "Default"
    },
    "description": "",
    "sourceType": "CONNECTION",
    "sourceConnection": {
        "id": "",
        "name": "",
        "type": "local"
},
        "sourceParameters": {
            "filePattern": "*.txt",
            "filePatternType": "WILDCARD",
            "includeSubfolder": "false",
            "sourceDirectory": "C:\\Monitor",
            "checkDuplicate": "false",
            "postPickupAction": "KEEP"
        
    },
    "targetConnection": {
        "id": "0100000B000000000002",
        "name": "ADLS",
        "type": "Azure Data Lake"
},
        "targetParameters": {
            "adlsTargetLocation": "/satyen/green"
        
    },
    "agentGroupId": "01000025000000000002",
    "createdTime": "2018-08-27T07:03:32Z",
    "updatedTime": "2018-08-29T12:14:58Z""
                    taskActions":[
                    { 
                                "action": "Compression", 
                                "actionType": "Zip", 
                                "properties": {}
                     } 
                    ]
}
}