Built.io Flow Enterprise is an integration platform that lets you connect your favorite cloud applications, platforms or IoT devices or enabling you to orchestrate and automate business workflows.
Apart from performing routine tasks like organizing meetings, analyzing sales and marketing data, and managing projects, you can leverage Built.io Flow Enterprise to automate complex tasks that would otherwise demand a lot of time and effort if performed manually.
As an example, we’ll show you how to migrate tens of thousands of records from one system to another automatically to avoid human error and to save time. The data can be migrated from cloud to cloud, on-prem to on-prem, cloud to on-prem or on-prem to cloud.
How It Works
Below we’ve shared how to build a workflow that migrates bulk data from one system to another. The workflow traverses through 3 main stages to migrate data.
Stage 1. Split all records in batches and send them for processing
Get the total number of records from the source system and split them in batches. For example, if there are 10,000 total records, you can split them in 20 batches so that each batch contains 500 records. Since the workflow can send the records of only one batch at a time (500 records) for processing, run the same flow recursively (20 times) till all the batches are sent for processing.
Stage 2. Process each batch using Batch API
If the destination system has an API to process data in batches, you can use it to process the whole set of records in a batch and upload the whole batch at the destination system at once
Stage 3. Process one record at a time for each batch
If the destination system doesn’t have a batch API, you will need to iterate through all the batch records (in this case, 500 records), one record at a time and call the same workflow recursively (500 times) till all the records are updated to the destination system.
Migrating Bulk Data From Different Environments
Now that we know how the process of data migration works in Built.io Flow Enterprise, let’s move on to creating the workflow. We will do this with the help of two examples.
Example 1: Migrating Data From A Cloud System
Let’s say you want to migrate 1000 records from your MongoDB instance hosted on cloud to any other system. To do so, you will need to create the following workflow in your Built.io Flow Enterprise account.
In this workflow, we first retrieve all the records from the MongoDB instance and split those records in batches. We will then set up conditions which will process these records one batch at a time and will update them to the destination system.
Configurator Variables To Look Out For
Here are all of the config variables used in this workflow along with their details:
- stage: Current stage of the workflow. By default it is set to ‘0’.
- stage_0_split: Maximum number of records to be processed in ‘stage_1’. By default it is set to ‘100’.
- stage_1_split: Maximum number of records to be uploaded in bulk at a time (stage_1). Alternatively you can also upload the records one by one using single upload (stage_2). For single upload, the default and maximum limit is set to ‘100’.
- initial_offset: Offset of the record from which the migration should start. By default, it is set to ‘0’.
- webhook_url: Webhook URL which will trigger the workflow initially. It will also be used to generate the restart URL and run workflow recursively in case of an error. You need to enable the Webhook trigger first.
- bulk_export: Specifies if the records should be migrated in bulk or one record at a time. Set this value to ‘False’ if you are migrating single record at a time. By default, it is set to ‘True’.
- error_mail_id: Email ID to which the error message and the stage restart URL should be sent.
- success_status_code: Expected status code response from upload http request.
Below are the step you’ll need to take to take advantage of this workflow. You can configure, use, and modify the steps as per your requirements:
In this stage, we will retrieve the count of the records we need to migrate and will split them in batches. We will then run a loop recursively to send these batches for processing, one batch at a time.
Get Count: Configure the ‘HTTP Request’ action with the required authentication and settings to get the total count of the records from the source system. You can replace this block with the in-built flow actions. If you already know the count of the records, you can remove this block and add the count to the next block instead.
Offset Calculator: This block determines the offset for each batch to be passed on to the next stage.
Required Config Parameter:
Total Objects: Insert the total count of records in this field or pass the total count from previous block.
Trigger Stage 1 Loop: This block triggers ‘stage_1’ for the current batch in processing.
In this stage, we will upload the records of the batch to destination system by using its bulk API.
Get Batch: Configure the ‘HTTP Request’ action with the required authentication and settings to get the batch you want to process. Also use ‘$config.params.stage_0_split’ to set limit on the records to be migrated at a time and use ‘$config.params.offset’ which is sent by the previous stage to use as ‘skip’ or ‘offset’ in the get batch call. You can replace this block by using in-built flow actions.
Transformation And Split Calculation: Add transformation logic within the marked function and set the path for data if the batch data is nested.
The next three blocks are executed in loop till all the the records are uploaded to the destination system via bulk API.
Pre-bulk Upload Transformation: Add pre-bulk upload transformation logic within the marked function, if required.
Upload Batch: Configure the bulk upload ‘HTTP Request’ and use the output from the previous block. You can replace this block by using the in-built flow actions.
Store Error Message: Add logic to capture the record offset and record index at which the error occurred. Also store the stage restart URL (which will be used to reprocess the record/batch at which the error occurred) and specify the error message and response code that you wish to send.
If the destination system doesn’t have a bulk API, you can remove the above three block from your workflow.
Trigger Stage 2 Loop: If it is not a bulk upload, this block will trigger ‘stage_2’ for the current batch in processing.
In this stage, we will iterate through all records of the batch, and will upload it to destination system one record at a time.
Single Upload: Configure the single upload HTTP block for a single record upload.
Store Error Message: Add logic to capture the record offset and record index at which the error occurred. Also store the stage restart URL and specify the error message and response code you wish to display.
After this, add a ‘JSON to HTML’ block to convert the JSON error message data (which can be retrieved through ‘Store Error Message’ block of ‘stage_1’ and ‘stage_2’) in HTML format and send this data to the intended user through ‘Send an Email’ action.
Once all the workflow configurations are done, enable your workflow’s webhook and save it.. You can now use this workflow to migrate data between two systems.
Example 2: Migrating Data From An On-Prem System Secured With A Firewall
If the data is behind a firewall on on-prem,you can use ‘Enterprise Gateway’. Enterprise Gateway creates a secure connection between Built.io Flow Enterprise and your source system so that you can access the data securely. (To enable ‘Enterprise Gateway’ for your account, contact us at firstname.lastname@example.org). Once Enterprise Gateway is enabled for your account, you can use the ‘Enterprise Connector’ to migrate data from your on-premise database or application. Here’s the workflow for migrating data using Enterprise Connector.
When you run any of these workflows, the specified records will be migrated to the destination system.
Here’s a quick video on how the workflow works.