Data Transformation Utilities

Utility functions for common data operations like encoding, merging, grouping, and formatting. These functions help you transform data between steps in your flows.

Quick Data Transformation: These utilities handle common data manipulation tasks without needing custom code. Use them for encoding, merging objects, formatting templates, and more.

Function List

base64-encode

Encode to base64

base64-decode

Decode from base64

json-xml

Convert JSON to XML

xml-json

Convert XML to JSON

Handlebars Template

Dynamic templates

Mapping

JSON path data mapping

secret-key-get-node

Retrieve secret keys

function-invoke

Invoke other functions

sftp

Upload files via SFTP

base64-encode

Encode text or objects to base64 format. Automatically handles both string and object inputs, converting objects to JSON before encoding.

Parameters

Takes a single input value directly (not wrapped in a params object):

input

string | object

required

The data to encode. Can be:

String: Encoded directly to base64
Object: Automatically converted to JSON string, then encoded to base64

Response

Returns the base64-encoded string directly (not wrapped in an object).

"Y29uc29sZS5sb2coImhlbGxvIHdvcmxkISIpOw=="

Example Usage

{
  "function": "base64-encode",
  "params": "Hello World!"
}

{
  "function": "base64-encode",
  "params": {
    "name": "John Doe",
    "email": "john@example.com"
  }
}

HTTP Request: Get file content
↓
Functions: Encode for transmission
  Function: base64-encode
  Input: (http_request.response)
↓
HTTP Request: Send encoded data

Common Use Cases

Encode Files for API

Encode files before sending to APIs

Get file from storage
Encode to base64
Send in HTTP request body
API receives and decodes

Store Binary Data

Store binary content as text

Receive binary file
Encode to base64
Store in database or KV storage
Decode when retrieved

Embed Data in Formats

Create data URLs or embed in JSON

Get image file
Encode to base64
Create data URL with encoded content
Use in HTML or email

Auto-Stringify: Objects are automatically converted to JSON before encoding. No need to manually stringify objects.

base64-decode

Decode base64-encoded data. Automatically detects and parses JSON objects in the decoded output.

Parameters

Takes a single input value directly:

input

string

required

The base64-encoded string to decode.

Response

Returns decoded data directly (not wrapped in an object). The return type depends on the decoded content:

Object: If the decoded string is valid JSON, returns the parsed object
String: If the decoded string is plain text, returns the string

// Example 1: Decodes to plain text
"Hello World!"

// Example 2: Decodes to object (auto-parsed JSON)
{
  "name": "John Doe",
  "email": "john@example.com"
}

Example Usage

{
  "function": "base64-decode",
  "params": "SGVsbG8gV29ybGQh"
}

{
  "function": "base64-decode",
  "params": "eyJuYW1lIjoiSm9obiBEb2UifQ=="
}

Trigger: Receive base64-encoded data
↓
Functions: Decode data
  Function: base64-decode
  Input: (trigger.encoded_data)
↓
Agent: Process decoded content
  (automatically parsed if JSON)

Common Patterns

Receive Encoded Files

Decode files received from APIs

HTTP Request: Receive base64 file
Decode from base64
Store or process decoded file

Retrieve Stored Data

Decode stored base64 data

Get base64 string from storage
Decode to original format
Use decoded content

Auto-Detection: This function automatically detects if the decoded string is valid JSON and parses it into an object. You don’t need to specify the output format or manually parse JSON.

json-xml

Transform JSON data to XML format. Useful for integrating with systems that require XML.

Parameters

json

object | string | Buffer

required

The JSON data to convert to XML. Can be:

Object: Converted directly to XML
String: Parsed as JSON first, then converted
Buffer: Converted to string, parsed as JSON, then converted

options

object

XML conversion options:

Show View all options

useXmlJs (boolean): Use xml-js library instead of custom converter
compact (boolean, default: false): Produce compact XML format
spaces (number, default: 0): Number of spaces for indentation
fullTagEmptyElement (boolean): Use full tags instead of self-closing tags
indentCdata (boolean): Indent CDATA sections
indentAttributes (boolean): Print attributes on multiple lines
ignoreDeclaration (boolean): Omit XML declaration
ignoreInstruction (boolean): Omit processing instructions
ignoreAttributes (boolean): Omit element attributes
ignoreComment (boolean): Omit comments
ignoreCdata (boolean): Omit CDATA sections
ignoreDoctype (boolean): Omit DOCTYPE declaration
ignoreText (boolean): Omit text content

Response

Returns XML as a string with XML declaration.

<?xml version="1.0"?><root><name>John</name><age>30</age></root>

Example Usage

{
  "function": "json-xml",
  "params": {
    "json": {
      "customer": {
        "name": "John Doe",
        "email": "john@example.com",
        "orders": [
          {"id": "001", "total": 150},
          {"id": "002", "total": 200}
        ]
      }
    },
    "options": {
      "compact": true,
      "spaces": 2
    }
  }
}

{
  "function": "json-xml",
  "params": {
    "json": {
      "QuestionAnswer": [
        {
          "QuestionCd": "MOTOR_NCB",
          "Explanation": "1000"
        }
      ]
    },
    "options": {
      "compact": true
    }
  }
}

Agent: Generate order data (JSON)
↓
Functions: Convert to XML
  Function: json-xml
  JSON: (agent.order_data)
  Options: {compact: true}
↓
HTTP Request: Send XML to SOAP API

Common Use Cases

Legacy System Integration

Send data to XML-only systems

Collect data in JSON format
Convert to XML
Send to SOAP API or legacy system

XML File Generation

Generate XML files for export

Build data structure in JSON
Convert to XML with proper formatting
Store as .xml file

ACORD Formats

Generate insurance industry standard formats

Build ACORD data structure
Convert to XML with compact option
Submit to insurance systems

XML Declaration: The output always includes XML version declaration at the beginning. Use ignoreDeclaration option set to true to omit it.

xml-json

Convert XML to JSON format. This is the reverse operation of json-xml, useful for parsing XML responses from APIs.

Parameters

xml

string

required

The XML string to convert to JSON.

options

object

XML parsing options:

Show View all options

useXmlJs (boolean): Use xml-js library instead of custom parser
compact (boolean, default: false): Produce compact JSON format
trim (boolean): Trim whitespace from text content
sanitize (boolean): Replace special characters with HTML entities
nativeType (boolean): Convert numeric/boolean strings to native types
nativeTypeAttributes (boolean): Convert attribute values to native types
addParent (boolean): Add parent property to each element
alwaysArray (boolean | array): Force elements to be arrays
alwaysChildren (boolean): Always generate elements property
instructionHasAttributes (boolean): Parse processing instructions as attributes
captureSpacesBetweenElements (boolean): Capture whitespace between elements
ignoreDeclaration (boolean): Skip XML declaration
ignoreInstruction (boolean): Skip processing instructions
ignoreAttributes (boolean): Skip element attributes
ignoreText (boolean): Skip text content
ignoreComment (boolean): Skip comments
ignoreCdata (boolean): Skip CDATA sections
ignoreDoctype (boolean): Skip DOCTYPE declaration

Response

Returns a JSON string (you may need to parse it in subsequent steps).

"{\"Response\":{\"Result\":{\"Status\":\"Success\"}}}"

Example Usage

{
  "function": "xml-json",
  "params": {
    "xml": "<?xml version=\"1.0\"?><Response><Status>Success</Status></Response>",
    "options": {
      "compact": true,
      "ignoreDeclaration": true
    }
  }
}

{
  "function": "xml-json",
  "params": {
    "xml": "(http_request.response)",
    "options": {
      "compact": true,
      "trim": true,
      "nativeType": true
    }
  }
}

HTTP Request: Call SOAP API
↓
Functions: Parse XML response
  Function: xml-json
  XML: (http_request.response)
  Options: {compact: true, trim: true}
↓
Agent: Process JSON data

Common Use Cases

Parse SOAP Responses

Convert SOAP API responses to JSON

Call SOAP API endpoint
Receive XML response
Convert to JSON
Process as structured data

Legacy System Integration

Parse XML from legacy systems

Receive XML from legacy system
Convert to JSON
Process with modern tools

XML File Processing

Process XML files

Read XML file
Parse to JSON
Transform and process data

Return Type: This function returns a JSON string, not a parsed object. You may need to use JSON.parse() or another step to convert the string to an object for further processing.

Handlebars Template

Use Handlebars templating engine to create dynamic templates with variables that are replaced at runtime.

Parameters

template

string

required

The Handlebars template string with variables in double curly braces.

variables

object

required

The data object containing values to replace in the template.

Response

Returns the rendered template as a string.

"Hello John Doe, your order #12345 has been confirmed!"

Example Usage

{
  "function": "handlebars",
  "params": {
    "template": "Hello {{customer.name}}, order {{order.id}} confirmed!",
    "variables": {
      "customer": {
        "name": "John Doe"
      },
      "order": {
        "id": "12345"
      }
    }
  }
}

{
  "function": "handlebars",
  "params": {
    "template": "{{#if premium}}Premium{{else}}Standard{{/if}} customer {{name}}",
    "variables": {
      "name": "John",
      "premium": true
    }
  }
}

Agent: Gather customer data
↓
Functions: Generate personalized email
  Function: handlebars
  Template: Email body with {{variables}}
  Variables: (agent.customer_data)
↓
HTTP Request: Send email

Template Features

Variables

Insert dynamic values

Hello {{name}}, you have {{count}} new messages.

Nested Properties

Access nested object properties

{{user.profile.firstName}} {{user.profile.lastName}}
Email: {{user.contact.email}}

Conditionals

Show content based on conditions

{{#if isPremium}}
  Thank you for being a premium member!
{{else}}
  Upgrade to premium for more features.
{{/if}}

Loops

Iterate over arrays

Your orders:
{{#each orders}}
  - Order {{this.id}}
{{/each}}

Common Use Cases

Email Generation

Create personalized emails

Email template with variables
+ Customer data
= Personalized email content

Report Generation

Generate dynamic reports

Report template
+ Data from multiple sources
= Formatted report

Notification Messages

Create dynamic notifications

Message template
+ Event data
= Personalized notification

Mapping

Use JSONPath expressions to select, transform, and map data into new structures. Powerful for data transformation and restructuring with support for filters, array operations, and conditional selection.

Parameters

data

object

required

The source data object to query and transform.

path

string | array | object

required

JSONPath expression(s) defining how to map the data:

String: Single JSONPath query
Array: For array transformations with mapping
Object: Map of output keys to JSONPath expressions

lookupData

object

Optional lookup data accessible as variables in path expressions for filtering and conditional selection.

Response

Returns the mapped/extracted data in the specified structure.

{
  "customerName": "John Doe",
  "orderTotal": 150.00,
  "itemCount": 3
}

Path Syntax

Basic Property Access

Access properties using JSONPath

"$.user.name"           // Get user.name
"$.order.total"         // Get order.total
"$.items[0].price"      // Get first item price
"$"                     // Get root object

String Concatenation

Combine values using pipe delimiter

"$.firstName|' '|$.lastName"  // Join with space
// Result: "John Doe"

"$.street|', '|$.city"        // Join with comma
// Result: "123 Main St, San Francisco"

Array Mapping

Transform arrays with custom mapping

[
  "$.items",                          // Source array path
  { id: "$.id", name: "$.name" },    // Mapping for each item
  { merge: false }                    // Options
]

Object Mapping

Create new object structure

{
  userName: "$.user.name",
  userEmail: "$.user.email",
  orderTotal: "$.order.total"
}

Filters with Lookup Data

Filter using external variables

// In path:
"$.items[?(@.id === targetId)]"

// In lookupData:
{ targetId: "abc123" }

// Result: Item where id equals abc123

Deep Property Search

Search recursively through nested structures

"$..price.duty"         // Find duty in any price object
"$..covers..limits"     // Find all limits in any covers

Example Usage

{
  "function": "mapping",
  "params": {
    "data": {
      "user": {
        "firstName": "John",
        "lastName": "Doe"
      },
      "order": {
        "total": 150.00,
        "items": [1, 2, 3]
      }
    },
    "path": {
      "customerName": "$.user.firstName|' '|$.user.lastName",
      "orderTotal": "$.order.total",
      "itemCount": "$.order.items.length"
    }
  }
}

{
  "function": "mapping",
  "params": {
    "data": {
      "products": [
        {"id": "p1", "name": "Phone", "price": 599},
        {"id": "p2", "name": "Laptop", "price": 1299}
      ]
    },
    "path": [
      "$.products",
      {
        "productId": "$.id",
        "productName": "$.name",
        "productPrice": "$.price"
      },
      { "merge": false }
    ]
  }
}

{
  "function": "mapping",
  "params": {
    "data": {
      "orders": [
        {"id": "001", "status": "completed"},
        {"id": "002", "status": "pending"}
      ]
    },
    "path": "$.orders[?(@.status === targetStatus)]",
    "lookupData": {
      "targetStatus": "completed"
    }
  }
}

HTTP Request: Get complex API response
↓
Functions: Extract and restructure data
  Function: mapping
  Data: (http_request.response)
  Path: {custom mapping definition}
↓
Agent: Use clean, structured data

Common Use Cases

API Response Transformation

Transform API responses to your format

External API format (complex)
→ Map to internal format (simplified)
→ Use in your system

Data Extraction

Extract specific fields from complex objects

Large nested object
→ Extract only needed fields
→ Simplified object for processing

Format Conversion

Convert between data formats

Source format A
→ Map fields to format B
→ Compatible with target system

Conditional Data Selection

Select data based on conditions

Large dataset
→ Filter by criteria using lookupData
→ Only matching records

JSONPath Plus: This function uses the jsonpath-plus library with full support for complex queries, filters, recursive descent, and array operations.

secret-key-get-node

Retrieve secret values stored in QuivaWorks’ secret manager. Use this to securely access API keys, tokens, and other sensitive configuration.

Parameters

key

string

required

The name/identifier of the secret to retrieve.

Response

Returns an object with either a value property (on success) or an error property (if not found).

// Success
{
  "value": "sk_live_abc123..."
}

// Not Found
{
  "error": "stripe_api_key not found"
}

Example Usage

{
  "function": "secret-key-get-node",
  "params": {
    "key": "stripe_api_key"
  }
}

{
  "function": "secret-key-get-node",
  "params": {
    "key": "postgres_password"
  }
}

Functions: Get API key from secrets
  Function: secret-key-get-node
  Key: "external_api_key"
↓
HTTP Request: Call external API
  Authorization: Bearer (functions.value)
↓
Process API response

Error Handling

Always check for the error property in the response to handle missing keys gracefully:

// In your flow logic
if (response.error) {
  // Key not found
  console.log(response.error);
} else {
  // Use response.value
  const apiKey = response.value;
}

Common Use Cases

API Authentication

Securely access API keys for external services

Get secret API key
Use in HTTP request headers
Keep key secure, never in code

Database Connections

Retrieve database credentials

Get database password
Connect to database
Execute queries securely

Third-Party Services

Access service tokens

Get OAuth token
Get webhook secret
Use for integrations

Security Best Practice: Never hardcode secrets in flows. Always use the secret manager and retrieve secrets at runtime using this function. Secrets are stored in the quiva-secrets KV bucket.

function-invoke

Invoke other QuivaWorks functions programmatically from within your flow. Useful for orchestrating complex workflows and modular function composition.

Parameters

invocation_type

string

required

Type of invocation:

"Async": Fire and forget (returns immediately, doesn’t wait for result)
"RequestResponse": Synchronous (waits for function to complete and returns result)

payload

object

required

Input data to pass to the invoked function. Structure depends on the target function’s requirements.

subject

string

required

The identifier/name of the function to invoke.

Response

Returns an object. Structure depends on the invoked function and invocation type.

// For RequestResponse
{
  "result": "...",
  "status": "success"
}

// For Async
{
  "invoked": true,
  "function": "function-name"
}

Example Usage

{
  "function": "function-invoke",
  "params": {
    "invocation_type": "RequestResponse",
    "payload": {
      "data": "process this",
      "options": {
        "validate": true
      }
    },
    "subject": "data-processor-function"
  }
}

{
  "function": "function-invoke",
  "params": {
    "invocation_type": "Async",
    "payload": {
      "notification": "send email",
      "recipient": "user@example.com"
    },
    "subject": "email-sender-function"
  }
}

HTTP Request: Receive webhook
↓
Functions: Invoke data processor
  Function: function-invoke
  Type: RequestResponse
  Payload: (webhook.data)
  Subject: "webhook-processor"
↓
Functions: Async notification
  Function: function-invoke
  Type: Async
  Subject: "notification-sender"

Invocation Types

RequestResponse (Synchronous)

Wait for function completion and get result

Invoke function
Wait for execution
Receive result
Continue with result

Use when: You need the result to continue processing

Async (Fire and Forget)

Invoke function without waiting

Invoke function
Return immediately
Function runs in background
Continue without waiting

Use when: Result not needed, or for triggering side effects

Common Use Cases

Function Orchestration

Chain multiple functions together

Main flow receives request
→ Invoke validation function
→ Invoke processing function
→ Invoke notification function

Modular Architecture

Break complex logic into reusable functions

Create specialized functions
Invoke them as needed
Reuse across multiple flows

Background Tasks

Trigger long-running operations

Receive request
Invoke async background function
Return immediate response
Background task completes later

Implementation: Uses QuivaWorks SDK func.invoke() method. Ensures proper function calling within the platform’s execution environment.

sftp

Upload files to SFTP servers securely. Supports both password and key-based authentication with configurable timeouts.

Parameters

host

string

required

SFTP server hostname or IP address.

port

number

required

SFTP server port number (typically 22).

username

string

required

Username for authentication.

fileContents

string

required

The file contents to upload (as a string).

filePath

string

required

Remote file path where the file should be uploaded.

privateKey

string

SSH private key for key-based authentication. Alternative to password authentication.

password

string

Password for password-based authentication. Alternative to key-based authentication.

algorithms

object

SSH algorithm configuration. Specify allowed server host key algorithms.

{
  "serverHostKey": ["ssh-rsa", "ssh-ed25519"]
}

connectionTimeout

number

default:"10000"

Connection timeout in milliseconds. Default is 10 seconds.

transferTimeout

number

default:"30000"

File transfer timeout in milliseconds. Default is 30 seconds.

Response

Returns a success message string on completion, or throws an error on failure.

"File uploaded successfully"

Example Usage

{
  "function": "sftp",
  "params": {
    "host": "sftp.example.com",
    "port": 22,
    "username": "ftpuser",
    "password": "secure_password",
    "fileContents": "Invoice data here...",
    "filePath": "/invoices/2025/invoice_001.txt"
  }
}

{
  "function": "sftp",
  "params": {
    "host": "secure-sftp.example.com",
    "port": 22,
    "username": "deploy",
    "privateKey": "-----BEGIN RSA PRIVATE KEY-----...",
    "fileContents": "(file_data)",
    "filePath": "/production/app.zip",
    "connectionTimeout": 15000,
    "transferTimeout": 60000
  }
}

{
  "function": "sftp",
  "params": {
    "host": "sftp.company.com",
    "port": 2222,
    "username": "backup",
    "password": "backup_pass",
    "fileContents": "(backup_data)",
    "filePath": "/backups/daily/backup.sql",
    "connectionTimeout": 20000,
    "transferTimeout": 120000,
    "algorithms": {
      "serverHostKey": ["ssh-rsa"]
    }
  }
}

Agent: Generate report file
↓
Functions: Upload to SFTP
  Function: sftp
  Host: sftp.example.com
  FileContents: (agent.report)
  FilePath: /reports/daily/report.pdf
↓
Functions: Send notification
  Message: "Report uploaded successfully"

Timeout Errors

The function provides specific error messages for timeout scenarios:

Connection Timeout

If server takes too long to accept connection:

Error: "Server took too long to connect. 
        Make sure the host and the port are correct."

Solutions:

Verify host and port are correct
Check network connectivity
Increase connectionTimeout if server is slow

Transfer Timeout

If file upload takes too long:

Error: "File took too long to send."

Solutions:

Increase transferTimeout for large files
Check network bandwidth
Verify server is accepting uploads

Authentication Methods

Password Authentication

Use username and password

{
  "username": "user",
  "password": "secure_password"
}

When to use: Simple setups, testing

Key-Based Authentication

Use SSH private key

{
  "username": "user",
  "privateKey": "-----BEGIN RSA PRIVATE KEY-----..."
}

When to use: Production environments, automated systems, enhanced security

Common Use Cases

File Delivery

Deliver files to partners or systems

Generate report/export
Upload via SFTP
Partner retrieves file

Backup Systems

Send backups to remote storage

Create backup
Upload to SFTP backup server
Verify upload success

Data Integration

Exchange data with external systems

Export data from your system
Upload to partner's SFTP
Partner processes file

Automated Deployments

Deploy files to remote servers

Build application files
Upload to production SFTP
Trigger deployment process

Connection Management: The SFTP connection is automatically closed after the upload completes or if an error occurs. No manual cleanup required.

Large Files: For large files, increase the transferTimeout parameter appropriately. As a guideline, allow approximately 1 second per MB plus overhead.

Best Practices

Auto-Detection Benefits

base64-decode automatically parses JSON - leverage this for cleaner flows

Always Handle Errors

Check for error properties in responses, especially with secret-key-get-node

Template Dynamic Content

Use Handlebars for emails, notifications, and reports with the variables parameter

Restructure with Mapping

Use Mapping with JSONPath for powerful data transformations

Secure SFTP Uploads

Prefer key-based authentication over passwords for production SFTP

Store Secrets Securely

Always use secret manager for sensitive data - never hardcode secrets

Function Orchestration

Use function-invoke for modular, reusable flow architectures

Set Appropriate Timeouts

Configure SFTP timeouts based on file sizes and network conditions

Migration Notes

If you’re updating existing flows that use these functions, note these breaking changes:

Breaking Changes:

base64-encode: Remove encoding parameter - pass data directly
base64-decode: Remove output_encoding parameter - auto-detection enabled
handlebars: Rename data parameter to variables
json-xml: Restructure to use json and options object instead of separate parameters
secret-key-get-node: Rename key_name to key, expect simplified response structure

Next Steps

Stream Functions

Real-time event processing

Key-Value Storage

Fast key-based storage

Object Storage

Store large files

Functions Step

Using Functions in flows

​Data Transformation Utilities

​Function List

base64-encode

base64-decode

json-xml

xml-json

Handlebars Template

Mapping

secret-key-get-node

function-invoke

sftp

​base64-encode

​Parameters

​Response

​Example Usage

​Common Use Cases

​base64-decode

​Parameters

​Response

​Example Usage

​Common Patterns

​json-xml

​Parameters

​Response

​Example Usage

​Common Use Cases

​xml-json

​Parameters

​Response

​Example Usage

​Common Use Cases

​Handlebars Template

​Parameters

​Response

​Example Usage

​Template Features

​Common Use Cases

​Mapping

​Parameters

​Response

​Path Syntax

​Example Usage

​Common Use Cases

​secret-key-get-node

​Parameters

​Response

​Example Usage

​Error Handling

​Common Use Cases

​function-invoke

​Parameters

​Response

​Example Usage

​Invocation Types

​Common Use Cases

​sftp

​Parameters

​Response

​Example Usage

​Timeout Errors

​Authentication Methods

​Common Use Cases

​Best Practices

Auto-Detection Benefits

Always Handle Errors

Template Dynamic Content

Restructure with Mapping

Secure SFTP Uploads

Store Secrets Securely

Function Orchestration

Set Appropriate Timeouts

​Migration Notes

​Next Steps

Stream Functions

Key-Value Storage

Object Storage

Functions Step

Data Transformation Utilities

Function List

base64-encode

Parameters

Response

Example Usage

Common Use Cases

base64-decode

Parameters

Response

Example Usage

Common Patterns

json-xml

Parameters

Response

Example Usage

Common Use Cases

xml-json

Parameters

Response

Example Usage

Common Use Cases

Handlebars Template

Parameters

Response

Example Usage

Template Features

Common Use Cases

Mapping

Parameters

Response

Path Syntax

Example Usage

Common Use Cases

secret-key-get-node

Parameters

Response

Example Usage

Error Handling

Common Use Cases

function-invoke

Parameters

Response

Example Usage

Invocation Types

Common Use Cases

sftp

Parameters

Response

Example Usage

Timeout Errors

Authentication Methods

Common Use Cases

Best Practices

Migration Notes

Next Steps