Skip to main content

Integrating External APIs

Learn how to integrate Stack9 with external services and APIs. This guide covers REST API connectors, authentication methods, error handling, and real-world integration patterns for common third-party services.

What You'll Build​

In this guide, you'll integrate with multiple external services:

  • Payment Gateway (Stripe-like API) with Bearer token auth
  • Email Service (SendGrid-like) with API key auth
  • Address Lookup (Google Maps-like) with query parameter auth
  • CRM System (Salesforce-like) with OAuth2
  • SMS Service (Twilio-like) with Basic auth

Time to complete: 60-75 minutes

Prerequisites​

  • Completed Building Workflows guide
  • Understanding of REST APIs
  • API credentials for services you want to integrate (or use test/sandbox APIs)

Understanding Stack9 Connectors​

Connectors in Stack9 provide a standardized way to integrate with external services:

Connector Types​

TypeUse CaseExample Services
rest_apiREST/HTTP APIsMost modern APIs
amazon_opensearchAWS OpenSearchElasticsearch alternative
amazon_s3File storageAmazon S3, MinIO
azure_blobFile storageAzure Blob Storage
postgresqlExternal databasesPostgreSQL databases
mongodbNoSQL databasesMongoDB Atlas
redisCachingRedis Cloud
graphqlGraphQL APIsGitHub, Shopify
sendgridEmail serviceSendGrid
open_apiOpenAPI/SwaggerAny OpenAPI spec
amazon_dynamodbNoSQL tablesDynamoDB
elastic_searchSearch engineElasticsearch

Note: Connector types must match the ConnectorType enum exactly (lowercase with underscores). For complete schema details, see the Connector Schema Reference.

Authentication Methods​

MethodConfigurationUse Case
bearerauthentication.type: "bearer"Modern APIs (OAuth2)
apiKeyauthentication.type: "apiKey"API key authentication
basicauthentication.type: "basic"Basic authentication

Note: For custom authentication headers, add them directly to the headers object in the configuration.

Step 1: Create Payment Gateway Connector​

Create src/connectors/payment_gateway.json:

{
  "key": "payment_gateway",
  "name": "Payment Gateway",
  "description": "Stripe-compatible payment processing",
  "type": "rest_api",
  "configuration": {
    "baseUrl": "https://api.payment-gateway.com/v1",
    "authentication": {
      "type": "bearer",
      "bearer": "%%PAYMENT_GATEWAY_API_KEY%%"
    },
    "headers": {
      "Content-Type": "application/json",
      "Accept": "application/json"
    }
  }
}

Environment Variable:

Add to .env:

PAYMENT_GATEWAY_API_KEY=sk_test_your_secret_key_here

Using the Connector in Action Type​

Create src/action-types/processPayment.ts:

import { Record, Number, String } from 'runtypes';
import { S9AutomationActionType } from '@april9au/stack9-sdk';

const ProcessPaymentParams = Record({
  amount: Number,
  currency: String,
  customer_email: String,
  order_id: Number,
});

export class ProcessPayment extends S9AutomationActionType {
  key = 'process_payment';
  name = 'Process Payment';

  async exec(params: any) {
    const validated = ProcessPaymentParams.check(params);
    const { connectors, logger } = this.context;

    try {
      const paymentGateway = connectors['payment_gateway'];

      // Create payment intent
      const paymentIntent = await paymentGateway.call({
        method: 'POST',
        path: '/payment_intents',
        body: {
          amount: Math.round(validated.amount * 100), // Convert to cents
          currency: validated.currency,
          customer_email: validated.customer_email,
          metadata: {
            order_id: validated.order_id,
          },
        },
      });

      logger.info(
        `Payment intent created: ${paymentIntent.id} for order ${validated.order_id}`
      );

      // Confirm payment
      const confirmedPayment = await paymentGateway.call({
        method: 'POST',
        path: `/payment_intents/${paymentIntent.id}/confirm`,
        body: {
          payment_method: 'pm_card_visa', // In real app, from customer
        },
      });

      if (confirmedPayment.status === 'succeeded') {
        return {
          success: true,
          transaction_id: confirmedPayment.id,
          amount: validated.amount,
          message: 'Payment processed successfully',
        };
      } else {
        return {
          success: false,
          error: `Payment status: ${confirmedPayment.status}`,
        };
      }
    } catch (error) {
      logger.error(`Payment processing failed: ${error.message}`);

      // Parse error response
      const errorMessage =
        error.response?.data?.error?.message || error.message;

      return {
        success: false,
        error: errorMessage,
        message: 'Payment failed',
      };
    }
  }
}

Step 2: Create Email Service Connector​

Create src/connectors/email_service.json:

{
  "key": "email_service",
  "name": "Email Service",
  "description": "SendGrid-compatible email delivery",
  "type": "sendgrid",
  "configuration": {
    "accessToken": "%%SENDGRID_API_KEY%%"
  }
}

Environment Variable:

SENDGRID_API_KEY=SG.your_sendgrid_api_key_here

Using Email Connector​

Create src/action-types/sendEmail.ts:

import { Record, String, Optional } from 'runtypes';
import { S9AutomationActionType } from '@april9au/stack9-sdk';

const SendEmailParams = Record({
  to: String,
  subject: String,
  template: String,
  data: Optional(Record({})),
});

export class SendEmail extends S9AutomationActionType {
  key = 'send_email';
  name = 'Send Email';

  async exec(params: any) {
    const validated = SendEmailParams.check(params);
    const { connectors, logger } = this.context;

    try {
      const emailService = connectors['email_service'];

      // Fetch template from database
      const template = await this.context.db('email_template')
        .where({ key: validated.template })
        .first();

      if (!template) {
        throw new Error(`Email template '${validated.template}' not found`);
      }

      // Replace template variables
      let htmlContent = template.html_content;
      let textContent = template.text_content;

      if (validated.data) {
        for (const [key, value] of Object.entries(validated.data)) {
          const regex = new RegExp(`{{${key}}}`, 'g');
          htmlContent = htmlContent.replace(regex, String(value));
          textContent = textContent.replace(regex, String(value));
        }
      }

      // Send email
      const response = await emailService.call({
        method: 'POST',
        path: '/mail/send',
        body: {
          personalizations: [
            {
              to: [{ email: validated.to }],
            },
          ],
          from: {
            email: 'noreply@yourapp.com',
            name: 'Your App Name',
          },
          subject: validated.subject,
          content: [
            {
              type: 'text/plain',
              value: textContent,
            },
            {
              type: 'text/html',
              value: htmlContent,
            },
          ],
        },
      });

      logger.info(`Email sent to ${validated.to}: ${validated.subject}`);

      return {
        success: true,
        message_id: response.headers['x-message-id'],
        message: 'Email sent successfully',
      };
    } catch (error) {
      logger.error(`Email sending failed: ${error.message}`);
      throw error;
    }
  }
}

Step 3: Create Address Lookup Connector​

Create src/connectors/address_lookup.json:

{
  "key": "address_lookup",
  "name": "Address Lookup Service",
  "description": "Google Maps-compatible address geocoding",
  "type": "rest_api",
  "configuration": {
    "baseUrl": "https://maps.googleapis.com/maps/api",
    "headers": {
      "Accept": "application/json"
    },
    "urlParameters": {
      "key": "%%GOOGLE_MAPS_API_KEY%%"
    }
  }
}

Environment Variable:

GOOGLE_MAPS_API_KEY=your_google_maps_api_key

Using Address Lookup​

Create src/action-types/validateAddress.ts:

import { Record, String } from 'runtypes';
import { S9AutomationActionType } from '@april9au/stack9-sdk';

export class ValidateAddress extends S9AutomationActionType {
  key = 'validate_address';
  name = 'Validate Address';

  async exec(params: any) {
    const { address } = params;
    const { connectors, logger } = this.context;

    try {
      const addressLookup = connectors['address_lookup'];

      // Geocode address
      const response = await addressLookup.call({
        method: 'GET',
        path: '/geocode/json',
        params: {
          address: address,
        },
      });

      if (response.status === 'OK' && response.results.length > 0) {
        const result = response.results[0];

        // Extract address components
        const components = result.address_components;
        const parsed = {
          formatted_address: result.formatted_address,
          street_number: this.getComponent(components, 'street_number'),
          route: this.getComponent(components, 'route'),
          city: this.getComponent(components, 'locality'),
          state: this.getComponent(components, 'administrative_area_level_1'),
          postal_code: this.getComponent(components, 'postal_code'),
          country: this.getComponent(components, 'country'),
          latitude: result.geometry.location.lat,
          longitude: result.geometry.location.lng,
        };

        logger.info(`Address validated: ${parsed.formatted_address}`);

        return {
          success: true,
          valid: true,
          parsed_address: parsed,
        };
      } else {
        return {
          success: true,
          valid: false,
          error: 'Address not found',
        };
      }
    } catch (error) {
      logger.error(`Address validation failed: ${error.message}`);
      throw error;
    }
  }

  private getComponent(components: any[], type: string): string {
    const component = components.find((c) => c.types.includes(type));
    return component?.long_name || '';
  }
}

Step 4: Create SMS Service Connector (Basic Auth)​

Create src/connectors/sms_service.json:

{
  "key": "sms_service",
  "name": "SMS Service",
  "description": "Twilio-compatible SMS delivery",
  "type": "rest_api",
  "configuration": {
    "baseUrl": "https://api.twilio.com/2010-04-01",
    "authentication": {
      "type": "basic",
      "username": "%%TWILIO_ACCOUNT_SID%%",
      "password": "%%TWILIO_AUTH_TOKEN%%"
    },
    "headers": {
      "Content-Type": "application/x-www-form-urlencoded"
    }
  }
}

Environment Variables:

TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_auth_token_here

Using SMS Connector​

Create src/action-types/sendSMS.ts:

import { Record, String } from 'runtypes';
import { S9AutomationActionType } from '@april9au/stack9-sdk';
import querystring from 'querystring';

const SendSMSParams = Record({
  to: String,
  message: String,
});

export class SendSMS extends S9AutomationActionType {
  key = 'send_sms';
  name = 'Send SMS';

  async exec(params: any) {
    const validated = SendSMSParams.check(params);
    const { connectors, logger } = this.context;

    try {
      const smsService = connectors['sms_service'];
      const accountSid = process.env.TWILIO_ACCOUNT_SID;

      // Twilio uses form-encoded data
      const body = querystring.stringify({
        To: validated.to,
        From: process.env.TWILIO_PHONE_NUMBER,
        Body: validated.message,
      });

      const response = await smsService.call({
        method: 'POST',
        path: `/Accounts/${accountSid}/Messages.json`,
        body: body,
        headers: {
          'Content-Type': 'application/x-www-form-urlencoded',
        },
      });

      logger.info(`SMS sent to ${validated.to}`);

      return {
        success: true,
        message_sid: response.sid,
        message: 'SMS sent successfully',
      };
    } catch (error) {
      logger.error(`SMS sending failed: ${error.message}`);
      throw error;
    }
  }
}

Step 5: Create CRM Connector (OAuth2)​

Create src/connectors/crm_api.json:

{
  "key": "crm_api",
  "name": "CRM API",
  "description": "Salesforce-compatible CRM integration",
  "type": "rest_api",
  "configuration": {
    "baseUrl": "https://api.crm-system.com/v2",
    "headers": {
      "Content-Type": "application/json",
      "Accept": "application/json"
    },
    "authentication": {
      "type": "bearer",
      "bearer": "%%CRM_ACCESS_TOKEN%%"
    }
  }
}

Environment Variables:

CRM_ACCESS_TOKEN=your_oauth2_access_token

Note: OAuth2 client credentials flow is not directly supported in the connector configuration. You'll need to obtain and manage the access token separately.

Using CRM Connector​

Create src/action-types/syncContactToCRM.ts:

import { Record, Number } from 'runtypes';
import { S9AutomationActionType } from '@april9au/stack9-sdk';

export class SyncContactToCRM extends S9AutomationActionType {
  key = 'sync_contact_to_crm';
  name = 'Sync Contact to CRM';

  async exec(params: any) {
    const { customer_id } = params;
    const { db, connectors, logger } = this.context;

    try {
      // Fetch customer
      const customer = await db('customer')
        .where({ id: customer_id })
        .first();

      if (!customer) {
        throw new Error(`Customer ${customer_id} not found`);
      }

      const crmApi = connectors['crm_api'];

      // Check if contact already exists in CRM
      const existingContacts = await crmApi.call({
        method: 'GET',
        path: '/contacts',
        params: {
          email: customer.email,
        },
      });

      if (existingContacts.data && existingContacts.data.length > 0) {
        // Update existing contact
        const contactId = existingContacts.data[0].id;

        await crmApi.call({
          method: 'PUT',
          path: `/contacts/${contactId}`,
          body: {
            first_name: customer.first_name,
            last_name: customer.last_name,
            email: customer.email,
            phone: customer.phone,
            company: customer.company,
            status: customer.status,
            custom_fields: {
              stack9_customer_id: customer.id,
              last_synced: new Date().toISOString(),
            },
          },
        });

        logger.info(`Updated CRM contact ${contactId} for customer ${customer.email}`);

        return {
          success: true,
          crm_contact_id: contactId,
          action: 'updated',
        };
      } else {
        // Create new contact
        const newContact = await crmApi.call({
          method: 'POST',
          path: '/contacts',
          body: {
            first_name: customer.first_name,
            last_name: customer.last_name,
            email: customer.email,
            phone: customer.phone,
            company: customer.company,
            status: customer.status,
            custom_fields: {
              stack9_customer_id: customer.id,
              created_in_stack9: new Date().toISOString(),
            },
          },
        });

        logger.info(`Created CRM contact ${newContact.id} for customer ${customer.email}`);

        return {
          success: true,
          crm_contact_id: newContact.id,
          action: 'created',
        };
      }
    } catch (error) {
      logger.error(`CRM sync failed: ${error.message}`);
      throw error;
    }
  }
}

Step 6: Error Handling Patterns​

Pattern 1: Retry with Exponential Backoff​

export class RetryableAction extends S9AutomationActionType {
  key = 'retryable_action';
  name = 'Retryable Action';

  async exec(params: any) {
    const maxRetries = 3;
    let lastError: Error;

    for (let attempt = 1; attempt <= maxRetries; attempt++) {
      try {
        const result = await this.makeApiCall(params);
        return result;
      } catch (error) {
        lastError = error;

        if (attempt < maxRetries) {
          // Exponential backoff: 1s, 2s, 4s
          const delay = Math.pow(2, attempt - 1) * 1000;
          this.context.logger.warn(
            `Attempt ${attempt} failed, retrying in ${delay}ms`
          );
          await this.sleep(delay);
        }
      }
    }

    throw new Error(`Failed after ${maxRetries} attempts: ${lastError.message}`);
  }

  private async makeApiCall(params: any) {
    // Your API call here
  }

  private sleep(ms: number): Promise<void> {
    return new Promise((resolve) => setTimeout(resolve, ms));
  }
}

Pattern 2: Circuit Breaker​

class CircuitBreaker {
  private failures = 0;
  private lastFailureTime = 0;
  private readonly threshold = 5;
  private readonly timeout = 60000; // 1 minute

  async call<T>(fn: () => Promise<T>): Promise<T> {
    // Check if circuit is open
    if (this.failures >= this.threshold) {
      const timeSinceLastFailure = Date.now() - this.lastFailureTime;
      if (timeSinceLastFailure < this.timeout) {
        throw new Error('Circuit breaker is open - too many failures');
      }
      // Reset after timeout
      this.failures = 0;
    }

    try {
      const result = await fn();
      this.failures = 0; // Reset on success
      return result;
    } catch (error) {
      this.failures++;
      this.lastFailureTime = Date.now();
      throw error;
    }
  }
}

export class ResilientAction extends S9AutomationActionType {
  private circuitBreaker = new CircuitBreaker();

  async exec(params: any) {
    return this.circuitBreaker.call(async () => {
      const { connectors } = this.context;
      return await connectors['external_api'].call({
        method: 'GET',
        path: '/data',
      });
    });
  }
}

Pattern 3: Timeout Handling​

async function withTimeout<T>(
  promise: Promise<T>,
  timeoutMs: number,
  errorMessage = 'Operation timed out'
): Promise<T> {
  let timeoutHandle: NodeJS.Timeout;

  const timeoutPromise = new Promise<T>((_, reject) => {
    timeoutHandle = setTimeout(() => {
      reject(new Error(errorMessage));
    }, timeoutMs);
  });

  try {
    return await Promise.race([promise, timeoutPromise]);
  } finally {
    clearTimeout(timeoutHandle!);
  }
}

export class TimeoutAction extends S9AutomationActionType {
  async exec(params: any) {
    const { connectors } = this.context;

    try {
      // 10 second timeout
      const result = await withTimeout(
        connectors['slow_api'].call({ method: 'GET', path: '/data' }),
        10000,
        'API call timed out after 10 seconds'
      );

      return { success: true, data: result };
    } catch (error) {
      if (error.message.includes('timed out')) {
        this.context.logger.error('API timeout - using cached data');
        // Fallback to cached data
        return { success: true, cached: true, data: await this.getCachedData() };
      }
      throw error;
    }
  }
}

Pattern 4: Rate Limiting​

class RateLimiter {
  private requests: number[] = [];
  private readonly limit: number;
  private readonly windowMs: number;

  constructor(limit: number, windowMs: number) {
    this.limit = limit;
    this.windowMs = windowMs;
  }

  async throttle(): Promise<void> {
    const now = Date.now();

    // Remove old requests outside the window
    this.requests = this.requests.filter((time) => now - time < this.windowMs);

    if (this.requests.length >= this.limit) {
      const oldestRequest = this.requests[0];
      const waitTime = this.windowMs - (now - oldestRequest);
      await new Promise((resolve) => setTimeout(resolve, waitTime));
      return this.throttle(); // Recursive retry
    }

    this.requests.push(now);
  }
}

export class RateLimitedAction extends S9AutomationActionType {
  // 100 requests per minute
  private rateLimiter = new RateLimiter(100, 60000);

  async exec(params: any) {
    await this.rateLimiter.throttle();

    const { connectors } = this.context;
    return await connectors['api'].call({
      method: 'GET',
      path: '/resource',
    });
  }
}

Step 7: Testing External Integrations​

Test Payment Processing​

# Test payment action
curl -X POST http://localhost:3000/api/test-action \
  -H "Content-Type: application/json" \
  -d '{
    "actionType": "process_payment",
    "params": {
      "amount": 99.99,
      "currency": "USD",
      "customer_email": "test@example.com",
      "order_id": 1
    }
  }'

Test Email Sending​

# Test email action
curl -X POST http://localhost:3000/api/test-action \
  -H "Content-Type: application/json" \
  -d '{
    "actionType": "send_email",
    "params": {
      "to": "recipient@example.com",
      "subject": "Test Email",
      "template": "welcome",
      "data": {
        "first_name": "John"
      }
    }
  }'

Test Address Validation​

# Test address lookup
curl -X POST http://localhost:3000/api/test-action \
  -H "Content-Type: application/json" \
  -d '{
    "actionType": "validate_address",
    "params": {
      "address": "1600 Amphitheatre Parkway, Mountain View, CA"
    }
  }'

Step 8: Advanced Patterns​

Pattern 1: Webhook Signature Verification​

import crypto from 'crypto';

export class VerifyWebhook extends S9AutomationActionType {
  key = 'verify_webhook';
  name = 'Verify Webhook Signature';

  async exec(params: any) {
    const { payload, signature, secret } = params;

    // Calculate expected signature
    const expectedSignature = crypto
      .createHmac('sha256', secret)
      .update(JSON.stringify(payload))
      .digest('hex');

    const isValid = crypto.timingSafeEqual(
      Buffer.from(signature),
      Buffer.from(expectedSignature)
    );

    if (!isValid) {
      throw new Error('Invalid webhook signature');
    }

    return {
      success: true,
      valid: true,
      message: 'Webhook signature verified',
    };
  }
}

Pattern 2: Pagination Handling​

export class FetchAllRecords extends S9AutomationActionType {
  async exec(params: any) {
    const { connectors, logger } = this.context;
    const api = connectors['external_api'];

    let allRecords = [];
    let page = 1;
    let hasMore = true;

    while (hasMore) {
      logger.info(`Fetching page ${page}`);

      const response = await api.call({
        method: 'GET',
        path: '/records',
        params: {
          page: page,
          per_page: 100,
        },
      });

      allRecords = allRecords.concat(response.data);
      hasMore = response.has_more;
      page++;

      // Safety limit
      if (page > 100) {
        logger.warn('Reached maximum page limit (100)');
        break;
      }
    }

    logger.info(`Fetched total of ${allRecords.length} records`);

    return {
      success: true,
      total_records: allRecords.length,
      records: allRecords,
    };
  }
}

Pattern 3: Bulk API Operations​

export class BulkSync extends S9AutomationActionType {
  async exec(params: any) {
    const { db, connectors, logger } = this.context;

    // Fetch records that need syncing
    const records = await db('customer')
      .where('needs_sync', true)
      .limit(1000);

    logger.info(`Syncing ${records.length} customers`);

    // Batch records (API allows max 100 per request)
    const batchSize = 100;
    const batches = [];

    for (let i = 0; i < records.length; i += batchSize) {
      batches.push(records.slice(i, i + batchSize));
    }

    const api = connectors['external_api'];
    let successCount = 0;
    let errorCount = 0;

    for (const [index, batch] of batches.entries()) {
      try {
        logger.info(`Processing batch ${index + 1}/${batches.length}`);

        await api.call({
          method: 'POST',
          path: '/bulk/customers',
          body: {
            customers: batch.map((c) => ({
              email: c.email,
              name: `${c.first_name} ${c.last_name}`,
              status: c.status,
            })),
          },
        });

        // Mark as synced
        await db('customer')
          .whereIn('id', batch.map((c) => c.id))
          .update({ needs_sync: false, last_synced: new Date() });

        successCount += batch.length;
      } catch (error) {
        logger.error(`Batch ${index + 1} failed: ${error.message}`);
        errorCount += batch.length;
      }
    }

    return {
      success: true,
      synced: successCount,
      failed: errorCount,
      total: records.length,
    };
  }
}

Best Practices​

Security​

  1. Never hardcode credentials - Always use environment variables
  2. Validate webhook signatures - Verify requests from external services
  3. Use HTTPS only - Never send credentials over HTTP
  4. Rotate API keys regularly - Implement key rotation policies
  5. Limit API key permissions - Use least privilege principle

Error Handling​

  1. Implement retries - Transient failures are common with APIs
  2. Use circuit breakers - Prevent cascading failures
  3. Set appropriate timeouts - Don't let operations hang forever
  4. Log all errors - Include request/response details
  5. Provide fallbacks - Gracefully degrade when APIs are down

Performance​

  1. Cache responses - Cache frequently accessed data
  2. Use batch operations - Reduce number of API calls
  3. Implement rate limiting - Respect API rate limits
  4. Paginate large datasets - Don't fetch everything at once
  5. Use async operations - Don't block on long-running calls

Monitoring​

  1. Track API usage - Monitor request counts and quotas
  2. Log response times - Identify slow APIs
  3. Alert on failures - Set up alerts for critical integrations
  4. Monitor rate limits - Don't hit rate limit ceilings
  5. Track costs - Many APIs charge per request

Troubleshooting​

Authentication Failures​

Problem: 401 Unauthorized responses

Solutions:

  1. Verify API key/token is correct
  2. Check environment variable is set
  3. Ensure auth header format matches API requirements
  4. Check if token has expired (OAuth2)
  5. Verify API key has required permissions

Timeout Errors​

Problem: Requests timing out

Solutions:

  1. Increase timeout in connector config
  2. Check if external API is down (status page)
  3. Implement retry logic with exponential backoff
  4. Use async operations for long requests
  5. Contact API support if persistent

Rate Limit Exceeded​

Problem: 429 Too Many Requests

Solutions:

  1. Implement rate limiting in your code
  2. Add delays between requests
  3. Use batch endpoints when available
  4. Cache responses to reduce calls
  5. Upgrade API plan if needed

Invalid Response Format​

Problem: Unexpected API response structure

Solutions:

  1. Check API documentation for response format
  2. Log full response for debugging
  3. Handle different response formats
  4. Check API version hasn't changed
  5. Validate response schema

SSL/TLS Errors​

Problem: Certificate verification failures

Solutions:

  1. Ensure correct base URL (https://)
  2. Check server certificate is valid
  3. Update Node.js version if outdated
  4. For testing only: disable cert verification (not production!)
  5. Contact API provider about certificate issues

Next Steps​

You've mastered external API integration! Continue learning: