intermediateintegration Featured

API Polling with Change Detection

Periodically check an external system for changes and trigger workflows when new or updated records are detected. The reliable alternative when webhooks aren't available.

BPMN 2.0

On this page

Visual Flow

Rendering diagram…

When to Use This Pattern

Use API polling when:

The external system doesn't support webhooks or event notifications
You need to detect new records, updates, or deletions in a remote system
Webhook delivery is unreliable and you need a guaranteed catch-up mechanism
You're monitoring a batch file drop (SFTP, SharePoint folder, email inbox)

How It Works

Rendering diagram…

Implementation Guide

Step 1: Choose the Change Detection Strategy

Strategy	How It Works	Pros	Cons
Modified timestamp	Query records where last_modified > last_poll_time	Simple, widely supported	Clock skew between systems
Incremental ID	Query records where id > last_seen_id	No clock issues	Only works for new records, not updates
Hash comparison	Pull all records, compare hash to stored hashes	Catches any change	Expensive for large datasets
Change token/delta API	Use the system's built-in change tracking	Most efficient	Not all systems offer this

Step 2: Implement the Polling Loop

state = read_state()  // { last_poll: "2025-03-15T10:00:00Z" }

records = api.get("/records", {
    modified_after: state.last_poll,
    order_by: "modified_date ASC",
    limit: 100
})

FOR EACH record IN records:
    IF NOT already_processed(record.id, record.version):
        trigger_workflow(record)
        mark_processed(record.id, record.version)

state.last_poll = now()
save_state(state)

Step 3: Handle Overlapping Windows

API timestamps can have issues:

Use overlapping windows. Subtract 5 minutes from last_poll to catch records that were being written during the previous poll.
Deduplicate. Since you're over-fetching, you'll see some records twice. Keep a processed-records log and skip duplicates.
Use server-side timestamps. Don't trust the workflow's clock — use the API's response timestamp or the record's modified date.

Step 4: Manage the State Store

You need durable state between poll runs:

Option	Best For
SharePoint list item	Simple, visible
Database record	Reliable, fast
Workflow variable (K2)	Built-in to the platform
File on shared drive	Quick but fragile

The state should contain:

last_poll_timestamp — when the last successful poll completed
last_record_id — the highest ID seen (if using incremental IDs)
poll_status — running / complete / failed
items_processed_last_run — for monitoring

Step 5: Set the Polling Interval

Data Sensitivity	Poll Interval	Notes
Real-time critical	1–5 minutes	Consider webhooks instead
Business-day important	15–30 minutes	Good balance
Daily batch	Once per day	Match business schedule
Weekly reporting	Once per week	Monday morning batch

Tips & Best Practices

Warning

Always process records in chronological order (oldest first) and commit the state after each successful record. If you process newest first and crash halfway through, you'll skip the older ones forever.

Rate limit yourself. Don't poll every 30 seconds if the API has a 100-requests/minute limit and you have 50 other workflows calling the same API.
Handle empty polls gracefully. Most polls return zero changes. Don't log warnings for empty results — it's normal. Only alert if polling itself fails.
Implement circuit breaker. If the API returns errors for 3 consecutive polls, back off to a slower interval and alert an admin. Don't hammer a broken API.
Log the delta. Track how many changes each poll finds. A sudden spike (10x normal) might indicate a data migration or bulk update happening in the source system.

Related patterns

integration

Change Data Capture Stream

Stream row-level changes out of a database in near real-time using the transaction log. No polling, no app changes — downstream systems get inserts, updates, and deletes as they happen.

Advanced

integration

Reverse ETL

Push modelled data from your warehouse back into the SaaS tools that business teams use every day — CRM, marketing, support — so they can act on analytics without a BI detour.

Advanced

integration

Data Sync Bridge

Keep records synchronized between two or more systems. Handles create/update/delete propagation with conflict resolution. Essential for multi-system environments.

Advanced