API Polling with Change Detection

Loading patterns…

When to Use This Pattern

Use API polling when:

The external system doesn't support webhooks or event notifications

You need to detect new records, updates, or deletions in a remote system

Webhook delivery is unreliable and you need a guaranteed catch-up mechanism

You're monitoring a batch file drop (SFTP, SharePoint folder, email inbox)

Implementation Guide

Step 1: Choose the Change Detection Strategy

Strategy	How It Works	Pros	Cons
Modified timestamp	Query records where last_modified > last_poll_time	Simple, widely supported	Clock skew between systems
Incremental ID	Query records where id > last_seen_id	No clock issues	Only works for new records, not updates
Hash comparison	Pull all records, compare hash to stored hashes	Catches any change	Expensive for large datasets
Change token/delta API	Use the system's built-in change tracking	Most efficient	Not all systems offer this

Step 2: Implement the Polling Loop

state = read_state()  // { last_poll: "2025-03-15T10:00:00Z" }

records = api.get("/records", {
    modified_after: state.last_poll,
    order_by: "modified_date ASC",
    limit: 100
})

FOR EACH record IN records:
    IF NOT already_processed(record.id, record.version):
        trigger_workflow(record)
        mark_processed(record.id, record.version)

state.last_poll = now()
save_state(state)

Step 3: Handle Overlapping Windows

API timestamps can have issues:

Use overlapping windows. Subtract 5 minutes from last_poll to catch records that were being written during the previous poll.
Deduplicate. Since you're over-fetching, you'll see some records twice. Keep a processed-records log and skip duplicates.
Use server-side timestamps. Don't trust the workflow's clock — use the API's response timestamp or the record's modified date.

Step 4: Manage the State Store

You need durable state between poll runs:

Option	Best For
SharePoint list item	Simple, visible
Database record	Reliable, fast
Workflow variable (K2)	Built-in to the platform
File on shared drive	Quick but fragile

The state should contain:

last_poll_timestamp — when the last successful poll completed
last_record_id — the highest ID seen (if using incremental IDs)
poll_status — running / complete / failed
items_processed_last_run — for monitoring

Step 5: Set the Polling Interval

Data Sensitivity	Poll Interval	Notes
Real-time critical	1–5 minutes	Consider webhooks instead
Business-day important	15–30 minutes	Good balance
Daily batch	Once per day	Match business schedule
Weekly reporting	Once per week	Monday morning batch

Tips & Best Practices

Warning

Always process records in chronological order (oldest first) and commit the state after each successful record. If you process newest first and crash halfway through, you'll skip the older ones forever.

Rate limit yourself. Don't poll every 30 seconds if the API has a 100-requests/minute limit and you have 50 other workflows calling the same API.

Handle empty polls gracefully. Most polls return zero changes. Don't log warnings for empty results — it's normal. Only alert if polling itself fails.

Implement circuit breaker. If the API returns errors for 3 consecutive polls, back off to a slower interval and alert an admin. Don't hammer a broken API.

Log the delta. Track how many changes each poll finds. A sudden spike (10x normal) might indicate a data migration or bulk update happening in the source system.

Visual Flow

When to Use This Pattern

How It Works

Implementation Guide

Tips & Best Practices

Related patterns

Event-Driven Webhook Handler

Change Data Capture Stream

Reverse ETL