Background Jobs Architecture: Cron vs Queue vs Workflow

Photo by Alberto Rodríguez on Unsplash

Your background jobs architecture comes down to one question: what happens when the job fails halfway through? A cron job answers "nothing, we try again at the next tick." A queue answers "we redeliver the message and you run it again from the top." A durable workflow answers "we resume from the step that broke." Those three answers cost wildly different amounts of setup, and picking the expensive one too early is the most common over-engineering mistake I see in indie SaaS. Most founders need a cron job. Some need a queue. Very few need a workflow engine on day one.

TL;DR

• Cron is for work on a schedule that nobody is waiting on. If a missed run self-heals at the next tick, cron is enough.
• A queue is for work triggered by an event that must not be lost. It buys you durability and retries; it costs you idempotency work.
• A durable workflow is for multi-step processes where re-running step one is expensive or wrong. It buys you resumability from the failure point.
• Queue delivery is at-least-once, not exactly-once. Every consumer you write must be safe to run twice.
• Start with cron. Move to a queue when a lost job costs you money. Move to a workflow when a retry costs you money.

What each of these three things actually guarantees

The names get used loosely, so here are the definitions I hold people to.

A cron job is a schedule that fires a function at a fixed time. That's the entire contract. There is no queue behind it, no retry, no memory of the last run. If the function crashes, the run is gone. On Vercel, cron jobs invoke a normal Vercel Function, so the function's timeout and pricing limits are the cron job's limits too.

A queue is a durable log of messages that are delivered to a consumer at least once. Producers publish; consumers process and acknowledge. If the consumer crashes or times out before acknowledging, the message becomes visible again and gets redelivered. The message survives your deploy, your crash, and your function timeout.

A durable workflow is a function whose completed steps are checkpointed, so a retry skips the work that already succeeded. Each step's result is persisted outside the function's execution context. When the function is re-executed after a failure, the SDK injects the stored results for finished steps and only runs the one that broke.

The distinction that matters most is the third one, and it's easiest to see with a concrete failure. Say you have a five-step onboarding job: create the account record, provision storage, charge the card, send the welcome email, notify Slack. Step three succeeds. Step four fails because Resend is having a bad afternoon.

With cron or a plain queue consumer, the retry runs the whole handler again — and you just charged the card twice. With a durable workflow, steps one through three are memoized and skipped, and only the email step retries. That is the entire value proposition, and it's why "just add retries" is not the same thing as durability.

When is a cron job enough?

More often than founders assume. Cron is the right answer when the work is schedule-driven and self-healing — meaning nobody triggered it, nobody is waiting on it, and a missed run causes no permanent damage because the next run picks up whatever was left behind.

Real examples from my own app that are correctly cron jobs:

• Expiring stale trial records nightly.
• Recomputing a dashboard metrics rollup every hour.
• Sweeping abandoned upload files older than 24 hours.
• Pinging a health-check endpoint.

Every one of those is idempotent by construction — running it twice produces the same state — and every one self-heals. If tonight's trial sweep doesn't run, tomorrow's catches the same records plus one more day's worth.

The plan limits are worth knowing before you design around them, because they're a real constraint on Vercel's free tier. Every plan gets 100 cron jobs per project, but the frequency differs sharply:

| | Cron jobs per project | Minimum interval | Scheduling precision | | --- | --- | --- | --- | | Hobby | 100 | Once per day | Per-hour (±59 min) | | Pro | 100 | Once per minute | Per-minute | | Enterprise | 100 | Once per minute | Per-minute |

On Hobby, an expression like */30 * * * * doesn't just get throttled — it fails deployment outright with "Hobby accounts are limited to daily cron jobs." And a daily job set for 0 1 * * * fires somewhere between 1:00 and 1:59 am. If you're building on the free tier, design for one imprecise daily sweep, not a tight polling loop. The mechanics of wiring one up are in the Vercel cron jobs tutorial.

Where cron stops being enough is when it becomes a polling mechanism — a job that runs every minute to check whether something happened. That's the pattern I argue against in webhooks vs polling for integrations, and the same logic applies internally. If you're polling your own database for new rows to process, you've built a bad queue. Build a real one.

What a queue buys you, and what it charges

You move to a queue when a job is event-driven and losing it costs you something. A user uploaded a CSV. A payment succeeded. A webhook arrived. Nobody can wait for the next cron tick, and dropping the work means a customer notices.

The core guarantee is durability. Vercel Queues writes every accepted message synchronously to three availability zones before the publish call returns, which means once you get an acknowledgment, the work is going to happen even if your function dies mid-processing, even if you deploy over it, even if a zone goes down.

The lifecycle is worth internalizing because it explains every weird behavior you'll hit later:

flowchart TD
    S["Publish message"] --> P["Pending (delay)"]
    P --> V["Visible"]
    V -->|"Consumer receives"| IF["In-flight (leased)"]
    IF -->|"Acknowledge"| D["Removed"]
    IF -->|"Lease expires"| V
    V -->|"TTL expires"| X["Deleted, never processed"]

Three things fall out of that diagram that trip people up.

Delivery is at-least-once, not exactly-once. A message can be delivered more than once — most commonly when your consumer finishes the work but doesn't acknowledge before the visibility timeout expires. The system assumes the delivery failed and sends it again. The default visibility timeout is 60 seconds, configurable from 0 to 3,600. This is the price of the durability guarantee, and it is not optional: every queue consumer you write must be idempotent. Deduplicate on a message ID, or make the operation naturally idempotent by setting a value rather than incrementing one.

Messages expire. Retention is configurable per message from 60 seconds to 7 days, defaulting to 24 hours. A message that hits its TTL is deleted whether or not anyone processed it. If your consumer has been broken for two days, you didn't queue up two days of work — you lost it.

Ordering is approximate. Vercel Queues delivers in approximate write order with no FIFO guarantee, and retried messages get lower priority than new ones. If your job assumes "process these in sequence," a queue is the wrong shape and you want a workflow.

The other thing to plan for is poison messages — a message that fails every single time because the payload is malformed. Vercel Queues has no built-in dead-letter queue; you handle it in the consumer's retry callback by acknowledging the message once the delivery count crosses your threshold. Retries respect your configured delay for the first 32 attempts, then the system forces exponential backoff. The saving grace is that new messages are always prioritized over retried ones, so a poisoned message drifts to the back on its own instead of blocking the line.

One genuinely nice detail if you're deploying continuously: topics are partitioned by deployment ID by default, so a new deployment produces and consumes its own messages. You can change a payload schema without worrying that the old consumer will choke on the new format while the rollout drains.

When you actually need a durable workflow

You need a workflow engine when re-running a completed step is expensive, irreversible, or wrong — or when the process spans more time than any single function invocation can cover.

The canonical cases:

• Anything that charges money mid-sequence. Retrying a five-step job that includes a Stripe charge means retrying the charge. The dunning and failed-payment recovery flows I run are workflows for exactly this reason.
• Anything with a wait in the middle. "Send the welcome email, wait three days, send the tips email if they haven't logged in." Vercel Queues caps message delay at 7 days; a workflow's sleep() has no such practical ceiling. My lifecycle email sequences live here.
• Fan-out with a join. Process 500 CSV rows in parallel, then send one summary email when all of them finish. Doing this with raw queue messages means building your own completion tracking.
• Long-running imports. A CSV import that parses, normalizes, validates, and writes is the textbook durable-workflow example.

The mechanism is memoization. In Inngest's execution model, each step.run() gets a unique ID, its result is persisted after it succeeds, and every subsequent execution of the function looks up that ID and injects the stored result instead of re-running the code. Each step also carries its own independent retry counter — so a flaky third-party API in step four can retry five times without touching steps one through three.

The catch, and it's a real one: because completed steps are skipped by looking up their IDs, any non-deterministic work has to live inside a step.run() call. A database query or an API call sitting loose in the function body will execute on every replay. This is the single most common bug founders hit with durable workflows, and it's silent — the code looks fine and just does the work three extra times. I walk through the setup and the gotchas in background jobs with Inngest.

It's also worth knowing that "durable workflow" isn't one design. Inngest uses step-based memoization with standard language primitives; Temporal uses deterministic replay, re-executing your whole workflow function from the top on each step and relying on an event history to skip completed work — which means strict determinism rules you have to follow, plus a Temporal Server cluster and separate worker processes to run. For a solo founder, the memoization model on your existing compute is the far cheaper entry point.

Putting it together

Here's the decision I'd actually make, in order:

| | Cron | Queue | Durable workflow | | --- | --- | --- | --- | | Trigger | Schedule | Event | Event or schedule | | Survives a crash | No | Yes | Yes | | Retries | No | Yes, whole handler | Yes, per step | | Resumes from failure point | No | No | Yes | | Handles waits over days | No | Up to 7 days | Yes | | Ordering | N/A | Approximate | Sequential by construction | | Idempotency required | Yes (by design) | Yes (at-least-once) | Per step | | Setup cost | Minutes | An afternoon | A day, plus new concepts |

And the flow I'd follow:

flowchart TD
    A["New background job"] --> B{"Triggered by a schedule<br/>or an event?"}
    B -->|Schedule| C{"Does a missed run<br/>self-heal next tick?"}
    C -->|Yes| D["Cron job"]
    C -->|No| E["Queue with a cron producer"]
    B -->|Event| F{"Is re-running the whole<br/>handler safe and cheap?"}
    F -->|Yes| G["Queue"]
    F -->|No| H{"Multiple steps, money,<br/>or waits over minutes?"}
    H -->|Yes| I["Durable workflow"]
    H -->|No| G

Two anti-patterns to name explicitly, because I've shipped both.

Don't do slow work inside the request. If a signup handler sends three emails and calls two APIs before returning, your signup is as slow and as fragile as the slowest third party in that chain. Push it to a queue and return immediately. This is the same reasoning behind acknowledging Stripe webhooks fast and processing after, which I unpack in Stripe webhook hell.

Don't reach for a workflow engine because it sounds more serious. A workflow engine is a new dependency, a new dashboard, a new failure mode, and a new set of determinism rules for your AI assistant to get subtly wrong. If your job is one step, a queue is the correct tool and adding a workflow buys you nothing.

The honest read on cost: cron is free-ish because it's just a function invocation. Queues and workflows add per-message or per-run pricing on top of the compute you were already paying for. For a pre-revenue app doing hundreds of jobs a day, all three are rounding errors. At tens of thousands a day it starts to matter, which is the same threshold where rate limiting starts to matter — and for the same reason.

Frequently asked questions

Can I just use a database table as a queue?

You can, and for low volume it works fine — insert rows, have a cron job claim and process them. The problems show up at scale: you have to build your own visibility timeout, your own retry backoff, and your own locking to stop two workers grabbing the same row. That's a weekend of work to rebuild badly what a managed queue gives you in an afternoon.

What's the difference between a queue and a durable workflow, in one sentence?

A queue makes sure a unit of work happens; a durable workflow makes sure a sequence of work happens without redoing the parts that already succeeded.

Do I need idempotency if my queue promises at-least-once delivery?

Yes — that's exactly why you need it. At-least-once means "one or more times," so your consumer will occasionally see the same message twice during timeouts or infrastructure failovers. Deduplicate on a message ID or make the operation naturally idempotent.

How do I test background jobs locally?

Cron is easiest: expose the handler as a route and hit it manually. Queues and workflows generally ship a local dev server that runs the same delivery semantics against your machine. The thing worth actually testing is the retry path — deliberately throw an error in the middle and confirm the job recovers the way you expect.

What about long-running jobs that exceed my function timeout?

Split them. On a queue, publish one message per unit of work rather than one message for the whole batch. On a workflow, put each chunk in its own step — the step boundary is also the checkpoint boundary, so a timeout mid-batch only loses the current chunk.

Should I let my AI assistant pick the architecture?

Let it implement, not decide. The choice between cron, queue, and workflow depends on business consequences an assistant can't see — whether a duplicate charge matters, whether a missed run self-heals. Decide that yourself, then hand over a specific instruction. Vague prompts here produce plausible code with the wrong failure semantics.

The short version

Good background jobs architecture is mostly about resisting the upgrade. Start with cron, because most scheduled work self-heals and a cron job takes five minutes. Move to a queue the first time losing a job would cost you a customer, and write every consumer as if it will run twice, because it will. Move to a durable workflow only when re-running a completed step is expensive, irreversible, or spans days.

The question that routes you correctly every time is still the one at the top: what happens when this fails halfway through? Answer that honestly and the tool picks itself.

The Coding Capybaras marketplace has copy-paste AI prompts for wiring queues and background jobs into a Next.js, Supabase, and Stripe app — including the idempotency and retry handling that the tutorials usually skip.