Zeebe job retries and incidents in Camunda 8, explained

A job that “keeps retrying” is one of the most common things new Camunda 8 developers get stuck on. The behaviour is predictable once you know the rules Zeebe follows. This guide walks through them, with Java examples.

The job lifecycle

When a process reaches a service task, Zeebe creates a job of the task’s type. A worker subscribed to that type activates the job — taking a lock on it for a timeout period — does the work, and then reports one of three outcomes:

• Complete — the work succeeded; the process moves on, optionally with new variables.
• Fail — a technical problem; Zeebe decrements the job’s retries and makes it available again.
• Throw a BPMN error — a business outcome the process model handles, such as “payment declined”.

If the worker does none of these before the timeout expires, Zeebe assumes the worker died and makes the job available to be activated again.

Configuring retries

The number of attempts is set on the task definition in your BPMN model. If you don’t set it, Zeebe uses a default of three.

<bpmn:serviceTask id="charge" name="Charge card">
  <bpmn:extensionElements>
    <zeebe:taskDefinition type="charge-card" retries="5" />
  </bpmn:extensionElements>
</bpmn:serviceTask>

In Modeler you’ll find this as the Retries field on the task. Pick a number based on how transient your failures are: a flaky downstream API deserves more attempts than a validation error that will never succeed.

Failing a job with backoff

When you fail a job you decide how many retries remain and, optionally, how long Zeebe should wait before offering it again. Without a backoff, a failing job is retried immediately — which is how a short outage turns into all retries burned in seconds.

client.newFailCommand(job)
    .retries(job.getRetries() - 1)
    .retryBackoff(Duration.ofSeconds(30))
    .errorMessage("Payment API timed out")
    .send()
    .join();

With the Spring integration’s @JobWorker, throwing an exception from your handler fails the job and decrements its retries for you. Always include a clear error message: it is what you will see in Operate when the retries run out.

Timeouts and duplicate work

Zeebe guarantees a job is delivered at least once, not exactly once. If your handler takes longer than the job timeout — or your pod is killed after doing the work but before completing the job — the job is handed out again and the work runs twice. Two rules follow:

1. Set the job timeout comfortably above your slowest realistic execution time.
2. Make workers idempotent: check a business key (order ID, payment reference) before repeating side effects such as charging a card or sending an email.

BPMN errors vs technical failures

Not every problem is a failure. If the card is declined, retrying won’t help — the process should take a different path. Throw a BPMN error and catch it with an error boundary event in the model:

client.newThrowErrorCommand(job)
    .errorCode("PAYMENT_DECLINED")
    .errorMessage("Card declined by issuer")
    .send()
    .join();

The error code must match the error defined on the boundary event. Rule of thumb: fail for things that might succeed on retry (timeouts, 503s), throw a BPMN error for business outcomes the process should handle.

Incidents and how to resolve them

An incident means the process instance is stuck and needs a human. Common causes:

• A job failed and its retries reached zero.
• A FEEL expression failed — for example, it referenced a variable that doesn’t exist.
• A BPMN error was thrown that no boundary event catches.

To resolve one in Operate:

1. Open the instance and read the incident’s error message.
2. Fix the root cause — deploy the code fix, correct the data, or update a variable.
3. For job failures, give the job retries again, then retry the incident.

Retrying without fixing the cause simply creates the same incident again.

Debugging a job that keeps retrying

• Is the worker running and subscribed to the exact job type? A typo in the type means jobs are never activated at all.
• Is the handler exceeding the timeout? Look for the same job being activated repeatedly without a failure in your logs.
• Is it failing with no backoff? Retries disappear in seconds; add a backoff.
• Should it be a BPMN error instead? Retrying a business rejection only delays the incident.
• Are variables what you expect? Check the instance’s variables in Operate before and after the task.