Control the retry policy

Using Golem’s retry mechanism

Golem applies a retry mechanism to all agents. In case of a failure, Golem will automatically recover the agent to the point before the failure and retry the operation. An exponential backoff and an upper limit on the number of retries are applied.

If the maximum number of retries is reached, the agent will be marked as failed and no further invocations will be possible on it.

This mechanism is automatic and applied to all kinds of failures. To rely on it, throw an unhandled exception.

Golem will retry from the point the unhandled exception was thrown. It is possible that a previous operation’s unwanted result (if it did not end in an exception) has been already persisted, in which case it won’t be retried.

Customizing the retry policy

The retry policy which controls the maximum number of retries and the exponential backoff is a global configuration of the Golem servers, but it can be customized for each agent.

TypeScript

The @golemcloud/golem-ts-sdk package exports the withRetryPolicy function to temporarily install a named retry policy for a block of code:


import {withRetryPolicy, type NamedRetryPolicy} from "@golemcloud/golem-ts-sdk"
 
// Durations are expected as nanoseconds
const policy: NamedRetryPolicy = {
  name: "temporary-policy",
  priority: 1000,
  predicate: {
    nodes: [{tag: "pred-true"}],
  },
  policy: {
    nodes: [
      {
        tag: "count-box",
        val: {
          maxRetries: 3,
          inner: 1,
        },
      },
      {
        tag: "clamp-delay",
        val: {
          minDelay: 100_000_000n, // 100 milliseconds
          maxDelay: 2_000_000_000n, // 2 seconds
          inner: 2,
        },
      },
      {
        tag: "exponential",
        val: {
          baseDelay: 100_000_000n,
          factor: 1.5,
        },
      },
    ],
  },
}
 
const result: string = withRetryPolicy(policy, () => {
  // this closure runs with the custom retry policy
  return "hello"
})

The policy passed to withRetryPolicy is a NamedRetryPolicy, which contains a name, priority, predicate, and policy tree:


export type RetryPredicate = {
  nodes: PredicateNode[];
};
 
export type RetryPolicy = {
  nodes: PolicyNode[];
};
 
export type NamedRetryPolicy = {
  name: string;
  priority: number;
  predicate: RetryPredicate;
  policy: RetryPolicy;
};

Rust

The golem_rust crate exports the with_retry_policy function to temporarily install a named retry policy for a block of code:


use golem_rust::bindings::golem::api::retry::{
    ClampConfig, CountBoxConfig, ExponentialConfig, PolicyNode, PredicateNode,
};
use golem_rust::retry::{with_retry_policy, NamedRetryPolicy, RetryPolicy, RetryPredicate};
 
let policy = NamedRetryPolicy {
    name: "temporary-policy".to_string(),
    priority: 1000,
    predicate: RetryPredicate {
        nodes: vec![PredicateNode::PredTrue],
    },
    policy: RetryPolicy {
        nodes: vec![
            PolicyNode::CountBox(CountBoxConfig {
                max_retries: 3,
                inner: 1,
            }),
            PolicyNode::ClampDelay(ClampConfig {
                min_delay: 100_000_000u64, // 100 milliseconds
                max_delay: 2_000_000_000u64, // 2 seconds
                inner: 2,
            }),
            PolicyNode::Exponential(ExponentialConfig {
                base_delay: 100_000_000u64,
                factor: 1.5,
            }),
        ],
    },
};
 
with_retry_policy(policy, || {
    "hello".to_string()
})

The policy passed to with_retry_policy is a NamedRetryPolicy, which contains a name, priority, predicate, and policy tree:


pub struct RetryPredicate {
    pub nodes: Vec<PredicateNode>,
}
 
pub struct RetryPolicy {
    pub nodes: Vec<PolicyNode>,
}
 
pub struct NamedRetryPolicy {
    pub name: String,
    pub priority: u32,
    pub predicate: RetryPredicate,
    pub policy: RetryPolicy,
}

Scala

In Scala, golem.Guards exposes withRetryPolicy to temporarily install a named retry policy for a Future-returning block:


import golem.Guards
import golem.host.js.JsNamedRetryPolicy
 
import scala.concurrent.Future
import scala.scalajs.js
 
val policy = js.Dynamic
  .literal(
    "name" -> "temporary-policy",
    "priority" -> 1000,
    "predicate" -> js.Dynamic.literal(
      "nodes" -> js.Array(
        js.Dynamic.literal("tag" -> "pred-true")
      )
    ),
    "policy" -> js.Dynamic.literal(
      "nodes" -> js.Array(
        js.Dynamic.literal(
          "tag" -> "count-box",
          "val" -> js.Dynamic.literal(
            "maxRetries" -> 3,
            "inner" -> 1
          )
        ),
        js.Dynamic.literal(
          "tag" -> "clamp-delay",
          "val" -> js.Dynamic.literal(
            "minDelay" -> js.BigInt("100000000"),
            "maxDelay" -> js.BigInt("2000000000"),
            "inner" -> 2
          )
        ),
        js.Dynamic.literal(
          "tag" -> "exponential",
          "val" -> js.Dynamic.literal(
            "baseDelay" -> js.BigInt("100000000"),
            "factor" -> 1.5
          )
        )
      )
    )
  )
  .asInstanceOf[JsNamedRetryPolicy]
 
val result: Future[String] =
  Guards.withRetryPolicy(policy) {
    Future.successful("hello")
  }

The policy passed to Guards.withRetryPolicy is a JsNamedRetryPolicy facade:


import golem.host.js.{JsNamedRetryPolicy, JsRetryPolicyTree, JsRetryPredicate}
 
import scala.scalajs.js
 
@js.native
sealed trait JsNamedRetryPolicy extends js.Object {
  def name: String = js.native
  def priority: Int = js.native
  def predicate: JsRetryPredicate = js.native
  def policy: JsRetryPolicyTree = js.native
}

MoonBit

Assuming the golemcloud/golem_sdk/api package is imported as @api, use the with_retry_policy function to temporarily install a retry policy for a block of code:


let policy : @api.RetryPolicy = @api.RetryPolicy::{
  max_attempts: 3U,
  min_delay: 100_000_000UL, // 100 milliseconds
  max_delay: 2_000_000_000UL, // 2 seconds
  multiplier: 1.5,
  max_jitter_factor: None,
}
 
let result : String = @api.with_retry_policy(policy, fn() {
  // this callback runs with the custom retry policy
  "hello"
})

The policy passed to with_retry_policy is a RetryPolicy, which contains the retry limit and backoff settings:


pub(all) struct RetryPolicy {
  max_attempts : UInt
  min_delay : UInt64
  max_delay : UInt64
  multiplier : Double
  max_jitter_factor : Double?
} derive(Show, Eq)

MoonBit uses a simpler RetryPolicy struct with max_attempts, min_delay, max_delay, multiplier, and max_jitter_factor fields, rather than the tree-based NamedRetryPolicy used by TypeScript, Rust, and Scala. The named retry policy system described below applies to TypeScript, Rust, and Scala only.

Named retry policies in golem.yaml

Retry policies can also be defined declaratively in the application manifest (golem.yaml). These named retry policies are created in the environment during deployment, making them available to all agents without requiring code changes.


retryPolicyDefaults:
  local:
    - name: default-retry
      priority: 10
      predicate: "true"
      policy:
        countBox:
          maxRetries: 3
          inner:
            exponential:
              baseDelay: { secs: 1, nanos: 0 }
              factor: 2.0

Policy types reference

The following policy types can be used to construct retry policy trees:

Type	Fields	Description
`"immediate"`	—	Retry immediately
`"never"`	—	Never retry
`periodic`	`{ secs, nanos }`	Fixed delay between retries
`exponential`	`{ baseDelay, factor }`	Exponentially increasing delay
`fibonacci`	`{ first, second }`	Fibonacci-sequence delays
`countBox`	`{ maxRetries, inner }`	Limit total retry count
`timeBox`	`{ limit, inner }`	Limit total retry time
`clamp`	`{ minDelay, maxDelay, inner }`	Clamp delay range
`addDelay`	`{ delay, inner }`	Add fixed delay to inner policy
`jitter`	`{ factor, inner }`	Add random jitter
`filteredOn`	`{ predicate, inner }`	Apply only when predicate matches
`andThen`	`[policy1, policy2]`	Sequential composition
`union`	`[policy1, policy2]`	Union of two policies
`intersect`	`[policy1, policy2]`	Intersection of two policies

Predicates reference

Predicates control when a retry policy is applied. The following predicate types are available:

Type	Fields	Description
`"true"`	—	Always matches
`"false"`	—	Never matches
`propEq`	`{ property, value }`	Property equals value
`propNeq`	`{ property, value }`	Property not equal
`propGt` / `propGte`	`{ property, value }`	Greater than / greater or equal
`propLt` / `propLte`	`{ property, value }`	Less than / less or equal
`propExists`	string	Property exists
`propIn`	`{ property, values }`	Property in set
`propMatches`	`{ property, pattern }`	Regex match
`propStartsWith`	`{ property, prefix }`	Starts with prefix
`propContains`	`{ property, substring }`	Contains substring
`and`	`[pred1, pred2]`	Logical AND
`or`	`[pred1, pred2]`	Logical OR
`not`	predicate	Logical NOT

Predicate values are typed: { text: "..." }, { integer: 42 }, or { boolean: true }.