UiPath Documentation
automation-suite
2.2510
true
Automation Suite on OpenShift installation guide

Durable Execution Service troubleshooting

Troubleshoot issues with the Durable Execution Service in Automation Suite on OpenShift.

The Kubernetes resources of the Durable Execution Service use the taas prefix, for example the taas-temporal-history deployment and the taas-temporal-app-secrets secret.

Maestro workflows are stuck or not starting​

Symptom: Maestro process instances remain in a pending or running state without progressing.

Cause: Durable Execution Service pods are unavailable or unhealthy, preventing workflow tasks from being dispatched or picked up by workers.

Resolution​

  1. Check the status of all Durable Execution Service pods:
    kubectl get pods -n uipath | grep taas
    kubectl get pods -n uipath | grep taas
    
  2. Identify any pods that are not in Running state and review their logs:
    kubectl logs -n uipath <pod-name> --tail=100
    kubectl logs -n uipath <pod-name> --tail=100
    
  3. Look for warn and error level entries. Common causes include database connectivity failures and resource exhaustion.
  4. If the taas-temporal-history pod is restarting, check memory usage. The history service is memory-intensive and its autoscaler is configured on memory utilization.

Durable Execution Service pods are crash-looping​

Symptom: One or more taas-temporal-* pods are in CrashLoopBackOff state.

Cause: The most common cause is a failure to connect to the PostgreSQL database on startup.

Resolution​

  1. Review the logs of the affected pod:
    kubectl logs -n uipath <pod-name> --previous
    kubectl logs -n uipath <pod-name> --previous
    
  2. Check for connection errors referencing the persistence or visibility store.
  3. Verify the PostgreSQL server is reachable from the cluster and that credentials in the taas-temporal-app-secrets secret are correct.
  4. Confirm that the PostgreSQL max_connections value meets the minimum requirement for your Durable Execution Service profile. The default PostgreSQL value of 100 is insufficient. For the required values per profile, refer to SQL database.

Workflow execution is slow​

Symptom: Maestro workflows start but tasks take longer than expected to be picked up or completed.

Cause: The Durable Execution Service is under-resourced, the PostgreSQL connection pool is exhausted, or the number of worker pollers is insufficient for the current load.

Resolution​

  1. Check Prometheus metrics for temporal_activity_schedule_to_start_latency_bucket and temporal_workflow_task_schedule_to_start_latency_bucket. High values indicate tasks are waiting in the queue longer than expected.
  2. Check that the PostgreSQL instance has sufficient available connections. Connection pool exhaustion causes persistence timeouts under load.
  3. Review pod resource usage for taas-temporal-history and taas-temporal-matching. If pods are CPU- or memory-constrained, consider adjusting resource limits or scaling the deployment.

Accessing the Durable Execution Service web UI​

The Durable Execution Service includes a web UI for diagnosing workflow executions. The web UI is based on the open-source Temporal web UI, so it shows Temporal branding in your browser. It runs in the taas-temporal-web deployment and is not exposed outside the cluster by default. Persistent or developer access to the web UI is not supported.

Note:

The web UI is a diagnostic tool for Automation Suite administrators, not a development interface. You can't use the web UI or the Durable Execution Service to run your own workflows, create namespaces, or connect your own Temporal SDK workers. To build, run, and monitor processes, use Maestro.

To open the web UI:

  1. Port-forward to the deployment:

    kubectl port-forward -n uipath deployment/taas-temporal-web 8008:8008
    kubectl port-forward -n uipath deployment/taas-temporal-web 8008:8008
    
  2. Open http://localhost:8008 in a browser. No sign-in is required when you access the web UI through port-forward.

Use the web UI to inspect individual workflow executions, view their execution history, and identify failed or timed-out activities.

Lock contention errors in Durable Execution Service logs​

Symptom: Durable Execution Service logs contain errors referencing lock contention or conflicting database writes.

Cause: Two Kubernetes clusters are connected to the same PostgreSQL persistence store simultaneously. Only one cluster can actively run the Durable Execution Service at a time.

Resolution​

In an active/passive deployment, scale all Durable Execution Service deployments on the passive cluster to zero replicas. For details on active/passive requirements for the Durable Execution Service, refer to Overview.

Durable Execution Service init job fails in a proxy environment​

Description​

When Maestro or the Durable Execution Service is enabled in a proxy environment, the Durable Execution Service init job fails with the following error:

Unable to get namespace default info ... context deadline exceeded
Unable to get namespace default info ... context deadline exceeded

The taas-temporal-frontend, taas-temporal-history, and taas-temporal-internal-frontend pod logs contain the following error:

failed to do connect handshake ... HTTP/1.1 503
failed to do connect handshake ... HTTP/1.1 503

Durable Execution Service components connect to each other by pod IP address. If the no_proxy list does not cover a pod IP address, the connection goes to the proxy, which cannot reach pod IP addresses and rejects the request. This happens when pod IP addresses fall outside the ranges in the no_proxy list, for example when the clusterNetwork CIDR falls outside 10.0.0.0/8.

Solution​

To address the issue, take the following steps:

  1. Add the pod CIDR ranges and the node subnet CIDR ranges to proxy.no_proxy in input.json. For details, refer to Proxy.

  2. Apply the configuration:

    uipathctl manifest apply input.json --versions versions.json
    uipathctl manifest apply input.json --versions versions.json
    

Was this page helpful?

Connect

Need help? Support(opens in new window)

Want to learn? UiPath Academy(opens in new window)

Have questions? UiPath Forum(opens in new window)

Stay updated