Deadlock in Automated Manufacturing System

#Automation

Deep Chauhan@deepJun 28, 2026

A production line with several robotic arms and conveyor belts. Each machine can request exclusive access to shared resources such as tool changers, part feeders, and transport tracks. The control software runs on a PLC network using a custom scheduler. Resources are allocated dynamically based on job queues. The system operates 24/7 with high throughput requirements and minimal downtime tolerance.

The Bug / Incident

Symptom: At 14:32:07 the line halted. Machine A reported "waiting for resource ToolChanger1". Machine B reported "waiting for resource Conveyor3". Machine C reported "waiting for resource PartFeeder2". The resource monitor displayed a cycle: ToolChanger1 -> Conveyor3 -> PartFeeder2 -> ToolChanger1. No further commands were executed and the PLC error log showed repeated timeout messages without any explicit exception.

The Investigation / Logic

1. Identify the resources involved: each machine requests exclusive locks on the resources it needs for its current task. 2. Observe that the waiting messages form a circular wait condition, which is one of the four Coffman conditions for deadlock. 3. Verify that the other conditions (mutual exclusion, hold and wait, no preemption) are also present in the system, confirming a classic deadlock scenario. 4. Examine the resource allocation graph at the moment of failure; the graph contains a single strongly connected component with three nodes, indicating a cycle. 5. Consider why the deadlock was not prevented: the scheduling algorithm allows machines to request resources in any order, so the safe ordering rule is violated. 6. Determine that detection is possible because the monitor can periodically scan the allocation graph for cycles. 7. Conclude that recovery must break the cycle, either by preempting a resource, rolling back a job, or forcing a timeout, to allow the remaining machines to continue.

The Fix / Resolution

The Fix / Solution

Implement a three‑layer strategy. First, enforce a global ordering of resource acquisition (e.g., assign each resource a numeric rank and require machines to request resources in ascending order) to prevent circular wait. Second, add a deadlock detection thread that periodically builds the resource allocation graph and runs a depth‑first search to locate cycles; when a cycle is found, log the involved machines and resources. Third, apply a recovery policy: choose the lowest‑priority job in the cycle, abort or roll back its current task, release its held resources, and notify the operator. Optionally, configure a timeout on each lock request so that a machine automatically releases its resources after a configurable period, forcing the cycle to break without manual intervention.