---
title: "Follow up on failed runs"
description: "Find out why a script run or software deployment failed on some endpoints, fix it, retry only those endpoints, and get told about the next failure."
url: "https://support.outpostplatform.com/automation/follow-up-on-failed-runs/"
product: "depot"
type: "how-to"
status: "draft"
ai_summary: "Triage a failed script run or software deployment: open it, select Failed, read the failure groups (exit code and first error line), read Details, fix and approve the script, then Retry failed, which starts a linked run on the current approved version. For next time: On failure alerts or tickets, auto-retry, a staged rollout, or a Script or deployment fails workflow."
availability:
  plans: ["home-lab", "essentials", "enterprise", "msp"]
  platforms: ["windows", "linux", "macos", "freebsd", "openbsd"]
  permission: "command:execute"
  preview: false
source: "https://gitlab.com/outpostplatform/docs/-/edit/main/src/content/docs/automation/follow-up-on-failed-runs.md"
license: "CC BY 4.0"
---

# Follow up on failed runs

This page takes a script run or software deployment that failed on some
endpoints and gets it working on them, without running it again on the
endpoints where it already worked. It then sets up the next run so a failure
reaches you instead of waiting for you to look.

## Before you begin

- For a script run you need `command:execute`. To change and approve the
  script you also need `script:manage` and `script:approve`.
- For a software deployment you need `software:deploy`. To change the catalog
  entry you need `software:manage`.
- On the MSP plan, open the client's workspace first.

## Find out what failed

1. Open the run:
   - **A script run**: go to **Scripts** > **Batches** and select **Open** on
     the run.
   - **A software deployment**: go to **Software** > **Deployments**, choose
     **Rollouts**, and select the package name.
2. Select the **Failed** tile. The table lists only the endpoints whose latest
   attempt failed. Select **Timed out** next if that tile is not zero.
3. Read **What the failures have in common**. Each line is a group of
   endpoints that failed with the same exit code and the same first error
   line, with how many endpoints are in it, such as
   `Exit 1603: Fatal error during installation - 41 endpoints`. One large group
   usually means one cause.
4. Select the largest group. The table narrows to its endpoints.
5. Select **Details** on one of its rows to read the endpoint's full stdout and
   stderr.
6. To work through the list elsewhere, select **Export CSV**. The file holds
   the rows the current filters show.

The exit code is the script's own, or the installer's for software. A script
that exits with a code other than `0` when it works shows **Failed** until you
list that code in **Also counts as success**. **Succeeded (reboot)** is not a
failure: the endpoint needs a restart to finish.

## Fix the cause

- **A script**: open it in **Scripts** > **Library**, change it, select
  **Save**, and approve the new version. See
  [Write and approve a script](/automation/write-and-approve-scripts/).
- **A package**: fix the catalog entry, such as its silent switches,
  **Exit codes that mean success**, or detection checks. See
  [Software catalog and self-service](/automation/software/).
- **Endpoints that never came online**: these show **Expired**, not **Failed**.
  Nothing needs fixing. Retry them when they are online, or turn on **Wake
  offline endpoints first** for the retry's next run.

## Retry the failed endpoints

1. On the run's page, select **Retry failed**.
2. Under **Retry endpoints that ended as**, keep **Failed**, **Timed out**,
   and **Expired** selected. Select **Canceled** only if you want to send the
   endpoints you canceled as well.
3. In **Wait for offline endpoints**, choose how long to wait.
4. Select **Retry**.

The retry is a new run, linked to the first one, and its page opens. It runs
the script's current approved version, so the fix you approved is what goes
out. The page says so when the version changed, such as **Script changed from
v3 to v4; the retry uses v4.** It copies every other option from the first run
except a staged rollout, so it goes to every chosen endpoint at once.

If **Retry failed** cannot be selected, the console says why:
no endpoint failed, timed out, expired, or was canceled, or you lack the
permission.

## Hear about the next failure

Choose one or more of these under **Follow-up** the next time you start the
run. Each is described in
[Follow-up options](/automation/run-scripts/#follow-up-options).

| You want | Choose |
| --- | --- |
| An alert for each endpoint that fails, which resolves itself when that endpoint later succeeds | **On failure**: **Raise an alert per failed endpoint** |
| One ticket for the run, with each failed endpoint listed in internal notes (Enterprise and MSP) | **On failure**: **Open one ticket for the run** |
| Installs that collide with another installer to try again on their own | **Auto-retry transient failures**, with `1618` in **Exit codes to retry** |
| A notice in the bell when the run is done, whether or not it failed | **Notify me when it finishes** |
| A bad script or package to stop on a few endpoints instead of all of them | **Staged rollout** |

:::caution
**Raise an alert per failed endpoint** raises one alert for every endpoint
that fails. For a run on many endpoints, prefer one ticket or a staged
rollout.
:::

For a rule that covers every run without choosing it each time, build a
workflow with the **Script or deployment fails** trigger (Enterprise and MSP,
`automation:manage`). It fires once per run that finished with failures or
paused its staged rollout, and can create a ticket or notify people. For
example, a **Create a ticket** step with the subject
`{{batch.kind}} failed: {{batch.name}}` and the message
`{{batch.failed}} of {{batch.total}} endpoints {{batch.reason}}. {{batch.link}}`.
See [Build a workflow](/service-desk/build-a-workflow/) and the
[placeholders](/service-desk/workflow-reference/#placeholders).

## Verify

Open the retry. Once every endpoint has run it, the **Failed** and **Timed
out** tiles read `0`, and **What the failures have in common** is gone. The
first run keeps its own counts, and links to the retry under **Retried as**.

## Next steps

- [Run scripts on endpoints](/automation/run-scripts/)
- [Deploy software](/automation/deploy-software/)
- [Alert rules](/service-desk/alert-rules/)