Follow up on failed runs

This page takes a script run or software deployment that failed on some endpoints and gets it working on them, without running it again on the endpoints where it already worked. It then sets up the next run so a failure reaches you instead of waiting for you to look.

Before you begin

• For a script run you need command:execute. To change and approve the script you also need script:manage and script:approve.
• For a software deployment you need software:deploy. To change the catalog entry you need software:manage.
• On the MSP plan, open the client’s workspace first.

Find out what failed

1. Open the run:
   • A script run: go to Scripts > Batches and select Open on the run.
   • A software deployment: go to Software > Deployments, choose Rollouts, and select the package name.
2. Select the Failed tile. The table lists only the endpoints whose latest attempt failed. Select Timed out next if that tile is not zero.
3. Read What the failures have in common. Each line is a group of endpoints that failed with the same exit code and the same first error line, with how many endpoints are in it, such as Exit 1603: Fatal error during installation - 41 endpoints. One large group usually means one cause.
4. Select the largest group. The table narrows to its endpoints.
5. Select Details on one of its rows to read the endpoint’s full stdout and stderr.
6. To work through the list elsewhere, select Export CSV. The file holds the rows the current filters show.

The exit code is the script’s own, or the installer’s for software. A script that exits with a code other than 0 when it works shows Failed until you list that code in Also counts as success. Succeeded (reboot) is not a failure: the endpoint needs a restart to finish.

Fix the cause

• A script: open it in Scripts > Library, change it, select Save, and approve the new version. See Write and approve a script.
• A package: fix the catalog entry, such as its silent switches, Exit codes that mean success, or detection checks. See Software catalog and self-service.
• Endpoints that never came online: these show Expired, not Failed. Nothing needs fixing. Retry them when they are online, or turn on Wake offline endpoints first for the retry’s next run.

Retry the failed endpoints

1. On the run’s page, select Retry failed.
2. Under Retry endpoints that ended as, keep Failed, Timed out, and Expired selected. Select Canceled only if you want to send the endpoints you canceled as well.
3. In Wait for offline endpoints, choose how long to wait.
4. Select Retry.

The retry is a new run, linked to the first one, and its page opens. It runs the script’s current approved version, so the fix you approved is what goes out. The page says so when the version changed, such as Script changed from v3 to v4; the retry uses v4. It copies every other option from the first run except a staged rollout, so it goes to every chosen endpoint at once.

If Retry failed cannot be selected, the console says why: no endpoint failed, timed out, expired, or was canceled, or you lack the permission.

Hear about the next failure

Choose one or more of these under Follow-up the next time you start the run. Each is described in Follow-up options.

You want	Choose
An alert for each endpoint that fails, which resolves itself when that endpoint later succeeds	On failure: Raise an alert per failed endpoint
One ticket for the run, with each failed endpoint listed in internal notes (Enterprise and MSP)	On failure: Open one ticket for the run
Installs that collide with another installer to try again on their own	Auto-retry transient failures, with 1618 in Exit codes to retry
A notice in the bell when the run is done, whether or not it failed	Notify me when it finishes
A bad script or package to stop on a few endpoints instead of all of them	Staged rollout

Caution

Raise an alert per failed endpoint raises one alert for every endpoint that fails. For a run on many endpoints, prefer one ticket or a staged rollout.

For a rule that covers every run without choosing it each time, build a workflow with the Script or deployment fails trigger (Enterprise and MSP, automation:manage). It fires once per run that finished with failures or paused its staged rollout, and can create a ticket or notify people. For example, a Create a ticket step with the subject {{batch.kind}} failed: {{batch.name}} and the message {{batch.failed}} of {{batch.total}} endpoints {{batch.reason}}. {{batch.link}}. See Build a workflow and the placeholders.

Verify

Open the retry. Once every endpoint has run it, the Failed and Timed out tiles read 0, and What the failures have in common is gone. The first run keeps its own counts, and links to the retry under Retried as.

Next steps

• Run scripts on endpoints
• Deploy software
• Alert rules