Scancel All Jobs: The Ultimate Guide to Canceling Every Slurm Job Fast
If you have ever typed scancel all jobs into a terminal on a SLURM cluster and hit enter without a second thought, you already know the strange mix of relief and panic that follows. The command is the fastest way to clear a queue, but it is also one of the quickest ways to destroy hours of compute time if you misjudge its scope. Understanding exactly what happens when you cancel everything is essential for researchers, HPC administrators, and anyone who shares a scheduler with other people.
This guide walks through the mechanics of scancel, the filters that control which jobs actually get terminated, and the safety habits that separate a clean queue reset from an accidental catastrophe. Whether you are clearing your own pending work before a maintenance window or auditing a shared partition, the details below will help you act with precision instead of guesswork.
Understanding What scancel Actually Does
SLURM ships with a family of client commands, and scancel is the one responsible for terminating jobs. It does not simply delete a row from a database; it signals the controller to stop the job, kill the associated job steps, and release the allocated resources back to the pool. Depending on the job's state, that process can be nearly instantaneous for pending work or involve a grace period for running processes.
There is an important nuance buried in the phrase "scancel all jobs." SLURM has no literal keyword called "all." Instead, you construct the effect by combining scancel with filters such as user, partition, account, or job state. If you leave the filters off and you happen to be an administrator, you can genuinely wipe the entire cluster queue. For a regular user, the scope is automatically limited to jobs you own.
The Anatomy of a scancel Command
The basic syntax follows a predictable pattern: the command name, then a set of optional flags, then zero or more job identifiers. Passing explicit numeric IDs is the safest approach because the blast radius is visible before you press enter. For example, cancelling a single job means typing scancel followed by that job's ID, and cancelling a comma-separated list of IDs works the same way.
The real power comes from filter flags. The --user flag limits the operation to one account, --partition restricts it to a specific queue, --state narrows it to pending or running jobs, and --name matches the job name you supplied at submission time. Flags such as --signal and --full control how forcefully the termination is delivered, which matters when your application needs a chance to write a checkpoint before disappearing.
Job Arrays, Job Steps, and Heterogeneous Jobs
Job arrays complicate the picture because a single array ID can represent hundreds of independent tasks. Cancelling the parent ID terminates the entire array, while appending an underscore and an index cancels one specific element. If you are trying to stop a runaway sweep across a thousand parameter combinations, the parent ID is almost always the tool you want, but it pays to confirm the array range first.
Job steps add another layer. A batch script typically spawns multiple steps, and terminating the parent job does not always guarantee every step dies immediately. Using the --full option forces the controller to terminate all steps associated with a job rather than waiting for them to exit gracefully. Heterogeneous jobs, which span multiple node groups, behave similarly and should be cancelled at the parent level unless you have a specific reason to target a component.
Practical Ways to Cancel Every Job You Own
The most common real-world scenario is straightforward: you have a backlog of experiments in the queue, you spot a bug in your submission script, and you want everything gone before it consumes a single node-hour. In that case, you are only ever cancelling your own work, and the risk of upsetting colleagues is minimal. The command is short, but the verification step before it matters just as much as the command itself.
Before issuing a mass cancellation, run a queue listing filtered to your username. That single check tells you how many jobs are pending, how many are running, and which partitions are affected. It takes three seconds and can save you from cancelling a job that has been running for two days and is minutes away from finishing.
Cancelling All Jobs for a Specific User
To clear your own queue in one shot, the canonical form is scancel with the user flag pointed at your account. On most systems, scancel --user=$USER or the shorthand scancel -u $USER does exactly what you expect: every job attributed to that username, in every state, is terminated. Administrators can substitute any username to clear another person's queue, though doing so without warning is a fast route to an awkward email thread.
The key habit here is to precede the cancellation with a filtered listing, for instance a queue query restricted to the same user. If the count matches what you expect, proceed. If the number is larger than you remember submitting, stop and investigate. Memory is unreliable when you have been debugging for six hours, and that extra job may be a colleague's submission that landed under a shared service account.
Filtering by Partition, Account, State, or Name
Filters turn a blunt instrument into a scalpel. Restricting by state lets you cancel only the pending backlog while leaving running jobs untouched, which is often exactly what you want when a broken script has queued hundreds of entries but the currently executing job is still producing valid results. Restricting by partition keeps your cleanup confined to a test queue rather than the production GPU pool.
Combining filters is where precision really shows. A command that targets a specific account, a specific partition, and the pending state will touch far less than an unfiltered sweep. Be aware that filters are additive rather than forgiving: if you specify a partition that also hosts other users' jobs, an administrative account can cancel their work along with yours. When in doubt, narrow the filter rather than broadening it.
| Filter | Purpose | Typical Use Case |
|---|---|---|
| --user | Limits to one account | Clearing your own queue |
| --state | Targets pending or running jobs | Dropping a broken backlog |
| --partition | Confines to one queue | Test or debug partitions |
| --name | Matches job name | Removing a specific sweep |
Risk Management, Alternatives, and Recovery
Mass cancellation is destructive by nature. Running jobs lose whatever progress exists only in memory, temporary files may be left half-written, and downstream pipeline steps that depend on the cancelled work will fail in confusing ways. On shared clusters, repeatedly killing large batches of jobs can also draw the attention of administrators who track queue churn and wasted allocations.
That does not mean you should avoid cancellation when it is warranted. It means you should treat it the way you would treat any irreversible operation: confirm the scope, understand the consequences, and have a plan for what happens after the queue is empty. A little discipline here pays off in preserved data and preserved goodwill.
Guardrails and Safer Habits
Start every cleanup with a listing command and read the output. If the job count or the partition names look unfamiliar, pause. Shell aliases that append a confirmation prompt to your cancellation command are a cheap and effective safeguard, particularly for anyone who works on a cluster late at night. Wrapping the operation inside a small script that prints the affected job list first gives you the same benefit with an audit trail.
Graceful termination is another option worth considering. The signal flag lets you send a softer signal before the controller resorts to a hard kill, giving well-written applications a window to flush buffers or write a restart file. You can also hold pending jobs rather than deleting them, or insert dependencies so a replacement job only starts after the current one exits cleanly.
After the Cancel: Cleanup and Communication
Once the queue is clear, verify the result with a fresh listing. Jobs can linger briefly in a completing state, and a step that refuses to die may need an explicit step-level termination. Accounting tools let you review what ran, how long it lasted, and what exit status it recorded, which is invaluable when you are trying to understand why a batch of jobs failed before you killed them.
Then communicate. If you share a project allocation, tell your collaborators that the queue was cleared and why. Re-submit corrected jobs with improved logging, reduced array sizes, or tighter resource requests so the same failure does not repeat. A clean queue is a good moment to fix the underlying script rather than simply restarting the same broken workflow.
Mastering scancel is less about memorizing flags and more about developing the instinct to check before you act. Once you can comfortably inspect a queue, target the right subset of jobs, and terminate them without collateral damage, you gain real control over your time on a shared cluster. The next time a submission script goes sideways, you will know exactly which command to reach for and, more importantly, which one to avoid.