Job Scheduling

A cluster is shared, and the decision about which job runs next is where a lot of a research team's effective throughput is won or lost. A scheduler that packs jobs tightly keeps the accelerators busy; one that leaves a job waiting for a whole node to free up leaves expensive hardware idle while somebody waits.

Large training jobs make this harder than ordinary batch scheduling because they are gang-scheduled: a job needing 512 accelerators cannot start with 511, so partial availability is worthless and fragmentation is the enemy. The scheduling literature that helps here optimises goodput — useful work per unit time, accounting for how a job's own efficiency changes with the allocation it is given — rather than raw utilisation, which a badly-shaped allocation can keep high while achieving little.

Queueing time is the term users feel and utilisation is the term operators report, and pushing either toward its optimum worsens the other: a cluster kept near full has long queues by definition, because there is no slack for an arriving job to land in. Which of the two to give up is a policy question rather than a technical one, and stating it as a policy is the useful step — a cluster that has not decided is optimising for whoever complains loudest.