Skip to content

fix(sched): do not dequeue a lower-priority task on the PrioritizedRR tick (#729) - #809

Open
ytakano wants to merge 2 commits into
tier4:mainfrom
ytakano:fix/prioritized-rr-tick-requeue-729
Open

ytakano wants to merge 2 commits into
tier4:mainfrom
ytakano:fix/prioritized-rr-tick-requeue-729

Conversation

@ytakano

@ytakano ytakano commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #729.

PrioritizedRRScheduler::invoke_preemption_tick() dequeued the head of the run queues whenever the quantum of the running task had expired, without comparing priorities. A lower-priority head was sent to the target CPU, rejected by do_preemption(), and re-queued at the tail of its priority level. Nothing restarted the quantum, so this repeated on every primary loop iteration.

With this change the tick dequeues a task only if it can actually replace the running one, and restarts the quantum otherwise.

Changes

  • Conditional dequeue. Add Scheduler::get_next_if() and scheduler::get_next_task_if(). They dequeue the head of the run queue only if a predicate accepts it, and the check and the dequeue are done under the run queue lock. get_next_task_if() passes the dequeued task to an on_taken callback while it still holds GLOBAL_WAKE_GET_MUTEX, and returns whether a task was dequeued. PriorityQueue::peek() is added for this. PrioritizedFIFO and PrioritizedRR share the implementation (take_head_if()).
  • Priority check in the tick. The tick dequeues the next task only if its priority is at least that of the running task (equal: quantum rotation, higher: plain preemption), no preemption is pending on that CPU, and the task still runs on that CPU. These checks run under GLOBAL_WAKE_GET_MUTEX, and the dequeued task is pushed to the preemption-pending heap before the mutex is released. So a wake path cannot push a pending task between the check and the dequeue, and the idle check in run_main sees the push, so the task is not left pending on a sleeping CPU.
  • Quantum restart. If there is no task to rotate with, the tick updates last_executed_time of the running task, so the run queues are not examined again on every iteration.
  • IPI retry. If a preemption is already pending on the CPU, the tick dequeues nothing and only sends the IPI again.
  • Panicked tasks. The tick never takes a task from the Panicked scheduler. Task::priority keeps the priority a task had before it panicked, so the priority comparison cannot judge such a task, and it could displace a healthy PrioritizedRR task.
  • Sleep of the primary CPU. The tick returns the time until the quantum expires, and scheduler::wake_task() caps its return value with it while a task is queued. Without this, a task of equal priority that arrives just after a quantum restart waits until something else wakes the primary CPU (up to 1 s when the timer is enabled).
  • Start of the quantum. last_executed_time is now updated before a task becomes visible as the running task: in do_preemption() just before set_current_task(), and in PrioritizedRRScheduler::get_next(). Before, do_preemption() did not update it when it resumed a preempted task, and the other paths updated it only later in run_main. In both cases the tick could see a stale time and rotate the task out before it ran.
  • Docs. mdbook/src/internal/scheduler.md and the comments on Task::preempts() and do_preemption() are updated.

Behavior changes to be aware of

  • While a PrioritizedRR task is running and a task is queued, the primary CPU wakes at least once per quantum (4 ms).
  • last_executed_time is also what awkernel_shell prints as "Last Executed". For a long-running PrioritizedRR task it is now the time of the last quantum restart, not the time of the last dispatch.

Not addressed

A task that the tick (or a wake path) moved to a preemption-pending heap is still re-queued at the tail of its level if the running task changes before the IPI is handled. This PR removes the case that happened on every iteration; the remaining window is the one described in #703.

Verification

  • make check BSP=aarch64_virt and make clippy: no warnings or errors.
  • Host unit tests with the pinned toolchain (awkernel_lib, awkernel_async_lib, awkernel_drivers): all pass. A test for PriorityQueue::peek() is added.
  • QEMU x86_64, test_prioritized_rr (45 s): the 24 priority-1 tasks ran 359 to 369 times each; priority-0 tasks ran only while the tasks were being spawned.
  • QEMU x86_64, test_multi_prioritized_scheduler (30 s): all 30 tasks started and finished, on this branch and on main.

Comparison with main

test_prioritized_rr always has an equal-priority task in the queue, so it does not hit #729. I measured a dedicated scenario instead: QEMU x86_64 with 16 CPUs (15 workers), 7 CPU-bound PrioritizedRR(1) tasks and 30 CPU-bound PrioritizedRR(0) tasks, so that only PrioritizedRR(0) tasks wait in the run queue. Temporary counters (not part of this PR) counted the IPIs sent by the tick and the pending tasks that do_preemption() rejected and re-queued.

main is fd13859, and this PR is 00fed6c, which includes the fixes for the review comments.

Cumulative values after 20 s:

main this PR
IPIs sent by the tick 83,348 38,335
Pending tasks rejected and re-queued 40,910 6
Preemptions performed 43,662 36,857
Work done by the PrioritizedRR(1) tasks (total) 1,137,796 1,183,807
Work done by a PrioritizedRR(0) task (min / max) 616 / 126,736 39,487 / 43,972
  • On main the tick sent about 7,900 IPIs per second for the first 10 s, and about half of them were rejected and re-queued. With this PR the rate is steady at about 1,900 per second, close to the 2,000 per second expected for 8 CPUs rotating every 4 ms.
  • On main at least one PrioritizedRR(0) task made no progress after the first 5 s: its count stayed at 616 until the end of the run (65 s). With this PR the minimum and the maximum differ by about 11%, and the values stay steady for the whole 65 s.

This is one run per branch on QEMU without KVM, so the absolute values will differ on real hardware. On main the IPI and preemption rates dropped to about 400 and 180 per second after about 10 s, and the rotation of the PrioritizedRR(0) tasks almost stopped. I did not investigate why; the missing EOI that #798 fixes may be related.

Related links

How was this PR tested?

  • scripts/ci.sh passes on my machine.

Notes for reviewers

… tick (tier4#729)

`invoke_preemption_tick()` dequeued the head of the run queues whenever
the quantum of the running task had expired, without comparing
priorities. A lower-priority head was sent to the target CPU, rejected
by `do_preemption()`, and re-queued at the tail of its level. Nothing
restarted the quantum, so this repeated on every primary loop iteration:
one IPI per iteration, and the FIFO order of the lower level was lost.

- Add `Scheduler::get_next_if()` and `get_next_task_if()`, which dequeue
  the head only if a predicate accepts it, under the run queue lock.
  `PriorityQueue::peek()` is added for it.
- The tick dequeues a task only if its priority is at least that of the
  running task and no preemption is pending on the CPU. Otherwise it
  restarts the quantum, or sends the IPI again if one is pending.
- Panicked tasks are never taken by the tick: `Task::priority` keeps the
  priority they had before the panic.
- The tick returns the time until the quantum expires, and
  `scheduler::wake_task()` caps the sleep of the primary CPU with it
  while a task is queued, so a restarted quantum is still checked on time.
- Start the quantum of a preempted task that `do_preemption()` resumes
  directly; it did not pass through `run_main`, so its quantum looked
  expired at once.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ytakano
ytakano marked this pull request as ready for review November 18, 2026 07:37
@ytakano
ytakano requested a review from atsushi421 November 18, 2026 07:37
// A queued task may be waiting for a time quantum to expire. Nothing
// else wakes the primary CPU at that time, so do not sleep beyond it.
// If no task is queued, the next enqueue wakes the primary CPU.
if NUM_TASK_IN_QUEUE.load(Ordering::Relaxed) == 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check runs before guard.wake_task(), so a task that a sleep handler enqueues in this same call is not counted, and quantum_wait is already dropped. wake_cpu(0) does nothing on CPU 0, so a timer-woken task with the same priority as the running PrioritizedRR tasks then waits for the next sleeper, or up to 1 s. Reading NUM_TASK_IN_QUEUE after guard.wake_task() closes this case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. Fixed in 00fed6c: the check now runs after guard.wake_task(), so a task enqueued by a sleep handler on CPU 0 is counted.

I measured it on QEMU x86_64 (KVM, 8 CPUs): all workers run CPU-bound PrioritizedRR(1) tasks, and one more task of the same priority sleeps for 30 ms in a loop. The delay of its resume was 10.6 ms (median) / 23.4 ms (max) before the fix, and 4.4 ms / 8.5 ms after it.

});

if let Some(next_task) = next_task {
push_preemption_pending(cpu_id, next_task);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_next_task_if releases GLOBAL_WAKE_GET_MUTEX before this push, and nothing checks again that task_id still runs on cpu_id. If that task finishes in this window, the worker can see an empty queue and no pending task, and go to sleep. The task pushed here then stays in the pending heap of an idle CPU, because the preempt IPI does nothing there and wake_workers() does not count pending tasks. main has the same window, but the idle check in run_main assumes that every pin is pushed under the mutex, as the wake paths do. Adding get_current_task(cpu_id) == Some(task_id) to the predicate and pushing before the mutex is released (for example in a callback on Taken) closes it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 00fed6c as you suggested. The predicate now also checks get_current_task(cpu_id) == Some(task_id), and get_next_task_if() takes an on_taken callback, so the task is pushed to the pending heap before GLOBAL_WAKE_GET_MUTEX is released.

The window is too narrow to hit as is, so I forced the interleaving with temporary delays (100 us between the dequeue and the push, 30 us between the idle check and sleep_cpu) on QEMU x86_64. Before the fix, two CPUs kept a pending task and never recovered in 40 s. After the fix, with the same delays, no task stayed pending.

The CPUs never recovered because of the missing EOI that #798 fixes. With that EOI added, the pending tasks stayed for at most 1.15 ms before the fix. So this change is still needed with #798.

Comment thread awkernel_async_lib/src/task/preempt.rs Outdated
if ctx.is_some() {
// `next` resumes on its own thread without passing through
// `run_main`, so start its time quantum here.
task_info.update_last_executed();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This covers only the resume branch, and it runs after set_current_task(cpu_id, next.id) has made next visible to the tick. On the branch without a saved context, last_executed_time stays stale until run_main updates it, so a tick in that window sees an expired quantum and an empty pending heap, and rotates next out before it runs. One update_last_executed() call just before set_current_task covers both branches.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 00fed6c: update_last_executed() is now called once just before set_current_task(), so it covers both branches.

The normal dispatch has the same window, between set_current_task() in PrioritizedRRScheduler::get_next() and the update in run_main, so I added the update to get_next() too.

I counted the equal-priority rotations of a task that had run for less than 2 ms (QEMU x86_64, 7 workers, 14 tasks, 30 s): 11,026 of 44,620 before, 1,406 of 35,014 with the change in do_preemption() only, and 7 of 33,670 with both.

Comment thread awkernel_async_lib/src/scheduler.rs Outdated
/// Get the next executable task.
fn get_next(&self, execution_ensured: bool) -> Option<Arc<Task>>;

/// Dequeue the task that `get_next(false)` would return, but only if

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After this change, only run_main calls get_next_task, and it always passes true, so the execution_ensured == false branches of the four schedulers are dead code. This doc, the doc of get_next_task_if, and the comment in edf.rs that the new EDF get_next_if points to still describe a get_next_task(false) path on the primary CPU. Rewording them, or removing the execution_ensured parameter, keeps them accurate.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. In 00fed6c I reworded the three comments that referred to get_next(false) / get_next_task(false) (the docs of get_next_if and get_next_task_if, and the comment in edf.rs).

I kept the execution_ensured parameter in this PR, because removing it touches all four schedulers. I will remove it in a separate PR.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I create a new issue #814.

task::{
get_last_executed_by_task_id, get_task, get_tasks_running, set_current_task,
set_need_preemption, State, NUM_TASK_IN_QUEUE,
get_task, get_tasks_running, set_current_task, set_need_preemption, State,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This removes the only use of task::get_last_executed_by_task_id. The function is pub, so the compiler does not warn about it, and it can go in this PR too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in 00fed6c.

- wake_task(): read NUM_TASK_IN_QUEUE after the sleep handlers have run,
  so that a task enqueued by a handler on the primary CPU keeps the
  quantum cap on the sleep of the primary CPU.
- invoke_preemption_tick(): check under GLOBAL_WAKE_GET_MUTEX that the
  task still runs on the CPU, and push the dequeued task to the
  preemption-pending heap before the mutex is released. The idle check in
  run_main() then sees the push, and the task is not left pending on a
  sleeping CPU. get_next_task_if() takes an on_taken callback for this.
- Start the time quantum before a task becomes visible as the running
  task, in do_preemption() (both branches) and in
  PrioritizedRRScheduler::get_next(). A stale last_executed_time let the
  tick rotate the task out before it ran.
- Reword the comments that referred to get_next(false) and
  get_next_task(false), which have no callers anymore.
- Remove the unused task::get_last_executed_by_task_id().

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PrioritizedRR: invoke_preemption_tick() pops and re-queues a lower-priority queue head on every primary loop iteration

2 participants