Repository navigation
Conversation
… tick (tier4#729) `invoke_preemption_tick()` dequeued the head of the run queues whenever the quantum of the running task had expired, without comparing priorities. A lower-priority head was sent to the target CPU, rejected by `do_preemption()`, and re-queued at the tail of its level. Nothing restarted the quantum, so this repeated on every primary loop iteration: one IPI per iteration, and the FIFO order of the lower level was lost. - Add `Scheduler::get_next_if()` and `get_next_task_if()`, which dequeue the head only if a predicate accepts it, under the run queue lock. `PriorityQueue::peek()` is added for it. - The tick dequeues a task only if its priority is at least that of the running task and no preemption is pending on the CPU. Otherwise it restarts the quantum, or sends the IPI again if one is pending. - Panicked tasks are never taken by the tick: `Task::priority` keeps the priority they had before the panic. - The tick returns the time until the quantum expires, and `scheduler::wake_task()` caps the sleep of the primary CPU with it while a task is queued, so a restarted quantum is still checked on time. - Start the quantum of a preempted task that `do_preemption()` resumes directly; it did not pass through `run_main`, so its quantum looked expired at once. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
| // A queued task may be waiting for a time quantum to expire. Nothing | ||
| // else wakes the primary CPU at that time, so do not sleep beyond it. | ||
| // If no task is queued, the next enqueue wakes the primary CPU. | ||
| if NUM_TASK_IN_QUEUE.load(Ordering::Relaxed) == 0 { |
There was a problem hiding this comment.
This check runs before guard.wake_task(), so a task that a sleep handler enqueues in this same call is not counted, and quantum_wait is already dropped. wake_cpu(0) does nothing on CPU 0, so a timer-woken task with the same priority as the running PrioritizedRR tasks then waits for the next sleeper, or up to 1 s. Reading NUM_TASK_IN_QUEUE after guard.wake_task() closes this case.
There was a problem hiding this comment.
You are right. Fixed in 00fed6c: the check now runs after guard.wake_task(), so a task enqueued by a sleep handler on CPU 0 is counted.
I measured it on QEMU x86_64 (KVM, 8 CPUs): all workers run CPU-bound PrioritizedRR(1) tasks, and one more task of the same priority sleeps for 30 ms in a loop. The delay of its resume was 10.6 ms (median) / 23.4 ms (max) before the fix, and 4.4 ms / 8.5 ms after it.
| }); | ||
|
|
||
| if let Some(next_task) = next_task { | ||
| push_preemption_pending(cpu_id, next_task); |
There was a problem hiding this comment.
get_next_task_if releases GLOBAL_WAKE_GET_MUTEX before this push, and nothing checks again that task_id still runs on cpu_id. If that task finishes in this window, the worker can see an empty queue and no pending task, and go to sleep. The task pushed here then stays in the pending heap of an idle CPU, because the preempt IPI does nothing there and wake_workers() does not count pending tasks. main has the same window, but the idle check in run_main assumes that every pin is pushed under the mutex, as the wake paths do. Adding get_current_task(cpu_id) == Some(task_id) to the predicate and pushing before the mutex is released (for example in a callback on Taken) closes it.
There was a problem hiding this comment.
Fixed in 00fed6c as you suggested. The predicate now also checks get_current_task(cpu_id) == Some(task_id), and get_next_task_if() takes an on_taken callback, so the task is pushed to the pending heap before GLOBAL_WAKE_GET_MUTEX is released.
The window is too narrow to hit as is, so I forced the interleaving with temporary delays (100 us between the dequeue and the push, 30 us between the idle check and sleep_cpu) on QEMU x86_64. Before the fix, two CPUs kept a pending task and never recovered in 40 s. After the fix, with the same delays, no task stayed pending.
The CPUs never recovered because of the missing EOI that #798 fixes. With that EOI added, the pending tasks stayed for at most 1.15 ms before the fix. So this change is still needed with #798.
| if ctx.is_some() { | ||
| // `next` resumes on its own thread without passing through | ||
| // `run_main`, so start its time quantum here. | ||
| task_info.update_last_executed(); |
There was a problem hiding this comment.
This covers only the resume branch, and it runs after set_current_task(cpu_id, next.id) has made next visible to the tick. On the branch without a saved context, last_executed_time stays stale until run_main updates it, so a tick in that window sees an expired quantum and an empty pending heap, and rotates next out before it runs. One update_last_executed() call just before set_current_task covers both branches.
There was a problem hiding this comment.
Fixed in 00fed6c: update_last_executed() is now called once just before set_current_task(), so it covers both branches.
The normal dispatch has the same window, between set_current_task() in PrioritizedRRScheduler::get_next() and the update in run_main, so I added the update to get_next() too.
I counted the equal-priority rotations of a task that had run for less than 2 ms (QEMU x86_64, 7 workers, 14 tasks, 30 s): 11,026 of 44,620 before, 1,406 of 35,014 with the change in do_preemption() only, and 7 of 33,670 with both.
| /// Get the next executable task. | ||
| fn get_next(&self, execution_ensured: bool) -> Option<Arc<Task>>; | ||
|
|
||
| /// Dequeue the task that `get_next(false)` would return, but only if |
There was a problem hiding this comment.
After this change, only run_main calls get_next_task, and it always passes true, so the execution_ensured == false branches of the four schedulers are dead code. This doc, the doc of get_next_task_if, and the comment in edf.rs that the new EDF get_next_if points to still describe a get_next_task(false) path on the primary CPU. Rewording them, or removing the execution_ensured parameter, keeps them accurate.
There was a problem hiding this comment.
Thanks. In 00fed6c I reworded the three comments that referred to get_next(false) / get_next_task(false) (the docs of get_next_if and get_next_task_if, and the comment in edf.rs).
I kept the execution_ensured parameter in this PR, because removing it touches all four schedulers. I will remove it in a separate PR.
| task::{ | ||
| get_last_executed_by_task_id, get_task, get_tasks_running, set_current_task, | ||
| set_need_preemption, State, NUM_TASK_IN_QUEUE, | ||
| get_task, get_tasks_running, set_current_task, set_need_preemption, State, |
There was a problem hiding this comment.
This removes the only use of task::get_last_executed_by_task_id. The function is pub, so the compiler does not warn about it, and it can go in this PR too.
- wake_task(): read NUM_TASK_IN_QUEUE after the sleep handlers have run, so that a task enqueued by a handler on the primary CPU keeps the quantum cap on the sleep of the primary CPU. - invoke_preemption_tick(): check under GLOBAL_WAKE_GET_MUTEX that the task still runs on the CPU, and push the dequeued task to the preemption-pending heap before the mutex is released. The idle check in run_main() then sees the push, and the task is not left pending on a sleeping CPU. get_next_task_if() takes an on_taken callback for this. - Start the time quantum before a task becomes visible as the running task, in do_preemption() (both branches) and in PrioritizedRRScheduler::get_next(). A stale last_executed_time let the tick rotate the task out before it ran. - Reword the comments that referred to get_next(false) and get_next_task(false), which have no callers anymore. - Remove the unused task::get_last_executed_by_task_id(). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Description
Fixes #729.
PrioritizedRRScheduler::invoke_preemption_tick()dequeued the head of the run queues whenever the quantum of the running task had expired, without comparing priorities. A lower-priority head was sent to the target CPU, rejected bydo_preemption(), and re-queued at the tail of its priority level. Nothing restarted the quantum, so this repeated on every primary loop iteration.With this change the tick dequeues a task only if it can actually replace the running one, and restarts the quantum otherwise.
Changes
last_executed_timeof the running task, so the run queues are not examined again on every iteration.Task::prioritykeeps the priority a task had before it panicked, so the priority comparison cannot judge such a task, and it could displace a healthy PrioritizedRR task.scheduler::wake_task()caps its return value with it while a task is queued. Without this, a task of equal priority that arrives just after a quantum restart waits until something else wakes the primary CPU (up to 1 s when the timer is enabled).mdbook/src/internal/scheduler.mdand the comments onTask::preempts()anddo_preemption()are updated.Behavior changes to be aware of
last_executed_timeis also whatawkernel_shellprints as "Last Executed". For a long-running PrioritizedRR task it is now the time of the last quantum restart, not the time of the last dispatch.Not addressed
A task that the tick (or a wake path) moved to a preemption-pending heap is still re-queued at the tail of its level if the running task changes before the IPI is handled. This PR removes the case that happened on every iteration; the remaining window is the one described in #703.
Verification
make check BSP=aarch64_virtandmake clippy: no warnings or errors.awkernel_lib,awkernel_async_lib,awkernel_drivers): all pass. A test forPriorityQueue::peek()is added.test_prioritized_rr(45 s): the 24 priority-1 tasks ran 359 to 369 times each; priority-0 tasks ran only while the tasks were being spawned.test_multi_prioritized_scheduler(30 s): all 30 tasks started and finished, on this branch and onmain.Comparison with
maintest_prioritized_rralways has an equal-priority task in the queue, so it does not hit #729. I measured a dedicated scenario instead: QEMU x86_64 with 16 CPUs (15 workers), 7 CPU-boundPrioritizedRR(1)tasks and 30 CPU-boundPrioritizedRR(0)tasks, so that onlyPrioritizedRR(0)tasks wait in the run queue. Temporary counters (not part of this PR) counted the IPIs sent by the tick and the pending tasks thatdo_preemption()rejected and re-queued.mainis fd13859, and this PR is 00fed6c, which includes the fixes for the review comments.Cumulative values after 20 s:
mainPrioritizedRR(1)tasks (total)PrioritizedRR(0)task (min / max)mainthe tick sent about 7,900 IPIs per second for the first 10 s, and about half of them were rejected and re-queued. With this PR the rate is steady at about 1,900 per second, close to the 2,000 per second expected for 8 CPUs rotating every 4 ms.mainat least onePrioritizedRR(0)task made no progress after the first 5 s: its count stayed at 616 until the end of the run (65 s). With this PR the minimum and the maximum differ by about 11%, and the values stay steady for the whole 65 s.This is one run per branch on QEMU without KVM, so the absolute values will differ on real hardware. On
mainthe IPI and preemption rates dropped to about 400 and 180 per second after about 10 s, and the rotation of thePrioritizedRR(0)tasks almost stopped. I did not investigate why; the missing EOI that #798 fixes may be related.Related links
How was this PR tested?
scripts/ci.shpasses on my machine.Notes for reviewers