Brain Fuck Scheduler

The virtual deadline formula is a future deadline time that is the scaled round robin timeslice based on the nice level offset by the current time (in niffy units or nanosecond jiffies aka an internal kernel time counter).[7]^{:ln 4023, 4063} The virtual deadline only suggests the order but doesn't guarantee that a task will run exactly on the future scheduled niffy.[7]^{:ln 161-163}

First a prio ratios lookup table is created.[7]^{:ln 8042-8044} It is based on a recursive sequence. It increases 10% each nice level.[7]^{:ln 161} It follows a parabolic pattern if graphed, and the niced tasks are distributed as a moving squared function from 0 to 39 (corresponding from highest to lowest nice priority) as the domain and 128 to 5089 as the range.[7]^{:ln 177-179, 120, 184-185} The moving part comes from the ${\displaystyle t}$ variable in the virtual deadline formula that Kolivas hinted.

${\displaystyle g_{0}=128}$

${\displaystyle g_{i}=INT(g_{i-1}*11/10)}$

The task's nice-to-index mapping function ${\displaystyle f(n)}$ is mapped from nice −20…19 to index 0…39 to be used as the input to the prio ratio lookup table. This mapping function is the TASK_USER_PRIO() macro in sched.h in the kernel header. The internal kernel implementation slightly differs with range between 100–140 static priority but users will see it as −20…19 nice.

The virtual deadline is based on this exact formula:[7]^{:ln 4063, 4036, 4033, 1180}

${\displaystyle T=6}$

${\displaystyle N=1<<20}$

${\displaystyle d(n,t)=t+g(f(n))*T*(N/128)}$

Alternatively,

${\displaystyle d(n,t)={\frac {g(f(n))}{128}}*T*N+t}$

where ${\displaystyle d(n,t)}$ is the virtual deadline in u64 integer nanoseconds as a function of nice ${\displaystyle n}$ and ${\displaystyle t}$ which is the current time in niffies (as in nanosecond jiffies), ${\displaystyle g(i)}$ is the prio ratio table lookup as a function of index, ${\displaystyle f(n)}$ is the task's nice-to-index mapping function, ${\displaystyle T}$ is the round robin timeslice in milliseconds, ${\displaystyle N}$ is a constant of 1 millisecond in terms of nanoseconds as a latency reducing approximation of the conversion factor of ${\displaystyle {\frac {10^{9}ns}{10^{3}ms}}}$ but Kolivas uses a base 2 constant ${\displaystyle N}$ with approximately that scale.[7]^{:ln 1173–1174} Smaller values of ${\displaystyle d}$ mean that the virtual deadline is earlier corresponding to negative nice values. Larger values of ${\displaystyle d}$ indicate the virtual deadline is pushed back later corresponding to positive nice values. It uses this formula whenever the timeslice expires.[7]^{:ln 5087}

128 in base 2 corresponds to 100 in base 10 and possibly a "pseudo 100."[7]^{:ln 3415} 115 in base 2 corresponds to 90 in base 10. Kolivas uses 128 for "fast shifts,"[7]^{:ln 3846, 1648, 3525} as in division is right shift base 2.

* The alternative formula is presented for ease of understanding. All math is done in integer math so precision loss would be great. It is possibly why Kolivas deferred the division by 128 to one of the largest numbers as a multiple of 128 resulting in no remainder.

Nice	Virtual Deadline in timeslices relative to ${\displaystyle t}$	Virtual Deadline in exact seconds relative to ${\displaystyle t}$
−20	1.0	0.006
−19	1.09	0.006562
−18	1.2	0.007219
−17	1.3	0.007922
−16	1.4	0.008672
−15	1.5	0.009516
−14	1.7	0.010453
−13	1.9	0.011484
−12	2.1	0.012609
−11	2.3	0.013828
−10	2.5	0.015187
−9	2.7	0.016688
−8	3.0	0.018328
−7	3.3	0.020156
−6	3.6	0.022172
−5	4.0	0.024375
−4	4.4	0.026812
−3	4.9	0.029484
−2	5.3	0.032391
−1	5.9	0.035625
0	6.5	0.039188
1	7.1	0.043078
2	7.8	0.047344
3	8.6	0.052078
4	9.5	0.057281
5	10.5	0.063000
6	11.5	0.069281
7	12.6	0.076172
8	13.9	0.083766
9	15.3	0.092109
10	16.8	0.101297
11	18.5	0.111422
12	20.4	0.122531
13	22.4	0.134766
14	24.7	0.148219
15	27.1	0.163031
16	29.8	0.179297
17	32.8	0.197203
18	36.1	0.216891
19	39.7	0.238547

BFS uses scheduling policies to determine how much of the CPU tasks may use. BFS uses 4 scheduling tiers (called scheduling policies or scheduling classes) ordered from best to worst which determines how tasks are selected[7]^{:ln 4146-4161} with the ones on top being executed first.

Each task has a special value called a prio. In the v0.462 edition (used in the -ck 4.0 kernel patchset), there are total of 103 "priority queues" (aka prio) or allowed values that it can take. No actual special data structure was used as the priority queue but only the doubly linked list runqueue itself. The lower prio value means it is more important and gets executed first.

The realtime policy was designed for realtime tasks. This policy implies that the running tasks cannot be interrupted (i.e. preempted) by the lower prio-ed task or lower priority policy tiers. Priority classes considered under the realtime policy by the scheduler are those marked SCHED_RR and SCHED_FIFO.[7]^{:ln 351, 1150} The scheduler treats realtime round robin (SCHED_RR) and realtime FIFO (SCHED_FIFO) differently.[7]^{:ln 3881-3934}

The design laid out first 100 static priority queues.[7]^{:ln 189}

The task that will get chosen for execution is based on task availability of the lowest value of prio of the 100 queues and FIFO scheduling.[7]^{:ln 4146–4161}

On forks, the process priority will be demoted to normal policy.[7]^{:ln 2708}

On unprivileged use (i.e. non-root user) of sched_setscheduler called with a request for realtime policy class, the scheduler will demote the task to Isochronous policy.[7]^{:ln 350–359, 5023–5025}

The Isochronous policy was designed for near realtime performance for non-root users.[7]^{:ln 325}

The design laid out 1 priority queue that by default ran as pseudo-realtime tasks, but can be tuned as a degree of realtime.[7]^{:ln 1201, 346–348} .

The behavior of the policy can allow a task can be demoted to normal policy[7]^{:ln 336} when it exceeds a tuneable resource handling percentage (70% by default[7]^{:ln 343, 432}) of 5 seconds scaled to the number of online CPUs and the timer resolution plus 1 tick.[7]^{:ln 343, 3844–3859, 1167, 338}[10]^{:ln 1678, 4770–4783, 734} The formula was altered in MuQSS due to the multi-runqueue design. The exact formulas are:

${\displaystyle T_{BFS}=((5*F*n)+1)*R}$
${\displaystyle T_{MuQSS}=(5*F)*R}$

where ${\displaystyle T}$ is the total number of isochronous ticks, ${\displaystyle F}$ is the timer frequency, ${\displaystyle n}$ is the number of online CPUs, ${\displaystyle R}$ is the tuneable resource handling percentage not in decimal but as a whole number. The timer frequency is set to 250 by default and editable in the kernel, but usually tuned to 100 Hz for servers and 1000 Hz for interactive desktops. 250 is the balanced value. Setting ${\displaystyle R}$ to 100 made tasks behave as realtime and 0 made it not pseudo-realtime and anything in the middle was pseudo-realtime.[7]^{:ln 346–348}

The task that had an earliest virtual deadline was chosen for execution, but when multiple Isochronous tasks are in existence, they schedule as round robin allowing tasks to run the tuneable round robin value (with 6 ms as the default) one after another in a fair equal chance without considering the nice level.[7]^{:ln 334}

This behavior of the Isochronous policy is unique to only BFS and MuQSS and may not be implemented in other CPU schedulers.[7]^{:ln 324, 8484–8485}[10]^{:ln 1287–1288}

The normal policy was designed for regular use and is the default policy. Newly created tasks are typically marked normal.[7]^{:ln 502}

The design laid out one priority queue and tasks are chosen to be executed first based on earliest virtual deadline.

The idle priority was designed for background processes such as distributed programs and transcoders so that foreground processes or those above this scheduling policy can run uninterrupted.[7]^{:ln 363–368}

The design laid out 1 priority queue and tasks can be promoted to normal policy automatically to prevent indefinite resource hold.[7]^{:ln 368}

The next executed task with Idle priority with others residing in the same priority policy is selected by the earliest virtual deadline.[7]^{:ln 4160–4161}

Preemption can occur when a newly ready task with a higher priority policy (i.e. higher prio) has an earlier virtual deadline than the currently running task - which will be descheduled and put at the back of the queue.[7]^{:ln 169–175} Descheduled means that its virtual deadline is updated.[7]^{:ln 165–166} The task's time gets refilled to max round robin quantum when it has used up all its time.[7]^{:ln 4057–4062, 5856} If the scheduler found the task at the higher prio with the earliest virtual deadline, it will execute in place of the less important currently running task only if all logical CPUs (including hyperthreaded cores / SMT threads) are busy. The scheduler will delay preemption as long as possible if there are unused logical CPUs.

If a task is marked idle priority policy, it cannot preempt at all even other idle policy marked tasks but rather use cooperative multitasking.[7]^{:ln 2341–2344}

When the scheduler discovers a waking task, it will need to determine which logical CPU to run the wakening task on the non-unicore system. The scheduler favors most the idle hyperthreaded cores (or idle SMT threads) first on the same CPU that the task executed on,[7]^{:ln 261} then the other idle core of a multicore CPU,[7]^{:ln 262} then the other CPUs on the same NUMA node,[7]^{:ln 267, 263–266, 255–258} then all busy hyperthreaded cores / SMT threads / logical CPUs to be preempted on the same NUMA node,[7]^{:ln 265–267} then the other (remote) NUMA node[7]^{:ln 268–270} and is ranked on a preference list.[7]^{:ln 255–274} This special scan exists to minimize latency overhead resulting of migrating the task.[7]^{:ln 245, 268–272}

The preemption order is similar to the above paragraph. The preemption order is hyperthreaded core / SMT units on the same multicore first, then the other core in the multicore, then the other CPU on the same NUMA node.[7]^{:ln 265-267} When it goes scanning for a task to preempt in the other remote NUMA node, the preemption is just any busy threads with lower to equal prio or later virtual deadline assuming that all logical CPUs (including hyperthreaded core / SMT threads) in the machine are all busy.[7]^{:ln 270} The scheduler will have to scan for a suitable task with a lower or maybe equal priority policy task (with a later virtual deadline if necessary) to preempt and avoid logical CPUs with a task with a higher priority policy which it cannot preempt. Local preemption has a higher rank than scanning for a remote idle NUMA unit.[7]^{:ln 265–269}

When a task is involuntary preempted at the time the CPU is slowed down as a result of kernel mediated CPU frequency scaling (aka CPU frequency governor), the task is specially marked "sticky" except those marked as realtime policy.[7]^{:ln 2085} Marked sticky indicates that the task still has unused time and the task is restricted executing to the same CPU.[7]^{:ln 233–243} The task will be marked sticky whenever the CPU scaling governor has scaled the CPU at a slower speed.[7]^{:ln 2082–2107, 8840–8848} The idled stickied task will return to either executing at full Ghz speed by chance or to be rescheduled to execute on the best idle CPU that is not the same CPU that the task ran on.[7]^{:ln 2082–2086, 239–242, 2068–2079} It is not desirable to migrate the task to other places but make it idle instead because of increased latency brought about of overhead to migrating the task to another CPU or NUMA node.[7]^{:ln 228, 245} This sticky feature was removed in the last iteration of BFS (v0.512) corresponding to Kolivas' patchset 4.8-ck1 and did not exist in MuQSS.

MuQSS uses a bidirectional static arrayed 8 level skip list and tasks are ordered by static priority [queues] (referring to the scheduling policy) and a virtual deadline.[10]^{:ln 519, 525, 537, 588, 608} 8 was chosen to fit the array in the cacheline.[10]^{:ln 523} Doubly linked data structure design was chosen to speed up task removal. Removing a task takes only O(1) with a doubly skip list versus the original design by William Pugh which takes O(n) worst case.[10]^{:ln 458}

Task insertion is O(log n).[10]^{:ln 458} The next task for execution lookup is O(k), where k is the number of CPUs.[10]^{:ln 589–590, 603, 5125} The next task for execution is O(1) per runqueue,[10]^{:ln 5124} but the scheduler examines every other runqueues to maintain task fairness among CPUs, for latency or balancing (to maximize CPU usage and cache coherency on the same NUMA node over those that access across NUMA nodes), so ultimately O(k).[10]^{:ln 591–593, 497–501, 633–656} The max number of tasks it can handle are 64k tasks per runqueue per CPU.[10]^{:ln 521} It uses multiple task runqueues in some configurations one runqueue per CPU, whereas its predecessor BFS only used one task runqueue for all CPUs.

Tasks are ordered as a gradient in the skip list in a way that realtime policy priority comes first and idle policy priority comes last.[10]^{:ln 2356-2358} Normal and idle priority policy still get sorted by virtual deadline which uses nice values.[10]^{:ln 2353} Realtime and Isochronous policy tasks are run in FIFO order ignoring nice values.[10]^{:ln 2350–2351} New tasks with same key are placed in FIFO order meaning that newer tasks get placed at the end of the list (i.e. top most node vertically), and tasks at 0th level or at the front-bottom get execution first before those at nearest to the top vertically and those furthest away from the head node.[10]^{:ln 2351–2352, 590} The key used for inserted sorting is either the static priority[10]^{:ln 2345, 2365,} or the virtual deadline[10]^{:ln 2363}.

The user can choose to share runqueues among multicore or have a runqueue per logical CPU.[10]^{:ln 947–1006} The speculation of sharing runqueues design was to reduce latency with a tradeoff of throughput.[10]^{:ln 947–1006}

A new behavior introduced by MuQSS was the use of the high resolution timer for below millisecond accuracy when timeslices were used up resulting in rescheduling tasks.[10]^{:ln 618-630, 3829–3851, 3854-3865, 5316}