lastdragon/linux-kernel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Resguardo inicial

Browse Source
This commit is contained in:
lastdragon
2026-09-06 12:53:21 -06:00
parent 662efe3c2a
commit ad2088d178
94853 changed files with 43200404 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
linux-7.2.3/tools/memory-model/Documentation/README
+105
View File
@@ -0,0 +1,105 @@
It has been said that successful communication requires first identifying
what your audience knows and then building a bridge from their current
knowledge to what they need to know. Unfortunately, the expected
Linux-kernel memory model (LKMM) audience might be anywhere from novice
to expert both in kernel hacking and in understanding LKMM.
This document therefore points out a number of places to start reading,
depending on what you know and what you would like to learn. Please note
that the documents later in this list assume that the reader understands
the material provided by documents earlier in this list.
If LKMM-specific terms lost you, glossary.txt might help you.
o You are new to Linux-kernel concurrency: simple.txt
o You have some background in Linux-kernel concurrency, and would
like an overview of the types of low-level concurrency primitives
that the Linux kernel provides: ordering.txt
Here, "low level" means atomic operations to single variables.
o You are familiar with the Linux-kernel concurrency primitives
that you need, and just want to get started with LKMM litmus
tests: litmus-tests.txt
o You need to locklessly access shared variables that are otherwise
protected by a lock: locking.txt
This locking.txt file expands on the "Locking" section in
recipes.txt, but is self-contained.
o You are familiar with Linux-kernel concurrency, and would
like a detailed intuitive understanding of LKMM, including
situations involving more than two threads: recipes.txt
o You would like a detailed understanding of what your compiler can
and cannot do to control dependencies: control-dependencies.txt
o You would like to mark concurrent normal accesses to shared
variables so that intentional "racy" accesses can be properly
documented, especially when you are responding to complaints
from KCSAN: access-marking.txt
o You are familiar with Linux-kernel concurrency and the use of
LKMM, and would like a quick reference: cheatsheet.txt
o You are familiar with Linux-kernel concurrency and the use
of LKMM, and would like to learn about LKMM's requirements,
rationale, and implementation: explanation.txt and
herd-representation.txt
o You are interested in the publications related to LKMM, including
hardware manuals, academic literature, standards-committee
working papers, and LWN articles: references.txt
====================
DESCRIPTION OF FILES
====================
README
This file.
access-marking.txt
Guidelines for marking intentionally concurrent accesses to
shared memory.
cheatsheet.txt
Quick-reference guide to the Linux-kernel memory model.
control-dependencies.txt
Guide to preventing compiler optimizations from destroying
your control dependencies.
explanation.txt
Detailed description of the memory model.
glossary.txt
Brief definitions of LKMM-related terms.
herd-representation.txt
The (abstract) representation of the Linux-kernel concurrency
primitives in terms of events.
litmus-tests.txt
The format, features, capabilities, and limitations of the litmus
tests that LKMM can evaluate.
locking.txt
Rules for accessing lock-protected shared variables outside of
their corresponding critical sections.
ordering.txt
Overview of the Linux kernel's low-level memory-ordering
primitives by category.
recipes.txt
Common memory-ordering patterns.
references.txt
Background information.
simple.txt
Starting point for someone new to Linux-kernel concurrency.
And also a reminder of the simpler approaches to concurrency!
linux-7.2.3/tools/memory-model/Documentation/access-marking.txt
+624
View File
@@ -0,0 +1,624 @@
MARKING SHARED-MEMORY ACCESSES
==============================
This document provides guidelines for marking intentionally concurrent
normal accesses to shared memory, that is "normal" as in accesses that do
not use read-modify-write atomic operations. It also describes how to
document these accesses, both with comments and with special assertions
processed by the Kernel Concurrency Sanitizer (KCSAN). This discussion
builds on an earlier LWN article [1] and Linux Foundation mentorship
session [2].
ACCESS-MARKING OPTIONS
======================
The Linux kernel provides the following access-marking options:
1. Plain C-language accesses (unmarked), for example, "a = b;"
2. Data-race marking, for example, "data_race(a = b);"
3. READ_ONCE(), for example, "a = READ_ONCE(b);"
The various forms of atomic_read() also fit in here.
4. WRITE_ONCE(), for example, "WRITE_ONCE(a, b);"
The various forms of atomic_set() also fit in here.
5. __data_racy, for example "int __data_racy a;"
6. KCSAN's negative-marking assertions, ASSERT_EXCLUSIVE_ACCESS()
and ASSERT_EXCLUSIVE_WRITER(), are described in the
"ACCESS-DOCUMENTATION OPTIONS" section below.
These may be used in combination, as shown in this admittedly improbable
example:
WRITE_ONCE(a, b + data_race(c + d) + READ_ONCE(e));
Neither plain C-language accesses nor data_race() (#1 and #2 above) place
any sort of constraint on the compiler's choice of optimizations [3].
In contrast, READ_ONCE() and WRITE_ONCE() (#3 and #4 above) restrict the
compiler's use of code-motion and common-subexpression optimizations.
Therefore, if a given access is involved in an intentional data race,
using READ_ONCE() for loads and WRITE_ONCE() for stores is usually
preferable to data_race(), which in turn is usually preferable to plain
C-language accesses. It is permissible to combine #2 and #3, for example,
data_race(READ_ONCE(a)), which will both restrict compiler optimizations
and disable KCSAN diagnostics.
KCSAN will complain about many types of data races involving plain
C-language accesses, but marking all accesses involved in a given data
race with one of data_race(), READ_ONCE(), or WRITE_ONCE(), will prevent
KCSAN from complaining. Of course, lack of KCSAN complaints does not
imply correct code. Therefore, please take a thoughtful approach
when responding to KCSAN complaints. Churning the code base with
ill-considered additions of data_race(), READ_ONCE(), and WRITE_ONCE()
is unhelpful.
In fact, the following sections describe situations where use of
data_race() and even plain C-language accesses is preferable to
READ_ONCE() and WRITE_ONCE().
Use of the data_race() Macro
----------------------------
Here are some situations where data_race() should be used instead of
READ_ONCE() and WRITE_ONCE():
1. Data-racy loads from shared variables whose values are used only
for diagnostic purposes.
2. Data-racy reads whose values are checked against marked reload.
3. Reads whose values feed into error-tolerant heuristics.
4. Writes setting values that feed into error-tolerant heuristics.
Data-Racy Reads for Approximate Diagnostics
Approximate diagnostics include lockdep reports, monitoring/statistics
(including /proc and /sys output), WARN*()/BUG*() checks whose return
values are ignored, and other situations where reads from shared variables
are not an integral part of the core concurrency design.
In fact, use of data_race() instead READ_ONCE() for these diagnostic
reads can enable better checking of the remaining accesses implementing
the core concurrency design. For example, suppose that the core design
prevents any non-diagnostic reads from shared variable x from running
concurrently with updates to x. Then using plain C-language writes
to x allows KCSAN to detect reads from x from within regions of code
that fail to exclude the updates. In this case, it is important to use
data_race() for the diagnostic reads because otherwise KCSAN would give
false-positive warnings about these diagnostic reads.
If it is necessary to both restrict compiler optimizations and disable
KCSAN diagnostics, use both data_race() and READ_ONCE(), for example,
data_race(READ_ONCE(a)).
In theory, plain C-language loads can also be used for this use case.
However, in practice this will have the disadvantage of causing KCSAN
to generate false positives because KCSAN will have no way of knowing
that the resulting data race was intentional.
Data-Racy Reads That Are Checked Against Marked Reload
The values from some reads are not implicitly trusted. They are instead
fed into some operation that checks the full value against a later marked
load from memory, which means that the occasional arbitrarily bogus value
is not a problem. For example, if a bogus value is fed into cmpxchg(),
all that happens is that this cmpxchg() fails, which normally results
in a retry. Unless the race condition that resulted in the bogus value
recurs, this retry will with high probability succeed, so no harm done.
However, please keep in mind that a data_race() load feeding into
a cmpxchg_relaxed() might still be subject to load fusing on some
architectures. Therefore, it is best to capture the return value from
the failing cmpxchg() for the next iteration of the loop, an approach
that provides the compiler much less scope for mischievous optimizations.
Capturing the return value from cmpxchg() also saves a memory reference
in many cases.
In theory, plain C-language loads can also be used for this use case.
However, in practice this will have the disadvantage of causing KCSAN
to generate false positives because KCSAN will have no way of knowing
that the resulting data race was intentional.
Reads Feeding Into Error-Tolerant Heuristics
Values from some reads feed into heuristics that can tolerate occasional
errors. Such reads can use data_race(), thus allowing KCSAN to focus on
the other accesses to the relevant shared variables. But please note
that data_race() loads are subject to load fusing, which can result in
consistent errors, which in turn are quite capable of breaking heuristics.
Therefore use of data_race() should be limited to cases where some other
code (such as a barrier() call) will force the occasional reload.
Note that this use case requires that the heuristic be able to handle
any possible error. In contrast, if the heuristics might be fatally
confused by one or more of the possible erroneous values, use READ_ONCE()
instead of data_race().
In theory, plain C-language loads can also be used for this use case.
However, in practice this will have the disadvantage of causing KCSAN
to generate false positives because KCSAN will have no way of knowing
that the resulting data race was intentional.
Writes Setting Values Feeding Into Error-Tolerant Heuristics
The values read into error-tolerant heuristics come from somewhere,
for example, from sysfs. This means that some code in sysfs writes
to this same variable, and these writes can also use data_race().
After all, if the heuristic can tolerate the occasional bogus value
due to compiler-mangled reads, it can also tolerate the occasional
compiler-mangled write, at least assuming that the proper value is in
place once the write completes.
Plain C-language stores can also be used for this use case. However,
in kernels built with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n, this
will have the disadvantage of causing KCSAN to generate false positives
because KCSAN will have no way of knowing that the resulting data race
was intentional.
Use of Plain C-Language Accesses
--------------------------------
Here are some example situations where plain C-language accesses should
used instead of READ_ONCE(), WRITE_ONCE(), and data_race():
1. Accesses protected by mutual exclusion, including strict locking
and sequence locking.
2. Initialization-time and cleanup-time accesses. This covers a
wide variety of situations, including the uniprocessor phase of
system boot, variables to be used by not-yet-spawned kthreads,
structures not yet published to reference-counted or RCU-protected
data structures, and the cleanup side of any of these situations.
3. Per-CPU variables that are not accessed from other CPUs.
4. Private per-task variables, including on-stack variables, some
fields in the task_struct structure, and task-private heap data.
5. Any other loads for which there is not supposed to be a concurrent
store to that same variable.
6. Any other stores for which there should be neither concurrent
loads nor concurrent stores to that same variable.
But note that KCSAN makes two explicit exceptions to this rule
by default, refraining from flagging plain C-language stores:
a. No matter what. You can override this default by building
with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n.
b. When the store writes the value already contained in
that variable. You can override this default by building
with CONFIG_KCSAN_REPORT_VALUE_CHANGE_ONLY=n.
c. When one of the stores is in an interrupt handler and
the other in the interrupted code. You can override this
default by building with CONFIG_KCSAN_INTERRUPT_WATCHER=y.
Note that it is important to use plain C-language accesses in these cases,
because doing otherwise prevents KCSAN from detecting violations of your
code's synchronization rules.
Use of __data_racy
------------------
Adding the __data_racy type qualifier to the declaration of a variable
causes KCSAN to treat all accesses to that variable as if they were
enclosed by data_race(). However, __data_racy does not affect the
compiler, though one could imagine hardened kernel builds treating the
__data_racy type qualifier as if it was the volatile keyword.
Note well that __data_racy is subject to the same pointer-declaration
rules as are other type qualifiers such as const and volatile.
For example:
int __data_racy *p; // Pointer to data-racy data.
int *__data_racy p; // Data-racy pointer to non-data-racy data.
ACCESS-DOCUMENTATION OPTIONS
============================
It is important to comment marked accesses so that people reading your
code, yourself included, are reminded of the synchronization design.
However, it is even more important to comment plain C-language accesses
that are intentionally involved in data races. Such comments are
needed to remind people reading your code, again, yourself included,
of how the compiler has been prevented from optimizing those accesses
into concurrency bugs.
It is also possible to tell KCSAN about your synchronization design.
For example, ASSERT_EXCLUSIVE_ACCESS(foo) tells KCSAN that any
concurrent access to variable foo by any other CPU is an error, even
if that concurrent access is marked with READ_ONCE(). In addition,
ASSERT_EXCLUSIVE_WRITER(foo) tells KCSAN that although it is OK for there
to be concurrent reads from foo from other CPUs, it is an error for some
other CPU to be concurrently writing to foo, even if that concurrent
write is marked with data_race() or WRITE_ONCE().
Note that although KCSAN will call out data races involving either
ASSERT_EXCLUSIVE_ACCESS() or ASSERT_EXCLUSIVE_WRITER() on the one hand
and data_race() writes on the other, KCSAN will not report the location
of these data_race() writes.
EXAMPLES
========
As noted earlier, the goal is to prevent the compiler from destroying
your concurrent algorithm, to help the human reader, and to inform
KCSAN of aspects of your concurrency design. This section looks at a
few examples showing how this can be done.
Lock Protection With Lockless Diagnostic Access
-----------------------------------------------
For example, suppose a shared variable "foo" is read only while a
reader-writer spinlock is read-held, written only while that same
spinlock is write-held, except that it is also read locklessly for
diagnostic purposes. The code might look as follows:
int foo;
DEFINE_RWLOCK(foo_rwlock);
void update_foo(int newval)
{
write_lock(&foo_rwlock);
foo = newval;
do_something(newval);
write_unlock(&foo_rwlock);
}
int read_foo(void)
{
int ret;
read_lock(&foo_rwlock);
do_something_else();
ret = foo;
read_unlock(&foo_rwlock);
return ret;
}
void read_foo_diagnostic(void)
{
pr_info("Current value of foo: %d\n", data_race(foo));
}
The reader-writer lock prevents the compiler from introducing concurrency
bugs into any part of the main algorithm using foo, which means that
the accesses to foo within both update_foo() and read_foo() can (and
should) be plain C-language accesses. One benefit of making them be
plain C-language accesses is that KCSAN can detect any erroneous lockless
reads from or updates to foo. The data_race() in read_foo_diagnostic()
tells KCSAN that data races are expected, and should be silently
ignored. This data_race() also tells the human reading the code that
read_foo_diagnostic() might sometimes return a bogus value.
If it is necessary to suppress compiler optimization and also detect
buggy lockless writes, read_foo_diagnostic() can be updated as follows:
void read_foo_diagnostic(void)
{
pr_info("Current value of foo: %d\n", data_race(READ_ONCE(foo)));
}
Alternatively, given that KCSAN is to ignore all accesses in this function,
this function can be marked __no_kcsan and the data_race() can be dropped:
void __no_kcsan read_foo_diagnostic(void)
{
pr_info("Current value of foo: %d\n", READ_ONCE(foo));
}
However, in order for KCSAN to detect buggy lockless writes, your kernel
must be built with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n. If you
need KCSAN to detect such a write even if that write did not change
the value of foo, you also need CONFIG_KCSAN_REPORT_VALUE_CHANGE_ONLY=n.
If you need KCSAN to detect such a write happening in an interrupt handler
running on the same CPU doing the legitimate lock-protected write, you
also need CONFIG_KCSAN_INTERRUPT_WATCHER=y. With some or all of these
Kconfig options set properly, KCSAN can be quite helpful, although
it is not necessarily a full replacement for hardware watchpoints.
On the other hand, neither are hardware watchpoints a full replacement
for KCSAN because it is not always easy to tell hardware watchpoint to
conditionally trap on accesses.
Lock-Protected Writes With Lockless Reads
-----------------------------------------
For another example, suppose a shared variable "foo" is updated only
while holding a spinlock, but is read locklessly. The code might look
as follows:
int foo;
DEFINE_SPINLOCK(foo_lock);
void update_foo(int newval)
{
spin_lock(&foo_lock);
WRITE_ONCE(foo, newval);
ASSERT_EXCLUSIVE_WRITER(foo);
do_something(newval);
spin_unlock(&foo_wlock);
}
int read_foo(void)
{
do_something_else();
return READ_ONCE(foo);
}
Because foo is read locklessly, all accesses are marked. The purpose
of the ASSERT_EXCLUSIVE_WRITER() is to allow KCSAN to check for a buggy
concurrent write, whether marked or not.
Lock-Protected Writes With Heuristic Lockless Reads
---------------------------------------------------
For another example, suppose that the code can normally make use of
a per-data-structure lock, but there are times when a global lock
is required. These times are indicated via a global flag. The code
might look as follows, and is based loosely on nf_conntrack_lock(),
nf_conntrack_all_lock(), and nf_conntrack_all_unlock():
bool global_flag;
DEFINE_SPINLOCK(global_lock);
struct foo {
spinlock_t f_lock;
int f_data;
};
/* All foo structures are in the following array. */
int nfoo;
struct foo *foo_array;
void do_something_locked(struct foo *fp)
{
/* This works even if data_race() returns nonsense. */
if (!data_race(global_flag)) {
spin_lock(&fp->f_lock);
if (!smp_load_acquire(&global_flag)) {
do_something(fp);
spin_unlock(&fp->f_lock);
return;
}
spin_unlock(&fp->f_lock);
}
spin_lock(&global_lock);
/* global_lock held, thus global flag cannot be set. */
spin_lock(&fp->f_lock);
spin_unlock(&global_lock);
/*
* global_flag might be set here, but begin_global()
* will wait for ->f_lock to be released.
*/
do_something(fp);
spin_unlock(&fp->f_lock);
}
void begin_global(void)
{
int i;
spin_lock(&global_lock);
WRITE_ONCE(global_flag, true);
for (i = 0; i < nfoo; i++) {
/*
* Wait for pre-existing local locks. One at
* a time to avoid lockdep limitations.
*/
spin_lock(&fp->f_lock);
spin_unlock(&fp->f_lock);
}
}
void end_global(void)
{
smp_store_release(&global_flag, false);
spin_unlock(&global_lock);
}
All code paths leading from the do_something_locked() function's first
read from global_flag acquire a lock, so endless load fusing cannot
happen.
If the value read from global_flag is true, then global_flag is
rechecked while holding ->f_lock, which, if global_flag is now false,
prevents begin_global() from completing. It is therefore safe to invoke
do_something().
Otherwise, if either value read from global_flag is true, then after
global_lock is acquired global_flag must be false. The acquisition of
->f_lock will prevent any call to begin_global() from returning, which
means that it is safe to release global_lock and invoke do_something().
For this to work, only those foo structures in foo_array[] may be passed
to do_something_locked(). The reason for this is that the synchronization
with begin_global() relies on momentarily holding the lock of each and
every foo structure.
The smp_load_acquire() and smp_store_release() are required because
changes to a foo structure between calls to begin_global() and
end_global() are carried out without holding that structure's ->f_lock.
The smp_load_acquire() and smp_store_release() ensure that the next
invocation of do_something() from do_something_locked() will see those
changes.
Lockless Reads and Writes
-------------------------
For another example, suppose a shared variable "foo" is both read and
updated locklessly. The code might look as follows:
int foo;
int update_foo(int newval)
{
int ret;
ret = xchg(&foo, newval);
do_something(newval);
return ret;
}
int read_foo(void)
{
do_something_else();
return READ_ONCE(foo);
}
Because foo is accessed locklessly, all accesses are marked. It does
not make sense to use ASSERT_EXCLUSIVE_WRITER() in this case because
there really can be concurrent lockless writers. KCSAN would
flag any concurrent plain C-language reads from foo, and given
CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n, also any concurrent plain
C-language writes to foo.
Lockless Reads and Writes, But With Single-Threaded Initialization
------------------------------------------------------------------
For yet another example, suppose that foo is initialized in a
single-threaded manner, but that a number of kthreads are then created
that locklessly and concurrently access foo. Some snippets of this code
might look as follows:
int foo;
void initialize_foo(int initval, int nkthreads)
{
int i;
foo = initval;
ASSERT_EXCLUSIVE_ACCESS(foo);
for (i = 0; i < nkthreads; i++)
kthread_run(access_foo_concurrently, ...);
}
/* Called from access_foo_concurrently(). */
int update_foo(int newval)
{
int ret;
ret = xchg(&foo, newval);
do_something(newval);
return ret;
}
/* Also called from access_foo_concurrently(). */
int read_foo(void)
{
do_something_else();
return READ_ONCE(foo);
}
The initialize_foo() uses a plain C-language write to foo because there
are not supposed to be concurrent accesses during initialization. The
ASSERT_EXCLUSIVE_ACCESS() allows KCSAN to flag buggy concurrent unmarked
reads, and the ASSERT_EXCLUSIVE_ACCESS() call further allows KCSAN to
flag buggy concurrent writes, even if: (1) Those writes are marked or
(2) The kernel was built with CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=y.
Checking Stress-Test Race Coverage
----------------------------------
When designing stress tests it is important to ensure that race conditions
of interest really do occur. For example, consider the following code
fragment:
int foo;
int update_foo(int newval)
{
return xchg(&foo, newval);
}
int xor_shift_foo(int shift, int mask)
{
int old, new, newold;
newold = data_race(foo); /* Checked by cmpxchg(). */
do {
old = newold;
new = (old << shift) ^ mask;
newold = cmpxchg(&foo, old, new);
} while (newold != old);
return old;
}
int read_foo(void)
{
return READ_ONCE(foo);
}
If it is possible for update_foo(), xor_shift_foo(), and read_foo() to be
invoked concurrently, the stress test should force this concurrency to
actually happen. KCSAN can evaluate the stress test when the above code
is modified to read as follows:
int foo;
int update_foo(int newval)
{
ASSERT_EXCLUSIVE_ACCESS(foo);
return xchg(&foo, newval);
}
int xor_shift_foo(int shift, int mask)
{
int old, new, newold;
newold = data_race(foo); /* Checked by cmpxchg(). */
do {
old = newold;
new = (old << shift) ^ mask;
ASSERT_EXCLUSIVE_ACCESS(foo);
newold = cmpxchg(&foo, old, new);
} while (newold != old);
return old;
}
int read_foo(void)
{
ASSERT_EXCLUSIVE_ACCESS(foo);
return READ_ONCE(foo);
}
If a given stress-test run does not result in KCSAN complaints from
each possible pair of ASSERT_EXCLUSIVE_ACCESS() invocations, the
stress test needs improvement. If the stress test was to be evaluated
on a regular basis, it would be wise to place the above instances of
ASSERT_EXCLUSIVE_ACCESS() under #ifdef so that they did not result in
false positives when not evaluating the stress test.
REFERENCES
==========
[1] "Concurrency bugs should fear the big bad data-race detector (part 2)"
https://lwn.net/Articles/816854/
[2] "The Kernel Concurrency Sanitizer"
https://www.linuxfoundation.org/webinars/the-kernel-concurrency-sanitizer
[3] "Who's afraid of a big bad optimizing compiler?"
https://lwn.net/Articles/793253/
linux-7.2.3/tools/memory-model/Documentation/cheatsheet.txt
+35
View File
@@ -0,0 +1,35 @@
Prior Operation Subsequent Operation
--------------- ---------------------------
C Self R W RMW Self R W DR DW RMW SV
-- ---- - - --- ---- - - -- -- --- --
Relaxed store Y Y
Relaxed load Y Y Y Y
Relaxed RMW operation Y Y Y Y
rcu_dereference() Y Y Y Y
Successful *_acquire() R Y Y Y Y Y Y
Successful *_release() C Y Y Y W Y
smp_rmb() Y R Y Y R
smp_wmb() Y W Y Y W
smp_mb() & synchronize_rcu() CP Y Y Y Y Y Y Y Y
Successful full non-void RMW CP Y Y Y Y Y Y Y Y Y Y Y
smp_mb__before_atomic() CP Y Y Y a a a a Y
smp_mb__after_atomic() CP a a Y Y Y Y Y Y
Key: Relaxed: A relaxed operation is either READ_ONCE(), WRITE_ONCE(),
a *_relaxed() RMW operation, an unsuccessful RMW
operation, a non-value-returning RMW operation such
as atomic_inc(), or one of the atomic*_read() and
atomic*_set() family of operations.
C: Ordering is cumulative
P: Ordering propagates
R: Read, for example, READ_ONCE(), or read portion of RMW
W: Write, for example, WRITE_ONCE(), or write portion of RMW
Y: Provides ordering
a: Provides ordering given intervening RMW atomic operation
DR: Dependent read (address dependency)
DW: Dependent write (address, data, or control dependency)
RMW: Atomic read-modify-write operation
SELF: Orders self, as opposed to accesses before and/or after
SV: Orders later accesses to the same variable
linux-7.2.3/tools/memory-model/Documentation/control-dependencies.txt
+258
View File
@@ -0,0 +1,258 @@
CONTROL DEPENDENCIES
====================
A major difficulty with control dependencies is that current compilers
do not support them. One purpose of this document is therefore to
help you prevent your compiler from breaking your code. However,
control dependencies also pose other challenges, which leads to the
second purpose of this document, namely to help you to avoid breaking
your own code, even in the absence of help from your compiler.
One such challenge is that control dependencies order only later stores.
Therefore, a load-load control dependency will not preserve ordering
unless a read memory barrier is provided. Consider the following code:
q = READ_ONCE(a);
if (q)
p = READ_ONCE(b);
This is not guaranteed to provide any ordering because some types of CPUs
are permitted to predict the result of the load from "b". This prediction
can cause other CPUs to see this load as having happened before the load
from "a". This means that an explicit read barrier is required, for example
as follows:
q = READ_ONCE(a);
if (q) {
smp_rmb();
p = READ_ONCE(b);
}
However, stores are not speculated. This means that ordering is
(usually) guaranteed for load-store control dependencies, as in the
following example:
q = READ_ONCE(a);
if (q)
WRITE_ONCE(b, 1);
Control dependencies can pair with each other and with other types
of ordering. But please note that neither the READ_ONCE() nor the
WRITE_ONCE() are optional. Without the READ_ONCE(), the compiler might
fuse the load from "a" with other loads. Without the WRITE_ONCE(),
the compiler might fuse the store to "b" with other stores. Worse yet,
the compiler might convert the store into a load and a check followed
by a store, and this compiler-generated load would not be ordered by
the control dependency.
Furthermore, if the compiler is able to prove that the value of variable
"a" is always non-zero, it would be well within its rights to optimize
the original example by eliminating the "if" statement as follows:
q = a;
b = 1; /* BUG: Compiler and CPU can both reorder!!! */
So don't leave out either the READ_ONCE() or the WRITE_ONCE().
In particular, although READ_ONCE() does force the compiler to emit a
load, it does *not* force the compiler to actually use the loaded value.
It is tempting to try use control dependencies to enforce ordering on
identical stores on both branches of the "if" statement as follows:
q = READ_ONCE(a);
if (q) {
barrier();
WRITE_ONCE(b, 1);
do_something();
} else {
barrier();
WRITE_ONCE(b, 1);
do_something_else();
}
Unfortunately, current compilers will transform this as follows at high
optimization levels:
q = READ_ONCE(a);
barrier();
WRITE_ONCE(b, 1); /* BUG: No ordering vs. load from a!!! */
if (q) {
/* WRITE_ONCE(b, 1); -- moved up, BUG!!! */
do_something();
} else {
/* WRITE_ONCE(b, 1); -- moved up, BUG!!! */
do_something_else();
}
Now there is no conditional between the load from "a" and the store to
"b", which means that the CPU is within its rights to reorder them: The
conditional is absolutely required, and must be present in the final
assembly code, after all of the compiler and link-time optimizations
have been applied. Therefore, if you need ordering in this example,
you must use explicit memory ordering, for example, smp_store_release():
q = READ_ONCE(a);
if (q) {
smp_store_release(&b, 1);
do_something();
} else {
smp_store_release(&b, 1);
do_something_else();
}
Without explicit memory ordering, control-dependency-based ordering is
guaranteed only when the stores differ, for example:
q = READ_ONCE(a);
if (q) {
WRITE_ONCE(b, 1);
do_something();
} else {
WRITE_ONCE(b, 2);
do_something_else();
}
The initial READ_ONCE() is still required to prevent the compiler from
knowing too much about the value of "a".
But please note that you need to be careful what you do with the local
variable "q", otherwise the compiler might be able to guess the value
and again remove the conditional branch that is absolutely required to
preserve ordering. For example:
q = READ_ONCE(a);
if (q % MAX) {
WRITE_ONCE(b, 1);
do_something();
} else {
WRITE_ONCE(b, 2);
do_something_else();
}
If MAX is compile-time defined to be 1, then the compiler knows that
(q % MAX) must be equal to zero, regardless of the value of "q".
The compiler is therefore within its rights to transform the above code
into the following:
q = READ_ONCE(a);
WRITE_ONCE(b, 2);
do_something_else();
Given this transformation, the CPU is not required to respect the ordering
between the load from variable "a" and the store to variable "b". It is
tempting to add a barrier(), but this does not help. The conditional
is gone, and the barrier won't bring it back. Therefore, if you need
to relying on control dependencies to produce this ordering, you should
make sure that MAX is greater than one, perhaps as follows:
q = READ_ONCE(a);
BUILD_BUG_ON(MAX <= 1); /* Order load from a with store to b. */
if (q % MAX) {
WRITE_ONCE(b, 1);
do_something();
} else {
WRITE_ONCE(b, 2);
do_something_else();
}
Please note once again that each leg of the "if" statement absolutely
must store different values to "b". As in previous examples, if the two
values were identical, the compiler could pull this store outside of the
"if" statement, destroying the control dependency's ordering properties.
You must also be careful avoid relying too much on boolean short-circuit
evaluation. Consider this example:
q = READ_ONCE(a);
if (q || 1 > 0)
WRITE_ONCE(b, 1);
Because the first condition cannot fault and the second condition is
always true, the compiler can transform this example as follows, again
destroying the control dependency's ordering:
q = READ_ONCE(a);
WRITE_ONCE(b, 1);
This is yet another example showing the importance of preventing the
compiler from out-guessing your code. Again, although READ_ONCE() really
does force the compiler to emit code for a given load, the compiler is
within its rights to discard the loaded value.
In addition, control dependencies apply only to the then-clause and
else-clause of the "if" statement in question. In particular, they do
not necessarily order the code following the entire "if" statement:
q = READ_ONCE(a);
if (q) {
WRITE_ONCE(b, 1);
} else {
WRITE_ONCE(b, 2);
}
WRITE_ONCE(c, 1); /* BUG: No ordering against the read from "a". */
It is tempting to argue that there in fact is ordering because the
compiler cannot reorder volatile accesses and also cannot reorder
the writes to "b" with the condition. Unfortunately for this line
of reasoning, the compiler might compile the two writes to "b" as
conditional-move instructions, as in this fanciful pseudo-assembly
language:
ld r1,a
cmp r1,$0
cmov,ne r4,$1
cmov,eq r4,$2
st r4,b
st $1,c
The control dependencies would then extend only to the pair of cmov
instructions and the store depending on them. This means that a weakly
ordered CPU would have no dependency of any sort between the load from
"a" and the store to "c". In short, control dependencies provide ordering
only to the stores in the then-clause and else-clause of the "if" statement
in question (including functions invoked by those two clauses), and not
to code following that "if" statement.
In summary:
(*) Control dependencies can order prior loads against later stores.
However, they do *not* guarantee any other sort of ordering:
Not prior loads against later loads, nor prior stores against
later anything. If you need these other forms of ordering, use
smp_load_acquire(), smp_store_release(), or, in the case of prior
stores and later loads, smp_mb().
(*) If both legs of the "if" statement contain identical stores to
the same variable, then you must explicitly order those stores,
either by preceding both of them with smp_mb() or by using
smp_store_release(). Please note that it is *not* sufficient to use
barrier() at beginning and end of each leg of the "if" statement
because, as shown by the example above, optimizing compilers can
destroy the control dependency while respecting the letter of the
barrier() law.
(*) Control dependencies require at least one run-time conditional
between the prior load and the subsequent store, and this
conditional must involve the prior load. If the compiler is able
to optimize the conditional away, it will have also optimized
away the ordering. Careful use of READ_ONCE() and WRITE_ONCE()
can help to preserve the needed conditional.
(*) Control dependencies require that the compiler avoid reordering the
dependency into nonexistence. Careful use of READ_ONCE() or
atomic{,64}_read() can help to preserve your control dependency.
(*) Control dependencies apply only to the then-clause and else-clause
of the "if" statement containing the control dependency, including
any functions that these two clauses call. Control dependencies
do *not* apply to code beyond the end of that "if" statement.
(*) Control dependencies pair normally with other types of barriers.
(*) Control dependencies do *not* provide multicopy atomicity. If you
need all the CPUs to agree on the ordering of a given store against
all other accesses, use smp_mb().
(*) Compilers do not understand control dependencies. It is therefore
your job to ensure that they do not break your code.
linux-7.2.3/tools/memory-model/Documentation/explanation.txt
+2810
View File
File diff suppressed because it is too large Load Diff
linux-7.2.3/tools/memory-model/Documentation/glossary.txt
+178
View File
@@ -0,0 +1,178 @@
This document contains brief definitions of LKMM-related terms. Like most
glossaries, it is not intended to be read front to back (except perhaps
as a way of confirming a diagnosis of OCD), but rather to be searched
for specific terms.
Address Dependency: When the address of a later memory access is computed
based on the value returned by an earlier load, an "address
dependency" extends from that load extending to the later access.
Address dependencies are quite common in RCU read-side critical
sections:
1 rcu_read_lock();
2 p = rcu_dereference(gp);
3 do_something(p->a);
4 rcu_read_unlock();
In this case, because the address of "p->a" on line 3 is computed
from the value returned by the rcu_dereference() on line 2, the
address dependency extends from that rcu_dereference() to that
"p->a". In rare cases, optimizing compilers can destroy address
dependencies. Please see Documentation/RCU/rcu_dereference.rst
for more information.
See also "Control Dependency" and "Data Dependency".
Acquire: With respect to a lock, acquiring that lock, for example,
using spin_lock(). With respect to a non-lock shared variable,
a special operation that includes a load and which orders that
load before later memory references running on that same CPU.
An example special acquire operation is smp_load_acquire(),
but atomic_read_acquire() and atomic_xchg_acquire() also include
acquire loads.
When an acquire load returns the value stored by a release store
to that same variable, (in other words, the acquire load "reads
from" the release store), then all operations preceding that
store "happen before" any operations following that load acquire.
See also "Happens-Before", "Reads-From", "Relaxed", and "Release".
Coherence (co): When one CPU's store to a given variable overwrites
either the value from another CPU's store or some later value,
there is said to be a coherence link from the second CPU to
the first.
It is also possible to have a coherence link within a CPU, which
is a "coherence internal" (coi) link. The term "coherence
external" (coe) link is used when it is necessary to exclude
the coi case.
See also "From-reads" and "Reads-from".
Control Dependency: When a later store's execution depends on a test
of a value computed from a value returned by an earlier load,
a "control dependency" extends from that load to that store.
For example:
1 if (READ_ONCE(x))
2 WRITE_ONCE(y, 1);
Here, the control dependency extends from the READ_ONCE() on
line 1 to the WRITE_ONCE() on line 2. Control dependencies are
fragile, and can be easily destroyed by optimizing compilers.
Please see control-dependencies.txt for more information.
See also "Address Dependency" and "Data Dependency".
Cycle: Memory-barrier pairing is restricted to a pair of CPUs, as the
name suggests. And in a great many cases, a pair of CPUs is all
that is required. In other cases, the notion of pairing must be
extended to additional CPUs, and the result is called a "cycle".
In a cycle, each CPU's ordering interacts with that of the next:
CPU 0 CPU 1 CPU 2
WRITE_ONCE(x, 1); WRITE_ONCE(y, 1); WRITE_ONCE(z, 1);
smp_mb(); smp_mb(); smp_mb();
r0 = READ_ONCE(y); r1 = READ_ONCE(z); r2 = READ_ONCE(x);
CPU 0's smp_mb() interacts with that of CPU 1, which interacts
with that of CPU 2, which in turn interacts with that of CPU 0
to complete the cycle. Because of the smp_mb() calls between
each pair of memory accesses, the outcome where r0, r1, and r2
are all equal to zero is forbidden by LKMM.
See also "Pairing".
Data Dependency: When the data written by a later store is computed based
on the value returned by an earlier load, a "data dependency"
extends from that load to that later store. For example:
1 r1 = READ_ONCE(x);
2 WRITE_ONCE(y, r1 + 1);
In this case, the data dependency extends from the READ_ONCE()
on line 1 to the WRITE_ONCE() on line 2. Data dependencies are
fragile and can be easily destroyed by optimizing compilers.
Because optimizing compilers put a great deal of effort into
working out what values integer variables might have, this is
especially true in cases where the dependency is carried through
an integer.
See also "Address Dependency" and "Control Dependency".
From-Reads (fr): When one CPU's store to a given variable happened
too late to affect the value returned by another CPU's
load from that same variable, there is said to be a from-reads
link from the load to the store.
It is also possible to have a from-reads link within a CPU, which
is a "from-reads internal" (fri) link. The term "from-reads
external" (fre) link is used when it is necessary to exclude
the fri case.
See also "Coherence" and "Reads-from".
Fully Ordered: An operation such as smp_mb() that orders all of
its CPU's prior accesses with all of that CPU's subsequent
accesses, or a marked access such as atomic_add_return()
that orders all of its CPU's prior accesses, itself, and
all of its CPU's subsequent accesses.
Happens-Before (hb): A relation between two accesses in which LKMM
guarantees the first access precedes the second. For more
detail, please see the "THE HAPPENS-BEFORE RELATION: hb"
section of explanation.txt.
Marked Access: An access to a variable that uses an special function or
macro such as "r1 = READ_ONCE(x)" or "smp_store_release(&a, 1)".
See also "Unmarked Access".
Pairing: "Memory-barrier pairing" reflects the fact that synchronizing
data between two CPUs requires that both CPUs their accesses.
Memory barriers thus tend to come in pairs, one executed by
one of the CPUs and the other by the other CPU. Of course,
pairing also occurs with other types of operations, so that a
smp_store_release() pairs with an smp_load_acquire() that reads
the value stored.
See also "Cycle".
Reads-From (rf): When one CPU's load returns the value stored by some other
CPU, there is said to be a reads-from link from the second
CPU's store to the first CPU's load. Reads-from links have the
nice property that time must advance from the store to the load,
which means that algorithms using reads-from links can use lighter
weight ordering and synchronization compared to algorithms using
coherence and from-reads links.
It is also possible to have a reads-from link within a CPU, which
is a "reads-from internal" (rfi) link. The term "reads-from
external" (rfe) link is used when it is necessary to exclude
the rfi case.
See also Coherence" and "From-reads".
Relaxed: A marked access that does not imply ordering, for example, a
READ_ONCE(), WRITE_ONCE(), a non-value-returning read-modify-write
operation, or a value-returning read-modify-write operation whose
name ends in "_relaxed".
See also "Acquire" and "Release".
Release: With respect to a lock, releasing that lock, for example,
using spin_unlock(). With respect to a non-lock shared variable,
a special operation that includes a store and which orders that
store after earlier memory references that ran on that same CPU.
An example special release store is smp_store_release(), but
atomic_set_release() and atomic_cmpxchg_release() also include
release stores.
See also "Acquire" and "Relaxed".
Unmarked Access: An access to a variable that uses normal C-language
syntax, for example, "a = b[2]";
See also "Marked Access".
linux-7.2.3/tools/memory-model/Documentation/herd-representation.txt
+113
View File
@@ -0,0 +1,113 @@
#
# Legend:
# R, a Load event
# W, a Store event
# F, a Fence event
# LKR, a Lock-Read event
# LKW, a Lock-Write event
# UL, an Unlock event
# LF, a Lock-Fail event
# RL, a Read-Locked event
# RU, a Read-Unlocked event
# R*, a Load event included in RMW
# W*, a Store event included in RMW
# SRCU, a Sleepable-Read-Copy-Update event
#
# po, a Program-Order link
# rmw, a Read-Modify-Write link - every rmw link is a po link
#
# By convention, a blank line in a cell means "same as the preceding line".
#
# Note that the syntactic representation does not always match the sets and
# relations in linux-kernel.cat, due to redefinitions in linux-kernel.bell and
# lock.cat. For example, the po link between LKR and LKW is upgraded to an rmw
# link, and W[ACQUIRE] are not included in the Acquire set.
#
# Disclaimer. The table includes representations of "add" and "and" operations;
# corresponding/identical representations of "sub", "inc", "dec" and "or", "xor",
# "andnot" operations are omitted.
#
------------------------------------------------------------------------------
| C macro | Events |
------------------------------------------------------------------------------
| Non-RMW ops | |
------------------------------------------------------------------------------
| READ_ONCE | R[ONCE] |
| atomic_read | |
| WRITE_ONCE | W[ONCE] |
| atomic_set | |
| smp_load_acquire | R[ACQUIRE] |
| atomic_read_acquire | |
| smp_store_release | W[RELEASE] |
| atomic_set_release | |
| smp_store_mb | W[ONCE] ->po F[MB] |
| smp_mb | F[MB] |
| smp_rmb | F[rmb] |
| smp_wmb | F[wmb] |
| smp_mb__before_atomic | F[before-atomic] |
| smp_mb__after_atomic | F[after-atomic] |
| spin_unlock | UL |
| spin_is_locked | On success: RL |
| | On failure: RU |
| smp_mb__after_spinlock | F[after-spinlock] |
| smp_mb__after_unlock_lock | F[after-unlock-lock] |
| rcu_read_lock | F[rcu-lock] |
| rcu_read_unlock | F[rcu-unlock] |
| synchronize_rcu | F[sync-rcu] |
| rcu_dereference | R[ONCE] |
| rcu_assign_pointer | W[RELEASE] |
| srcu_read_lock | R[srcu-lock] |
| srcu_down_read | |
| srcu_read_unlock | W[srcu-unlock] |
| srcu_up_read | |
| synchronize_srcu | SRCU[sync-srcu] |
| smp_mb__after_srcu_read_unlock | F[after-srcu-read-unlock] |
------------------------------------------------------------------------------
| RMW ops w/o return value | |
------------------------------------------------------------------------------
| atomic_add | R*[NORETURN] ->rmw W*[NORETURN] |
| atomic_and | |
| spin_lock | LKR ->po LKW |
------------------------------------------------------------------------------
| RMW ops w/ return value | |
------------------------------------------------------------------------------
| atomic_add_return | R*[MB] ->rmw W*[MB] |
| atomic_fetch_add | |
| atomic_fetch_and | |
| atomic_xchg | |
| xchg | |
| atomic_add_negative | |
| atomic_add_return_relaxed | R*[ONCE] ->rmw W*[ONCE] |
| atomic_fetch_add_relaxed | |
| atomic_fetch_and_relaxed | |
| atomic_xchg_relaxed | |
| xchg_relaxed | |
| atomic_add_negative_relaxed | |
| atomic_add_return_acquire | R*[ACQUIRE] ->rmw W*[ACQUIRE] |
| atomic_fetch_add_acquire | |
| atomic_fetch_and_acquire | |
| atomic_xchg_acquire | |
| xchg_acquire | |
| atomic_add_negative_acquire | |
| atomic_add_return_release | R*[RELEASE] ->rmw W*[RELEASE] |
| atomic_fetch_add_release | |
| atomic_fetch_and_release | |
| atomic_xchg_release | |
| xchg_release | |
| atomic_add_negative_release | |
------------------------------------------------------------------------------
| Conditional RMW ops | |
------------------------------------------------------------------------------
| atomic_cmpxchg | On success: R*[MB] ->rmw W*[MB] |
| | On failure: R*[MB] |
| cmpxchg | |
| atomic_add_unless | |
| atomic_cmpxchg_relaxed | On success: R*[ONCE] ->rmw W*[ONCE] |
| | On failure: R*[ONCE] |
| atomic_cmpxchg_acquire | On success: R*[ACQUIRE] ->rmw W*[ACQUIRE] |
| | On failure: R*[ACQUIRE] |
| atomic_cmpxchg_release | On success: R*[RELEASE] ->rmw W*[RELEASE] |
| | On failure: R*[RELEASE] |
| spin_trylock | On success: LKR ->po LKW |
| | On failure: LF |
------------------------------------------------------------------------------
linux-7.2.3/tools/memory-model/Documentation/litmus-tests.txt
+1083
View File
File diff suppressed because it is too large Load Diff
linux-7.2.3/tools/memory-model/Documentation/locking.txt
+303
View File
@@ -0,0 +1,303 @@
[!] Note:
This file expands on the "Locking" section of recipes.txt,
focusing on locklessly accessing shared variables that are
otherwise protected by a lock.
Locking
=======
Locking is well-known and the common use cases are straightforward: Any
CPU holding a given lock sees any changes previously seen or made by any
CPU before it previously released that same lock. This last sentence
is the only part of this document that most developers will need to read.
However, developers who would like to also access lock-protected shared
variables outside of their corresponding locks should continue reading.
Locking and Prior Accesses
--------------------------
The basic rule of locking is worth repeating:
Any CPU holding a given lock sees any changes previously seen
or made by any CPU before it previously released that same lock.
Note that this statement is a bit stronger than "Any CPU holding a
given lock sees all changes made by any CPU during the time that CPU was
previously holding this same lock". For example, consider the following
pair of code fragments:
/* See MP+polocks.litmus. */
void CPU0(void)
{
WRITE_ONCE(x, 1);
spin_lock(&mylock);
WRITE_ONCE(y, 1);
spin_unlock(&mylock);
}
void CPU1(void)
{
spin_lock(&mylock);
r0 = READ_ONCE(y);
spin_unlock(&mylock);
r1 = READ_ONCE(x);
}
The basic rule guarantees that if CPU0() acquires mylock before CPU1(),
then both r0 and r1 must be set to the value 1. This also has the
consequence that if the final value of r0 is equal to 1, then the final
value of r1 must also be equal to 1. In contrast, the weaker rule would
say nothing about the final value of r1.
Locking and Subsequent Accesses
-------------------------------
The converse to the basic rule also holds: Any CPU holding a given
lock will not see any changes that will be made by any CPU after it
subsequently acquires this same lock. This converse statement is
illustrated by the following litmus test:
/* See MP+porevlocks.litmus. */
void CPU0(void)
{
r0 = READ_ONCE(y);
spin_lock(&mylock);
r1 = READ_ONCE(x);
spin_unlock(&mylock);
}
void CPU1(void)
{
spin_lock(&mylock);
WRITE_ONCE(x, 1);
spin_unlock(&mylock);
WRITE_ONCE(y, 1);
}
This converse to the basic rule guarantees that if CPU0() acquires
mylock before CPU1(), then both r0 and r1 must be set to the value 0.
This also has the consequence that if the final value of r1 is equal
to 0, then the final value of r0 must also be equal to 0. In contrast,
the weaker rule would say nothing about the final value of r0.
These examples show only a single pair of CPUs, but the effects of the
locking basic rule extend across multiple acquisitions of a given lock
across multiple CPUs.
Double-Checked Locking
----------------------
It is well known that more than just a lock is required to make
double-checked locking work correctly, This litmus test illustrates
one incorrect approach:
/* See Documentation/litmus-tests/locking/DCL-broken.litmus. */
void CPU0(void)
{
r0 = READ_ONCE(flag);
if (r0 == 0) {
spin_lock(&lck);
r1 = READ_ONCE(flag);
if (r1 == 0) {
WRITE_ONCE(data, 1);
WRITE_ONCE(flag, 1);
}
spin_unlock(&lck);
}
r2 = READ_ONCE(data);
}
/* CPU1() is the exactly the same as CPU0(). */
There are two problems. First, there is no ordering between the first
READ_ONCE() of "flag" and the READ_ONCE() of "data". Second, there is
no ordering between the two WRITE_ONCE() calls. It should therefore be
no surprise that "r2" can be zero, and a quick herd7 run confirms this.
One way to fix this is to use smp_load_acquire() and smp_store_release()
as shown in this corrected version:
/* See Documentation/litmus-tests/locking/DCL-fixed.litmus. */
void CPU0(void)
{
r0 = smp_load_acquire(&flag);
if (r0 == 0) {
spin_lock(&lck);
r1 = READ_ONCE(flag);
if (r1 == 0) {
WRITE_ONCE(data, 1);
smp_store_release(&flag, 1);
}
spin_unlock(&lck);
}
r2 = READ_ONCE(data);
}
/* CPU1() is the exactly the same as CPU0(). */
The smp_load_acquire() guarantees that its load from "flags" will
be ordered before the READ_ONCE() from data, thus solving the first
problem. The smp_store_release() guarantees that its store will be
ordered after the WRITE_ONCE() to "data", solving the second problem.
The smp_store_release() pairs with the smp_load_acquire(), thus ensuring
that the ordering provided by each actually takes effect. Again, a
quick herd7 run confirms this.
In short, if you access a lock-protected variable without holding the
corresponding lock, you will need to provide additional ordering, in
this case, via the smp_load_acquire() and the smp_store_release().
Ordering Provided by a Lock to CPUs Not Holding That Lock
---------------------------------------------------------
It is not necessarily the case that accesses ordered by locking will be
seen as ordered by CPUs not holding that lock. Consider this example:
/* See Z6.0+pooncelock+pooncelock+pombonce.litmus. */
void CPU0(void)
{
spin_lock(&mylock);
WRITE_ONCE(x, 1);
WRITE_ONCE(y, 1);
spin_unlock(&mylock);
}
void CPU1(void)
{
spin_lock(&mylock);
r0 = READ_ONCE(y);
WRITE_ONCE(z, 1);
spin_unlock(&mylock);
}
void CPU2(void)
{
WRITE_ONCE(z, 2);
smp_mb();
r1 = READ_ONCE(x);
}
Counter-intuitive though it might be, it is quite possible to have
the final value of r0 be 1, the final value of z be 2, and the final
value of r1 be 0. The reason for this surprising outcome is that CPU2()
never acquired the lock, and thus did not fully benefit from the lock's
ordering properties.
Ordering can be extended to CPUs not holding the lock by careful use
of smp_mb__after_spinlock():
/* See Z6.0+pooncelock+poonceLock+pombonce.litmus. */
void CPU0(void)
{
spin_lock(&mylock);
WRITE_ONCE(x, 1);
WRITE_ONCE(y, 1);
spin_unlock(&mylock);
}
void CPU1(void)
{
spin_lock(&mylock);
smp_mb__after_spinlock();
r0 = READ_ONCE(y);
WRITE_ONCE(z, 1);
spin_unlock(&mylock);
}
void CPU2(void)
{
WRITE_ONCE(z, 2);
smp_mb();
r1 = READ_ONCE(x);
}
This addition of smp_mb__after_spinlock() strengthens the lock
acquisition sufficiently to rule out the counter-intuitive outcome.
In other words, the addition of the smp_mb__after_spinlock() prohibits
the counter-intuitive result where the final value of r0 is 1, the final
value of z is 2, and the final value of r1 is 0.
No Roach-Motel Locking!
-----------------------
This example requires familiarity with the herd7 "filter" clause, so
please read up on that topic in litmus-tests.txt.
It is tempting to allow memory-reference instructions to be pulled
into a critical section, but this cannot be allowed in the general case.
For example, consider a spin loop preceding a lock-based critical section.
Now, herd7 does not model spin loops, but we can emulate one with two
loads, with a "filter" clause to constrain the first to return the
initial value and the second to return the updated value, as shown below:
/* See Documentation/litmus-tests/locking/RM-fixed.litmus. */
void CPU0(void)
{
spin_lock(&lck);
r2 = atomic_inc_return(&y);
WRITE_ONCE(x, 1);
spin_unlock(&lck);
}
void CPU1(void)
{
r0 = READ_ONCE(x);
r1 = READ_ONCE(x);
spin_lock(&lck);
r2 = atomic_inc_return(&y);
spin_unlock(&lck);
}
filter (1:r0=0 /\ 1:r1=1)
exists (1:r2=1)
The variable "x" is the control variable for the emulated spin loop.
CPU0() sets it to "1" while holding the lock, and CPU1() emulates the
spin loop by reading it twice, first into "1:r0" (which should get the
initial value "0") and then into "1:r1" (which should get the updated
value "1").
The "filter" clause takes this into account, constraining "1:r0" to
equal "0" and "1:r1" to equal 1.
Then the "exists" clause checks to see if CPU1() acquired its lock first,
which should not happen given the filter clause because CPU0() updates
"x" while holding the lock. And herd7 confirms this.
But suppose that the compiler was permitted to reorder the spin loop
into CPU1()'s critical section, like this:
/* See Documentation/litmus-tests/locking/RM-broken.litmus. */
void CPU0(void)
{
int r2;
spin_lock(&lck);
r2 = atomic_inc_return(&y);
WRITE_ONCE(x, 1);
spin_unlock(&lck);
}
void CPU1(void)
{
spin_lock(&lck);
r0 = READ_ONCE(x);
r1 = READ_ONCE(x);
r2 = atomic_inc_return(&y);
spin_unlock(&lck);
}
filter (1:r0=0 /\ 1:r1=1)
exists (1:r2=1)
If "1:r0" is equal to "0", "1:r1" can never equal "1" because CPU0()
cannot update "x" while CPU1() holds the lock. And herd7 confirms this,
showing zero executions matching the "filter" criteria.
And this is why Linux-kernel lock and unlock primitives must prevent
code from entering critical sections. It is not sufficient to only
prevent code from leaving them.
linux-7.2.3/tools/memory-model/Documentation/ordering.txt
+556
View File
@@ -0,0 +1,556 @@
This document gives an overview of the categories of memory-ordering
operations provided by the Linux-kernel memory model (LKMM).
Categories of Ordering
======================
This section lists LKMM's three top-level categories of memory-ordering
operations in decreasing order of strength:
1. Barriers (also known as "fences"). A barrier orders some or
all of the CPU's prior operations against some or all of its
subsequent operations.
2. Ordered memory accesses. These operations order themselves
against some or all of the CPU's prior accesses or some or all
of the CPU's subsequent accesses, depending on the subcategory
of the operation.
3. Unordered accesses, as the name indicates, have no ordering
properties except to the extent that they interact with an
operation in the previous categories. This being the real world,
some of these "unordered" operations provide limited ordering
in some special situations.
Each of the above categories is described in more detail by one of the
following sections.
Barriers
========
Each of the following categories of barriers is described in its own
subsection below:
a. Full memory barriers.
b. Read-modify-write (RMW) ordering augmentation barriers.
c. Write memory barrier.
d. Read memory barrier.
e. Compiler barrier.
Note well that many of these primitives generate absolutely no code
in kernels built with CONFIG_SMP=n. Therefore, if you are writing
a device driver, which must correctly order accesses to a physical
device even in kernels built with CONFIG_SMP=n, please use the
ordering primitives provided for that purpose. For example, instead of
smp_mb(), use mb(). See the "Linux Kernel Device Drivers" book or the
https://lwn.net/Articles/698014/ article for more information.
Full Memory Barriers
--------------------
The Linux-kernel primitives that provide full ordering include:
o The smp_mb() full memory barrier.
o Value-returning RMW atomic operations whose names do not end in
_acquire, _release, or _relaxed.
o RCU's grace-period primitives.
First, the smp_mb() full memory barrier orders all of the CPU's prior
accesses against all subsequent accesses from the viewpoint of all CPUs.
In other words, all CPUs will agree that any earlier action taken
by that CPU happened before any later action taken by that same CPU.
For example, consider the following:
WRITE_ONCE(x, 1);
smp_mb(); // Order store to x before load from y.
r1 = READ_ONCE(y);
All CPUs will agree that the store to "x" happened before the load
from "y", as indicated by the comment. And yes, please comment your
memory-ordering primitives. It is surprisingly hard to remember their
purpose after even a few months.
Second, some RMW atomic operations provide full ordering. These
operations include value-returning RMW atomic operations (that is, those
with non-void return types) whose names do not end in _acquire, _release,
or _relaxed. Examples include atomic_add_return(), atomic_dec_and_test(),
cmpxchg(), and xchg(). Note that conditional RMW atomic operations such
as cmpxchg() are only guaranteed to provide ordering when they succeed.
When RMW atomic operations provide full ordering, they partition the
CPU's accesses into three groups:
1. All code that executed prior to the RMW atomic operation.
2. The RMW atomic operation itself.
3. All code that executed after the RMW atomic operation.
All CPUs will agree that any operation in a given partition happened
before any operation in a higher-numbered partition.
In contrast, non-value-returning RMW atomic operations (that is, those
with void return types) do not guarantee any ordering whatsoever. Nor do
value-returning RMW atomic operations whose names end in _relaxed.
Examples of the former include atomic_inc() and atomic_dec(),
while examples of the latter include atomic_cmpxchg_relaxed() and
atomic_xchg_relaxed(). Similarly, value-returning non-RMW atomic
operations such as atomic_read() do not guarantee full ordering, and
are covered in the later section on unordered operations.
Value-returning RMW atomic operations whose names end in _acquire or
_release provide limited ordering, and will be described later in this
document.
Finally, RCU's grace-period primitives provide full ordering. These
primitives include synchronize_rcu(), synchronize_rcu_expedited(),
synchronize_srcu() and so on. However, these primitives have orders
of magnitude greater overhead than smp_mb(), atomic_xchg(), and so on.
Furthermore, RCU's grace-period primitives can only be invoked in
sleepable contexts. Therefore, RCU's grace-period primitives are
typically instead used to provide ordering against RCU read-side critical
sections, as documented in their comment headers. But of course if you
need a synchronize_rcu() to interact with readers, it costs you nothing
to also rely on its additional full-memory-barrier semantics. Just please
carefully comment this, otherwise your future self will hate you.
RMW Ordering Augmentation Barriers
----------------------------------
As noted in the previous section, non-value-returning RMW operations
such as atomic_inc() and atomic_dec() guarantee no ordering whatsoever.
Nevertheless, a number of popular CPU families, including x86, provide
full ordering for these primitives. One way to obtain full ordering on
all architectures is to add a call to smp_mb():
WRITE_ONCE(x, 1);
atomic_inc(&my_counter);
smp_mb(); // Inefficient on x86!!!
r1 = READ_ONCE(y);
This works, but the added smp_mb() adds needless overhead for
x86, on which atomic_inc() provides full ordering all by itself.
The smp_mb__after_atomic() primitive can be used instead:
WRITE_ONCE(x, 1);
atomic_inc(&my_counter);
smp_mb__after_atomic(); // Order store to x before load from y.
r1 = READ_ONCE(y);
The smp_mb__after_atomic() primitive emits code only on CPUs whose
atomic_inc() implementations do not guarantee full ordering, thus
incurring no unnecessary overhead on x86. There are a number of
variations on the smp_mb__*() theme:
o smp_mb__before_atomic(), which provides full ordering prior
to an unordered RMW atomic operation.
o smp_mb__after_atomic(), which, as shown above, provides full
ordering subsequent to an unordered RMW atomic operation.
o smp_mb__after_spinlock(), which provides full ordering subsequent
to a successful spinlock acquisition. Note that spin_lock() is
always successful but spin_trylock() might not be.
o smp_mb__after_srcu_read_unlock(), which provides full ordering
subsequent to an srcu_read_unlock().
It is bad practice to place code between the smp__*() primitive and the
operation whose ordering that it is augmenting. The reason is that the
ordering of this intervening code will differ from one CPU architecture
to another.
Write Memory Barrier
--------------------
The Linux kernel's write memory barrier is smp_wmb(). If a CPU executes
the following code:
WRITE_ONCE(x, 1);
smp_wmb();
WRITE_ONCE(y, 1);
Then any given CPU will see the write to "x" has having happened before
the write to "y". However, you are usually better off using a release
store, as described in the "Release Operations" section below.
Note that smp_wmb() might fail to provide ordering for unmarked C-language
stores because profile-driven optimization could determine that the
value being overwritten is almost always equal to the new value. Such a
compiler might then reasonably decide to transform "x = 1" and "y = 1"
as follows:
if (x != 1)
x = 1;
smp_wmb(); // BUG: does not order the reads!!!
if (y != 1)
y = 1;
Therefore, if you need to use smp_wmb() with unmarked C-language writes,
you will need to make sure that none of the compilers used to build
the Linux kernel carry out this sort of transformation, both now and in
the future.
Read Memory Barrier
-------------------
The Linux kernel's read memory barrier is smp_rmb(). If a CPU executes
the following code:
r0 = READ_ONCE(y);
smp_rmb();
r1 = READ_ONCE(x);
Then any given CPU will see the read from "y" as having preceded the read from
"x". However, you are usually better off using an acquire load, as described
in the "Acquire Operations" section below.
Compiler Barrier
----------------
The Linux kernel's compiler barrier is barrier(). This primitive
prohibits compiler code-motion optimizations that might move memory
references across the point in the code containing the barrier(), but
does not constrain hardware memory ordering. For example, this can be
used to prevent the compiler from moving code across an infinite loop:
WRITE_ONCE(x, 1);
while (dontstop)
barrier();
r1 = READ_ONCE(y);
Without the barrier(), the compiler would be within its rights to move the
WRITE_ONCE() to follow the loop. This code motion could be problematic
in the case where an interrupt handler terminates the loop. Another way
to handle this is to use READ_ONCE() for the load of "dontstop".
Note that the barriers discussed previously use barrier() or its low-level
equivalent in their implementations.
Ordered Memory Accesses
=======================
The Linux kernel provides a wide variety of ordered memory accesses:
a. Release operations.
b. Acquire operations.
c. RCU read-side ordering.
d. Control dependencies.
Each of the above categories has its own section below.
Release Operations
------------------
Release operations include smp_store_release(), atomic_set_release(),
rcu_assign_pointer(), and value-returning RMW operations whose names
end in _release. These operations order their own store against all
of the CPU's prior memory accesses. Release operations often provide
improved readability and performance compared to explicit barriers.
For example, use of smp_store_release() saves a line compared to the
smp_wmb() example above:
WRITE_ONCE(x, 1);
smp_store_release(&y, 1);
More important, smp_store_release() makes it easier to connect up the
different pieces of the concurrent algorithm. The variable stored to
by the smp_store_release(), in this case "y", will normally be used in
an acquire operation in other parts of the concurrent algorithm.
To see the performance advantages, suppose that the above example reads
from "x" instead of writing to it. Then an smp_wmb() could not guarantee
ordering, and an smp_mb() would be needed instead:
r1 = READ_ONCE(x);
smp_mb();
WRITE_ONCE(y, 1);
But smp_mb() often incurs much higher overhead than does
smp_store_release(), which still provides the needed ordering of "x"
against "y". On x86, the version using smp_store_release() might compile
to a simple load instruction followed by a simple store instruction.
In contrast, the smp_mb() compiles to an expensive instruction that
provides the needed ordering.
There is a wide variety of release operations:
o Store operations, including not only the aforementioned
smp_store_release(), but also atomic_set_release(), and
atomic_long_set_release().
o RCU's rcu_assign_pointer() operation. This is the same as
smp_store_release() except that: (1) It takes the pointer to
be assigned to instead of a pointer to that pointer, (2) It
is intended to be used in conjunction with rcu_dereference()
and similar rather than smp_load_acquire(), and (3) It checks
for an RCU-protected pointer in "sparse" runs.
o Value-returning RMW operations whose names end in _release,
such as atomic_fetch_add_release() and cmpxchg_release().
Note that release ordering is guaranteed only against the
memory-store portion of the RMW operation, and not against the
memory-load portion. Note also that conditional operations such
as cmpxchg_release() are only guaranteed to provide ordering
when they succeed.
As mentioned earlier, release operations are often paired with acquire
operations, which are the subject of the next section.
Acquire Operations
------------------
Acquire operations include smp_load_acquire(), atomic_read_acquire(),
and value-returning RMW operations whose names end in _acquire. These
operations order their own load against all of the CPU's subsequent
memory accesses. Acquire operations often provide improved performance
and readability compared to explicit barriers. For example, use of
smp_load_acquire() saves a line compared to the smp_rmb() example above:
r0 = smp_load_acquire(&y);
r1 = READ_ONCE(x);
As with smp_store_release(), this also makes it easier to connect
the different pieces of the concurrent algorithm by looking for the
smp_store_release() that stores to "y". In addition, smp_load_acquire()
improves upon smp_rmb() by ordering against subsequent stores as well
as against subsequent loads.
There are a couple of categories of acquire operations:
o Load operations, including not only the aforementioned
smp_load_acquire(), but also atomic_read_acquire(), and
atomic64_read_acquire().
o Value-returning RMW operations whose names end in _acquire,
such as atomic_xchg_acquire() and atomic_cmpxchg_acquire().
Note that acquire ordering is guaranteed only against the
memory-load portion of the RMW operation, and not against the
memory-store portion. Note also that conditional operations
such as atomic_cmpxchg_acquire() are only guaranteed to provide
ordering when they succeed.
Symmetry being what it is, acquire operations are often paired with the
release operations covered earlier. For example, consider the following
example, where task0() and task1() execute concurrently:
void task0(void)
{
WRITE_ONCE(x, 1);
smp_store_release(&y, 1);
}
void task1(void)
{
r0 = smp_load_acquire(&y);
r1 = READ_ONCE(x);
}
If "x" and "y" are both initially zero, then either r0's final value
will be zero or r1's final value will be one, thus providing the required
ordering.
RCU Read-Side Ordering
----------------------
This category includes read-side markers such as rcu_read_lock()
and rcu_read_unlock() as well as pointer-traversal primitives such as
rcu_dereference() and srcu_dereference().
Compared to locking primitives and RMW atomic operations, markers
for RCU read-side critical sections incur very low overhead because
they interact only with the corresponding grace-period primitives.
For example, the rcu_read_lock() and rcu_read_unlock() markers interact
with synchronize_rcu(), synchronize_rcu_expedited(), and call_rcu().
The way this works is that if a given call to synchronize_rcu() cannot
prove that it started before a given call to rcu_read_lock(), then
that synchronize_rcu() must block until the matching rcu_read_unlock()
is reached. For more information, please see the synchronize_rcu()
docbook header comment and the material in Documentation/RCU.
RCU's pointer-traversal primitives, including rcu_dereference() and
srcu_dereference(), order their load (which must be a pointer) against any
of the CPU's subsequent memory accesses whose address has been calculated
from the value loaded. There is said to be an *address dependency*
from the value returned by the rcu_dereference() or srcu_dereference()
to that subsequent memory access.
A call to rcu_dereference() for a given RCU-protected pointer is
usually paired with a call to rcu_assign_pointer() for that same pointer
in much the same way that a call to smp_load_acquire() is paired with
a call to smp_store_release(). Calls to rcu_dereference() and
rcu_assign_pointer() are often buried in other APIs, for example,
the RCU list API members defined in include/linux/rculist.h. For more
information, please see the docbook headers in that file, the most
recent LWN article on the RCU API (https://lwn.net/Articles/988638/),
and of course the material in Documentation/RCU.
If the pointer value is manipulated between the rcu_dereference()
that returned it and a later rcu_dereference(), please read
Documentation/RCU/rcu_dereference.rst. It can also be quite helpful to
review uses in the Linux kernel.
Control Dependencies
--------------------
A control dependency extends from a marked load (READ_ONCE() or stronger)
through an "if" condition to a marked store (WRITE_ONCE() or stronger)
that is executed only by one of the legs of that "if" statement.
Control dependencies are so named because they are mediated by
control-flow instructions such as comparisons and conditional branches.
In short, you can use a control dependency to enforce ordering between
an READ_ONCE() and a WRITE_ONCE() when there is an "if" condition
between them. The canonical example is as follows:
q = READ_ONCE(a);
if (q)
WRITE_ONCE(b, 1);
In this case, all CPUs would see the read from "a" as happening before
the write to "b".
However, control dependencies are easily destroyed by compiler
optimizations, so any use of control dependencies must take into account
all of the compilers used to build the Linux kernel. Please see the
"control-dependencies.txt" file for more information.
Unordered Accesses
==================
Each of these two categories of unordered accesses has a section below:
a. Unordered marked operations.
b. Unmarked C-language accesses.
Unordered Marked Operations
---------------------------
Unordered operations to different variables are just that, unordered.
However, if a group of CPUs apply these operations to a single variable,
all the CPUs will agree on the operation order. Of course, the ordering
of unordered marked accesses can also be constrained using the mechanisms
described earlier in this document.
These operations come in three categories:
o Marked writes, such as WRITE_ONCE() and atomic_set(). These
primitives require the compiler to emit the corresponding store
instructions in the expected execution order, thus suppressing
a number of destructive optimizations. However, they provide no
hardware ordering guarantees, and in fact many CPUs will happily
reorder marked writes with each other or with other unordered
operations, unless these operations are to the same variable.
o Marked reads, such as READ_ONCE() and atomic_read(). These
primitives require the compiler to emit the corresponding load
instructions in the expected execution order, thus suppressing
a number of destructive optimizations. However, they provide no
hardware ordering guarantees, and in fact many CPUs will happily
reorder marked reads with each other or with other unordered
operations, unless these operations are to the same variable.
o Unordered RMW atomic operations. These are non-value-returning
RMW atomic operations whose names do not end in _acquire or
_release, and also value-returning RMW operations whose names
end in _relaxed. Examples include atomic_add(), atomic_or(),
and atomic64_fetch_xor_relaxed(). These operations do carry
out the specified RMW operation atomically, for example, five
concurrent atomic_inc() operations applied to a given variable
will reliably increase the value of that variable by five.
However, many CPUs will happily reorder these operations with
each other or with other unordered operations.
This category of operations can be efficiently ordered using
smp_mb__before_atomic() and smp_mb__after_atomic(), as was
discussed in the "RMW Ordering Augmentation Barriers" section.
In short, these operations can be freely reordered unless they are all
operating on a single variable or unless they are constrained by one of
the operations called out earlier in this document.
Unmarked C-Language Accesses
----------------------------
Unmarked C-language accesses are normal variable accesses to normal
variables, that is, to variables that are not "volatile" and are not
C11 atomic variables. These operations provide no ordering guarantees,
and further do not guarantee "atomic" access. For example, the compiler
might (and sometimes does) split a plain C-language store into multiple
smaller stores. A load from that same variable running on some other
CPU while such a store is executing might see a value that is a mashup
of the old value and the new value.
Unmarked C-language accesses are unordered, and are also subject to
any number of compiler optimizations, many of which can break your
concurrent code. It is possible to use unmarked C-language accesses for
shared variables that are subject to concurrent access, but great care
is required on an ongoing basis. The compiler-constraining barrier()
primitive can be helpful, as can the various ordering primitives discussed
in this document. It nevertheless bears repeating that use of unmarked
C-language accesses requires careful attention to not just your code,
but to all the compilers that might be used to build it. Such compilers
might replace a series of loads with a single load, and might replace
a series of stores with a single store. Some compilers will even split
a single store into multiple smaller stores.
But there are some ways of using unmarked C-language accesses for shared
variables without such worries:
o Guard all accesses to a given variable by a particular lock,
so that there are never concurrent conflicting accesses to
that variable. (There are "conflicting accesses" when
(1) at least one of the concurrent accesses to a variable is an
unmarked C-language access and (2) when at least one of those
accesses is a write, whether marked or not.)
o As above, but using other synchronization primitives such
as reader-writer locks or sequence locks.
o Use locking or other means to ensure that all concurrent accesses
to a given variable are reads.
o Restrict use of a given variable to statistics or heuristics
where the occasional bogus value can be tolerated.
o Declare the accessed variables as C11 atomics.
https://lwn.net/Articles/691128/
o Declare the accessed variables as "volatile".
If you need to live more dangerously, please do take the time to
understand the compilers. One place to start is these two LWN
articles:
Who's afraid of a big bad optimizing compiler?
https://lwn.net/Articles/793253
Calibrating your fear of big bad optimizing compilers
https://lwn.net/Articles/799218
Used properly, unmarked C-language accesses can reduce overhead on
fastpaths. However, the price is great care and continual attention
to your compiler as new versions come out and as new optimizations
are enabled.
linux-7.2.3/tools/memory-model/Documentation/recipes.txt
+574
View File
@@ -0,0 +1,574 @@
This document provides "recipes", that is, litmus tests for commonly
occurring situations, as well as a few that illustrate subtly broken but
attractive nuisances. Many of these recipes include example code from
v5.7 of the Linux kernel.
The first section covers simple special cases, the second section
takes off the training wheels to cover more involved examples,
and the third section provides a few rules of thumb.
Simple special cases
====================
This section presents two simple special cases, the first being where
there is only one CPU or only one memory location is accessed, and the
second being use of that old concurrency workhorse, locking.
Single CPU or single memory location
------------------------------------
If there is only one CPU on the one hand or only one variable
on the other, the code will execute in order. There are (as
usual) some things to be careful of:
1. Some aspects of the C language are unordered. For example,
in the expression "f(x) + g(y)", the order in which f and g are
called is not defined; the object code is allowed to use either
order or even to interleave the computations.
2. Compilers are permitted to use the "as-if" rule. That is, a
compiler can emit whatever code it likes for normal accesses,
as long as the results of a single-threaded execution appear
just as if the compiler had followed all the relevant rules.
To see this, compile with a high level of optimization and run
the debugger on the resulting binary.
3. If there is only one variable but multiple CPUs, that variable
must be properly aligned and all accesses to that variable must
be full sized. Variables that straddle cachelines or pages void
your full-ordering warranty, as do undersized accesses that load
from or store to only part of the variable.
4. If there are multiple CPUs, accesses to shared variables should
use READ_ONCE() and WRITE_ONCE() or stronger to prevent load/store
tearing, load/store fusing, and invented loads and stores.
There are exceptions to this rule, including:
i. When there is no possibility of a given shared variable
being updated by some other CPU, for example, while
holding the update-side lock, reads from that variable
need not use READ_ONCE().
ii. When there is no possibility of a given shared variable
being either read or updated by other CPUs, for example,
when running during early boot, reads from that variable
need not use READ_ONCE() and writes to that variable
need not use WRITE_ONCE().
Locking
-------
[!] Note:
locking.txt expands on this section, providing more detail on
locklessly accessing lock-protected shared variables.
Locking is well-known and straightforward, at least if you don't think
about it too hard. And the basic rule is indeed quite simple: Any CPU that
has acquired a given lock sees any changes previously seen or made by any
CPU before it released that same lock. Note that this statement is a bit
stronger than "Any CPU holding a given lock sees all changes made by any
CPU during the time that CPU was holding this same lock". For example,
consider the following pair of code fragments:
/* See MP+polocks.litmus. */
void CPU0(void)
{
WRITE_ONCE(x, 1);
spin_lock(&mylock);
WRITE_ONCE(y, 1);
spin_unlock(&mylock);
}
void CPU1(void)
{
spin_lock(&mylock);
r0 = READ_ONCE(y);
spin_unlock(&mylock);
r1 = READ_ONCE(x);
}
The basic rule guarantees that if CPU0() acquires mylock before CPU1(),
then both r0 and r1 must be set to the value 1. This also has the
consequence that if the final value of r0 is equal to 1, then the final
value of r1 must also be equal to 1. In contrast, the weaker rule would
say nothing about the final value of r1.
The converse to the basic rule also holds, as illustrated by the
following litmus test:
/* See MP+porevlocks.litmus. */
void CPU0(void)
{
r0 = READ_ONCE(y);
spin_lock(&mylock);
r1 = READ_ONCE(x);
spin_unlock(&mylock);
}
void CPU1(void)
{
spin_lock(&mylock);
WRITE_ONCE(x, 1);
spin_unlock(&mylock);
WRITE_ONCE(y, 1);
}
This converse to the basic rule guarantees that if CPU0() acquires
mylock before CPU1(), then both r0 and r1 must be set to the value 0.
This also has the consequence that if the final value of r1 is equal
to 0, then the final value of r0 must also be equal to 0. In contrast,
the weaker rule would say nothing about the final value of r0.
These examples show only a single pair of CPUs, but the effects of the
locking basic rule extend across multiple acquisitions of a given lock
across multiple CPUs.
However, it is not necessarily the case that accesses ordered by
locking will be seen as ordered by CPUs not holding that lock.
Consider this example:
/* See Z6.0+pooncelock+pooncelock+pombonce.litmus. */
void CPU0(void)
{
spin_lock(&mylock);
WRITE_ONCE(x, 1);
WRITE_ONCE(y, 1);
spin_unlock(&mylock);
}
void CPU1(void)
{
spin_lock(&mylock);
r0 = READ_ONCE(y);
WRITE_ONCE(z, 1);
spin_unlock(&mylock);
}
void CPU2(void)
{
WRITE_ONCE(z, 2);
smp_mb();
r1 = READ_ONCE(x);
}
Counter-intuitive though it might be, it is quite possible to have
the final value of r0 be 1, the final value of z be 2, and the final
value of r1 be 0. The reason for this surprising outcome is that
CPU2() never acquired the lock, and thus did not benefit from the
lock's ordering properties.
Ordering can be extended to CPUs not holding the lock by careful use
of smp_mb__after_spinlock():
/* See Z6.0+pooncelock+poonceLock+pombonce.litmus. */
void CPU0(void)
{
spin_lock(&mylock);
WRITE_ONCE(x, 1);
WRITE_ONCE(y, 1);
spin_unlock(&mylock);
}
void CPU1(void)
{
spin_lock(&mylock);
smp_mb__after_spinlock();
r0 = READ_ONCE(y);
WRITE_ONCE(z, 1);
spin_unlock(&mylock);
}
void CPU2(void)
{
WRITE_ONCE(z, 2);
smp_mb();
r1 = READ_ONCE(x);
}
This addition of smp_mb__after_spinlock() strengthens the lock acquisition
sufficiently to rule out the counter-intuitive outcome.
Taking off the training wheels
==============================
This section looks at more complex examples, including message passing,
load buffering, release-acquire chains, store buffering.
Many classes of litmus tests have abbreviated names, which may be found
here: https://www.cl.cam.ac.uk/~pes20/ppc-supplemental/test6.pdf
Message passing (MP)
--------------------
The MP pattern has one CPU execute a pair of stores to a pair of variables
and another CPU execute a pair of loads from this same pair of variables,
but in the opposite order. The goal is to avoid the counter-intuitive
outcome in which the first load sees the value written by the second store
but the second load does not see the value written by the first store.
In the absence of any ordering, this goal may not be met, as can be seen
in the MP+poonceonces.litmus litmus test. This section therefore looks at
a number of ways of meeting this goal.
Release and acquire
~~~~~~~~~~~~~~~~~~~
Use of smp_store_release() and smp_load_acquire() is one way to force
the desired MP ordering. The general approach is shown below:
/* See MP+pooncerelease+poacquireonce.litmus. */
void CPU0(void)
{
WRITE_ONCE(x, 1);
smp_store_release(&y, 1);
}
void CPU1(void)
{
r0 = smp_load_acquire(&y);
r1 = READ_ONCE(x);
}
The smp_store_release() macro orders any prior accesses against the
store, while the smp_load_acquire macro orders the load against any
subsequent accesses. Therefore, if the final value of r0 is the value 1,
the final value of r1 must also be the value 1.
The init_stack_slab() function in lib/stackdepot.c uses release-acquire
in this way to safely initialize of a slab of the stack. Working out
the mutual-exclusion design is left as an exercise for the reader.
Assign and dereference
~~~~~~~~~~~~~~~~~~~~~~
Use of rcu_assign_pointer() and rcu_dereference() is quite similar to the
use of smp_store_release() and smp_load_acquire(), except that both
rcu_assign_pointer() and rcu_dereference() operate on RCU-protected
pointers. The general approach is shown below:
/* See MP+onceassign+derefonce.litmus. */
int z;
int *y = &z;
int x;
void CPU0(void)
{
WRITE_ONCE(x, 1);
rcu_assign_pointer(y, &x);
}
void CPU1(void)
{
rcu_read_lock();
r0 = rcu_dereference(y);
r1 = READ_ONCE(*r0);
rcu_read_unlock();
}
In this example, if the final value of r0 is &x then the final value of
r1 must be 1.
The rcu_assign_pointer() macro has the same ordering properties as does
smp_store_release(), but the rcu_dereference() macro orders the load only
against later accesses that depend on the value loaded. A dependency
is present if the value loaded determines the address of a later access
(address dependency, as shown above), the value written by a later store
(data dependency), or whether or not a later store is executed in the
first place (control dependency). Note that the term "data dependency"
is sometimes casually used to cover both address and data dependencies.
In lib/math/prime_numbers.c, the expand_to_next_prime() function invokes
rcu_assign_pointer(), and the next_prime_number() function invokes
rcu_dereference(). This combination mediates access to a bit vector
that is expanded as additional primes are needed.
Write and read memory barriers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It is usually better to use smp_store_release() instead of smp_wmb()
and to use smp_load_acquire() instead of smp_rmb(). However, the older
smp_wmb() and smp_rmb() APIs are still heavily used, so it is important
to understand their use cases. The general approach is shown below:
/* See MP+fencewmbonceonce+fencermbonceonce.litmus. */
void CPU0(void)
{
WRITE_ONCE(x, 1);
smp_wmb();
WRITE_ONCE(y, 1);
}
void CPU1(void)
{
r0 = READ_ONCE(y);
smp_rmb();
r1 = READ_ONCE(x);
}
The smp_wmb() macro orders prior stores against later stores, and the
smp_rmb() macro orders prior loads against later loads. Therefore, if
the final value of r0 is 1, the final value of r1 must also be 1.
The xlog_state_switch_iclogs() function in fs/xfs/xfs_log.c contains
the following write-side code fragment:
log->l_curr_block -= log->l_logBBsize;
ASSERT(log->l_curr_block >= 0);
smp_wmb();
log->l_curr_cycle++;
And the xlog_valid_lsn() function in fs/xfs/xfs_log_priv.h contains
the corresponding read-side code fragment:
cur_cycle = READ_ONCE(log->l_curr_cycle);
smp_rmb();
cur_block = READ_ONCE(log->l_curr_block);
Alternatively, consider the following comment in function
perf_output_put_handle() in kernel/events/ring_buffer.c:
* kernel user
*
* if (LOAD ->data_tail) { LOAD ->data_head
* (A) smp_rmb() (C)
* STORE $data LOAD $data
* smp_wmb() (B) smp_mb() (D)
* STORE ->data_head STORE ->data_tail
* }
The B/C pairing is an example of the MP pattern using smp_wmb() on the
write side and smp_rmb() on the read side.
Of course, given that smp_mb() is strictly stronger than either smp_wmb()
or smp_rmb(), any code fragment that would work with smp_rmb() and
smp_wmb() would also work with smp_mb() replacing either or both of the
weaker barriers.
Load buffering (LB)
-------------------
The LB pattern has one CPU load from one variable and then store to a
second, while another CPU loads from the second variable and then stores
to the first. The goal is to avoid the counter-intuitive situation where
each load reads the value written by the other CPU's store. In the
absence of any ordering it is quite possible that this may happen, as
can be seen in the LB+poonceonces.litmus litmus test.
One way of avoiding the counter-intuitive outcome is through the use of a
control dependency paired with a full memory barrier:
/* See LB+fencembonceonce+ctrlonceonce.litmus. */
void CPU0(void)
{
r0 = READ_ONCE(x);
if (r0)
WRITE_ONCE(y, 1);
}
void CPU1(void)
{
r1 = READ_ONCE(y);
smp_mb();
WRITE_ONCE(x, 1);
}
This pairing of a control dependency in CPU0() with a full memory
barrier in CPU1() prevents r0 and r1 from both ending up equal to 1.
The A/D pairing from the ring-buffer use case shown earlier also
illustrates LB. Here is a repeat of the comment in
perf_output_put_handle() in kernel/events/ring_buffer.c, showing a
control dependency on the kernel side and a full memory barrier on
the user side:
* kernel user
*
* if (LOAD ->data_tail) { LOAD ->data_head
* (A) smp_rmb() (C)
* STORE $data LOAD $data
* smp_wmb() (B) smp_mb() (D)
* STORE ->data_head STORE ->data_tail
* }
*
* Where A pairs with D, and B pairs with C.
The kernel's control dependency between the load from ->data_tail
and the store to data combined with the user's full memory barrier
between the load from data and the store to ->data_tail prevents
the counter-intuitive outcome where the kernel overwrites the data
before the user gets done loading it.
Release-acquire chains
----------------------
Release-acquire chains are a low-overhead, flexible, and easy-to-use
method of maintaining order. However, they do have some limitations that
need to be fully understood. Here is an example that maintains order:
/* See ISA2+pooncerelease+poacquirerelease+poacquireonce.litmus. */
void CPU0(void)
{
WRITE_ONCE(x, 1);
smp_store_release(&y, 1);
}
void CPU1(void)
{
r0 = smp_load_acquire(y);
smp_store_release(&z, 1);
}
void CPU2(void)
{
r1 = smp_load_acquire(z);
r2 = READ_ONCE(x);
}
In this case, if r0 and r1 both have final values of 1, then r2 must
also have a final value of 1.
The ordering in this example is stronger than it needs to be. For
example, ordering would still be preserved if CPU1()'s smp_load_acquire()
invocation was replaced with READ_ONCE().
It is tempting to assume that CPU0()'s store to x is globally ordered
before CPU1()'s store to z, but this is not the case:
/* See Z6.0+pooncerelease+poacquirerelease+mbonceonce.litmus. */
void CPU0(void)
{
WRITE_ONCE(x, 1);
smp_store_release(&y, 1);
}
void CPU1(void)
{
r0 = smp_load_acquire(y);
smp_store_release(&z, 1);
}
void CPU2(void)
{
WRITE_ONCE(z, 2);
smp_mb();
r1 = READ_ONCE(x);
}
One might hope that if the final value of r0 is 1 and the final value
of z is 2, then the final value of r1 must also be 1, but it really is
possible for r1 to have the final value of 0. The reason, of course,
is that in this version, CPU2() is not part of the release-acquire chain.
This situation is accounted for in the rules of thumb below.
Despite this limitation, release-acquire chains are low-overhead as
well as simple and powerful, at least as memory-ordering mechanisms go.
Store buffering
---------------
Store buffering can be thought of as upside-down load buffering, so
that one CPU first stores to one variable and then loads from a second,
while another CPU stores to the second variable and then loads from the
first. Preserving order requires nothing less than full barriers:
/* See SB+fencembonceonces.litmus. */
void CPU0(void)
{
WRITE_ONCE(x, 1);
smp_mb();
r0 = READ_ONCE(y);
}
void CPU1(void)
{
WRITE_ONCE(y, 1);
smp_mb();
r1 = READ_ONCE(x);
}
Omitting either smp_mb() will allow both r0 and r1 to have final
values of 0, but providing both full barriers as shown above prevents
this counter-intuitive outcome.
This pattern most famously appears as part of Dekker's locking
algorithm, but it has a much more practical use within the Linux kernel
of ordering wakeups. The following comment taken from waitqueue_active()
in include/linux/wait.h shows the canonical pattern:
* CPU0 - waker CPU1 - waiter
*
* for (;;) {
* @cond = true; prepare_to_wait(&wq_head, &wait, state);
* smp_mb(); // smp_mb() from set_current_state()
* if (waitqueue_active(wq_head)) if (@cond)
* wake_up(wq_head); break;
* schedule();
* }
* finish_wait(&wq_head, &wait);
On CPU0, the store is to @cond and the load is in waitqueue_active().
On CPU1, prepare_to_wait() contains both a store to wq_head and a call
to set_current_state(), which contains an smp_mb() barrier; the load is
"if (@cond)". The full barriers prevent the undesirable outcome where
CPU1 puts the waiting task to sleep and CPU0 fails to wake it up.
Note that use of locking can greatly simplify this pattern.
Rules of thumb
==============
There might seem to be no pattern governing what ordering primitives are
needed in which situations, but this is not the case. There is a pattern
based on the relation between the accesses linking successive CPUs in a
given litmus test. There are three types of linkage:
1. Write-to-read, where the next CPU reads the value that the
previous CPU wrote. The LB litmus-test patterns contain only
this type of relation. In formal memory-modeling texts, this
relation is called "reads-from" and is usually abbreviated "rf".
2. Read-to-write, where the next CPU overwrites the value that the
previous CPU read. The SB litmus test contains only this type
of relation. In formal memory-modeling texts, this relation is
often called "from-reads" and is sometimes abbreviated "fr".
3. Write-to-write, where the next CPU overwrites the value written
by the previous CPU. The Z6.0 litmus test pattern contains a
write-to-write relation between the last access of CPU1() and
the first access of CPU2(). In formal memory-modeling texts,
this relation is often called "coherence order" and is sometimes
abbreviated "co". In the C++ standard, it is instead called
"modification order" and often abbreviated "mo".
The strength of memory ordering required for a given litmus test to
avoid a counter-intuitive outcome depends on the types of relations
linking the memory accesses for the outcome in question:
o If all links are write-to-read links, then the weakest
possible ordering within each CPU suffices. For example, in
the LB litmus test, a control dependency was enough to do the
job.
o If all but one of the links are write-to-read links, then a
release-acquire chain suffices. Both the MP and the ISA2
litmus tests illustrate this case.
o If more than one of the links are something other than
write-to-read links, then a full memory barrier is required
between each successive pair of non-write-to-read links. This
case is illustrated by the Z6.0 litmus tests, both in the
locking and in the release-acquire sections.
However, if you find yourself having to stretch these rules of thumb
to fit your situation, you should consider creating a litmus test and
running it on the model.
linux-7.2.3/tools/memory-model/Documentation/references.txt
+130
View File
@@ -0,0 +1,130 @@
This document provides background reading for memory models and related
tools. These documents are aimed at kernel hackers who are interested
in memory models.
Hardware manuals and models
===========================
o SPARC International Inc. (Ed.). 1994. "The SPARC Architecture
Reference Manual Version 9". SPARC International Inc.
o Compaq Computer Corporation (Ed.). 2002. "Alpha Architecture
Reference Manual". Compaq Computer Corporation.
o Intel Corporation (Ed.). 2002. "A Formal Specification of Intel
Itanium Processor Family Memory Ordering". Intel Corporation.
o Intel Corporation (Ed.). 2002. "Intel 64 and IA-32 Architectures
Software Developer’s Manual". Intel Corporation.
o Peter Sewell, Susmit Sarkar, Scott Owens, Francesco Zappa Nardelli,
and Magnus O. Myreen. 2010. "x86-TSO: A Rigorous and Usable
Programmer's Model for x86 Multiprocessors". Commun. ACM 53, 7
(July, 2010), 89-97. http://doi.acm.org/10.1145/1785414.1785443
o IBM Corporation (Ed.). 2009. "Power ISA Version 2.06". IBM
Corporation.
o ARM Ltd. (Ed.). 2009. "ARM Barrier Litmus Tests and Cookbook".
ARM Ltd.
o Susmit Sarkar, Peter Sewell, Jade Alglave, Luc Maranget, and
Derek Williams. 2011. "Understanding POWER Multiprocessors". In
Proceedings of the 32Nd ACM SIGPLAN Conference on Programming
Language Design and Implementation (PLDI ’11). ACM, New York,
NY, USA, 175–186.
o Susmit Sarkar, Kayvan Memarian, Scott Owens, Mark Batty,
Peter Sewell, Luc Maranget, Jade Alglave, and Derek Williams.
2012. "Synchronising C/C++ and POWER". In Proceedings of the 33rd
ACM SIGPLAN Conference on Programming Language Design and
Implementation (PLDI '12). ACM, New York, NY, USA, 311-322.
o ARM Ltd. (Ed.). 2014. "ARM Architecture Reference Manual (ARMv8,
for ARMv8-A architecture profile)". ARM Ltd.
o Imagination Technologies, LTD. 2015. "MIPS(R) Architecture
For Programmers, Volume II-A: The MIPS64(R) Instruction,
Set Reference Manual". Imagination Technologies, LTD.
o Shaked Flur, Kathryn E. Gray, Christopher Pulte, Susmit
Sarkar, Ali Sezgin, Luc Maranget, Will Deacon, and Peter
Sewell. 2016. "Modelling the ARMv8 Architecture, Operationally:
Concurrency and ISA". In Proceedings of the 43rd Annual ACM
SIGPLAN-SIGACT Symposium on Principles of Programming Languages
(POPL ’16). ACM, New York, NY, USA, 608–621.
o Shaked Flur, Susmit Sarkar, Christopher Pulte, Kyndylan Nienhuis,
Luc Maranget, Kathryn E. Gray, Ali Sezgin, Mark Batty, and Peter
Sewell. 2017. "Mixed-size Concurrency: ARM, POWER, C/C++11,
and SC". In Proceedings of the 44th ACM SIGPLAN Symposium on
Principles of Programming Languages (POPL 2017). ACM, New York,
NY, USA, 429–442.
o Christopher Pulte, Shaked Flur, Will Deacon, Jon French,
Susmit Sarkar, and Peter Sewell. 2018. "Simplifying ARM concurrency:
multicopy-atomic axiomatic and operational models for ARMv8". In
Proceedings of the ACM on Programming Languages, Volume 2, Issue
POPL, Article No. 19. ACM, New York, NY, USA.
Linux-kernel memory model
=========================
o Jade Alglave, Will Deacon, Boqun Feng, David Howells, Daniel
Lustig, Luc Maranget, Paul E. McKenney, Andrea Parri, Nicholas
Piggin, Alan Stern, Akira Yokosawa, and Peter Zijlstra.
2019. "Calibrating your fear of big bad optimizing compilers"
Linux Weekly News. https://lwn.net/Articles/799218/
o Jade Alglave, Will Deacon, Boqun Feng, David Howells, Daniel
Lustig, Luc Maranget, Paul E. McKenney, Andrea Parri, Nicholas
Piggin, Alan Stern, Akira Yokosawa, and Peter Zijlstra.
2019. "Who's afraid of a big bad optimizing compiler?"
Linux Weekly News. https://lwn.net/Articles/793253/
o Jade Alglave, Luc Maranget, Paul E. McKenney, Andrea Parri, and
Alan Stern. 2018. "Frightening small children and disconcerting
grown-ups: Concurrency in the Linux kernel". In Proceedings of
the 23rd International Conference on Architectural Support for
Programming Languages and Operating Systems (ASPLOS 2018). ACM,
New York, NY, USA, 405-418. Webpage: http://diy.inria.fr/linux/.
o Jade Alglave, Luc Maranget, Paul E. McKenney, Andrea Parri, and
Alan Stern. 2017. "A formal kernel memory-ordering model (part 1)"
Linux Weekly News. https://lwn.net/Articles/718628/
o Jade Alglave, Luc Maranget, Paul E. McKenney, Andrea Parri, and
Alan Stern. 2017. "A formal kernel memory-ordering model (part 2)"
Linux Weekly News. https://lwn.net/Articles/720550/
o Jade Alglave, Luc Maranget, Paul E. McKenney, Andrea Parri, and
Alan Stern. 2017-2019. "A Formal Model of Linux-Kernel Memory
Ordering" (backup material for the LWN articles)
https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/LWNLinuxMM/
Memory-model tooling
====================
o Daniel Jackson. 2002. "Alloy: A Lightweight Object Modelling
Notation". ACM Trans. Softw. Eng. Methodol. 11, 2 (April 2002),
256–290. http://doi.acm.org/10.1145/505145.505149
o Jade Alglave, Luc Maranget, and Michael Tautschnig. 2014. "Herding
Cats: Modelling, Simulation, Testing, and Data Mining for Weak
Memory". ACM Trans. Program. Lang. Syst. 36, 2, Article 7 (July
2014), 7:1–7:74 pages.
o Jade Alglave, Patrick Cousot, and Luc Maranget. 2016. "Syntax and
semantics of the weak consistency model specification language
cat". CoRR abs/1608.07531 (2016). https://arxiv.org/abs/1608.07531
Memory-model comparisons
========================
o Paul E. McKenney, Ulrich Weigand, Andrea Parri, and Boqun
Feng. 2018. "Linux-Kernel Memory Model". (27 September 2018).
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0124r6.html.
linux-7.2.3/tools/memory-model/Documentation/simple.txt
+270
View File
@@ -0,0 +1,270 @@
This document provides options for those wishing to keep their
memory-ordering lives simple, as is necessary for those whose domain
is complex. After all, there are bugs other than memory-ordering bugs,
and the time spent gaining memory-ordering knowledge is not available
for gaining domain knowledge. Furthermore Linux-kernel memory model
(LKMM) is quite complex, with subtle differences in code often having
dramatic effects on correctness.
The options near the beginning of this list are quite simple. The idea
is not that kernel hackers don't already know about them, but rather
that they might need the occasional reminder.
Please note that this is a generic guide, and that specific subsystems
will often have special requirements or idioms. For example, developers
of MMIO-based device drivers will often need to use mb(), rmb(), and
wmb(), and therefore might find smp_mb(), smp_rmb(), and smp_wmb()
to be more natural than smp_load_acquire() and smp_store_release().
On the other hand, those coming in from other environments will likely
be more familiar with these last two.
Single-threaded code
====================
In single-threaded code, there is no reordering, at least assuming
that your toolchain and hardware are working correctly. In addition,
it is generally a mistake to assume your code will only run in a single
threaded context as the kernel can enter the same code path on multiple
CPUs at the same time. One important exception is a function that makes
no external data references.
In the general case, you will need to take explicit steps to ensure that
your code really is executed within a single thread that does not access
shared variables. A simple way to achieve this is to define a global lock
that you acquire at the beginning of your code and release at the end,
taking care to ensure that all references to your code's shared data are
also carried out under that same lock. Because only one thread can hold
this lock at a given time, your code will be executed single-threaded.
This approach is called "code locking".
Code locking can severely limit both performance and scalability, so it
should be used with caution, and only on code paths that execute rarely.
After all, a huge amount of effort was required to remove the Linux
kernel's old "Big Kernel Lock", so let's please be very careful about
adding new "little kernel locks".
One of the advantages of locking is that, in happy contrast with the
year 1981, almost all kernel developers are very familiar with locking.
The Linux kernel's lockdep (CONFIG_PROVE_LOCKING=y) is very helpful with
the formerly feared deadlock scenarios.
Please use the standard locking primitives provided by the kernel rather
than rolling your own. For one thing, the standard primitives interact
properly with lockdep. For another thing, these primitives have been
tuned to deal better with high contention. And for one final thing, it is
surprisingly hard to correctly code production-quality lock acquisition
and release functions. After all, even simple non-production-quality
locking functions must carefully prevent both the CPU and the compiler
from moving code in either direction across the locking function.
Despite the scalability limitations of single-threaded code, RCU
takes this approach for much of its grace-period processing and also
for early-boot operation. The reason RCU is able to scale despite
single-threaded grace-period processing is use of batching, where all
updates that accumulated during one grace period are handled by the
next one. In other words, slowing down grace-period processing makes
it more efficient. Nor is RCU unique: Similar batching optimizations
are used in many I/O operations.
Packaged code
=============
Even if performance and scalability concerns prevent your code from
being completely single-threaded, it is often possible to use library
functions that handle the concurrency nearly or entirely on their own.
This approach delegates any LKMM worries to the library maintainer.
In the kernel, what is the "library"? Quite a bit. It includes the
contents of the lib/ directory, much of the include/linux/ directory along
with a lot of other heavily used APIs. But heavily used examples include
the list macros (for example, include/linux/{,rcu}list.h), workqueues,
smp_call_function(), and the various hash tables and search trees.
Data locking
============
With code locking, we use single-threaded code execution to guarantee
serialized access to the data that the code is accessing. However,
we can also achieve this by instead associating the lock with specific
instances of the data structures. This creates a "critical section"
in the code execution that will execute as though it is single threaded.
By placing all the accesses and modifications to a shared data structure
inside a critical section, we ensure that the execution context that
holds the lock has exclusive access to the shared data.
The poster boy for this approach is the hash table, where placing a lock
in each hash bucket allows operations on different buckets to proceed
concurrently. This works because the buckets do not overlap with each
other, so that an operation on one bucket does not interfere with any
other bucket.
As the number of buckets increases, data locking scales naturally.
In particular, if the amount of data increases with the number of CPUs,
increasing the number of buckets as the number of CPUs increase results
in a naturally scalable data structure.
Per-CPU processing
==================
Partitioning processing and data over CPUs allows each CPU to take
a single-threaded approach while providing excellent performance and
scalability. Of course, there is no free lunch: The dark side of this
excellence is substantially increased memory footprint.
In addition, it is sometimes necessary to occasionally update some global
view of this processing and data, in which case something like locking
must be used to protect this global view. This is the approach taken
by the percpu_counter infrastructure. In many cases, there are already
generic/library variants of commonly used per-cpu constructs available.
Please use them rather than rolling your own.
RCU uses DEFINE_PER_CPU*() declaration to create a number of per-CPU
data sets. For example, each CPU does private quiescent-state processing
within its instance of the per-CPU rcu_data structure, and then uses data
locking to report quiescent states up the grace-period combining tree.
Packaged primitives: Sequence locking
=====================================
Lockless programming is considered by many to be more difficult than
lock-based programming, but there are a few lockless design patterns that
have been built out into an API. One of these APIs is sequence locking.
Although this API can be used in extremely complex ways, there are simple
and effective ways of using it that avoid the need to pay attention to
memory ordering.
The basic keep-things-simple rule for sequence locking is "do not write
in read-side code". Yes, you can do writes from within sequence-locking
readers, but it won't be so simple. For example, such writes will be
lockless and should be idempotent.
For more sophisticated use cases, LKMM can guide you, including use
cases involving combining sequence locking with other synchronization
primitives. (LKMM does not yet know about sequence locking, so it is
currently necessary to open-code it in your litmus tests.)
Additional information may be found in include/linux/seqlock.h.
Packaged primitives: RCU
========================
Another lockless design pattern that has been baked into an API
is RCU. The Linux kernel makes sophisticated use of RCU, but the
keep-things-simple rules for RCU are "do not write in read-side code"
and "do not update anything that is visible to and accessed by readers",
and "protect updates with locking".
These rules are illustrated by the functions foo_update_a() and
foo_get_a() shown in Documentation/RCU/whatisRCU.rst. Additional
RCU usage patterns maybe found in Documentation/RCU and in the
source code.
Packaged primitives: Atomic operations
======================================
Back in the day, the Linux kernel had three types of atomic operations:
1. Initialization and read-out, such as atomic_set() and atomic_read().
2. Operations that did not return a value and provided no ordering,
such as atomic_inc() and atomic_dec().
3. Operations that returned a value and provided full ordering, such as
atomic_add_return() and atomic_dec_and_test(). Note that some
value-returning operations provide full ordering only conditionally.
For example, cmpxchg() provides ordering only upon success.
More recent kernels have operations that return a value but do not
provide full ordering. These are flagged with either a _relaxed()
suffix (providing no ordering), or an _acquire() or _release() suffix
(providing limited ordering).
Additional information may be found in these files:
Documentation/atomic_t.txt
Documentation/atomic_bitops.txt
Documentation/core-api/refcount-vs-atomic.rst
Reading code using these primitives is often also quite helpful.
Lockless, fully ordered
=======================
When using locking, there often comes a time when it is necessary
to access some variable or another without holding the data lock
that serializes access to that variable.
If you want to keep things simple, use the initialization and read-out
operations from the previous section only when there are no racing
accesses. Otherwise, use only fully ordered operations when accessing
or modifying the variable. This approach guarantees that code prior
to a given access to that variable will be seen by all CPUs as having
happened before any code following any later access to that same variable.
Please note that per-CPU functions are not atomic operations and
hence they do not provide any ordering guarantees at all.
If the lockless accesses are frequently executed reads that are used
only for heuristics, or if they are frequently executed writes that
are used only for statistics, please see the next section.
Lockless statistics and heuristics
==================================
Unordered primitives such as atomic_read(), atomic_set(), READ_ONCE(), and
WRITE_ONCE() can safely be used in some cases. These primitives provide
no ordering, but they do prevent the compiler from carrying out a number
of destructive optimizations (for which please see the next section).
One example use for these primitives is statistics, such as per-CPU
counters exemplified by the rt_cache_stat structure's routing-cache
statistics counters. Another example use case is heuristics, such as
the jiffies_till_first_fqs and jiffies_till_next_fqs kernel parameters
controlling how often RCU scans for idle CPUs.
But be careful. "Unordered" really does mean "unordered". It is all
too easy to assume ordering, and this assumption must be avoided when
using these primitives.
Don't let the compiler trip you up
==================================
It can be quite tempting to use plain C-language accesses for lockless
loads from and stores to shared variables. Although this is both
possible and quite common in the Linux kernel, it does require a
surprising amount of analysis, care, and knowledge about the compiler.
Yes, some decades ago it was not unfair to consider a C compiler to be
an assembler with added syntax and better portability, but the advent of
sophisticated optimizing compilers mean that those days are long gone.
Today's optimizing compilers can profoundly rewrite your code during the
translation process, and have long been ready, willing, and able to do so.
Therefore, if you really need to use C-language assignments instead of
READ_ONCE(), WRITE_ONCE(), and so on, you will need to have a very good
understanding of both the C standard and your compiler. Here are some
introductory references and some tooling to start you on this noble quest:
Who's afraid of a big bad optimizing compiler?
https://lwn.net/Articles/793253/
Calibrating your fear of big bad optimizing compilers
https://lwn.net/Articles/799218/
Concurrency bugs should fear the big bad data-race detector (part 1)
https://lwn.net/Articles/816850/
Concurrency bugs should fear the big bad data-race detector (part 2)
https://lwn.net/Articles/816854/
More complex use cases
======================
If the alternatives above do not do what you need, please look at the
recipes.txt file to peel off the next layer of the memory-ordering
onion.