lastdragon/linux-kernel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Resguardo inicial

Browse Source
This commit is contained in:
lastdragon
2026-09-06 12:53:21 -06:00
parent 662efe3c2a
commit ad2088d178
94853 changed files with 43200404 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
linux-7.2.3/Documentation/admin-guide/LSM/LoadPin.rst
+31
View File
@@ -0,0 +1,31 @@
=======
LoadPin
=======
LoadPin is a Linux Security Module that ensures all kernel-loaded files
(modules, firmware, etc) all originate from the same filesystem, with
the expectation that such a filesystem is backed by a read-only device
such as dm-verity or CDROM. This allows systems that have a verified
and/or unchangeable filesystem to enforce module and firmware loading
restrictions without needing to sign the files individually.
The LSM is selectable at build-time with ``CONFIG_SECURITY_LOADPIN``, and
can be controlled at boot-time with the kernel command line option
"``loadpin.enforce``". By default, it is enabled, but can be disabled at
boot ("``loadpin.enforce=0``").
LoadPin starts pinning when it sees the first file loaded. If the
block device backing the filesystem is not read-only, a sysctl is
created to toggle pinning: ``/proc/sys/kernel/loadpin/enabled``. (Having
a mutable filesystem means pinning is mutable too, but having the
sysctl allows for easy testing on systems with a mutable filesystem.)
It's also possible to exclude specific file types from LoadPin using kernel
command line option "``loadpin.exclude``". By default, all files are
included, but they can be excluded using kernel command line option such
as "``loadpin.exclude=kernel-module,kexec-image``". This allows to use
different mechanisms such as ``CONFIG_MODULE_SIG`` and
``CONFIG_KEXEC_VERIFY_SIG`` to verify kernel module and kernel image while
still use LoadPin to protect the integrity of other files kernel loads. The
full list of valid file types can be found in ``kernel_read_file_str``
defined in ``include/linux/kernel_read_file.h``.
linux-7.2.3/Documentation/admin-guide/LSM/SELinux.rst
+44
View File
@@ -0,0 +1,44 @@
=======
SELinux
=======
Information about the SELinux kernel subsystem can be found at the
following links:
https://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/selinux.git/tree/README.md
https://github.com/selinuxproject/selinux-kernel/wiki
Information about the SELinux userspace can be found at:
https://github.com/SELinuxProject/selinux/wiki
If you want to use SELinux, chances are you will want
to use the distro-provided policies, or install the
latest reference policy release from
https://github.com/SELinuxProject/refpolicy
However, if you want to install a dummy policy for
testing, you can do using ``mdp`` provided under
scripts/selinux. Note that this requires the selinux
userspace to be installed - in particular you will
need checkpolicy to compile a kernel, and setfiles and
fixfiles to label the filesystem.
1. Compile the kernel with selinux enabled.
2. Type ``make`` to compile ``mdp``.
3. Make sure that you are not running with
SELinux enabled and a real policy. If
you are, reboot with selinux disabled
before continuing.
4. Run install_policy.sh::
cd scripts/selinux
sh install_policy.sh
Step 4 will create a new dummy policy valid for your
kernel, with a single selinux user, role, and type.
It will compile the policy, will set your ``SELINUXTYPE`` to
``dummy`` in ``/etc/selinux/config``, install the compiled policy
as ``dummy``, and relabel your filesystem.
linux-7.2.3/Documentation/admin-guide/LSM/SafeSetID.rst
+118
View File
@@ -0,0 +1,118 @@
=========
SafeSetID
=========
SafeSetID is an LSM module that gates the setid family of syscalls to restrict
UID/GID transitions from a given UID/GID to only those approved by a
system-wide allowlist. These restrictions also prohibit the given UIDs/GIDs
from obtaining auxiliary privileges associated with CAP_SET{U/G}ID, such as
allowing a user to set up user namespace UID/GID mappings.
Background
==========
In absence of file capabilities, processes spawned on a Linux system that need
to switch to a different user must be spawned with CAP_SETUID privileges.
CAP_SETUID is granted to programs running as root or those running as a non-root
user that have been explicitly given the CAP_SETUID runtime capability. It is
often preferable to use Linux runtime capabilities rather than file
capabilities, since using file capabilities to run a program with elevated
privileges opens up possible security holes since any user with access to the
file can exec() that program to gain the elevated privileges.
While it is possible to implement a tree of processes by giving full
CAP_SET{U/G}ID capabilities, this is often at odds with the goals of running a
tree of processes under non-root user(s) in the first place. Specifically,
since CAP_SETUID allows changing to any user on the system, including the root
user, it is an overpowered capability for what is needed in this scenario,
especially since programs often only call setuid() to drop privileges to a
lesser-privileged user -- not elevate privileges. Unfortunately, there is no
generally feasible way in Linux to restrict the potential UIDs that a user can
switch to through setuid() beyond allowing a switch to any user on the system.
This SafeSetID LSM seeks to provide a solution for restricting setid
capabilities in such a way.
The main use case for this LSM is to allow a non-root program to transition to
other untrusted uids without full blown CAP_SETUID capabilities. The non-root
program would still need CAP_SETUID to do any kind of transition, but the
additional restrictions imposed by this LSM would mean it is a "safer" version
of CAP_SETUID since the non-root program cannot take advantage of CAP_SETUID to
do any unapproved actions (e.g. setuid to uid 0 or create/enter new user
namespace). The higher level goal is to allow for uid-based sandboxing of system
services without having to give out CAP_SETUID all over the place just so that
non-root programs can drop to even-lesser-privileged uids. This is especially
relevant when one non-root daemon on the system should be allowed to spawn other
processes as different uids, but it's undesirable to give the daemon a
basically-root-equivalent CAP_SETUID.
Other Approaches Considered
===========================
Solve this problem in userspace
-------------------------------
For candidate applications that would like to have restricted setid capabilities
as implemented in this LSM, an alternative option would be to simply take away
setid capabilities from the application completely and refactor the process
spawning semantics in the application (e.g. by using a privileged helper program
to do process spawning and UID/GID transitions). Unfortunately, there are a
number of semantics around process spawning that would be affected by this, such
as fork() calls where the program doesn't immediately call exec() after the
fork(), parent processes specifying custom environment variables or command line
args for spawned child processes, or inheritance of file handles across a
fork()/exec(). Because of this, as solution that uses a privileged helper in
userspace would likely be less appealing to incorporate into existing projects
that rely on certain process-spawning semantics in Linux.
Use user namespaces
-------------------
Another possible approach would be to run a given process tree in its own user
namespace and give programs in the tree setid capabilities. In this way,
programs in the tree could change to any desired UID/GID in the context of their
own user namespace, and only approved UIDs/GIDs could be mapped back to the
initial system user namespace, affectively preventing privilege escalation.
Unfortunately, it is not generally feasible to use user namespaces in isolation,
without pairing them with other namespace types, which is not always an option.
Linux checks for capabilities based off of the user namespace that "owns" some
entity. For example, Linux has the notion that network namespaces are owned by
the user namespace in which they were created. A consequence of this is that
capability checks for access to a given network namespace are done by checking
whether a task has the given capability in the context of the user namespace
that owns the network namespace -- not necessarily the user namespace under
which the given task runs. Therefore spawning a process in a new user namespace
effectively prevents it from accessing the network namespace owned by the
initial namespace. This is a deal-breaker for any application that expects to
retain the CAP_NET_ADMIN capability for the purpose of adjusting network
configurations. Using user namespaces in isolation causes problems regarding
other system interactions, including use of pid namespaces and device creation.
Use an existing LSM
-------------------
None of the other in-tree LSMs have the capability to gate setid transitions, or
even employ the security_task_fix_setuid hook at all. SELinux says of that hook:
"Since setuid only affects the current process, and since the SELinux controls
are not based on the Linux identity attributes, SELinux does not need to control
this operation."
Directions for use
==================
This LSM hooks the setid syscalls to make sure transitions are allowed if an
applicable restriction policy is in place. Policies are configured through
securityfs by writing to the safesetid/uid_allowlist_policy and
safesetid/gid_allowlist_policy files at the location where securityfs is
mounted. The format for adding a policy is '<UID>:<UID>' or '<GID>:<GID>',
using literal numbers, and ending with a newline character such as '123:456\n'.
Writing an empty string "" will flush the policy. Again, configuring a policy
for a UID/GID will prevent that UID/GID from obtaining auxiliary setid
privileges, such as allowing a user to set up user namespace UID/GID mappings.
Note on GID policies and setgroups()
====================================
In v5.9 we are adding support for limiting CAP_SETGID privileges as was done
previously for CAP_SETUID. However, for compatibility with common sandboxing
related code conventions in userspace, we currently allow arbitrary
setgroups() calls for processes with CAP_SETGID restrictions. Until we add
support in a future release for restricting setgroups() calls, these GID
policies add no meaningful security. setgroups() restrictions will be enforced
once we have the policy checking code in place, which will rely on GID policy
configuration code added in v5.9.
linux-7.2.3/Documentation/admin-guide/LSM/Smack.rst
+871
View File
@@ -0,0 +1,871 @@
=====
Smack
=====
"Good for you, you've decided to clean the elevator!"
- The Elevator, from Dark Star
Smack is the Simplified Mandatory Access Control Kernel.
Smack is a kernel based implementation of mandatory access
control that includes simplicity in its primary design goals.
Smack is not the only Mandatory Access Control scheme
available for Linux. Those new to Mandatory Access Control
are encouraged to compare Smack with the other mechanisms
available to determine which is best suited to the problem
at hand.
Smack consists of three major components:
- The kernel
- Basic utilities, which are helpful but not required
- Configuration data
The kernel component of Smack is implemented as a Linux
Security Modules (LSM) module. It requires netlabel and
works best with file systems that support extended attributes,
although xattr support is not strictly required.
It is safe to run a Smack kernel under a "vanilla" distribution.
Smack kernels use the CIPSO IP option. Some network
configurations are intolerant of IP options and can impede
access to systems that use them as Smack does.
Smack is used in the Tizen operating system. Please
go to http://wiki.tizen.org for information about how
Smack is used in Tizen.
The current git repository for Smack user space is:
git://github.com/smack-team/smack.git
This should make and install on most modern distributions.
There are five commands included in smackutil:
chsmack:
display or set Smack extended attribute values
smackctl:
load the Smack access rules
smackaccess:
report if a process with one label has access
to an object with another
These two commands are obsolete with the introduction of
the smackfs/load2 and smackfs/cipso2 interfaces.
smackload:
properly formats data for writing to smackfs/load
smackcipso:
properly formats data for writing to smackfs/cipso
In keeping with the intent of Smack, configuration data is
minimal and not strictly required. The most important
configuration step is mounting the smackfs pseudo filesystem.
If smackutil is installed the startup script will take care
of this, but it can be manually as well.
Add this line to ``/etc/fstab``::
smackfs /sys/fs/smackfs smackfs defaults 0 0
The ``/sys/fs/smackfs`` directory is created by the kernel.
Smack uses extended attributes (xattrs) to store labels on filesystem
objects. The attributes are stored in the extended attribute security
name space. A process must have ``CAP_MAC_ADMIN`` to change any of these
attributes.
The extended attributes that Smack uses are:
SMACK64
Used to make access control decisions. In almost all cases
the label given to a new filesystem object will be the label
of the process that created it.
SMACK64EXEC
The Smack label of a process that execs a program file with
this attribute set will run with this attribute's value.
SMACK64MMAP
Don't allow the file to be mmapped by a process whose Smack
label does not allow all of the access permitted to a process
with the label contained in this attribute. This is a very
specific use case for shared libraries.
SMACK64TRANSMUTE
Can only have the value "TRUE". If this attribute is present
on a directory when an object is created in the directory and
the Smack rule (more below) that permitted the write access
to the directory includes the transmute ("t") mode the object
gets the label of the directory instead of the label of the
creating process. If the object being created is a directory
the SMACK64TRANSMUTE attribute is set as well.
SMACK64IPIN
This attribute is only available on file descriptors for sockets.
Use the Smack label in this attribute for access control
decisions on packets being delivered to this socket.
SMACK64IPOUT
This attribute is only available on file descriptors for sockets.
Use the Smack label in this attribute for access control
decisions on packets coming from this socket.
There are multiple ways to set a Smack label on a file::
# attr -S -s SMACK64 -V "value" path
# chsmack -a value path
A process can see the Smack label it is running with by
reading ``/proc/self/attr/current``. A process with ``CAP_MAC_ADMIN``
can set the process Smack by writing there.
Most Smack configuration is accomplished by writing to files
in the smackfs filesystem. This pseudo-filesystem is mounted
on ``/sys/fs/smackfs``.
access
Provided for backward compatibility. The access2 interface
is preferred and should be used instead.
This interface reports whether a subject with the specified
Smack label has a particular access to an object with a
specified Smack label. Write a fixed format access rule to
this file. The next read will indicate whether the access
would be permitted. The text will be either "1" indicating
access, or "0" indicating denial.
access2
This interface reports whether a subject with the specified
Smack label has a particular access to an object with a
specified Smack label. Write a long format access rule to
this file. The next read will indicate whether the access
would be permitted. The text will be either "1" indicating
access, or "0" indicating denial.
ambient
This contains the Smack label applied to unlabeled network
packets.
change-rule
This interface allows modification of existing access control rules.
The format accepted on write is::
"%s %s %s %s"
where the first string is the subject label, the second the
object label, the third the access to allow and the fourth the
access to deny. The access strings may contain only the characters
"rwxat-". If a rule for a given subject and object exists it will be
modified by enabling the permissions in the third string and disabling
those in the fourth string. If there is no such rule it will be
created using the access specified in the third and the fourth strings.
cipso
Provided for backward compatibility. The cipso2 interface
is preferred and should be used instead.
This interface allows a specific CIPSO header to be assigned
to a Smack label. The format accepted on write is::
"%24s%4d%4d"["%4d"]...
The first string is a fixed Smack label. The first number is
the level to use. The second number is the number of categories.
The following numbers are the categories::
"level-3-cats-5-19 3 2 5 19"
cipso2
This interface allows a specific CIPSO header to be assigned
to a Smack label. The format accepted on write is::
"%s%4d%4d"["%4d"]...
The first string is a long Smack label. The first number is
the level to use. The second number is the number of categories.
The following numbers are the categories::
"level-3-cats-5-19 3 2 5 19"
direct
This contains the CIPSO level used for Smack direct label
representation in network packets.
doi
This contains the CIPSO domain of interpretation used in
network packets.
ipv6host
This interface allows specific IPv6 internet addresses to be
treated as single label hosts. Packets are sent to single
label hosts only from processes that have Smack write access
to the host label. All packets received from single label hosts
are given the specified label. The format accepted on write is::
"%h:%h:%h:%h:%h:%h:%h:%h label" or
"%h:%h:%h:%h:%h:%h:%h:%h/%d label".
The "::" address shortcut is not supported.
If label is "-DELETE" a matched entry will be deleted.
load
Provided for backward compatibility. The load2 interface
is preferred and should be used instead.
This interface allows access control rules in addition to
the system defined rules to be specified. The format accepted
on write is::
"%24s%24s%5s"
where the first string is the subject label, the second the
object label, and the third the requested access. The access
string may contain only the characters "rwxat-", and specifies
which sort of access is allowed. The "-" is a placeholder for
permissions that are not allowed. The string "r-x--" would
specify read and execute access. Labels are limited to 23
characters in length.
load2
This interface allows access control rules in addition to
the system defined rules to be specified. The format accepted
on write is::
"%s %s %s"
where the first string is the subject label, the second the
object label, and the third the requested access. The access
string may contain only the characters "rwxat-", and specifies
which sort of access is allowed. The "-" is a placeholder for
permissions that are not allowed. The string "r-x--" would
specify read and execute access.
load-self
Provided for backward compatibility. The load-self2 interface
is preferred and should be used instead.
This interface allows process specific access rules to be
defined. These rules are only consulted if access would
otherwise be permitted, and are intended to provide additional
restrictions on the process. The format is the same as for
the load interface.
load-self2
This interface allows process specific access rules to be
defined. These rules are only consulted if access would
otherwise be permitted, and are intended to provide additional
restrictions on the process. The format is the same as for
the load2 interface.
logging
This contains the Smack logging state.
mapped
This contains the CIPSO level used for Smack mapped label
representation in network packets.
netlabel
This interface allows specific internet addresses to be
treated as single label hosts. Packets are sent to single
label hosts without CIPSO headers, but only from processes
that have Smack write access to the host label. All packets
received from single label hosts are given the specified
label. The format accepted on write is::
"%d.%d.%d.%d label" or "%d.%d.%d.%d/%d label".
If the label specified is "-CIPSO" the address is treated
as a host that supports CIPSO headers.
onlycap
This contains labels processes must have for CAP_MAC_ADMIN
and ``CAP_MAC_OVERRIDE`` to be effective. If this file is empty
these capabilities are effective at for processes with any
label. The values are set by writing the desired labels, separated
by spaces, to the file or cleared by writing "-" to the file.
ptrace
This is used to define the current ptrace policy
0 - default:
this is the policy that relies on Smack access rules.
For the ``PTRACE_READ`` a subject needs to have a read access on
object. For the ``PTRACE_ATTACH`` a read-write access is required.
1 - exact:
this is the policy that limits ``PTRACE_ATTACH``. Attach is
only allowed when subject's and object's labels are equal.
``PTRACE_READ`` is not affected. Can be overridden with ``CAP_SYS_PTRACE``.
2 - draconian:
this policy behaves like the 'exact' above with an
exception that it can't be overridden with ``CAP_SYS_PTRACE``.
revoke-subject
Writing a Smack label here sets the access to '-' for all access
rules with that subject label.
unconfined
If the kernel is configured with ``CONFIG_SECURITY_SMACK_BRINGUP``
a process with ``CAP_MAC_ADMIN`` can write a label into this interface.
Thereafter, accesses that involve that label will be logged and
the access permitted if it wouldn't be otherwise. Note that this
is dangerous and can ruin the proper labeling of your system.
It should never be used in production.
relabel-self
This interface contains a list of labels to which the process can
transition to, by writing to ``/proc/self/attr/current``.
Normally a process can change its own label to any legal value, but only
if it has ``CAP_MAC_ADMIN``. This interface allows a process without
``CAP_MAC_ADMIN`` to relabel itself to one of labels from predefined list.
A process without ``CAP_MAC_ADMIN`` can change its label only once. When it
does, this list will be cleared.
The values are set by writing the desired labels, separated
by spaces, to the file or cleared by writing "-" to the file.
If you are using the smackload utility
you can add access rules in ``/etc/smack/accesses``. They take the form::
subjectlabel objectlabel access
access is a combination of the letters rwxatb which specify the
kind of access permitted a subject with subjectlabel on an
object with objectlabel. If there is no rule no access is allowed.
Look for additional programs on http://schaufler-ca.com
The Simplified Mandatory Access Control Kernel (Whitepaper)
===========================================================
Casey Schaufler
casey@schaufler-ca.com
Mandatory Access Control
------------------------
Computer systems employ a variety of schemes to constrain how information is
shared among the people and services using the machine. Some of these schemes
allow the program or user to decide what other programs or users are allowed
access to pieces of data. These schemes are called discretionary access
control mechanisms because the access control is specified at the discretion
of the user. Other schemes do not leave the decision regarding what a user or
program can access up to users or programs. These schemes are called mandatory
access control mechanisms because you don't have a choice regarding the users
or programs that have access to pieces of data.
Bell & LaPadula
---------------
From the middle of the 1980's until the turn of the century Mandatory Access
Control (MAC) was very closely associated with the Bell & LaPadula security
model, a mathematical description of the United States Department of Defense
policy for marking paper documents. MAC in this form enjoyed a following
within the Capital Beltway and Scandinavian supercomputer centers but was
often sited as failing to address general needs.
Domain Type Enforcement
-----------------------
Around the turn of the century Domain Type Enforcement (DTE) became popular.
This scheme organizes users, programs, and data into domains that are
protected from each other. This scheme has been widely deployed as a component
of popular Linux distributions. The administrative overhead required to
maintain this scheme and the detailed understanding of the whole system
necessary to provide a secure domain mapping leads to the scheme being
disabled or used in limited ways in the majority of cases.
Smack
-----
Smack is a Mandatory Access Control mechanism designed to provide useful MAC
while avoiding the pitfalls of its predecessors. The limitations of Bell &
LaPadula are addressed by providing a scheme whereby access can be controlled
according to the requirements of the system and its purpose rather than those
imposed by an arcane government policy. The complexity of Domain Type
Enforcement and avoided by defining access controls in terms of the access
modes already in use.
Smack Terminology
-----------------
The jargon used to talk about Smack will be familiar to those who have dealt
with other MAC systems and shouldn't be too difficult for the uninitiated to
pick up. There are four terms that are used in a specific way and that are
especially important:
Subject:
A subject is an active entity on the computer system.
On Smack a subject is a task, which is in turn the basic unit
of execution.
Object:
An object is a passive entity on the computer system.
On Smack files of all types, IPC, and tasks can be objects.
Access:
Any attempt by a subject to put information into or get
information from an object is an access.
Label:
Data that identifies the Mandatory Access Control
characteristics of a subject or an object.
These definitions are consistent with the traditional use in the security
community. There are also some terms from Linux that are likely to crop up:
Capability:
A task that possesses a capability has permission to
violate an aspect of the system security policy, as identified by
the specific capability. A task that possesses one or more
capabilities is a privileged task, whereas a task with no
capabilities is an unprivileged task.
Privilege:
A task that is allowed to violate the system security
policy is said to have privilege. As of this writing a task can
have privilege either by possessing capabilities or by having an
effective user of root.
Smack Basics
------------
Smack is an extension to a Linux system. It enforces additional restrictions
on what subjects can access which objects, based on the labels attached to
each of the subject and the object.
Labels
~~~~~~
Smack labels are ASCII character strings. They can be up to 255 characters
long, but keeping them to twenty-three characters is recommended.
Single character labels using special characters, that being anything
other than a letter or digit, are reserved for use by the Smack development
team. Smack labels are unstructured, case sensitive, and the only operation
ever performed on them is comparison for equality. Smack labels cannot
contain unprintable characters, the "/" (slash), the "\" (backslash), the "'"
(quote) and '"' (double-quote) characters.
Smack labels cannot begin with a '-'. This is reserved for special options.
There are some predefined labels::
_ Pronounced "floor", a single underscore character.
^ Pronounced "hat", a single circumflex character.
* Pronounced "star", a single asterisk character.
? Pronounced "huh", a single question mark character.
@ Pronounced "web", a single at sign character.
Every task on a Smack system is assigned a label. The Smack label
of a process will usually be assigned by the system initialization
mechanism.
Access Rules
~~~~~~~~~~~~
Smack uses the traditional access modes of Linux. These modes are read,
execute, write, and occasionally append. There are a few cases where the
access mode may not be obvious. These include:
Signals:
A signal is a write operation from the subject task to
the object task.
Internet Domain IPC:
Transmission of a packet is considered a
write operation from the source task to the destination task.
Smack restricts access based on the label attached to a subject and the label
attached to the object it is trying to access. The rules enforced are, in
order:
1. Any access requested by a task labeled "*" is denied.
2. A read or execute access requested by a task labeled "^"
is permitted.
3. A read or execute access requested on an object labeled "_"
is permitted.
4. Any access requested on an object labeled "*" is permitted.
5. Any access requested by a task on an object with the same
label is permitted.
6. Any access requested that is explicitly defined in the loaded
rule set is permitted.
7. Any other access is denied.
Smack Access Rules
~~~~~~~~~~~~~~~~~~
With the isolation provided by Smack access separation is simple. There are
many interesting cases where limited access by subjects to objects with
different labels is desired. One example is the familiar spy model of
sensitivity, where a scientist working on a highly classified project would be
able to read documents of lower classifications and anything she writes will
be "born" highly classified. To accommodate such schemes Smack includes a
mechanism for specifying rules allowing access between labels.
Access Rule Format
~~~~~~~~~~~~~~~~~~
The format of an access rule is::
subject-label object-label access
Where subject-label is the Smack label of the task, object-label is the Smack
label of the thing being accessed, and access is a string specifying the sort
of access allowed. The access specification is searched for letters that
describe access modes:
a: indicates that append access should be granted.
r: indicates that read access should be granted.
w: indicates that write access should be granted.
x: indicates that execute access should be granted.
t: indicates that the rule requests transmutation.
b: indicates that the rule should be reported for bring-up.
Uppercase values for the specification letters are allowed as well.
Access mode specifications can be in any order. Examples of acceptable rules
are::
TopSecret Secret rx
Secret Unclass R
Manager Game x
User HR w
Snap Crackle rwxatb
New Old rRrRr
Closed Off -
Examples of unacceptable rules are::
Top Secret Secret rx
Ace Ace r
Odd spells waxbeans
Spaces are not allowed in labels. Since a subject always has access to files
with the same label specifying a rule for that case is pointless. Only
valid letters (rwxatbRWXATB) and the dash ('-') character are allowed in
access specifications. The dash is a placeholder, so "a-r" is the same
as "ar". A lone dash is used to specify that no access should be allowed.
Applying Access Rules
~~~~~~~~~~~~~~~~~~~~~
The developers of Linux rarely define new sorts of things, usually importing
schemes and concepts from other systems. Most often, the other systems are
variants of Unix. Unix has many endearing properties, but consistency of
access control models is not one of them. Smack strives to treat accesses as
uniformly as is sensible while keeping with the spirit of the underlying
mechanism.
File system objects including files, directories, named pipes, symbolic links,
and devices require access permissions that closely match those used by mode
bit access. To open a file for reading read access is required on the file. To
search a directory requires execute access. Creating a file with write access
requires both read and write access on the containing directory. Deleting a
file requires read and write access to the file and to the containing
directory. It is possible that a user may be able to see that a file exists
but not any of its attributes by the circumstance of having read access to the
containing directory but not to the differently labeled file. This is an
artifact of the file name being data in the directory, not a part of the file.
If a directory is marked as transmuting (SMACK64TRANSMUTE=TRUE) and the
access rule that allows a process to create an object in that directory
includes 't' access the label assigned to the new object will be that
of the directory, not the creating process. This makes it much easier
for two processes with different labels to share data without granting
access to all of their files.
IPC objects, message queues, semaphore sets, and memory segments exist in flat
namespaces and access requests are only required to match the object in
question.
Process objects reflect tasks on the system and the Smack label used to access
them is the same Smack label that the task would use for its own access
attempts. Sending a signal via the kill() system call is a write operation
from the signaler to the recipient. Debugging a process requires both reading
and writing. Creating a new task is an internal operation that results in two
tasks with identical Smack labels and requires no access checks.
Sockets are data structures attached to processes and sending a packet from
one process to another requires that the sender have write access to the
receiver. The receiver is not required to have read access to the sender.
Setting Access Rules
~~~~~~~~~~~~~~~~~~~~
The configuration file /etc/smack/accesses contains the rules to be set at
system startup. The contents are written to the special file
/sys/fs/smackfs/load2. Rules can be added at any time and take effect
immediately. For any pair of subject and object labels there can be only
one rule, with the most recently specified overriding any earlier
specification.
Task Attribute
~~~~~~~~~~~~~~
The Smack label of a process can be read from ``/proc/<pid>/attr/current``. A
process can read its own Smack label from ``/proc/self/attr/current``. A
privileged process can change its own Smack label by writing to
``/proc/self/attr/current`` but not the label of another process.
Format of writing is : only the label or the label followed by one of the
3 trailers: ``\n`` (by common agreement for ``/proc/...`` interfaces),
``\0`` (because some applications incorrectly include it),
``\n\0`` (because we think some applications may incorrectly include it).
File Attribute
~~~~~~~~~~~~~~
The Smack label of a filesystem object is stored as an extended attribute
named SMACK64 on the file. This attribute is in the security namespace. It can
only be changed by a process with privilege.
Privilege
~~~~~~~~~
A process with CAP_MAC_OVERRIDE or CAP_MAC_ADMIN is privileged.
CAP_MAC_OVERRIDE allows the process access to objects it would
be denied otherwise. CAP_MAC_ADMIN allows a process to change
Smack data, including rules and attributes.
Smack Networking
~~~~~~~~~~~~~~~~
As mentioned before, Smack enforces access control on network protocol
transmissions. Every packet sent by a Smack process is tagged with its Smack
label. This is done by adding a CIPSO tag to the header of the IP packet. Each
packet received is expected to have a CIPSO tag that identifies the label and
if it lacks such a tag the network ambient label is assumed. Before the packet
is delivered a check is made to determine that a subject with the label on the
packet has write access to the receiving process and if that is not the case
the packet is dropped.
CIPSO Configuration
~~~~~~~~~~~~~~~~~~~
It is normally unnecessary to specify the CIPSO configuration. The default
values used by the system handle all internal cases. Smack will compose CIPSO
label values to match the Smack labels being used without administrative
intervention. Unlabeled packets that come into the system will be given the
ambient label.
Smack requires configuration in the case where packets from a system that is
not Smack that speaks CIPSO may be encountered. Usually this will be a Trusted
Solaris system, but there are other, less widely deployed systems out there.
CIPSO provides 3 important values, a Domain Of Interpretation (DOI), a level,
and a category set with each packet. The DOI is intended to identify a group
of systems that use compatible labeling schemes, and the DOI specified on the
Smack system must match that of the remote system or packets will be
discarded. The DOI is 3 by default. The value can be read from
/sys/fs/smackfs/doi and can be changed by writing to /sys/fs/smackfs/doi.
The label and category set are mapped to a Smack label as defined in
/etc/smack/cipso.
A Smack/CIPSO mapping has the form::
smack level [category [category]*]
Smack does not expect the level or category sets to be related in any
particular way and does not assume or assign accesses based on them. Some
examples of mappings::
TopSecret 7
TS:A,B 7 1 2
SecBDE 5 2 4 6
RAFTERS 7 12 26
The ":" and "," characters are permitted in a Smack label but have no special
meaning.
The mapping of Smack labels to CIPSO values is defined by writing to
/sys/fs/smackfs/cipso2.
In addition to explicit mappings Smack supports direct CIPSO mappings. One
CIPSO level is used to indicate that the category set passed in the packet is
in fact an encoding of the Smack label. The level used is 250 by default. The
value can be read from /sys/fs/smackfs/direct and changed by writing to
/sys/fs/smackfs/direct.
Socket Attributes
~~~~~~~~~~~~~~~~~
There are two attributes that are associated with sockets. These attributes
can only be set by privileged tasks, but any task can read them for their own
sockets.
SMACK64IPIN:
The Smack label of the task object. A privileged
program that will enforce policy may set this to the star label.
SMACK64IPOUT:
The Smack label transmitted with outgoing packets.
A privileged program may set this to match the label of another
task with which it hopes to communicate.
UNIX domain socket (UDS) with a BSD address functions both as a file in a
filesystem and as a socket. As a file, it carries the SMACK64 attribute. This
attribute is not involved in Smack security enforcement and is immutably
assigned the label "*".
Smack Netlabel Exceptions
~~~~~~~~~~~~~~~~~~~~~~~~~
You will often find that your labeled application has to talk to the outside,
unlabeled world. To do this there's a special file /sys/fs/smackfs/netlabel
where you can add some exceptions in the form of::
@IP1 LABEL1 or
@IP2/MASK LABEL2
It means that your application will have unlabeled access to @IP1 if it has
write access on LABEL1, and access to the subnet @IP2/MASK if it has write
access on LABEL2.
Entries in the /sys/fs/smackfs/netlabel file are matched by longest mask
first, like in classless IPv4 routing.
A special label '@' and an option '-CIPSO' can be used there::
@ means Internet, any application with any label has access to it
-CIPSO means standard CIPSO networking
If you don't know what CIPSO is and don't plan to use it, you can just do::
echo 127.0.0.1 -CIPSO > /sys/fs/smackfs/netlabel
echo 0.0.0.0/0 @ > /sys/fs/smackfs/netlabel
If you use CIPSO on your 192.168.0.0/16 local network and need also unlabeled
Internet access, you can have::
echo 127.0.0.1 -CIPSO > /sys/fs/smackfs/netlabel
echo 192.168.0.0/16 -CIPSO > /sys/fs/smackfs/netlabel
echo 0.0.0.0/0 @ > /sys/fs/smackfs/netlabel
Writing Applications for Smack
------------------------------
There are three sorts of applications that will run on a Smack system. How an
application interacts with Smack will determine what it will have to do to
work properly under Smack.
Smack Ignorant Applications
---------------------------
By far the majority of applications have no reason whatever to care about the
unique properties of Smack. Since invoking a program has no impact on the
Smack label associated with the process the only concern likely to arise is
whether the process has execute access to the program.
Smack Relevant Applications
---------------------------
Some programs can be improved by teaching them about Smack, but do not make
any security decisions themselves. The utility ls(1) is one example of such a
program.
Smack Enforcing Applications
----------------------------
These are special programs that not only know about Smack, but participate in
the enforcement of system policy. In most cases these are the programs that
set up user sessions. There are also network services that provide information
to processes running with various labels.
File System Interfaces
----------------------
Smack maintains labels on file system objects using extended attributes. The
Smack label of a file, directory, or other file system object can be obtained
using getxattr(2)::
len = getxattr("/", "security.SMACK64", value, sizeof (value));
will put the Smack label of the root directory into value. A privileged
process can set the Smack label of a file system object with setxattr(2)::
len = strlen("Rubble");
rc = setxattr("/foo", "security.SMACK64", "Rubble", len, 0);
will set the Smack label of /foo to "Rubble" if the program has appropriate
privilege.
Socket Interfaces
-----------------
The socket attributes can be read using fgetxattr(2).
A privileged process can set the Smack label of outgoing packets with
fsetxattr(2)::
len = strlen("Rubble");
rc = fsetxattr(fd, "security.SMACK64IPOUT", "Rubble", len, 0);
will set the Smack label "Rubble" on packets going out from the socket if the
program has appropriate privilege::
rc = fsetxattr(fd, "security.SMACK64IPIN, "*", strlen("*"), 0);
will set the Smack label "*" as the object label against which incoming
packets will be checked if the program has appropriate privilege.
Administration
--------------
Smack supports some mount options:
smackfsdef=label:
specifies the label to give files that lack
the Smack label extended attribute.
smackfsroot=label:
specifies the label to assign the root of the
file system if it lacks the Smack extended attribute.
smackfshat=label:
specifies a label that must have read access to
all labels set on the filesystem. Not yet enforced.
smackfsfloor=label:
specifies a label to which all labels set on the
filesystem must have read access. Not yet enforced.
smackfstransmute=label:
behaves exactly like smackfsroot except that it also
sets the transmute flag on the root of the mount
These mount options apply to all file system types.
Smack auditing
--------------
If you want Smack auditing of security events, you need to set CONFIG_AUDIT
in your kernel configuration.
By default, all denied events will be audited. You can change this behavior by
writing a single character to the /sys/fs/smackfs/logging file::
0 : no logging
1 : log denied (default)
2 : log accepted
3 : log denied & accepted
Events are logged as 'key=value' pairs, for each event you at least will get
the subject, the object, the rights requested, the action, the kernel function
that triggered the event, plus other pairs depending on the type of event
audited.
Bringup Mode
------------
Bringup mode provides logging features that can make application
configuration and system bringup easier. Configure the kernel with
CONFIG_SECURITY_SMACK_BRINGUP to enable these features. When bringup
mode is enabled accesses that succeed due to rules marked with the "b"
access mode will logged. When a new label is introduced for processes
rules can be added aggressively, marked with the "b". The logging allows
tracking of which rules actual get used for that label.
Another feature of bringup mode is the "unconfined" option. Writing
a label to /sys/fs/smackfs/unconfined makes subjects with that label
able to access any object, and objects with that label accessible to
all subjects. Any access that is granted because a label is unconfined
is logged. This feature is dangerous, as files and directories may
be created in places they couldn't if the policy were being enforced.
linux-7.2.3/Documentation/admin-guide/LSM/Yama.rst
+75
View File
@@ -0,0 +1,75 @@
====
Yama
====
Yama is a Linux Security Module that collects system-wide DAC security
protections that are not handled by the core kernel itself. This is
selectable at build-time with ``CONFIG_SECURITY_YAMA``, and can be controlled
at run-time through sysctls in ``/proc/sys/kernel/yama``:
ptrace_scope
============
As Linux grows in popularity, it will become a larger target for
malware. One particularly troubling weakness of the Linux process
interfaces is that a single user is able to examine the memory and
running state of any of their processes. For example, if one application
(e.g. Pidgin) was compromised, it would be possible for an attacker to
attach to other running processes (e.g. Firefox, SSH sessions, GPG agent,
etc) to extract additional credentials and continue to expand the scope
of their attack without resorting to user-assisted phishing.
This is not a theoretical problem. `SSH session hijacking
<https://www.blackhat.com/presentations/bh-usa-05/bh-us-05-boileau.pdf>`_
and `arbitrary code injection
<https://c-skills.blogspot.com/2007/05/injectso.html>`_ attacks already
exist and remain possible if ptrace is allowed to operate as before.
Since ptrace is not commonly used by non-developers and non-admins, system
builders should be allowed the option to disable this debugging system.
For a solution, some applications use ``prctl(PR_SET_DUMPABLE, ...)`` to
specifically disallow such ptrace attachment (e.g. ssh-agent), but many
do not. A more general solution is to only allow ptrace directly from a
parent to a child process (i.e. direct "gdb EXE" and "strace EXE" still
work), or with ``CAP_SYS_PTRACE`` (i.e. "gdb --pid=PID", and "strace -p PID"
still work as root).
In mode 1, software that has defined application-specific relationships
between a debugging process and its inferior (crash handlers, etc),
``prctl(PR_SET_PTRACER, pid, ...)`` can be used. An inferior can declare which
other process (and its descendants) are allowed to call ``PTRACE_ATTACH``
against it. Only one such declared debugging process can exists for
each inferior at a time. For example, this is used by KDE, Chromium, and
Firefox's crash handlers, and by Wine for allowing only Wine processes
to ptrace each other. If a process wishes to entirely disable these ptrace
restrictions, it can call ``prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY, ...)``
so that any otherwise allowed process (even those in external pid namespaces)
may attach.
The sysctl settings (writable only with ``CAP_SYS_PTRACE``) are:
0 - classic ptrace permissions:
a process can ``PTRACE_ATTACH`` to any other
process running under the same uid, as long as it is dumpable (i.e.
did not transition uids, start privileged, or have called
``prctl(PR_SET_DUMPABLE...)`` already). Similarly, ``PTRACE_TRACEME`` is
unchanged.
1 - restricted ptrace:
a process must have a predefined relationship
with the inferior it wants to call ``PTRACE_ATTACH`` on. By default,
this relationship is that of only its descendants when the above
classic criteria is also met. To change the relationship, an
inferior can call ``prctl(PR_SET_PTRACER, debugger, ...)`` to declare
an allowed debugger PID to call ``PTRACE_ATTACH`` on the inferior.
Using ``PTRACE_TRACEME`` is unchanged.
2 - admin-only attach:
only processes with ``CAP_SYS_PTRACE`` may use ptrace, either with
``PTRACE_ATTACH`` or through children calling ``PTRACE_TRACEME``.
3 - no attach:
no processes may use ptrace with ``PTRACE_ATTACH`` nor via
``PTRACE_TRACEME``. Once set, this sysctl value cannot be changed.
The original children-only logic was based on the restrictions in grsecurity.
linux-7.2.3/Documentation/admin-guide/LSM/apparmor.rst
+54
View File
@@ -0,0 +1,54 @@
========
AppArmor
========
What is AppArmor?
=================
AppArmor is MAC style security extension for the Linux kernel. It implements
a task centered policy, with task "profiles" being created and loaded
from user space. Tasks on the system that do not have a profile defined for
them run in an unconfined state which is equivalent to standard Linux DAC
permissions.
How to enable/disable
=====================
set ``CONFIG_SECURITY_APPARMOR=y``
If AppArmor should be selected as the default security module then set::
CONFIG_DEFAULT_SECURITY_APPARMOR=y
The CONFIG_LSM parameter manages the order and selection of LSMs.
Specify apparmor as the first "major" module (e.g. AppArmor, SELinux, Smack)
in the list.
Build the kernel
If AppArmor is not the default security module it can be enabled by passing
``security=apparmor`` on the kernel's command line.
If AppArmor is the default security module it can be disabled by passing
``apparmor=0, security=XXXX`` (where ``XXXX`` is valid security module), on the
kernel's command line.
For AppArmor to enforce any restrictions beyond standard Linux DAC permissions
policy must be loaded into the kernel from user space (see the Documentation
and tools links).
Documentation
=============
Documentation can be found on the wiki, linked below.
Links
=====
Mailing List - apparmor@lists.ubuntu.com
Wiki - http://wiki.apparmor.net
User space tools - https://gitlab.com/apparmor
Kernel module - git://git.kernel.org/pub/scm/linux/kernel/git/jj/linux-apparmor
linux-7.2.3/Documentation/admin-guide/LSM/index.rst
+51
View File
@@ -0,0 +1,51 @@
===========================
Linux Security Module Usage
===========================
The Linux Security Module (LSM) framework provides a mechanism for
various security checks to be hooked by new kernel extensions. The name
"module" is a bit of a misnomer since these extensions are not actually
loadable kernel modules. Instead, they are selectable at build-time via
CONFIG_DEFAULT_SECURITY and can be overridden at boot-time via the
``"security=..."`` kernel command line argument, in the case where multiple
LSMs were built into a given kernel.
The primary users of the LSM interface are Mandatory Access Control
(MAC) extensions which provide a comprehensive security policy. Examples
include SELinux, Smack, Tomoyo, and AppArmor. In addition to the larger
MAC extensions, other extensions can be built using the LSM to provide
specific changes to system operation when these tweaks are not available
in the core functionality of Linux itself.
The Linux capabilities modules will always be included. This may be
followed by any number of "minor" modules and at most one "major" module.
For more details on capabilities, see ``capabilities(7)`` in the Linux
man-pages project.
A list of the active security modules can be found by reading
``/sys/kernel/security/lsm``. This is a comma separated list, and
will always include the capability module. The list reflects the
order in which checks are made. The capability module will always
be first, followed by any "minor" modules (e.g. Yama) and then
the one "major" module (e.g. SELinux) if there is one configured.
Process attributes associated with "major" security modules should
be accessed and maintained using the special files in ``/proc/.../attr``.
A security module may maintain a module specific subdirectory there,
named after the module. ``/proc/.../attr/smack`` is provided by the Smack
security module and contains all its special files. The files directly
in ``/proc/.../attr`` remain as legacy interfaces for modules that provide
subdirectories.
.. toctree::
:maxdepth: 1
apparmor
LoadPin
SELinux
Smack
tomoyo
Yama
SafeSetID
ipe
landlock
linux-7.2.3/Documentation/admin-guide/LSM/ipe.rst
+835
View File
@@ -0,0 +1,835 @@
.. SPDX-License-Identifier: GPL-2.0
Integrity Policy Enforcement (IPE)
==================================
.. NOTE::
This is the documentation for admins, system builders, or individuals
attempting to use IPE. If you're looking for more developer-focused
documentation about IPE please see :doc:`the design docs </security/ipe>`.
Overview
--------
Integrity Policy Enforcement (IPE) is a Linux Security Module that takes a
complementary approach to access control. Unlike traditional access control
mechanisms that rely on labels and paths for decision-making, IPE focuses
on the immutable security properties inherent to system components. These
properties are fundamental attributes or features of a system component
that cannot be altered, ensuring a consistent and reliable basis for
security decisions.
To elaborate, in the context of IPE, system components primarily refer to
files or the devices these files reside on. However, this is just a
starting point. The concept of system components is flexible and can be
extended to include new elements as the system evolves. The immutable
properties include the origin of a file, which remains constant and
unchangeable over time. For example, IPE policies can be crafted to trust
files originating from the initramfs. Since initramfs is typically verified
by the bootloader, its files are deemed trustworthy; "file is from
initramfs" becomes an immutable property under IPE's consideration.
The immutable property concept extends to the security features enabled on
a file's origin, such as dm-verity or fs-verity, which provide a layer of
integrity and trust. For example, IPE allows the definition of policies
that trust files from a dm-verity protected device. dm-verity ensures the
integrity of an entire device by providing a verifiable and immutable state
of its contents. Similarly, fs-verity offers filesystem-level integrity
checks, allowing IPE to enforce policies that trust files protected by
fs-verity. These two features cannot be turned off once established, so
they are considered immutable properties. These examples demonstrate how
IPE leverages immutable properties, such as a file's origin and its
integrity protection mechanisms, to make access control decisions.
For the IPE policy, specifically, it grants the ability to enforce
stringent access controls by assessing security properties against
reference values defined within the policy. This assessment can be based on
the existence of a security property (e.g., verifying if a file originates
from initramfs) or evaluating the internal state of an immutable security
property. The latter includes checking the roothash of a dm-verity
protected device, determining whether dm-verity possesses a valid
signature, assessing the digest of a fs-verity protected file, or
determining whether fs-verity possesses a valid built-in signature. This
nuanced approach to policy enforcement enables a highly secure and
customizable system defense mechanism, tailored to specific security
requirements and trust models.
To enable IPE, ensure that ``CONFIG_SECURITY_IPE`` (under
:menuselection:`Security -> Integrity Policy Enforcement (IPE)`) config
option is enabled.
Use Cases
---------
IPE works best in fixed-function devices: devices in which their purpose
is clearly defined and not supposed to be changed (e.g. network firewall
device in a data center, an IoT device, etcetera), where all software and
configuration is built and provisioned by the system owner.
IPE is a long-way off for use in general-purpose computing: the Linux
community as a whole tends to follow a decentralized trust model (known as
the web of trust), which IPE has no support for it yet. Instead, IPE
supports PKI (public key infrastructure), which generally designates a
set of trusted entities that provide a measure of absolute trust.
Additionally, while most packages are signed today, the files inside
the packages (for instance, the executables), tend to be unsigned. This
makes it difficult to utilize IPE in systems where a package manager is
expected to be functional, without major changes to the package manager
and ecosystem behind it.
The digest_cache LSM [#digest_cache_lsm]_ is a system that when combined with IPE,
could be used to enable and support general-purpose computing use cases.
Known Limitations
-----------------
IPE cannot verify the integrity of anonymous executable memory, such as
the trampolines created by gcc closures and libffi (<3.4.2), or JIT'd code.
Unfortunately, as this is dynamically generated code, there is no way
for IPE to ensure the integrity of this code to form a trust basis.
IPE cannot verify the integrity of programs written in interpreted
languages when these scripts are invoked by passing these program files
to the interpreter. This is because the way interpreters execute these
files; the scripts themselves are not evaluated as executable code
through one of IPE's hooks, but they are merely text files that are read
(as opposed to compiled executables). However, with the introduction of the
``AT_EXECVE_CHECK`` flag (:doc:`AT_EXECVE_CHECK </userspace-api/check_exec>`),
interpreters can use it to signal the kernel that a script file will be executed,
and request the kernel to perform LSM security checks on it.
IPE's EXECUTE operation enforcement differs between compiled executables and
interpreted scripts: For compiled executables, enforcement is triggered
automatically by the kernel during ``execve()``, ``execveat()``, ``mmap()``
and ``mprotect()`` syscalls when loading executable content. For interpreted
scripts, enforcement requires explicit interpreter integration using
``execveat()`` with ``AT_EXECVE_CHECK`` flag. Unlike exec syscalls that IPE
intercepts during the execution process, this mechanism needs the interpreter
to take the initiative, and existing interpreters won't be automatically
supported unless the signal call is added.
Threat Model
------------
IPE specifically targets the risk of tampering with user-space executable
code after the kernel has initially booted, including the kernel modules
loaded from userspace via ``modprobe`` or ``insmod``.
To illustrate, consider a scenario where an untrusted binary, possibly
malicious, is downloaded along with all necessary dependencies, including a
loader and libc. The primary function of IPE in this context is to prevent
the execution of such binaries and their dependencies.
IPE achieves this by verifying the integrity and authenticity of all
executable code before allowing them to run. It conducts a thorough
check to ensure that the code's integrity is intact and that they match an
authorized reference value (digest, signature, etc) as per the defined
policy. If a binary does not pass this verification process, either
because its integrity has been compromised or it does not meet the
authorization criteria, IPE will deny its execution. Additionally, IPE
generates audit logs which may be utilized to detect and analyze failures
resulting from policy violation.
Tampering threat scenarios include modification or replacement of
executable code by a range of actors including:
- Actors with physical access to the hardware
- Actors with local network access to the system
- Actors with access to the deployment system
- Compromised internal systems under external control
- Malicious end users of the system
- Compromised end users of the system
- Remote (external) compromise of the system
IPE does not mitigate threats arising from malicious but authorized
developers (with access to a signing certificate), or compromised
developer tools used by them (i.e. return-oriented programming attacks).
Additionally, IPE draws hard security boundary between userspace and
kernelspace. As a result, kernel-level exploits are considered outside
the scope of IPE and mitigation is left to other mechanisms.
Policy
------
IPE policy is a plain-text [#devdoc]_ policy composed of multiple statements
over several lines. There is one required line, at the top of the
policy, indicating the policy name, and the policy version, for
instance::
policy_name=Ex_Policy policy_version=0.0.0
The policy name is a unique key identifying this policy in a human
readable name. This is used to create nodes under securityfs as well as
uniquely identify policies to deploy new policies vs update existing
policies.
The policy version indicates the current version of the policy (NOT the
policy syntax version). This is used to prevent rollback of policy to
potentially insecure previous versions of the policy.
The next portion of IPE policy are rules. Rules are formed by key=value
pairs, known as properties. IPE rules require two properties: ``action``,
which determines what IPE does when it encounters a match against the
rule, and ``op``, which determines when the rule should be evaluated.
The ordering is significant, a rule must start with ``op``, and end with
``action``. Thus, a minimal rule is::
op=EXECUTE action=ALLOW
This example will allow any execution. Additional properties are used to
assess immutable security properties about the files being evaluated.
These properties are intended to be descriptions of systems within the
kernel that can provide a measure of integrity verification, such that IPE
can determine the trust of the resource based on the value of the property.
Rules are evaluated top-to-bottom. As a result, any revocation rules,
or denies should be placed early in the file to ensure that these rules
are evaluated before a rule with ``action=ALLOW``.
IPE policy supports comments. The character '#' will function as a
comment, ignoring all characters to the right of '#' until the newline.
The default behavior of IPE evaluations can also be expressed in policy,
through the ``DEFAULT`` statement. This can be done at a global level,
or a per-operation level::
# Global
DEFAULT action=ALLOW
# Operation Specific
DEFAULT op=EXECUTE action=ALLOW
A default must be set for all known operations in IPE. If you want to
preserve older policies being compatible with newer kernels that can introduce
new operations, set a global default of ``ALLOW``, then override the
defaults on a per-operation basis (as above).
With configurable policy-based LSMs, there's several issues with
enforcing the configurable policies at startup, around reading and
parsing the policy:
1. The kernel *should* not read files from userspace, so directly reading
the policy file is prohibited.
2. The kernel command line has a character limit, and one kernel module
should not reserve the entire character limit for its own
configuration.
3. There are various boot loaders in the kernel ecosystem, so handing
off a memory block would be costly to maintain.
As a result, IPE has addressed this problem through a concept of a "boot
policy". A boot policy is a minimal policy which is compiled into the
kernel. This policy is intended to get the system to a state where
userspace is set up and ready to receive commands, at which point a more
complex policy can be deployed via securityfs. The boot policy can be
specified via ``SECURITY_IPE_BOOT_POLICY`` config option, which accepts
a path to a plain-text version of the IPE policy to apply. This policy
will be compiled into the kernel. If not specified, IPE will be disabled
until a policy is deployed and activated through securityfs.
Deploying Policies
~~~~~~~~~~~~~~~~~~
Policies can be deployed from userspace through securityfs. These policies
are signed through the PKCS#7 message format to enforce some level of
authorization of the policies (prohibiting an attacker from gaining
unconstrained root, and deploying an "allow all" policy). These
policies must be signed by a certificate that chains to the
``SYSTEM_TRUSTED_KEYRING``, or to the secondary and/or platform keyrings if
``CONFIG_IPE_POLICY_SIG_SECONDARY_KEYRING`` and/or
``CONFIG_IPE_POLICY_SIG_PLATFORM_KEYRING`` are enabled, respectively.
With openssl, the policy can be signed by::
openssl smime -sign \
-in "$MY_POLICY" \
-signer "$MY_CERTIFICATE" \
-inkey "$MY_PRIVATE_KEY" \
-noattr \
-nodetach \
-nosmimecap \
-outform der \
-out "$MY_POLICY.p7b"
Deploying the policies is done through securityfs, through the
``new_policy`` node. To deploy a policy, simply cat the file into the
securityfs node::
cat "$MY_POLICY.p7b" > /sys/kernel/security/ipe/new_policy
Upon success, this will create one subdirectory under
``/sys/kernel/security/ipe/policies/``. The subdirectory will be the
``policy_name`` field of the policy deployed, so for the example above,
the directory will be ``/sys/kernel/security/ipe/policies/Ex_Policy``.
Within this directory, there will be seven files: ``pkcs7``, ``policy``,
``name``, ``version``, ``active``, ``update``, and ``delete``.
The ``pkcs7`` file is read-only. Reading it returns the raw PKCS#7 data
that was provided to the kernel, representing the policy. If the policy being
read is the boot policy, this will return ``ENOENT``, as it is not signed.
The ``policy`` file is read only. Reading it returns the PKCS#7 inner
content of the policy, which will be the plain text policy.
The ``active`` file is used to set a policy as the currently active policy.
This file is rw, and accepts a value of ``"1"`` to set the policy as active.
Since only a single policy can be active at one time, all other policies
will be marked inactive. The policy being marked active must have a policy
version greater or equal to the currently-running version.
The ``update`` file is used to update a policy that is already present
in the kernel. This file is write-only and accepts a PKCS#7 signed
policy. Two checks will always be performed on this policy: First, the
``policy_names`` must match with the updated version and the existing
version. Second the updated policy must have a policy version greater than
the currently-running version. This is to prevent rollback attacks.
The ``delete`` file is used to remove a policy that is no longer needed.
This file is write-only and accepts a value of ``1`` to delete the policy.
On deletion, the securityfs node representing the policy will be removed.
However, delete the current active policy is not allowed and will return
an operation not permitted error.
Similarly, writing to both ``update`` and ``new_policy`` could result in
bad message(policy syntax error) or file exists error. The latter error happens
when trying to deploy a policy with a ``policy_name`` while the kernel already
has a deployed policy with the same ``policy_name``.
Deploying a policy will *not* cause IPE to start enforcing the policy. IPE will
only enforce the policy marked active. Note that only one policy can be active
at a time.
Once deployment is successful, the policy can be activated, by writing file
``/sys/kernel/security/ipe/policies/$policy_name/active``.
For example, the ``Ex_Policy`` can be activated by::
echo 1 > "/sys/kernel/security/ipe/policies/Ex_Policy/active"
From above point on, ``Ex_Policy`` is now the enforced policy on the
system.
IPE also provides a way to delete policies. This can be done via the
``delete`` securityfs node,
``/sys/kernel/security/ipe/policies/$policy_name/delete``.
Writing ``1`` to that file deletes the policy::
echo 1 > "/sys/kernel/security/ipe/policies/$policy_name/delete"
There is only one requirement to delete a policy: the policy being deleted
must be inactive.
.. NOTE::
If a traditional MAC system is enabled (SELinux, apparmor, smack), all
writes to ipe's securityfs nodes require ``CAP_MAC_ADMIN``.
Modes
~~~~~
IPE supports two modes of operation: permissive (similar to SELinux's
permissive mode) and enforced. In permissive mode, all events are
checked and policy violations are logged, but the policy is not really
enforced. This allows users to test policies before enforcing them.
The default mode is enforce, and can be changed via the kernel command
line parameter ``ipe.enforce=(0|1)``, or the securityfs node
``/sys/kernel/security/ipe/enforce``.
.. NOTE::
If a traditional MAC system is enabled (SELinux, apparmor, smack, etcetera),
all writes to ipe's securityfs nodes require ``CAP_MAC_ADMIN``.
Audit Events
~~~~~~~~~~~~
1420 AUDIT_IPE_ACCESS
^^^^^^^^^^^^^^^^^^^^^
Event Examples::
type=1420 audit(1653364370.067:61): ipe_op=EXECUTE ipe_hook=MMAP enforcing=1 pid=2241 comm="ld-linux.so" path="/deny/lib/libc.so.6" dev="sda2" ino=14549020 rule="DEFAULT action=DENY"
type=1300 audit(1653364370.067:61): SYSCALL arch=c000003e syscall=9 success=no exit=-13 a0=7f1105a28000 a1=195000 a2=5 a3=812 items=0 ppid=2219 pid=2241 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=2 comm="ld-linux.so" exe="/tmp/ipe-test/lib/ld-linux.so" subj=unconfined key=(null)
type=1327 audit(1653364370.067:61): 707974686F6E3300746573742F6D61696E2E7079002D6E00
type=1420 audit(1653364735.161:64): ipe_op=EXECUTE ipe_hook=MMAP enforcing=1 pid=2472 comm="mmap_test" path=? dev=? ino=? rule="DEFAULT action=DENY"
type=1300 audit(1653364735.161:64): SYSCALL arch=c000003e syscall=9 success=no exit=-13 a0=0 a1=1000 a2=4 a3=21 items=0 ppid=2219 pid=2472 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=2 comm="mmap_test" exe="/root/overlake_test/upstream_test/vol_fsverity/bin/mmap_test" subj=unconfined key=(null)
type=1327 audit(1653364735.161:64): 707974686F6E3300746573742F6D61696E2E7079002D6E00
This event indicates that IPE made an access control decision; the IPE
specific record (1420) is always emitted in conjunction with a
``AUDITSYSCALL`` record.
Determining whether IPE is in permissive or enforced mode can be derived
from ``success`` property and exit code of the ``AUDITSYSCALL`` record.
Field descriptions:
+-----------+------------+-----------+---------------------------------------------------------------------------------+
| Field | Value Type | Optional? | Description of Value |
+===========+============+===========+=================================================================================+
| ipe_op | string | No | The IPE operation name associated with the log |
+-----------+------------+-----------+---------------------------------------------------------------------------------+
| ipe_hook | string | No | The name of the LSM hook that triggered the IPE event |
+-----------+------------+-----------+---------------------------------------------------------------------------------+
| enforcing | integer | No | The current IPE enforcing state 1 is in enforcing mode, 0 is in permissive mode |
+-----------+------------+-----------+---------------------------------------------------------------------------------+
| pid | integer | No | The pid of the process that triggered the IPE event. |
+-----------+------------+-----------+---------------------------------------------------------------------------------+
| comm | string | No | The command line program name of the process that triggered the IPE event |
+-----------+------------+-----------+---------------------------------------------------------------------------------+
| path | string | Yes | The absolute path to the evaluated file |
+-----------+------------+-----------+---------------------------------------------------------------------------------+
| ino | integer | Yes | The inode number of the evaluated file |
+-----------+------------+-----------+---------------------------------------------------------------------------------+
| dev | string | Yes | The device name of the evaluated file, e.g. vda |
+-----------+------------+-----------+---------------------------------------------------------------------------------+
| rule | string | No | The matched policy rule |
+-----------+------------+-----------+---------------------------------------------------------------------------------+
1421 AUDIT_IPE_CONFIG_CHANGE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Event Example::
type=1421 audit(1653425583.136:54): old_active_pol_name="Allow_All" old_active_pol_version=0.0.0 old_policy_digest=sha256:E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855 new_active_pol_name="boot_verified" new_active_pol_version=0.0.0 new_policy_digest=sha256:820EEA5B40CA42B51F68962354BA083122A20BB846F26765076DD8EED7B8F4DB auid=4294967295 ses=4294967295 lsm=ipe res=1
type=1300 audit(1653425583.136:54): SYSCALL arch=c000003e syscall=1 success=yes exit=2 a0=3 a1=5596fcae1fb0 a2=2 a3=2 items=0 ppid=184 pid=229 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=4294967295 comm="python3" exe="/usr/bin/python3.10" key=(null)
type=1327 audit(1653425583.136:54): PROCTITLE proctitle=707974686F6E3300746573742F6D61696E2E7079002D66002E2
This event indicates that IPE switched the active poliy from one to another
along with the version and the hash digest of the two policies.
Note IPE can only have one policy active at a time, all access decision
evaluation is based on the current active policy.
The normal procedure to deploy a new policy is loading the policy to deploy
into the kernel first, then switch the active policy to it.
This record will always be emitted in conjunction with a ``AUDITSYSCALL`` record for the ``write`` syscall.
Field descriptions:
+------------------------+------------+-----------+---------------------------------------------------+
| Field | Value Type | Optional? | Description of Value |
+========================+============+===========+===================================================+
| old_active_pol_name | string | Yes | The name of previous active policy |
+------------------------+------------+-----------+---------------------------------------------------+
| old_active_pol_version | string | Yes | The version of previous active policy |
+------------------------+------------+-----------+---------------------------------------------------+
| old_policy_digest | string | Yes | The hash of previous active policy |
+------------------------+------------+-----------+---------------------------------------------------+
| new_active_pol_name | string | No | The name of current active policy |
+------------------------+------------+-----------+---------------------------------------------------+
| new_active_pol_version | string | No | The version of current active policy |
+------------------------+------------+-----------+---------------------------------------------------+
| new_policy_digest | string | No | The hash of current active policy |
+------------------------+------------+-----------+---------------------------------------------------+
| auid | integer | No | The login user ID |
+------------------------+------------+-----------+---------------------------------------------------+
| ses | integer | No | The login session ID |
+------------------------+------------+-----------+---------------------------------------------------+
| lsm | string | No | The lsm name associated with the event |
+------------------------+------------+-----------+---------------------------------------------------+
| res | integer | No | The result of the audited operation(success/fail) |
+------------------------+------------+-----------+---------------------------------------------------+
1422 AUDIT_IPE_POLICY_LOAD
^^^^^^^^^^^^^^^^^^^^^^^^^^
Event Example::
type=1422 audit(1653425529.927:53): policy_name="boot_verified" policy_version=0.0.0 policy_digest=sha256:820EEA5B40CA42B51F68962354BA083122A20BB846F26765076DD8EED7B8F4DB auid=4294967295 ses=4294967295 lsm=ipe res=1 errno=0
type=1300 audit(1653425529.927:53): arch=c000003e syscall=1 success=yes exit=2567 a0=3 a1=5596fcae1fb0 a2=a07 a3=2 items=0 ppid=184 pid=229 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=4294967295 comm="python3" exe="/usr/bin/python3.10" key=(null)
type=1327 audit(1653425529.927:53): PROCTITLE proctitle=707974686F6E3300746573742F6D61696E2E7079002D66002E2E
This record indicates a new policy has been loaded into the kernel with the policy name, policy version and policy hash.
This record will always be emitted in conjunction with a ``AUDITSYSCALL`` record for the ``write`` syscall.
Field descriptions:
+----------------+------------+-----------+-------------------------------------------------------------+
| Field | Value Type | Optional? | Description of Value |
+================+============+===========+=============================================================+
| policy_name | string | Yes | The policy_name |
+----------------+------------+-----------+-------------------------------------------------------------+
| policy_version | string | Yes | The policy_version |
+----------------+------------+-----------+-------------------------------------------------------------+
| policy_digest | string | Yes | The policy hash |
+----------------+------------+-----------+-------------------------------------------------------------+
| auid | integer | No | The login user ID |
+----------------+------------+-----------+-------------------------------------------------------------+
| ses | integer | No | The login session ID |
+----------------+------------+-----------+-------------------------------------------------------------+
| lsm | string | No | The lsm name associated with the event |
+----------------+------------+-----------+-------------------------------------------------------------+
| res | integer | No | The result of the audited operation(success/fail) |
+----------------+------------+-----------+-------------------------------------------------------------+
| errno | integer | No | Error code from policy loading operations (see table below) |
+----------------+------------+-----------+-------------------------------------------------------------+
Policy error codes (errno):
The following table lists the error codes that may appear in the errno field while loading or updating the policy:
+----------------+--------------------------------------------------------+
| Error Code | Description |
+================+========================================================+
| 0 | Success |
+----------------+--------------------------------------------------------+
| -EPERM | Insufficient permission |
+----------------+--------------------------------------------------------+
| -EEXIST | Same name policy already deployed |
+----------------+--------------------------------------------------------+
| -EBADMSG | Policy is invalid |
+----------------+--------------------------------------------------------+
| -ENOMEM | Out of memory (OOM) |
+----------------+--------------------------------------------------------+
| -ERANGE | Policy version number overflow |
+----------------+--------------------------------------------------------+
| -EINVAL | Policy version parsing error |
+----------------+--------------------------------------------------------+
| -ENOKEY | Key used to sign the IPE policy not found in keyring |
+----------------+--------------------------------------------------------+
| -EKEYREJECTED | Policy signature verification failed |
+----------------+--------------------------------------------------------+
| -ESTALE | Attempting to update an IPE policy with older version |
+----------------+--------------------------------------------------------+
| -ENOENT | Policy was deleted while updating |
+----------------+--------------------------------------------------------+
1404 AUDIT_MAC_STATUS
^^^^^^^^^^^^^^^^^^^^^
Event Examples::
type=1404 audit(1653425689.008:55): enforcing=0 old_enforcing=1 auid=4294967295 ses=4294967295 enabled=1 old-enabled=1 lsm=ipe res=1
type=1300 audit(1653425689.008:55): arch=c000003e syscall=1 success=yes exit=2 a0=1 a1=55c1065e5c60 a2=2 a3=0 items=0 ppid=405 pid=441 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=)
type=1327 audit(1653425689.008:55): proctitle="-bash"
type=1404 audit(1653425689.008:55): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295 enabled=1 old-enabled=1 lsm=ipe res=1
type=1300 audit(1653425689.008:55): arch=c000003e syscall=1 success=yes exit=2 a0=1 a1=55c1065e5c60 a2=2 a3=0 items=0 ppid=405 pid=441 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=)
type=1327 audit(1653425689.008:55): proctitle="-bash"
This record will always be emitted in conjunction with a ``AUDITSYSCALL`` record for the ``write`` syscall.
Field descriptions:
+---------------+------------+-----------+-------------------------------------------------------------------------------------------------+
| Field | Value Type | Optional? | Description of Value |
+===============+============+===========+=================================================================================================+
| enforcing | integer | No | The enforcing state IPE is being switched to, 1 is in enforcing mode, 0 is in permissive mode |
+---------------+------------+-----------+-------------------------------------------------------------------------------------------------+
| old_enforcing | integer | No | The enforcing state IPE is being switched from, 1 is in enforcing mode, 0 is in permissive mode |
+---------------+------------+-----------+-------------------------------------------------------------------------------------------------+
| auid | integer | No | The login user ID |
+---------------+------------+-----------+-------------------------------------------------------------------------------------------------+
| ses | integer | No | The login session ID |
+---------------+------------+-----------+-------------------------------------------------------------------------------------------------+
| enabled | integer | No | The new TTY audit enabled setting |
+---------------+------------+-----------+-------------------------------------------------------------------------------------------------+
| old-enabled | integer | No | The old TTY audit enabled setting |
+---------------+------------+-----------+-------------------------------------------------------------------------------------------------+
| lsm | string | No | The lsm name associated with the event |
+---------------+------------+-----------+-------------------------------------------------------------------------------------------------+
| res | integer | No | The result of the audited operation(success/fail) |
+---------------+------------+-----------+-------------------------------------------------------------------------------------------------+
Success Auditing
^^^^^^^^^^^^^^^^
IPE supports success auditing. When enabled, all events that pass IPE
policy and are not blocked will emit an audit event. This is disabled by
default, and can be enabled via the kernel command line
``ipe.success_audit=(0|1)`` or
``/sys/kernel/security/ipe/success_audit`` securityfs file.
This is *very* noisy, as IPE will check every userspace binary on the
system, but is useful for debugging policies.
.. NOTE::
If a traditional MAC system is enabled (SELinux, apparmor, smack, etcetera),
all writes to ipe's securityfs nodes require ``CAP_MAC_ADMIN``.
Properties
----------
As explained above, IPE properties are ``key=value`` pairs expressed in IPE
policy. Two properties are built-into the policy parser: 'op' and 'action'.
The other properties are used to restrict immutable security properties
about the files being evaluated. Currently those properties are:
'``boot_verified``', '``dmverity_signature``', '``dmverity_roothash``',
'``fsverity_signature``', '``fsverity_digest``'. A description of all
properties supported by IPE are listed below:
op
~~
Indicates the operation for a rule to apply to. Must be in every rule,
as the first token. IPE supports the following operations:
``EXECUTE``
Pertains to any file attempting to be executed, or loaded as an
executable.
``FIRMWARE``:
Pertains to firmware being loaded via the firmware_class interface.
This covers both the preallocated buffer and the firmware file
itself.
``KMODULE``:
Pertains to loading kernel modules via ``modprobe`` or ``insmod``.
``KEXEC_IMAGE``:
Pertains to kernel images loading via ``kexec``.
``KEXEC_INITRAMFS``
Pertains to initrd images loading via ``kexec --initrd``.
``POLICY``:
Controls loading policies via reading a kernel-space initiated read.
An example of such is loading IMA policies by writing the path
to the policy file to ``$securityfs/ima/policy``
``X509_CERT``:
Controls loading IMA certificates through the Kconfigs,
``CONFIG_IMA_X509_PATH`` and ``CONFIG_EVM_X509_PATH``.
action
~~~~~~
Determines what IPE should do when a rule matches. Must be in every
rule, as the final clause. Can be one of:
``ALLOW``:
If the rule matches, explicitly allow access to the resource to proceed
without executing any more rules.
``DENY``:
If the rule matches, explicitly prohibit access to the resource to
proceed without executing any more rules.
boot_verified
~~~~~~~~~~~~~
This property can be utilized for authorization of files from initramfs.
The format of this property is::
boot_verified=(TRUE|FALSE)
.. WARNING::
This property will trust files from initramfs(rootfs). It should
only be used during early booting stage. Before mounting the real
rootfs on top of the initramfs, initramfs script will recursively
remove all files and directories on the initramfs. This is typically
implemented by using switch_root(8) [#switch_root]_. Therefore the
initramfs will be empty and not accessible after the real
rootfs takes over. It is advised to switch to a different policy
that doesn't rely on the property after this point.
This ensures that the trust policies remain relevant and effective
throughout the system's operation.
dmverity_roothash
~~~~~~~~~~~~~~~~~
This property can be utilized for authorization or revocation of
specific dm-verity volumes, identified via their root hashes. It has a
dependency on the DM_VERITY module. This property is controlled by
the ``IPE_PROP_DM_VERITY`` config option, it will be automatically
selected when ``SECURITY_IPE`` and ``DM_VERITY`` are all enabled.
The format of this property is::
dmverity_roothash=DigestName:HexadecimalString
The supported DigestNames for dmverity_roothash are [#dmveritydigests]_
+ blake2b-512
+ blake2s-256
+ sha256
+ sha384
+ sha512
+ sha3-224
+ sha3-256
+ sha3-384
+ sha3-512
+ sm3
+ rmd160
dmverity_signature
~~~~~~~~~~~~~~~~~~
This property can be utilized for authorization of all dm-verity
volumes that have a signed roothash that validated by a keyring
specified by dm-verity's configuration, either the system trusted
keyring, or the secondary keyring. It depends on
``DM_VERITY_VERIFY_ROOTHASH_SIG`` config option and is controlled by
the ``IPE_PROP_DM_VERITY_SIGNATURE`` config option, it will be automatically
selected when ``SECURITY_IPE``, ``DM_VERITY`` and
``DM_VERITY_VERIFY_ROOTHASH_SIG`` are all enabled.
The format of this property is::
dmverity_signature=(TRUE|FALSE)
fsverity_digest
~~~~~~~~~~~~~~~
This property can be utilized for authorization of specific fsverity
enabled files, identified via their fsverity digests.
It depends on ``FS_VERITY`` config option and is controlled by
the ``IPE_PROP_FS_VERITY`` config option, it will be automatically
selected when ``SECURITY_IPE`` and ``FS_VERITY`` are all enabled.
The format of this property is::
fsverity_digest=DigestName:HexadecimalString
The supported DigestNames for fsverity_digest are [#fsveritydigest]_
+ sha256
+ sha512
fsverity_signature
~~~~~~~~~~~~~~~~~~
This property is used to authorize all fs-verity enabled files that have
been verified by fs-verity's built-in signature mechanism. The signature
verification relies on a key stored within the ".fs-verity" keyring. It
depends on ``FS_VERITY_BUILTIN_SIGNATURES`` config option and
it is controlled by the ``IPE_PROP_FS_VERITY`` config option,
it will be automatically selected when ``SECURITY_IPE``, ``FS_VERITY``
and ``FS_VERITY_BUILTIN_SIGNATURES`` are all enabled.
The format of this property is::
fsverity_signature=(TRUE|FALSE)
Policy Examples
---------------
Allow all
~~~~~~~~~
::
policy_name=Allow_All policy_version=0.0.0
DEFAULT action=ALLOW
Allow only initramfs
~~~~~~~~~~~~~~~~~~~~
::
policy_name=Allow_Initramfs policy_version=0.0.0
DEFAULT action=DENY
op=EXECUTE boot_verified=TRUE action=ALLOW
Allow any signed and validated dm-verity volume and the initramfs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
policy_name=Allow_Signed_DMV_And_Initramfs policy_version=0.0.0
DEFAULT action=DENY
op=EXECUTE boot_verified=TRUE action=ALLOW
op=EXECUTE dmverity_signature=TRUE action=ALLOW
Prohibit execution from a specific dm-verity volume
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
policy_name=Deny_DMV_By_Roothash policy_version=0.0.0
DEFAULT action=DENY
op=EXECUTE dmverity_roothash=sha256:cd2c5bae7c6c579edaae4353049d58eb5f2e8be0244bf05345bc8e5ed257baff action=DENY
op=EXECUTE boot_verified=TRUE action=ALLOW
op=EXECUTE dmverity_signature=TRUE action=ALLOW
Allow only a specific dm-verity volume
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
policy_name=Allow_DMV_By_Roothash policy_version=0.0.0
DEFAULT action=DENY
op=EXECUTE dmverity_roothash=sha256:401fcec5944823ae12f62726e8184407a5fa9599783f030dec146938 action=ALLOW
Allow any fs-verity file with a valid built-in signature
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
policy_name=Allow_Signed_And_Validated_FSVerity policy_version=0.0.0
DEFAULT action=DENY
op=EXECUTE fsverity_signature=TRUE action=ALLOW
Allow execution of a specific fs-verity file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
policy_name=ALLOW_FSV_By_Digest policy_version=0.0.0
DEFAULT action=DENY
op=EXECUTE fsverity_digest=sha256:fd88f2b8824e197f850bf4c5109bea5cf0ee38104f710843bb72da796ba5af9e action=ALLOW
Additional Information
----------------------
- `Github Repository <https://github.com/microsoft/ipe>`_
- :doc:`Developer and design docs for IPE </security/ipe>`
FAQ
---
Q:
What's the difference between other LSMs which provide a measure of
trust-based access control?
A:
In general, there's two other LSMs that can provide similar functionality:
IMA, and Loadpin.
IMA and IPE are functionally very similar. The significant difference between
the two is the policy. [#devdoc]_
Loadpin and IPE differ fairly dramatically, as Loadpin only covers the IPE's
kernel read operations, whereas IPE is capable of controlling execution
on top of kernel read. The trust model is also different; Loadpin roots its
trust in the initial super-block, whereas trust in IPE is stemmed from kernel
itself (via ``SYSTEM_TRUSTED_KEYS``).
-----------
.. [#digest_cache_lsm] https://lore.kernel.org/lkml/20240415142436.2545003-1-roberto.sassu@huaweicloud.com/
.. [#devdoc] Please see :doc:`the design docs </security/ipe>` for more on
this topic.
.. [#switch_root] https://man7.org/linux/man-pages/man8/switch_root.8.html
.. [#dmveritydigests] These hash algorithms are based on values accepted by
the Linux crypto API; IPE does not impose any
restrictions on the digest algorithm itself;
thus, this list may be out of date.
.. [#fsveritydigest] These hash algorithms are based on values accepted by the
kernel's fsverity support; IPE does not impose any
restrictions on the digest algorithm itself;
thus, this list may be out of date.
linux-7.2.3/Documentation/admin-guide/LSM/landlock.rst
+194
View File
@@ -0,0 +1,194 @@
.. SPDX-License-Identifier: GPL-2.0
.. Copyright © 2025 Microsoft Corporation
================================
Landlock: system-wide management
================================
:Author: Mickaël Salaün
:Date: June 2026
Landlock can leverage the audit framework to log events.
User space documentation can be found here:
Documentation/userspace-api/landlock.rst.
Audit
=====
Denied access requests are logged by default for a sandboxed program if `audit`
is enabled. This default behavior can be changed with the
sys_landlock_restrict_self() flags (cf.
Documentation/userspace-api/landlock.rst), or suppressed on a per-object
basis by using ``LANDLOCK_ADD_RULE_QUIET`` (ABI 10+). Landlock logs can
also be masked thanks to audit rules. Landlock can generate 2 audit
record types.
Record types
------------
AUDIT_LANDLOCK_ACCESS
This record type identifies a denied access request to a kernel resource.
The ``domain`` field indicates the ID of the domain that blocked the
request. The ``blockers`` field indicates the cause(s) of this denial
(separated by a comma), and the following fields identify the kernel object
(similar to SELinux). There may be more than one of this record type per
audit event.
Example with a file link request generating two records in the same event::
domain=195ba459b blockers=fs.refer path="/usr/bin" dev="vda2" ino=351
domain=195ba459b blockers=fs.make_reg,fs.refer path="/usr/local" dev="vda2" ino=365
The ``blockers`` field uses dot-separated prefixes to indicate the type of
restriction that caused the denial:
**fs.*** - Filesystem access rights (ABI 1+):
- fs.execute, fs.write_file, fs.read_file, fs.read_dir
- fs.remove_dir, fs.remove_file
- fs.make_char, fs.make_dir, fs.make_reg, fs.make_sock
- fs.make_fifo, fs.make_block, fs.make_sym
- fs.refer (ABI 2+)
- fs.truncate (ABI 3+)
- fs.ioctl_dev (ABI 5+)
**net.*** - Network access rights (ABI 4+):
- net.bind_tcp - TCP port binding was denied
- net.connect_tcp - TCP connection was denied
- net.bind_udp - UDP port binding was denied
- net.connect_send_udp - UDP connection and send was denied
**scope.*** - IPC scoping restrictions (ABI 6+):
- scope.abstract_unix_socket - Abstract UNIX socket connection denied
- scope.signal - Signal sending denied
Multiple blockers can appear in a single event (comma-separated) when
multiple access rights are missing. For example, creating a regular file
in a directory that lacks both ``make_reg`` and ``refer`` rights would show
``blockers=fs.make_reg,fs.refer``.
The object identification fields (path, dev, ino for filesystem; opid,
ocomm for signals) depend on the type of access being blocked and provide
context about what resource was involved in the denial.
AUDIT_LANDLOCK_DOMAIN
This record type describes the status of a Landlock domain. The ``status``
field can be either ``allocated`` or ``deallocated``.
The ``allocated`` status is part of the same audit event and follows
the first logged ``AUDIT_LANDLOCK_ACCESS`` record of a domain. It identifies
Landlock domain information at the time of the sys_landlock_restrict_self()
call with the following fields:
- the ``domain`` ID
- the enforcement ``mode``
- the domain creator's ``pid``
- the domain creator's ``uid``
- the domain creator's executable path (``exe``)
- the domain creator's command line (``comm``)
Example::
domain=195ba459b status=allocated mode=enforcing pid=300 uid=0 exe="/root/sandboxer" comm="sandboxer"
The ``deallocated`` status is an event on its own and it identifies a
Landlock domain release. After such event, it is guarantee that the
related domain ID will never be reused during the lifetime of the system.
The ``domain`` field indicates the ID of the domain which is released, and
the ``denials`` field indicates the total number of denied access request,
which might not have been logged according to the audit rules and
sys_landlock_restrict_self()'s flags.
Example::
domain=195ba459b status=deallocated denials=3
Event samples
--------------
Here are two examples of log events (see serial numbers).
In this example a sandboxed program (``kill``) tries to send a signal to the
init process, which is denied because of the signal scoping restriction
(``LL_SCOPED=s``)::
$ LL_FS_RO=/ LL_FS_RW=/ LL_SCOPED=s LL_FORCE_LOG=1 ./sandboxer kill 1
This command generates two events, each identified with a unique serial
number following a timestamp (``msg=audit(1729738800.268:30)``). The first
event (serial ``30``) contains 4 records. The first record
(``type=LANDLOCK_ACCESS``) shows an access denied by the domain `1a6fdc66f`.
The cause of this denial is signal scoping restriction
(``blockers=scope.signal``). The process that would have receive this signal
is the init process (``opid=1 ocomm="systemd"``).
The second record (``type=LANDLOCK_DOMAIN``) describes (``status=allocated``)
domain `1a6fdc66f`. This domain was created by process ``286`` executing the
``/root/sandboxer`` program launched by the root user.
The third record (``type=SYSCALL``) describes the syscall, its provided
arguments, its result (``success=no exit=-1``), and the process that called it.
The fourth record (``type=PROCTITLE``) shows the command's name as an
hexadecimal value. This can be translated with ``python -c
'print(bytes.fromhex("6B696C6C0031"))'``.
Finally, the last record (``type=LANDLOCK_DOMAIN``) is also the only one from
the second event (serial ``31``). It is not tied to a direct user space action
but an asynchronous one to free resources tied to a Landlock domain
(``status=deallocated``). This can be useful to know that the following logs
will not concern the domain ``1a6fdc66f`` anymore. This record also summarize
the number of requests this domain denied (``denials=1``), whether they were
logged or not.
.. code-block::
type=LANDLOCK_ACCESS msg=audit(1729738800.268:30): domain=1a6fdc66f blockers=scope.signal opid=1 ocomm="systemd"
type=LANDLOCK_DOMAIN msg=audit(1729738800.268:30): domain=1a6fdc66f status=allocated mode=enforcing pid=286 uid=0 exe="/root/sandboxer" comm="sandboxer"
type=SYSCALL msg=audit(1729738800.268:30): arch=c000003e syscall=62 success=no exit=-1 [..] ppid=272 pid=286 auid=0 uid=0 gid=0 [...] comm="kill" [...]
type=PROCTITLE msg=audit(1729738800.268:30): proctitle=6B696C6C0031
type=LANDLOCK_DOMAIN msg=audit(1729738800.324:31): domain=1a6fdc66f status=deallocated denials=1
Here is another example showcasing filesystem access control::
$ LL_FS_RO=/ LL_FS_RW=/tmp LL_FORCE_LOG=1 ./sandboxer sh -c "echo > /etc/passwd"
The related audit logs contains 8 records from 3 different events (serials 33,
34 and 35) created by the same domain `1a6fdc679`::
type=LANDLOCK_ACCESS msg=audit(1729738800.221:33): domain=1a6fdc679 blockers=fs.write_file path="/dev/tty" dev="devtmpfs" ino=9
type=LANDLOCK_DOMAIN msg=audit(1729738800.221:33): domain=1a6fdc679 status=allocated mode=enforcing pid=289 uid=0 exe="/root/sandboxer" comm="sandboxer"
type=SYSCALL msg=audit(1729738800.221:33): arch=c000003e syscall=257 success=no exit=-13 [...] ppid=272 pid=289 auid=0 uid=0 gid=0 [...] comm="sh" [...]
type=PROCTITLE msg=audit(1729738800.221:33): proctitle=7368002D63006563686F203E202F6574632F706173737764
type=LANDLOCK_ACCESS msg=audit(1729738800.221:34): domain=1a6fdc679 blockers=fs.write_file path="/etc/passwd" dev="vda2" ino=143821
type=SYSCALL msg=audit(1729738800.221:34): arch=c000003e syscall=257 success=no exit=-13 [...] ppid=272 pid=289 auid=0 uid=0 gid=0 [...] comm="sh" [...]
type=PROCTITLE msg=audit(1729738800.221:34): proctitle=7368002D63006563686F203E202F6574632F706173737764
type=LANDLOCK_DOMAIN msg=audit(1729738800.261:35): domain=1a6fdc679 status=deallocated denials=2
Event filtering
---------------
If you get spammed with audit logs related to Landlock, this is either an
attack attempt or a bug in the security policy. We can put in place some
filters to limit noise with two complementary ways:
- with sys_landlock_restrict_self()'s flags, or
``LANDLOCK_ADD_RULE_QUIET`` (ABI 10+) if we can fix the sandboxed
programs,
- or with audit rules (see :manpage:`auditctl(8)`).
Additional documentation
========================
* `Linux Audit Documentation`_
* Documentation/userspace-api/landlock.rst
* Documentation/security/landlock.rst
* https://landlock.io
.. Links
.. _Linux Audit Documentation:
https://github.com/linux-audit/audit-documentation/wiki
linux-7.2.3/Documentation/admin-guide/LSM/tomoyo.rst
+52
View File
@@ -0,0 +1,52 @@
======
TOMOYO
======
What is TOMOYO?
===============
TOMOYO is a name-based MAC extension (LSM module) for the Linux kernel.
LiveCD-based tutorials are available at
https://tomoyo.sourceforge.net/1.8/ubuntu12.04-live.html
https://tomoyo.sourceforge.net/1.8/centos6-live.html
Though these tutorials use non-LSM version of TOMOYO, they are useful for you
to know what TOMOYO is.
How to enable TOMOYO?
=====================
Build the kernel with ``CONFIG_SECURITY_TOMOYO=y`` and pass ``security=tomoyo`` on
kernel's command line.
Please see https://tomoyo.sourceforge.net/2.6/ for details.
Where is documentation?
=======================
User <-> Kernel interface documentation is available at
https://tomoyo.sourceforge.net/2.6/policy-specification/index.html .
Materials we prepared for seminars and symposiums are available at
https://sourceforge.net/projects/tomoyo/files/docs/ .
Below lists are chosen from three aspects.
What is TOMOYO?
TOMOYO Linux Overview
https://sourceforge.net/projects/tomoyo/files/docs/lca2009-takeda.pdf
TOMOYO Linux: pragmatic and manageable security for Linux
https://sourceforge.net/projects/tomoyo/files/docs/freedomhectaipei-tomoyo.pdf
TOMOYO Linux: A Practical Method to Understand and Protect Your Own Linux Box
https://sourceforge.net/projects/tomoyo/files/docs/PacSec2007-en-no-demo.pdf
What can TOMOYO do?
Deep inside TOMOYO Linux
https://sourceforge.net/projects/tomoyo/files/docs/lca2009-kumaneko.pdf
The role of "pathname based access control" in security.
https://sourceforge.net/projects/tomoyo/files/docs/lfj2008-bof.pdf
History of TOMOYO?
Realities of Mainlining
https://sourceforge.net/projects/tomoyo/files/docs/lfj2008.pdf
linux-7.2.3/Documentation/admin-guide/RAS/address-translation.rst
+24
View File
@@ -0,0 +1,24 @@
.. SPDX-License-Identifier: GPL-2.0
Address translation
===================
x86 AMD
-------
Zen-based AMD systems include a Data Fabric that manages the layout of
physical memory. Devices attached to the Fabric, like memory controllers,
I/O, etc., may not have a complete view of the system physical memory map.
These devices may provide a "normalized", i.e. device physical, address
when reporting memory errors. Normalized addresses must be translated to
a system physical address for the kernel to action on the memory.
AMD Address Translation Library (CONFIG_AMD_ATL) provides translation for
this case.
Glossary of acronyms used in address translation for Zen-based systems
* CCM = Cache Coherent Moderator
* COD = Cluster-on-Die
* COH_ST = Coherent Station
* DF = Data Fabric
linux-7.2.3/Documentation/admin-guide/RAS/error-decoding.rst
+21
View File
@@ -0,0 +1,21 @@
.. SPDX-License-Identifier: GPL-2.0
Error decoding
==============
x86
---
Error decoding on AMD systems should be done using the rasdaemon tool:
https://github.com/mchehab/rasdaemon/
While the daemon is running, it would automatically log and decode
errors. If not, one can still decode such errors by supplying the
hardware information from the error::
$ rasdaemon -p --status <STATUS> --ipid <IPID> --smca
Also, the user can pass particular family and model to decode the error
string::
$ rasdaemon -p --status <STATUS> --ipid <IPID> --smca --family <CPU Family> --model <CPU Model> --bank <BANK_NUM>
linux-7.2.3/Documentation/admin-guide/RAS/index.rst
+7
View File
@@ -0,0 +1,7 @@
.. SPDX-License-Identifier: GPL-2.0
.. toctree::
:maxdepth: 2
main
error-decoding
address-translation
linux-7.2.3/Documentation/admin-guide/RAS/main.rst
+1087
View File
File diff suppressed because it is too large Load Diff
linux-7.2.3/Documentation/admin-guide/README.rst
+360
View File
@@ -0,0 +1,360 @@
.. _readme:
Linux kernel release 6.x <http://kernel.org/>
=============================================
These are the release notes for Linux version 6. Read them carefully,
as they tell you what this is all about, explain how to install the
kernel, and what to do if something goes wrong.
What is Linux?
--------------
Linux is a clone of the operating system Unix, written from scratch by
Linus Torvalds with assistance from a loosely-knit team of hackers across
the Net. It aims towards POSIX and Single UNIX Specification compliance.
It has all the features you would expect in a modern fully-fledged Unix,
including true multitasking, virtual memory, shared libraries, demand
loading, shared copy-on-write executables, proper memory management,
and multistack networking including IPv4 and IPv6.
It is distributed under the GNU General Public License v2 - see the
accompanying COPYING file for more details.
On what hardware does it run?
-----------------------------
Although originally developed first for 32-bit x86-based PCs (386 or higher),
today Linux also runs on (at least) the Compaq Alpha AXP, Sun SPARC and
UltraSPARC, Motorola 68000, PowerPC, PowerPC64, ARM, Hitachi SuperH, Cell,
IBM S/390, MIPS, HP PA-RISC, Intel IA-64, DEC VAX, AMD x86-64 Xtensa, and
ARC architectures.
Linux is easily portable to most general-purpose 32- or 64-bit architectures
as long as they have a paged memory management unit (PMMU) and a port of the
GNU C compiler (gcc) (part of The GNU Compiler Collection, GCC). Linux has
also been ported to a number of architectures without a PMMU, although
functionality is then obviously somewhat limited.
Linux has also been ported to itself. You can now run the kernel as a
userspace application - this is called UserMode Linux (UML).
Documentation
-------------
- There is a lot of documentation available both in electronic form on
the Internet and in books, both Linux-specific and pertaining to
general UNIX questions. I'd recommend looking into the documentation
subdirectories on any Linux FTP site for the LDP (Linux Documentation
Project) books. This README is not meant to be documentation on the
system: there are much better sources available.
- There are various README files in the Documentation/ subdirectory:
these typically contain kernel-specific installation notes for some
drivers for example. Please read the
:ref:`Documentation/process/changes.rst <changes>` file, as it
contains information about the problems which may result from upgrading
your kernel.
Installing the kernel source
----------------------------
- If you install the full sources, put the kernel tarball in a
directory where you have permissions (e.g. your home directory) and
unpack it::
xz -cd linux-6.x.tar.xz | tar xvf -
Replace "X" with the version number of the latest kernel.
Do NOT use the /usr/src/linux area! This area has a (usually
incomplete) set of kernel headers that are used by the library header
files. They should match the library, and not get messed up by
whatever the kernel-du-jour happens to be.
- You can also upgrade between 6.x releases by patching. Patches are
distributed in the xz format. To install by patching, get all the
newer patch files, enter the top level directory of the kernel source
(linux-6.x) and execute::
xz -cd ../patch-6.x.xz | patch -p1
Replace "x" for all versions bigger than the version "x" of your current
source tree, **in_order**, and you should be ok. You may want to remove
the backup files (some-file-name~ or some-file-name.orig), and make sure
that there are no failed patches (some-file-name# or some-file-name.rej).
If there are, either you or I have made a mistake.
Unlike patches for the 6.x kernels, patches for the 6.x.y kernels
(also known as the -stable kernels) are not incremental but instead apply
directly to the base 6.x kernel. For example, if your base kernel is 6.0
and you want to apply the 6.0.3 patch, you must not first apply the 6.0.1
and 6.0.2 patches. Similarly, if you are running kernel version 6.0.2 and
want to jump to 6.0.3, you must first reverse the 6.0.2 patch (that is,
patch -R) **before** applying the 6.0.3 patch. You can read more on this in
:ref:`Documentation/process/applying-patches.rst <applying_patches>`.
Alternatively, the script patch-kernel can be used to automate this
process. It determines the current kernel version and applies any
patches found::
linux/scripts/patch-kernel linux
The first argument in the command above is the location of the
kernel source. Patches are applied from the current directory, but
an alternative directory can be specified as the second argument.
- Make sure you have no stale .o files and dependencies lying around::
cd linux
make mrproper
You should now have the sources correctly installed.
Software requirements
---------------------
Compiling and running the 6.x kernels requires up-to-date
versions of various software packages. Consult
:ref:`Documentation/process/changes.rst <changes>` for the minimum version numbers
required and how to get updates for these packages. Beware that using
excessively old versions of these packages can cause indirect
errors that are very difficult to track down, so don't assume that
you can just update packages when obvious problems arise during
build or operation.
Build directory for the kernel
------------------------------
When compiling the kernel, all output files will per default be
stored together with the kernel source code.
Using the option ``make O=output/dir`` allows you to specify an alternate
place for the output files (including .config).
Example::
kernel source code: /usr/src/linux-6.x
build directory: /home/name/build/kernel
To configure and build the kernel, use::
cd /usr/src/linux-6.x
make O=/home/name/build/kernel menuconfig
make O=/home/name/build/kernel
sudo make O=/home/name/build/kernel modules_install install
Please note: If the ``O=output/dir`` option is used, then it must be
used for all invocations of make.
Configuring the kernel
----------------------
Do not skip this step even if you are only upgrading one minor
version. New configuration options are added in each release, and
odd problems will turn up if the configuration files are not set up
as expected. If you want to carry your existing configuration to a
new version with minimal work, use ``make oldconfig``, which will
only ask you for the answers to new questions.
- Alternative configuration commands are::
"make config" Plain text interface.
"make menuconfig" Text based color menus, radiolists & dialogs.
"make nconfig" Enhanced text based color menus.
"make xconfig" Qt based configuration tool.
"make gconfig" GTK based configuration tool.
"make oldconfig" Default all questions based on the contents of
your existing ./.config file and asking about
new config symbols.
"make olddefconfig"
Like above, but sets new symbols to their default
values without prompting.
"make defconfig" Create a ./.config file by using the default
symbol values from either arch/$ARCH/configs/defconfig
or arch/$ARCH/configs/${PLATFORM}_defconfig,
depending on the architecture.
"make ${PLATFORM}_defconfig"
Create a ./.config file by using the default
symbol values from
arch/$ARCH/configs/${PLATFORM}_defconfig.
Use "make help" to get a list of all available
platforms of your architecture.
"make allyesconfig"
Create a ./.config file by setting symbol
values to 'y' as much as possible.
"make allmodconfig"
Create a ./.config file by setting symbol
values to 'm' as much as possible.
"make allnoconfig" Create a ./.config file by setting symbol
values to 'n' as much as possible.
"make randconfig" Create a ./.config file by setting symbol
values to random values.
"make localmodconfig" Create a config based on current config and
loaded modules (lsmod). Disables any module
option that is not needed for the loaded modules.
To create a localmodconfig for another machine,
store the lsmod of that machine into a file
and pass it in as a LSMOD parameter.
Also, you can preserve modules in certain folders
or kconfig files by specifying their paths in
parameter LMC_KEEP.
target$ lsmod > /tmp/mylsmod
target$ scp /tmp/mylsmod host:/tmp
host$ make LSMOD=/tmp/mylsmod \
LMC_KEEP="drivers/usb:drivers/gpu:fs" \
localmodconfig
The above also works when cross compiling.
"make localyesconfig" Similar to localmodconfig, except it will convert
all module options to built in (=y) options. You can
also preserve modules by LMC_KEEP.
"make kvm_guest.config" Enable additional options for kvm guest kernel
support.
"make xen.config" Enable additional options for xen dom0 guest kernel
support.
"make tinyconfig" Configure the tiniest possible kernel.
You can find more information on using the Linux kernel config tools
in Documentation/kbuild/kconfig.rst.
- NOTES on ``make config``:
- Having unnecessary drivers will make the kernel bigger, and can
under some circumstances lead to problems: probing for a
nonexistent controller card may confuse your other controllers.
- A kernel with math-emulation compiled in will still use the
coprocessor if one is present: the math emulation will just
never get used in that case. The kernel will be slightly larger,
but will work on different machines regardless of whether they
have a math coprocessor or not.
- The "kernel hacking" configuration details usually result in a
bigger or slower kernel (or both), and can even make the kernel
less stable by configuring some routines to actively try to
break bad code to find kernel problems (kmalloc()). Thus you
should probably answer 'n' to the questions for "development",
"experimental", or "debugging" features.
Compiling the kernel
--------------------
- Make sure you have at least gcc 8.1 available.
For more information, refer to :ref:`Documentation/process/changes.rst <changes>`.
- Do a ``make`` to create a compressed kernel image. It is also possible to do
``make install`` if you have lilo installed or if your distribution has an
install script recognised by the kernel's installer. Most popular
distributions will have a recognized install script. You may want to
check your distribution's setup first.
To do the actual install, you have to be root, but none of the normal
build should require that. Don't take the name of root in vain.
- If you configured any of the parts of the kernel as ``modules``, you
will also have to do ``make modules_install``.
- Verbose kernel compile/build output:
Normally, the kernel build system runs in a fairly quiet mode (but not
totally silent). However, sometimes you or other kernel developers need
to see compile, link, or other commands exactly as they are executed.
For this, use "verbose" build mode. This is done by passing
``V=1`` to the ``make`` command, e.g.::
make V=1 all
To have the build system also tell the reason for the rebuild of each
target, use ``V=2``. The default is ``V=0``.
- Keep a backup kernel handy in case something goes wrong. This is
especially true for the development releases, since each new release
contains new code which has not been debugged. Make sure you keep a
backup of the modules corresponding to that kernel, as well. If you
are installing a new kernel with the same version number as your
working kernel, make a backup of your modules directory before you
do a ``make modules_install``.
Alternatively, before compiling, use the kernel config option
"LOCALVERSION" to append a unique suffix to the regular kernel version.
LOCALVERSION can be set in the "General Setup" menu.
- In order to boot your new kernel, you'll need to copy the kernel
image (e.g. .../linux/arch/x86/boot/bzImage after compilation)
to the place where your regular bootable kernel is found.
- Booting a kernel directly from a storage device without the assistance
of a bootloader such as LILO or GRUB, is no longer supported in BIOS
(non-EFI systems). On UEFI/EFI systems, however, you can use EFISTUB
which allows the motherboard to boot directly to the kernel.
On modern workstations and desktops, it's generally recommended to use a
bootloader as difficulties can arise with multiple kernels and secure boot.
For more details on EFISTUB,
see "Documentation/admin-guide/efi-stub.rst".
- It's important to note that as of 2016 LILO (LInux LOader) is no longer in
active development, though as it was extremely popular, it often comes up
in documentation. Popular alternatives include GRUB2, rEFInd, Syslinux,
systemd-boot, or EFISTUB. For various reasons, it's not recommended to use
software that's no longer in active development.
- Chances are your distribution includes an install script and running
``make install`` will be all that's needed. Should that not be the case
you'll have to identify your bootloader and reference its documentation or
configure your EFI.
Legacy LILO Instructions
------------------------
- If you use LILO the kernel images are specified in the file /etc/lilo.conf.
The kernel image file is usually /vmlinuz, /boot/vmlinuz, /bzImage or
/boot/bzImage. To use the new kernel, save a copy of the old image and copy
the new image over the old one. Then, you MUST RERUN LILO to update the
loading map! If you don't, you won't be able to boot the new kernel image.
- Reinstalling LILO is usually a matter of running /sbin/lilo. You may wish
to edit /etc/lilo.conf to specify an entry for your old kernel image
(say, /vmlinux.old) in case the new one does not work. See the LILO docs
for more information.
- After reinstalling LILO, you should be all set. Shutdown the system,
reboot, and enjoy!
- If you ever need to change the default root device, video mode, etc. in the
kernel image, use your bootloader's boot options where appropriate. No need
to recompile the kernel to change these parameters.
- Reboot with the new kernel and enjoy.
If something goes wrong
-----------------------
If you have problems that seem to be due to kernel bugs, please follow the
instructions at 'Documentation/admin-guide/reporting-issues.rst'.
Hints on understanding kernel bug reports are in
'Documentation/admin-guide/bug-hunting.rst'. More on debugging the kernel
with gdb is in 'Documentation/process/debugging/gdb-kernel-debugging.rst' and
'Documentation/process/debugging/kgdb.rst'.
linux-7.2.3/Documentation/admin-guide/abi-obsolete-files.rst
+7
View File
@@ -0,0 +1,7 @@
.. SPDX-License-Identifier: GPL-2.0
Obsolete ABI Files
==================
.. kernel-abi:: obsolete
:no-symbols:
linux-7.2.3/Documentation/admin-guide/abi-obsolete.rst
+13
View File
@@ -0,0 +1,13 @@
.. SPDX-License-Identifier: GPL-2.0
ABI obsolete symbols
====================
Documents interfaces that are still remaining in the kernel, but are
marked to be removed at some later point in time.
The description of the interface will document the reason why it is
obsolete and when it can be expected to be removed.
.. kernel-abi:: obsolete
:no-files:
linux-7.2.3/Documentation/admin-guide/abi-removed-files.rst
+7
View File
@@ -0,0 +1,7 @@
.. SPDX-License-Identifier: GPL-2.0
Removed ABI Files
=================
.. kernel-abi:: removed
:no-symbols:
linux-7.2.3/Documentation/admin-guide/abi-removed.rst
+7
View File
@@ -0,0 +1,7 @@
.. SPDX-License-Identifier: GPL-2.0
ABI removed symbols
===================
.. kernel-abi:: removed
:no-files:
linux-7.2.3/Documentation/admin-guide/abi-stable-files.rst
+7
View File
@@ -0,0 +1,7 @@
.. SPDX-License-Identifier: GPL-2.0
Stable ABI Files
================
.. kernel-abi:: stable
:no-symbols:
linux-7.2.3/Documentation/admin-guide/abi-stable.rst
+16
View File
@@ -0,0 +1,16 @@
.. SPDX-License-Identifier: GPL-2.0
ABI stable symbols
==================
Documents the interfaces that the developer has defined to be stable.
Userspace programs are free to use these interfaces with no
restrictions, and backward compatibility for them will be guaranteed
for at least 2 years.
Most interfaces (like syscalls) are expected to never change and always
be available.
.. kernel-abi:: stable
:no-files:
linux-7.2.3/Documentation/admin-guide/abi-testing-files.rst
+7
View File
@@ -0,0 +1,7 @@
.. SPDX-License-Identifier: GPL-2.0
Testing ABI Files
=================
.. kernel-abi:: testing
:no-symbols:
linux-7.2.3/Documentation/admin-guide/abi-testing.rst
+22
View File
@@ -0,0 +1,22 @@
.. SPDX-License-Identifier: GPL-2.0
ABI testing symbols
===================
Documents interfaces that are felt to be stable,
as the main development of this interface has been completed.
The interface can be changed to add new features, but the
current interface will not break by doing this, unless grave
errors or security problems are found in them.
Userspace programs can start to rely on these interfaces, but they must
be aware of changes that can occur before these interfaces move to
be marked stable.
Programs that use these interfaces are strongly encouraged to add their
name to the description of these interfaces, so that the kernel
developers can easily notify them if any changes occur.
.. kernel-abi:: testing
:no-files:
linux-7.2.3/Documentation/admin-guide/abi.rst
+29
View File
@@ -0,0 +1,29 @@
.. SPDX-License-Identifier: GPL-2.0
=====================
Linux ABI description
=====================
.. kernel-abi:: README
ABI symbols
-----------
.. toctree::
:maxdepth: 2
abi-stable
abi-testing
abi-obsolete
abi-removed
ABI files
---------
.. toctree::
:maxdepth: 2
abi-stable-files
abi-testing-files
abi-obsolete-files
abi-removed-files
linux-7.2.3/Documentation/admin-guide/acpi/cppc_sysfs.rst
+78
View File
@@ -0,0 +1,78 @@
.. SPDX-License-Identifier: GPL-2.0
==================================================
Collaborative Processor Performance Control (CPPC)
==================================================
.. _cppc_sysfs:
CPPC
====
CPPC defined in the ACPI spec describes a mechanism for the OS to manage the
performance of a logical processor on a contiguous and abstract performance
scale. CPPC exposes a set of registers to describe abstract performance scale,
to request performance levels and to measure per-cpu delivered performance.
For more details on CPPC please refer to the ACPI specification at:
http://uefi.org/specifications
Some of the CPPC registers are exposed via sysfs under::
/sys/devices/system/cpu/cpuX/acpi_cppc/
for each cpu X::
$ ls -lR /sys/devices/system/cpu/cpu0/acpi_cppc/
/sys/devices/system/cpu/cpu0/acpi_cppc/:
total 0
-r--r--r-- 1 root root 65536 Mar 5 19:38 feedback_ctrs
-r--r--r-- 1 root root 65536 Mar 5 19:38 highest_perf
-r--r--r-- 1 root root 65536 Mar 5 19:38 lowest_freq
-r--r--r-- 1 root root 65536 Mar 5 19:38 lowest_nonlinear_perf
-r--r--r-- 1 root root 65536 Mar 5 19:38 lowest_perf
-r--r--r-- 1 root root 65536 Mar 5 19:38 nominal_freq
-r--r--r-- 1 root root 65536 Mar 5 19:38 nominal_perf
-r--r--r-- 1 root root 65536 Mar 5 19:38 reference_perf
-r--r--r-- 1 root root 65536 Mar 5 19:38 wraparound_time
* highest_perf : Highest performance of this processor (abstract scale).
* nominal_perf : Highest sustained performance of this processor
(abstract scale).
* lowest_nonlinear_perf : Lowest performance of this processor with nonlinear
power savings (abstract scale).
* lowest_perf : Lowest performance of this processor (abstract scale).
* lowest_freq : CPU frequency corresponding to lowest_perf (in MHz).
* nominal_freq : CPU frequency corresponding to nominal_perf (in MHz).
The above frequencies should only be used to report processor performance in
frequency instead of abstract scale. These values should not be used for any
functional decisions.
* feedback_ctrs : Includes both Reference and delivered performance counter.
Reference counter ticks up proportional to processor's reference performance.
Delivered counter ticks up proportional to processor's delivered performance.
* wraparound_time: Minimum time for the feedback counters to wraparound
(seconds).
* reference_perf : Performance level at which reference performance counter
accumulates (abstract scale).
Computing Average Delivered Performance
=======================================
Below describes the steps to compute the average performance delivered by
taking two different snapshots of feedback counters at time T1 and T2.
T1: Read feedback_ctrs as fbc_t1
Wait or run some workload
T2: Read feedback_ctrs as fbc_t2
::
delivered_counter_delta = fbc_t2[del] - fbc_t1[del]
reference_counter_delta = fbc_t2[ref] - fbc_t1[ref]
delivered_perf = (reference_perf x delivered_counter_delta) / reference_counter_delta
linux-7.2.3/Documentation/admin-guide/acpi/fan_performance_states.rst
+90
View File
@@ -0,0 +1,90 @@
.. SPDX-License-Identifier: GPL-2.0
===========================
ACPI Fan Performance States
===========================
When the optional _FPS object is present under an ACPI device representing a
fan (for example, PNP0C0B or INT3404), the ACPI fan driver creates additional
"state*" attributes in the sysfs directory of the ACPI device in question.
These attributes list properties of fan performance states.
For more information on _FPS refer to the ACPI specification at:
http://uefi.org/specifications
For instance, the contents of the INT3404 ACPI device sysfs directory
may look as follows::
$ ls -l /sys/bus/acpi/devices/INT3404:00/
total 0
...
-r--r--r-- 1 root root 4096 Dec 13 20:38 state0
-r--r--r-- 1 root root 4096 Dec 13 20:38 state1
-r--r--r-- 1 root root 4096 Dec 13 20:38 state10
-r--r--r-- 1 root root 4096 Dec 13 20:38 state11
-r--r--r-- 1 root root 4096 Dec 13 20:38 state2
-r--r--r-- 1 root root 4096 Dec 13 20:38 state3
-r--r--r-- 1 root root 4096 Dec 13 20:38 state4
-r--r--r-- 1 root root 4096 Dec 13 20:38 state5
-r--r--r-- 1 root root 4096 Dec 13 20:38 state6
-r--r--r-- 1 root root 4096 Dec 13 20:38 state7
-r--r--r-- 1 root root 4096 Dec 13 20:38 state8
-r--r--r-- 1 root root 4096 Dec 13 20:38 state9
-r--r--r-- 1 root root 4096 Dec 13 01:00 status
...
where each of the "state*" files represents one performance state of the fan
and contains a colon-separated list of 5 integer numbers (fields) with the
following interpretation::
control_percent:trip_point_index:speed_rpm:noise_level_mdb:power_mw
* ``control_percent``: The percent value to be used to set the fan speed to a
specific level using the _FSL object (0-100).
* ``trip_point_index``: The active cooling trip point number that corresponds
to this performance state (0-9).
* ``speed_rpm``: Speed of the fan in rotations per minute.
* ``noise_level_mdb``: Audible noise emitted by the fan in this state in
millidecibels.
* ``power_mw``: Power draw of the fan in this state in milliwatts.
For example::
$cat /sys/bus/acpi/devices/INT3404:00/state1
25:0:3200:12500:1250
When a given field is not populated or its value provided by the platform
firmware is invalid, the "not-defined" string is shown instead of the value.
ACPI Fan Fine Grain Control
=============================
When _FIF object specifies support for fine grain control, then fan speed
can be set from 0 to 100% with the recommended minimum "step size" via
_FSL object. User can adjust fan speed using thermal sysfs cooling device.
Here use can look at fan performance states for a reference speed (speed_rpm)
and set it by changing cooling device cur_state. If the fine grain control
is supported then user can also adjust to some other speeds which are
not defined in the performance states.
The support of fine grain control is presented via sysfs attribute
"fine_grain_control". If fine grain control is present, this attribute
will show "1" otherwise "0".
This sysfs attribute is presented in the same directory as performance states.
ACPI Fan Performance Feedback
=============================
The optional _FST object provides status information for the fan device.
This includes field to provide current fan speed in revolutions per minute
at which the fan is rotating.
This speed is presented in the sysfs using the attribute "fan_speed_rpm",
in the same directory as performance states.
linux-7.2.3/Documentation/admin-guide/acpi/index.rst
+14
View File
@@ -0,0 +1,14 @@
============
ACPI Support
============
Here we document in detail how to interact with various mechanisms in
the Linux ACPI support.
.. toctree::
:maxdepth: 1
initrd_table_override
ssdt-overlays
cppc_sysfs
fan_performance_states
linux-7.2.3/Documentation/admin-guide/acpi/initrd_table_override.rst
+115
View File
@@ -0,0 +1,115 @@
.. SPDX-License-Identifier: GPL-2.0
================================
Upgrading ACPI tables via initrd
================================
What is this about
==================
If the ACPI_TABLE_UPGRADE compile option is true, it is possible to
upgrade the ACPI execution environment that is defined by the ACPI tables
via upgrading the ACPI tables provided by the BIOS with an instrumented,
modified, more recent version one, or installing brand new ACPI tables.
When building initrd with kernel in a single image, option
ACPI_TABLE_OVERRIDE_VIA_BUILTIN_INITRD should also be true for this
feature to work.
For a full list of ACPI tables that can be upgraded/installed, take a look
at the char `*table_sigs[MAX_ACPI_SIGNATURE];` definition in
drivers/acpi/tables.c.
All ACPI tables iasl (Intel's ACPI compiler and disassembler) knows should
be overridable, except:
- ACPI_SIG_RSDP (has a signature of 6 bytes)
- ACPI_SIG_FACS (does not have an ordinary ACPI table header)
Both could get implemented as well.
What is this for
================
Complain to your platform/BIOS vendor if you find a bug which is so severe
that a workaround is not accepted in the Linux kernel. And this facility
allows you to upgrade the buggy tables before your platform/BIOS vendor
releases an upgraded BIOS binary.
This facility can be used by platform/BIOS vendors to provide a Linux
compatible environment without modifying the underlying platform firmware.
This facility also provides a powerful feature to easily debug and test
ACPI BIOS table compatibility with the Linux kernel by modifying old
platform provided ACPI tables or inserting new ACPI tables.
It can and should be enabled in any kernel because there is no functional
change with not instrumented initrds.
How does it work
================
::
# Extract the machine's ACPI tables:
cd /tmp
acpidump >acpidump
acpixtract -a acpidump
# Disassemble, modify and recompile them:
iasl -d *.dat
# For example add this statement into a _PRT (PCI Routing Table) function
# of the DSDT:
Store("HELLO WORLD", debug)
# And increase the OEM Revision. For example, before modification:
DefinitionBlock ("DSDT.aml", "DSDT", 2, "INTEL ", "TEMPLATE", 0x00000000)
# After modification:
DefinitionBlock ("DSDT.aml", "DSDT", 2, "INTEL ", "TEMPLATE", 0x00000001)
iasl -sa dsdt.dsl
# Add the raw ACPI tables to an uncompressed cpio archive.
# They must be put into a /kernel/firmware/acpi directory inside the cpio
# archive. Note that if the table put here matches a platform table
# (similar Table Signature, and similar OEMID, and similar OEM Table ID)
# with a more recent OEM Revision, the platform table will be upgraded by
# this table. If the table put here doesn't match a platform table
# (dissimilar Table Signature, or dissimilar OEMID, or dissimilar OEM Table
# ID), this table will be appended.
mkdir -p kernel/firmware/acpi
cp dsdt.aml kernel/firmware/acpi
# A maximum of "NR_ACPI_INITRD_TABLES (64)" tables are currently allowed
# (see osl.c):
iasl -sa facp.dsl
iasl -sa ssdt1.dsl
cp facp.aml kernel/firmware/acpi
cp ssdt1.aml kernel/firmware/acpi
# The uncompressed cpio archive must be the first. Other, typically
# compressed cpio archives, must be concatenated on top of the uncompressed
# one. Following command creates the uncompressed cpio archive and
# concatenates the original initrd on top:
find kernel | cpio -H newc --create > /boot/instrumented_initrd
cat /boot/initrd >>/boot/instrumented_initrd
# reboot with increased acpi debug level, e.g. boot params:
acpi.debug_level=0x2 acpi.debug_layer=0xFFFFFFFF
# and check your syslog:
[ 1.268089] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[ 1.272091] [ACPI Debug] String [0x0B] "HELLO WORLD"
iasl is able to disassemble and recompile quite a lot different,
also static ACPI tables.
Where to retrieve userspace tools
=================================
iasl and acpixtract are part of Intel's ACPICA project:
https://acpica.org/
and should be packaged by distributions (for example in the acpica package
on SUSE).
acpidump can be found in Len Browns pmtools:
ftp://kernel.org/pub/linux/kernel/people/lenb/acpi/utils/pmtools/acpidump
This tool is also part of the acpica package on SUSE.
Alternatively, used ACPI tables can be retrieved via sysfs in latest kernels:
/sys/firmware/acpi/tables
linux-7.2.3/Documentation/admin-guide/acpi/ssdt-overlays.rst
+181
View File
@@ -0,0 +1,181 @@
.. SPDX-License-Identifier: GPL-2.0
=============
SSDT Overlays
=============
In order to support ACPI open-ended hardware configurations (e.g. development
boards) we need a way to augment the ACPI configuration provided by the firmware
image. A common example is connecting sensors on I2C / SPI buses on development
boards.
Although this can be accomplished by creating a kernel platform driver or
recompiling the firmware image with updated ACPI tables, neither is practical:
the former proliferates board specific kernel code while the latter requires
access to firmware tools which are often not publicly available.
Because ACPI supports external references in AML code a more practical
way to augment firmware ACPI configuration is by dynamically loading
user defined SSDT tables that contain the board specific information.
For example, to enumerate a Bosch BMA222E accelerometer on the I2C bus of the
Minnowboard MAX development board exposed via the LSE connector [1], the
following ASL code can be used::
DefinitionBlock ("minnowmax.aml", "SSDT", 1, "Vendor", "Accel", 0x00000003)
{
External (\_SB.I2C6, DeviceObj)
Scope (\_SB.I2C6)
{
Device (STAC)
{
Name (_HID, "BMA222E")
Name (RBUF, ResourceTemplate ()
{
I2cSerialBus (0x0018, ControllerInitiated, 0x00061A80,
AddressingMode7Bit, "\\_SB.I2C6", 0x00,
ResourceConsumer, ,)
GpioInt (Edge, ActiveHigh, Exclusive, PullDown, 0x0000,
"\\_SB.GPO2", 0x00, ResourceConsumer, , )
{ // Pin list
0
}
})
Method (_CRS, 0, Serialized)
{
Return (RBUF)
}
}
}
}
which can then be compiled to AML binary format::
$ iasl minnowmax.asl
Intel ACPI Component Architecture
ASL Optimizing Compiler version 20140214-64 [Mar 29 2014]
Copyright (c) 2000 - 2014 Intel Corporation
ASL Input: minnomax.asl - 30 lines, 614 bytes, 7 keywords
AML Output: minnowmax.aml - 165 bytes, 6 named objects, 1 executable opcodes
[1] https://www.elinux.org/Minnowboard:MinnowMax#Low_Speed_Expansion_.28Top.29
The resulting AML code can then be loaded by the kernel using one of the methods
below.
Loading ACPI SSDTs from initrd
==============================
This option allows loading of user defined SSDTs from initrd and it is useful
when the system does not support EFI or when there is not enough EFI storage.
It works in a similar way with initrd based ACPI tables override/upgrade: SSDT
AML code must be placed in the first, uncompressed, initrd under the
"kernel/firmware/acpi" path. Multiple files can be used and this will translate
in loading multiple tables. Only SSDT and OEM tables are allowed. See
initrd_table_override.txt for more details.
Here is an example::
# Add the raw ACPI tables to an uncompressed cpio archive.
# They must be put into a /kernel/firmware/acpi directory inside the
# cpio archive.
# The uncompressed cpio archive must be the first.
# Other, typically compressed cpio archives, must be
# concatenated on top of the uncompressed one.
mkdir -p kernel/firmware/acpi
cp ssdt.aml kernel/firmware/acpi
# Create the uncompressed cpio archive and concatenate the original initrd
# on top:
find kernel | cpio -H newc --create > /boot/instrumented_initrd
cat /boot/initrd >>/boot/instrumented_initrd
Loading ACPI SSDTs from EFI variables
=====================================
This is the preferred method, when EFI is supported on the platform, because it
allows a persistent, OS independent way of storing the user defined SSDTs. There
is also work underway to implement EFI support for loading user defined SSDTs
and using this method will make it easier to convert to the EFI loading
mechanism when that will arrive. To enable it, the
CONFIG_EFI_CUSTOM_SSDT_OVERLAYS should be chosen to y.
In order to load SSDTs from an EFI variable the ``"efivar_ssdt=..."`` kernel
command line parameter can be used (the name has a limitation of 16 characters).
The argument for the option is the variable name to use. If there are multiple
variables with the same name but with different vendor GUIDs, all of them will
be loaded.
In order to store the AML code in an EFI variable the efivarfs filesystem can be
used. It is enabled and mounted by default in /sys/firmware/efi/efivars in all
recent distribution.
Creating a new file in /sys/firmware/efi/efivars will automatically create a new
EFI variable. Updating a file in /sys/firmware/efi/efivars will update the EFI
variable. Please note that the file name needs to be specially formatted as
"Name-GUID" and that the first 4 bytes in the file (little-endian format)
represent the attributes of the EFI variable (see EFI_VARIABLE_MASK in
include/linux/efi.h). Writing to the file must also be done with one write
operation.
For example, you can use the following bash script to create/update an EFI
variable with the content from a given file::
#!/bin/sh -e
while [ -n "$1" ]; do
case "$1" in
"-f") filename="$2"; shift;;
"-g") guid="$2"; shift;;
*) name="$1";;
esac
shift
done
usage()
{
echo "Syntax: ${0##*/} -f filename [ -g guid ] name"
exit 1
}
[ -n "$name" -a -f "$filename" ] || usage
EFIVARFS="/sys/firmware/efi/efivars"
[ -d "$EFIVARFS" ] || exit 2
if stat -tf $EFIVARFS | grep -q -v de5e81e4; then
mount -t efivarfs none $EFIVARFS
fi
# try to pick up an existing GUID
[ -n "$guid" ] || guid=$(find "$EFIVARFS" -name "$name-*" | head -n1 | cut -f2- -d-)
# use a randomly generated GUID
[ -n "$guid" ] || guid="$(cat /proc/sys/kernel/random/uuid)"
# efivarfs expects all of the data in one write
tmp=$(mktemp)
/bin/echo -ne "\007\000\000\000" | cat - $filename > $tmp
dd if=$tmp of="$EFIVARFS/$name-$guid" bs=$(stat -c %s $tmp)
rm $tmp
Loading ACPI SSDTs from configfs
================================
This option allows loading of user defined SSDTs from user space via the configfs
interface. The CONFIG_ACPI_CONFIGFS option must be select and configfs must be
mounted. In the following examples, we assume that configfs has been mounted in
/sys/kernel/config.
New tables can be loading by creating new directories in /sys/kernel/config/acpi/table
and writing the SSDT AML code in the aml attribute::
cd /sys/kernel/config/acpi/table
mkdir my_ssdt
cat ~/ssdt.aml > my_ssdt/aml
linux-7.2.3/Documentation/admin-guide/aoe/aoe.rst
+150
View File
@@ -0,0 +1,150 @@
Introduction
============
ATA over Ethernet is a network protocol that provides simple access to
block storage on the LAN.
http://support.coraid.com/documents/AoEr11.txt
The EtherDrive (R) HOWTO for 2.6 and 3.x kernels is found at ...
http://support.coraid.com/support/linux/EtherDrive-2.6-HOWTO.html
It has many tips and hints! Please see, especially, recommended
tunings for virtual memory:
http://support.coraid.com/support/linux/EtherDrive-2.6-HOWTO-5.html#ss5.19
The aoetools are userland programs that are designed to work with this
driver. The aoetools are on sourceforge.
http://aoetools.sourceforge.net/
The scripts in this Documentation/admin-guide/aoe directory are intended to
document the use of the driver and are not necessary if you install
the aoetools.
Creating Device Nodes
=====================
Users of udev should find the block device nodes created
automatically, but to create all the necessary device nodes, use the
udev configuration rules provided in udev.txt (in this directory).
There is a udev-install.sh script that shows how to install these
rules on your system.
There is also an autoload script that shows how to edit
/etc/modprobe.d/aoe.conf to ensure that the aoe module is loaded when
necessary. Preloading the aoe module is preferable to autoloading,
however, because AoE discovery takes a few seconds. It can be
confusing when an AoE device is not present the first time the a
command is run but appears a second later.
Using Device Nodes
==================
"cat /dev/etherd/err" blocks, waiting for error diagnostic output,
like any retransmitted packets.
"echo eth2 eth4 > /dev/etherd/interfaces" tells the aoe driver to
limit ATA over Ethernet traffic to eth2 and eth4. AoE traffic from
untrusted networks should be ignored as a matter of security. See
also the aoe_iflist driver option described below.
"echo > /dev/etherd/discover" tells the driver to find out what AoE
devices are available.
In the future these character devices may disappear and be replaced
by sysfs counterparts. Using the commands in aoetools insulates
users from these implementation details.
The block devices are named like this::
e{shelf}.{slot}
e{shelf}.{slot}p{part}
... so that "e0.2" is the third blade from the left (slot 2) in the
first shelf (shelf address zero). That's the whole disk. The first
partition on that disk would be "e0.2p1".
Using sysfs
===========
Each aoe block device in /sys/block has the extra attributes of
state, mac, and netif. The state attribute is "up" when the device
is ready for I/O and "down" if detected but unusable. The
"down,closewait" state shows that the device is still open and
cannot come up again until it has been closed.
The mac attribute is the ethernet address of the remote AoE device.
The netif attribute is the network interface on the localhost
through which we are communicating with the remote AoE device.
There is a script in this directory that formats this information in
a convenient way. Users with aoetools should use the aoe-stat
command::
root@makki root# sh Documentation/admin-guide/aoe/status.sh
e10.0 eth3 up
e10.1 eth3 up
e10.2 eth3 up
e10.3 eth3 up
e10.4 eth3 up
e10.5 eth3 up
e10.6 eth3 up
e10.7 eth3 up
e10.8 eth3 up
e10.9 eth3 up
e4.0 eth1 up
e4.1 eth1 up
e4.2 eth1 up
e4.3 eth1 up
e4.4 eth1 up
e4.5 eth1 up
e4.6 eth1 up
e4.7 eth1 up
e4.8 eth1 up
e4.9 eth1 up
Use /sys/module/aoe/parameters/aoe_iflist (or better, the driver
option discussed below) instead of /dev/etherd/interfaces to limit
AoE traffic to the network interfaces in the given
whitespace-separated list. Unlike the old character device, the
sysfs entry can be read from as well as written to.
It's helpful to trigger discovery after setting the list of allowed
interfaces. The aoetools package provides an aoe-discover script
for this purpose. You can also directly use the
/dev/etherd/discover special file described above.
Driver Options
==============
There is a boot option for the built-in aoe driver and a
corresponding module parameter, aoe_iflist. Without this option,
all network interfaces may be used for ATA over Ethernet. Here is a
usage example for the module parameter::
modprobe aoe_iflist="eth1 eth3"
The aoe_deadsecs module parameter determines the maximum number of
seconds that the driver will wait for an AoE device to provide a
response to an AoE command. After aoe_deadsecs seconds have
elapsed, the AoE device will be marked as "down". A value of zero
is supported for testing purposes and makes the aoe driver keep
trying AoE commands forever.
The aoe_maxout module parameter has a default of 128. This is the
maximum number of unresponded packets that will be sent to an AoE
target at one time.
The aoe_dyndevs module parameter defaults to 1, meaning that the
driver will assign a block device minor number to a discovered AoE
target based on the order of its discovery. With dynamic minor
device numbers in use, a greater range of AoE shelf and slot
addresses can be supported. Users with udev will never have to
think about minor numbers. Using aoe_dyndevs=0 allows device nodes
to be pre-created using a static minor-number scheme with the
aoe-mkshelf script in the aoetools.
linux-7.2.3/Documentation/admin-guide/aoe/autoload.sh
+17
View File
@@ -0,0 +1,17 @@
#!/bin/sh
# set aoe to autoload by installing the
# aliases in /etc/modprobe.d/
f=/etc/modprobe.d/aoe.conf
if test ! -r $f || test ! -w $f; then
echo "cannot configure $f for module autoloading" 1>&2
exit 1
fi
grep major-152 $f >/dev/null
if [ $? = 1 ]; then
echo alias block-major-152 aoe >> $f
echo alias char-major-152 aoe >> $f
fi
linux-7.2.3/Documentation/admin-guide/aoe/examples.rst
+23
View File
@@ -0,0 +1,23 @@
Example of udev rules
---------------------
.. include:: udev.txt
:literal:
Example of udev install rules script
------------------------------------
.. literalinclude:: udev-install.sh
:language: shell
Example script to get status
----------------------------
.. literalinclude:: status.sh
:language: shell
Example of AoE autoload script
------------------------------
.. literalinclude:: autoload.sh
:language: shell
linux-7.2.3/Documentation/admin-guide/aoe/index.rst
+10
View File
@@ -0,0 +1,10 @@
=======================
ATA over Ethernet (AoE)
=======================
.. toctree::
:maxdepth: 1
aoe
todo
examples
linux-7.2.3/Documentation/admin-guide/aoe/status.sh
+30
View File
@@ -0,0 +1,30 @@
#! /bin/sh
# collate and present sysfs information about AoE storage
#
# A more complete version of this script is aoe-stat, in the
# aoetools.
set -e
format="%8s\t%8s\t%8s\n"
me=`basename $0`
sysd=${sysfs_dir:-/sys}
# printf "$format" device mac netif state
# Suse 9.1 Pro doesn't put /sys in /etc/mtab
#test -z "`mount | grep sysfs`" && {
test ! -d "$sysd/block" && {
echo "$me Error: sysfs is not mounted" 1>&2
exit 1
}
for d in `ls -d $sysd/block/etherd* 2>/dev/null | grep -v p` end; do
# maybe ls comes up empty, so we use "end"
test $d = end && continue
dev=`echo "$d" | sed 's/.*!//'`
printf "$format" \
"$dev" \
"`cat \"$d/netif\"`" \
"`cat \"$d/state\"`"
done | sort
linux-7.2.3/Documentation/admin-guide/aoe/todo.rst
+17
View File
@@ -0,0 +1,17 @@
TODO
====
There is a potential for deadlock when allocating a struct sk_buff for
data that needs to be written out to aoe storage. If the data is
being written from a dirty page in order to free that page, and if
there are no other pages available, then deadlock may occur when a
free page is needed for the sk_buff allocation. This situation has
not been observed, but it would be nice to eliminate any potential for
deadlock under memory pressure.
Because ATA over Ethernet is not fragmented by the kernel's IP code,
the destructor member of the struct sk_buff is available to the aoe
driver. By using a mempool for allocating all but the first few
sk_buffs, and by registering a destructor, we should be able to
efficiently allocate sk_buffs without introducing any potential for
deadlock.
linux-7.2.3/Documentation/admin-guide/aoe/udev-install.sh
+33
View File
@@ -0,0 +1,33 @@
# install the aoe-specific udev rules from udev.txt into
# the system's udev configuration
#
me="`basename $0`"
# find udev.conf, often /etc/udev/udev.conf
# (or environment can specify where to find udev.conf)
#
if test -z "$conf"; then
if test -r /etc/udev/udev.conf; then
conf=/etc/udev/udev.conf
else
conf="`find /etc -type f -name udev.conf 2> /dev/null`"
if test -z "$conf" || test ! -r "$conf"; then
echo "$me Error: no udev.conf found" 1>&2
exit 1
fi
fi
fi
# find the directory where udev rules are stored, often
# /etc/udev/rules.d
#
rules_d="`sed -n '/^udev_rules=/{ s!udev_rules=!!; s!\"!!g; p; }' $conf`"
if test -z "$rules_d" ; then
rules_d=/etc/udev/rules.d
fi
if test ! -d "$rules_d"; then
echo "$me Error: cannot find udev rules directory" 1>&2
exit 1
fi
sh -xc "cp `dirname $0`/udev.txt $rules_d/60-aoe.rules"
linux-7.2.3/Documentation/admin-guide/aoe/udev.txt
+26
View File
@@ -0,0 +1,26 @@
# These rules tell udev what device nodes to create for aoe support.
# They may be installed along the following lines. Check the section
# 8 udev manpage to see whether your udev supports SUBSYSTEM, and
# whether it uses one or two equal signs for SUBSYSTEM and KERNEL.
#
# ecashin@makki ~$ su
# Password:
# bash# find /etc -type f -name udev.conf
# /etc/udev/udev.conf
# bash# grep udev_rules= /etc/udev/udev.conf
# udev_rules="/etc/udev/rules.d/"
# bash# ls /etc/udev/rules.d/
# 10-wacom.rules 50-udev.rules
# bash# cp /path/to/linux/Documentation/admin-guide/aoe/udev.txt \
# /etc/udev/rules.d/60-aoe.rules
#
# aoe char devices
SUBSYSTEM=="aoe", KERNEL=="discover", NAME="etherd/%k", GROUP="disk", MODE="0220"
SUBSYSTEM=="aoe", KERNEL=="err", NAME="etherd/%k", GROUP="disk", MODE="0440"
SUBSYSTEM=="aoe", KERNEL=="interfaces", NAME="etherd/%k", GROUP="disk", MODE="0220"
SUBSYSTEM=="aoe", KERNEL=="revalidate", NAME="etherd/%k", GROUP="disk", MODE="0220"
SUBSYSTEM=="aoe", KERNEL=="flush", NAME="etherd/%k", GROUP="disk", MODE="0220"
# aoe block devices
KERNEL=="etherd*", GROUP="disk"
linux-7.2.3/Documentation/admin-guide/auxdisplay/cfag12864b.rst
+98
View File
@@ -0,0 +1,98 @@
===================================
cfag12864b LCD Driver Documentation
===================================
:License: GPLv2
:Author & Maintainer: Miguel Ojeda <ojeda@kernel.org>
:Date: 2006-10-27
.. INDEX
1. DRIVER INFORMATION
2. DEVICE INFORMATION
3. WIRING
4. USERSPACE PROGRAMMING
1. Driver Information
---------------------
This driver supports a cfag12864b LCD.
2. Device Information
---------------------
:Manufacturer: Crystalfontz
:Device Name: Crystalfontz 12864b LCD Series
:Device Code: cfag12864b
:Webpage: http://www.crystalfontz.com
:Device Webpage: http://www.crystalfontz.com/products/12864b/
:Type: LCD (Liquid Crystal Display)
:Width: 128
:Height: 64
:Colors: 2 (B/N)
:Controller: ks0108
:Controllers: 2
:Pages: 8 each controller
:Addresses: 64 each page
:Data size: 1 byte each address
:Memory size: 2 * 8 * 64 * 1 = 1024 bytes = 1 Kbyte
3. Wiring
---------
The cfag12864b LCD Series don't have official wiring.
The common wiring is done to the parallel port as shown::
Parallel Port cfag12864b
Name Pin# Pin# Name
Strobe ( 1)------------------------------(17) Enable
Data 0 ( 2)------------------------------( 4) Data 0
Data 1 ( 3)------------------------------( 5) Data 1
Data 2 ( 4)------------------------------( 6) Data 2
Data 3 ( 5)------------------------------( 7) Data 3
Data 4 ( 6)------------------------------( 8) Data 4
Data 5 ( 7)------------------------------( 9) Data 5
Data 6 ( 8)------------------------------(10) Data 6
Data 7 ( 9)------------------------------(11) Data 7
(10) [+5v]---( 1) Vdd
(11) [GND]---( 2) Ground
(12) [+5v]---(14) Reset
(13) [GND]---(15) Read / Write
Line (14)------------------------------(13) Controller Select 1
(15)
Init (16)------------------------------(12) Controller Select 2
Select (17)------------------------------(16) Data / Instruction
Ground (18)---[GND] [+5v]---(19) LED +
Ground (19)---[GND]
Ground (20)---[GND] E A Values:
Ground (21)---[GND] [GND]---[P1]---(18) Vee - R = Resistor = 22 ohm
Ground (22)---[GND] | - P1 = Preset = 10 Kohm
Ground (23)---[GND] ---- S ------( 3) V0 - P2 = Preset = 1 Kohm
Ground (24)---[GND] | |
Ground (25)---[GND] [GND]---[P2]---[R]---(20) LED -
4. Userspace Programming
------------------------
The cfag12864bfb describes a framebuffer device (/dev/fbX).
It has a size of 1024 bytes = 1 Kbyte.
Each bit represents one pixel. If the bit is high, the pixel will
turn on. If the pixel is low, the pixel will turn off.
You can use the framebuffer as a file: fopen, fwrite, fclose...
Although the LCD won't get updated until the next refresh time arrives.
Also, you can mmap the framebuffer: open & mmap, munmap & close...
which is the best option for most uses.
Check samples/auxdisplay/cfag12864b-example.c
for a real working userspace complete program with usage examples.
linux-7.2.3/Documentation/admin-guide/auxdisplay/index.rst
+9
View File
@@ -0,0 +1,9 @@
=========================
Auxiliary Display Support
=========================
.. toctree::
:maxdepth: 1
ks0108.rst
cfag12864b.rst
linux-7.2.3/Documentation/admin-guide/auxdisplay/ks0108.rst
+50
View File
@@ -0,0 +1,50 @@
==========================================
ks0108 LCD Controller Driver Documentation
==========================================
:License: GPLv2
:Author & Maintainer: Miguel Ojeda <ojeda@kernel.org>
:Date: 2006-10-27
.. INDEX
1. DRIVER INFORMATION
2. DEVICE INFORMATION
3. WIRING
1. Driver Information
---------------------
This driver supports the ks0108 LCD controller.
2. Device Information
---------------------
:Manufacturer: Samsung
:Device Name: KS0108 LCD Controller
:Device Code: ks0108
:Webpage: -
:Device Webpage: -
:Type: LCD Controller (Liquid Crystal Display Controller)
:Width: 64
:Height: 64
:Colors: 2 (B/N)
:Pages: 8
:Addresses: 64 each page
:Data size: 1 byte each address
:Memory size: 8 * 64 * 1 = 512 bytes
3. Wiring
---------
The driver supports data parallel port wiring.
If you aren't building LCD related hardware, you should check
your LCD specific wiring information in the same folder.
For example, check Documentation/admin-guide/auxdisplay/cfag12864b.rst
linux-7.2.3/Documentation/admin-guide/bcache.rst
+644
View File
@@ -0,0 +1,644 @@
============================
A block layer cache (bcache)
============================
Say you've got a big slow raid 6, and an ssd or three. Wouldn't it be
nice if you could use them as cache... Hence bcache.
The bcache wiki can be found at:
https://bcache.evilpiepirate.org
This is the git repository of bcache-tools:
https://git.kernel.org/pub/scm/linux/kernel/git/colyli/bcache-tools.git/
The latest bcache kernel code can be found from mainline Linux kernel:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
It's designed around the performance characteristics of SSDs - it only allocates
in erase block sized buckets, and it uses a hybrid btree/log to track cached
extents (which can be anywhere from a single sector to the bucket size). It's
designed to avoid random writes at all costs.
Both writethrough and writeback caching are supported. Writeback defaults to
off, but can be switched on and off arbitrarily at runtime. Bcache goes to
great lengths to protect your data - it reliably handles unclean shutdown. (It
doesn't even have a notion of a clean shutdown; bcache simply doesn't return
writes as completed until they're on stable storage).
Writeback caching can use most of the cache for buffering writes - writing
dirty data to the backing device is always done sequentially, scanning from the
start to the end of the index.
Since random IO is what SSDs excel at, there generally won't be much benefit
to caching large sequential IO. Bcache detects sequential IO and skips it;
it also keeps a rolling average of the IO sizes per task, and as long as the
average is above the cutoff it will skip all IO from that task - instead of
caching the first 512k after every seek. Backups and large file copies should
thus entirely bypass the cache.
In the event of a data IO error on the flash it will try to recover by reading
from disk or invalidating cache entries. For unrecoverable errors (meta data
or dirty data), caching is automatically disabled; if dirty data was present
in the cache it first disables writeback caching and waits for all dirty data
to be flushed.
Getting started:
You'll need bcache util from the bcache-tools repository. Both the cache device
and backing device must be formatted before use::
bcache make -B /dev/sdb
bcache make -C /dev/sdc
`bcache make` has the ability to format multiple devices at the same time - if
you format your backing devices and cache device at the same time, you won't
have to manually attach::
bcache make -B /dev/sda /dev/sdb -C /dev/sdc
If your bcache-tools is not updated to latest version and does not have the
unified `bcache` utility, you may use the legacy `make-bcache` utility to format
bcache device with same -B and -C parameters.
bcache-tools now ships udev rules, and bcache devices are known to the kernel
immediately. Without udev, you can manually register devices like this::
echo /dev/sdb > /sys/fs/bcache/register
echo /dev/sdc > /sys/fs/bcache/register
Registering the backing device makes the bcache device show up in /dev; you can
now format it and use it as normal. But the first time using a new bcache
device, it'll be running in passthrough mode until you attach it to a cache.
If you are thinking about using bcache later, it is recommended to setup all your
slow devices as bcache backing devices without a cache, and you can choose to add
a caching device later.
See 'ATTACHING' section below.
The devices show up as::
/dev/bcache<N>
As well as (with udev)::
/dev/bcache/by-uuid/<uuid>
/dev/bcache/by-label/<label>
To get started::
mkfs.ext4 /dev/bcache0
mount /dev/bcache0 /mnt
You can control bcache devices through sysfs at /sys/block/bcache<N>/bcache .
You can also control them through /sys/fs//bcache/<cset-uuid>/ .
Cache devices are managed as sets; multiple caches per set isn't supported yet
but will allow for mirroring of metadata and dirty data in the future. Your new
cache set shows up as /sys/fs/bcache/<UUID>
Attaching
---------
After your cache device and backing device are registered, the backing device
must be attached to your cache set to enable caching. Attaching a backing
device to a cache set is done thusly, with the UUID of the cache set in
/sys/fs/bcache::
echo <CSET-UUID> > /sys/block/bcache0/bcache/attach
This only has to be done once. The next time you reboot, just reregister all
your bcache devices. If a backing device has data in a cache somewhere, the
/dev/bcache<N> device won't be created until the cache shows up - particularly
important if you have writeback caching turned on.
If you're booting up and your cache device is gone and never coming back, you
can force run the backing device::
echo 1 > /sys/block/sdb/bcache/running
(You need to use /sys/block/sdb (or whatever your backing device is called), not
/sys/block/bcache0, because bcache0 doesn't exist yet. If you're using a
partition, the bcache directory would be at /sys/block/sdb/sdb2/bcache)
The backing device will still use that cache set if it shows up in the future,
but all the cached data will be invalidated. If there was dirty data in the
cache, don't expect the filesystem to be recoverable - you will have massive
filesystem corruption, though ext4's fsck does work miracles.
Error Handling
--------------
Bcache tries to transparently handle IO errors to/from the cache device without
affecting normal operation; if it sees too many errors (the threshold is
configurable, and defaults to 0) it shuts down the cache device and switches all
the backing devices to passthrough mode.
- For reads from the cache, if they error we just retry the read from the
backing device.
- For writethrough writes, if the write to the cache errors we just switch to
invalidating the data at that lba in the cache (i.e. the same thing we do for
a write that bypasses the cache)
- For writeback writes, we currently pass that error back up to the
filesystem/userspace. This could be improved - we could retry it as a write
that skips the cache so we don't have to error the write.
- When we detach, we first try to flush any dirty data (if we were running in
writeback mode). It currently doesn't do anything intelligent if it fails to
read some of the dirty data, though.
Howto/cookbook
--------------
A) Starting a bcache with a missing caching device
If registering the backing device doesn't help, it's already there, you just need
to force it to run without the cache::
host:~# echo /dev/sdb1 > /sys/fs/bcache/register
[ 119.844831] bcache: register_bcache() error opening /dev/sdb1: device already registered
Next, you try to register your caching device if it's present. However
if it's absent, or registration fails for some reason, you can still
start your bcache without its cache, like so::
host:/sys/block/sdb/sdb1/bcache# echo 1 > running
Note that this may cause data loss if you were running in writeback mode.
B) Bcache does not find its cache::
host:/sys/block/md5/bcache# echo 0226553a-37cf-41d5-b3ce-8b1e944543a8 > attach
[ 1933.455082] bcache: bch_cached_dev_attach() Couldn't find uuid for md5 in set
[ 1933.478179] bcache: __cached_dev_store() Can't attach 0226553a-37cf-41d5-b3ce-8b1e944543a8
[ 1933.478179] : cache set not found
In this case, the caching device was simply not registered at boot
or disappeared and came back, and needs to be (re-)registered::
host:/sys/block/md5/bcache# echo /dev/sdh2 > /sys/fs/bcache/register
C) Corrupt bcache crashes the kernel at device registration time:
This should never happen. If it does happen, then you have found a bug!
Please report it to the bcache development list: linux-bcache@vger.kernel.org
Be sure to provide as much information that you can including kernel dmesg
output if available so that we may assist.
D) Recovering data without bcache:
If bcache is not available in the kernel, a filesystem on the backing
device is still available at an 8KiB offset. So either via a loopdev
of the backing device created with --offset 8K, or any value defined by
--data-offset when you originally formatted bcache with `bcache make`.
For example::
losetup -o 8192 /dev/loop0 /dev/your_bcache_backing_dev
This should present your unmodified backing device data in /dev/loop0
If your cache is in writethrough mode, then you can safely discard the
cache device without losing data.
E) Wiping a cache device
::
host:~# wipefs -a /dev/sdh2
16 bytes were erased at offset 0x1018 (bcache)
they were: c6 85 73 f6 4e 1a 45 ca 82 65 f5 7f 48 ba 6d 81
After you boot back with bcache enabled, you recreate the cache and attach it::
host:~# bcache make -C /dev/sdh2
UUID: 7be7e175-8f4c-4f99-94b2-9c904d227045
Set UUID: 5bc072a8-ab17-446d-9744-e247949913c1
version: 0
nbuckets: 106874
block_size: 1
bucket_size: 1024
nr_in_set: 1
nr_this_dev: 0
first_bucket: 1
[ 650.511912] bcache: run_cache_set() invalidating existing data
[ 650.549228] bcache: register_cache() registered cache device sdh2
start backing device with missing cache::
host:/sys/block/md5/bcache# echo 1 > running
attach new cache::
host:/sys/block/md5/bcache# echo 5bc072a8-ab17-446d-9744-e247949913c1 > attach
[ 865.276616] bcache: bch_cached_dev_attach() Caching md5 as bcache0 on set 5bc072a8-ab17-446d-9744-e247949913c1
F) Remove or replace a caching device::
host:/sys/block/sda/sda7/bcache# echo 1 > detach
[ 695.872542] bcache: cached_dev_detach_finish() Caching disabled for sda7
host:~# wipefs -a /dev/nvme0n1p4
wipefs: error: /dev/nvme0n1p4: probing initialization failed: Device or resource busy
Ooops, it's disabled, but not unregistered, so it's still protected
We need to go and unregister it::
host:/sys/fs/bcache/b7ba27a1-2398-4649-8ae3-0959f57ba128# ls -l cache0
lrwxrwxrwx 1 root root 0 Feb 25 18:33 cache0 -> ../../../devices/pci0000:00/0000:00:1d.0/0000:70:00.0/nvme/nvme0/nvme0n1/nvme0n1p4/bcache/
host:/sys/fs/bcache/b7ba27a1-2398-4649-8ae3-0959f57ba128# echo 1 > stop
kernel: [ 917.041908] bcache: cache_set_free() Cache set b7ba27a1-2398-4649-8ae3-0959f57ba128 unregistered
Now we can wipe it::
host:~# wipefs -a /dev/nvme0n1p4
/dev/nvme0n1p4: 16 bytes were erased at offset 0x00001018 (bcache): c6 85 73 f6 4e 1a 45 ca 82 65 f5 7f 48 ba 6d 81
G) dm-crypt and bcache
First setup bcache unencrypted and then install dmcrypt on top of
/dev/bcache<N> This will work faster than if you dmcrypt both the backing
and caching devices and then install bcache on top. [benchmarks?]
H) Stop/free a registered bcache to wipe and/or recreate it
Suppose that you need to free up all bcache references so that you can
fdisk run and re-register a changed partition table, which won't work
if there are any active backing or caching devices left on it:
1) Is it present in /dev/bcache* ? (there are times where it won't be)
If so, it's easy::
host:/sys/block/bcache0/bcache# echo 1 > stop
2) But if your backing device is gone, this won't work::
host:/sys/block/bcache0# cd bcache
bash: cd: bcache: No such file or directory
In this case, you may have to unregister the dmcrypt block device that
references this bcache to free it up::
host:~# dmsetup remove oldds1
bcache: bcache_device_free() bcache0 stopped
bcache: cache_set_free() Cache set 5bc072a8-ab17-446d-9744-e247949913c1 unregistered
This causes the backing bcache to be removed from /sys/fs/bcache and
then it can be reused. This would be true of any block device stacking
where bcache is a lower device.
3) In other cases, you can also look in /sys/fs/bcache/::
host:/sys/fs/bcache# ls -l */{cache?,bdev?}
lrwxrwxrwx 1 root root 0 Mar 5 09:39 0226553a-37cf-41d5-b3ce-8b1e944543a8/bdev1 -> ../../../devices/virtual/block/dm-1/bcache/
lrwxrwxrwx 1 root root 0 Mar 5 09:39 0226553a-37cf-41d5-b3ce-8b1e944543a8/cache0 -> ../../../devices/virtual/block/dm-4/bcache/
lrwxrwxrwx 1 root root 0 Mar 5 09:39 5bc072a8-ab17-446d-9744-e247949913c1/cache0 -> ../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdl/sdl2/bcache/
The device names will show which UUID is relevant, cd in that directory
and stop the cache::
host:/sys/fs/bcache/5bc072a8-ab17-446d-9744-e247949913c1# echo 1 > stop
This will free up bcache references and let you reuse the partition for
other purposes.
Troubleshooting performance
---------------------------
Bcache has a bunch of config options and tunables. The defaults are intended to
be reasonable for typical desktop and server workloads, but they're not what you
want for getting the best possible numbers when benchmarking.
- Backing device alignment
The default metadata size in bcache is 8k. If your backing device is
RAID based, then be sure to align this by a multiple of your stride
width using `bcache make --data-offset`. If you intend to expand your
disk array in the future, then multiply a series of primes by your
raid stripe size to get the disk multiples that you would like.
For example: If you have a 64k stripe size, then the following offset
would provide alignment for many common RAID5 data spindle counts::
64k * 2*2*2*3*3*5*7 bytes = 161280k
That space is wasted, but for only 157.5MB you can grow your RAID 5
volume to the following data-spindle counts without re-aligning::
3,4,5,6,7,8,9,10,12,14,15,18,20,21 ...
- Bad write performance
If write performance is not what you expected, you probably wanted to be
running in writeback mode, which isn't the default (not due to a lack of
maturity, but simply because in writeback mode you'll lose data if something
happens to your SSD)::
# echo writeback > /sys/block/bcache0/bcache/cache_mode
- Bad performance, or traffic not going to the SSD that you'd expect
By default, bcache doesn't cache everything. It tries to skip sequential IO -
because you really want to be caching the random IO, and if you copy a 10
gigabyte file you probably don't want that pushing 10 gigabytes of randomly
accessed data out of your cache.
But if you want to benchmark reads from cache, and you start out with fio
writing an 8 gigabyte test file - so you want to disable that::
# echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
To set it back to the default (4 mb), do::
# echo 4M > /sys/block/bcache0/bcache/sequential_cutoff
- Traffic's still going to the spindle/still getting cache misses
In the real world, SSDs don't always keep up with disks - particularly with
slower SSDs, many disks being cached by one SSD, or mostly sequential IO. So
you want to avoid being bottlenecked by the SSD and having it slow everything
down.
To avoid that bcache tracks latency to the cache device, and gradually
throttles traffic if the latency exceeds a threshold (it does this by
cranking down the sequential bypass).
You can disable this if you need to by setting the thresholds to 0::
# echo 0 > /sys/fs/bcache/<cache set>/congested_read_threshold_us
# echo 0 > /sys/fs/bcache/<cache set>/congested_write_threshold_us
The default is 2000 us (2 milliseconds) for reads, and 20000 for writes.
- Still getting cache misses, of the same data
One last issue that sometimes trips people up is actually an old bug, due to
the way cache coherency is handled for cache misses. If a btree node is full,
a cache miss won't be able to insert a key for the new data and the data
won't be written to the cache.
In practice this isn't an issue because as soon as a write comes along it'll
cause the btree node to be split, and you need almost no write traffic for
this to not show up enough to be noticeable (especially since bcache's btree
nodes are huge and index large regions of the device). But when you're
benchmarking, if you're trying to warm the cache by reading a bunch of data
and there's no other traffic - that can be a problem.
Solution: warm the cache by doing writes, or use the testing branch (there's
a fix for the issue there).
Sysfs - backing device
----------------------
Available at /sys/block/<bdev>/bcache, /sys/block/bcache*/bcache and
(if attached) /sys/fs/bcache/<cset-uuid>/bdev*
attach
Echo the UUID of a cache set to this file to enable caching.
cache_mode
Can be one of either writethrough, writeback, writearound or none.
clear_stats
Writing to this file resets the running total stats (not the day/hour/5 minute
decaying versions).
detach
Write to this file to detach from a cache set. If there is dirty data in the
cache, it will be flushed first.
dirty_data
Amount of dirty data for this backing device in the cache. Continuously
updated unlike the cache set's version, but may be slightly off.
label
Name of underlying device.
readahead
Size of readahead that should be performed. Defaults to 0. If set to e.g.
1M, it will round cache miss reads up to that size, but without overlapping
existing cache entries.
running
1 if bcache is running (i.e. whether the /dev/bcache device exists, whether
it's in passthrough mode or caching).
sequential_cutoff
A sequential IO will bypass the cache once it passes this threshold; the
most recent 128 IOs are tracked so sequential IO can be detected even when
it isn't all done at once.
sequential_merge
If non zero, bcache keeps a list of the last 128 requests submitted to compare
against all new requests to determine which new requests are sequential
continuations of previous requests for the purpose of determining sequential
cutoff. This is necessary if the sequential cutoff value is greater than the
maximum acceptable sequential size for any single request.
state
The backing device can be in one of four different states:
no cache: Has never been attached to a cache set.
clean: Part of a cache set, and there is no cached dirty data.
dirty: Part of a cache set, and there is cached dirty data.
inconsistent: The backing device was forcibly run by the user when there was
dirty data cached but the cache set was unavailable; whatever data was on the
backing device has likely been corrupted.
stop
Write to this file to shut down the bcache device and close the backing
device.
writeback_delay
When dirty data is written to the cache and it previously did not contain
any, waits some number of seconds before initiating writeback. Defaults to
30.
writeback_percent
If nonzero, bcache tries to keep around this percentage of the cache dirty by
throttling background writeback and using a PD controller to smoothly adjust
the rate.
writeback_rate
Rate in sectors per second - if writeback_percent is nonzero, background
writeback is throttled to this rate. Continuously adjusted by bcache but may
also be set by the user.
writeback_running
If off, writeback of dirty data will not take place at all. Dirty data will
still be added to the cache until it is mostly full; only meant for
benchmarking. Defaults to on.
Sysfs - backing device stats
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are directories with these numbers for a running total, as well as
versions that decay over the past day, hour and 5 minutes; they're also
aggregated in the cache set directory as well.
bypassed
Amount of IO (both reads and writes) that has bypassed the cache
cache_hits, cache_misses, cache_hit_ratio
Hits and misses are counted per individual IO as bcache sees them; a
partial hit is counted as a miss.
cache_bypass_hits, cache_bypass_misses
Hits and misses for IO that is intended to skip the cache are still counted,
but broken out here.
cache_miss_collisions
Counts instances where data was going to be inserted into the cache from a
cache miss, but raced with a write and data was already present (usually 0
since the synchronization for cache misses was rewritten)
Sysfs - cache set
~~~~~~~~~~~~~~~~~
Available at /sys/fs/bcache/<cset-uuid>
average_key_size
Average data per key in the btree.
bdev<0..n>
Symlink to each of the attached backing devices.
block_size
Block size of the cache devices.
btree_cache_size
Amount of memory currently used by the btree cache
bucket_size
Size of buckets
cache<0..n>
Symlink to each of the cache devices comprising this cache set.
cache_available_percent
Percentage of cache device which doesn't contain dirty data, and could
potentially be used for writeback. This doesn't mean this space isn't used
for clean cached data; the unused statistic (in priority_stats) is typically
much lower.
clear_stats
Clears the statistics associated with this cache
dirty_data
Amount of dirty data is in the cache (updated when garbage collection runs).
flash_vol_create
Echoing a size to this file (in human readable units, k/M/G) creates a thinly
provisioned volume backed by the cache set.
io_error_halflife, io_error_limit
These determines how many errors we accept before disabling the cache.
Each error is decayed by the half life (in # ios). If the decaying count
reaches io_error_limit dirty data is written out and the cache is disabled.
journal_delay_ms
Journal writes will delay for up to this many milliseconds, unless a cache
flush happens sooner. Defaults to 100.
root_usage_percent
Percentage of the root btree node in use. If this gets too high the node
will split, increasing the tree depth.
stop
Write to this file to shut down the cache set - waits until all attached
backing devices have been shut down.
tree_depth
Depth of the btree (A single node btree has depth 0).
unregister
Detaches all backing devices and closes the cache devices; if dirty data is
present it will disable writeback caching and wait for it to be flushed.
Sysfs - cache set internal
~~~~~~~~~~~~~~~~~~~~~~~~~~
This directory also exposes timings for a number of internal operations, with
separate files for average duration, average frequency, last occurrence and max
duration: garbage collection, btree read, btree node sorts and btree splits.
active_journal_entries
Number of journal entries that are newer than the index.
btree_nodes
Total nodes in the btree.
btree_used_percent
Average fraction of btree in use.
bset_tree_stats
Statistics about the auxiliary search trees
btree_cache_max_chain
Longest chain in the btree node cache's hash table
cache_read_races
Counts instances where while data was being read from the cache, the bucket
was reused and invalidated - i.e. where the pointer was stale after the read
completed. When this occurs the data is reread from the backing device.
trigger_gc
Writing to this file forces garbage collection to run.
Sysfs - Cache device
~~~~~~~~~~~~~~~~~~~~
Available at /sys/block/<cdev>/bcache
block_size
Minimum granularity of writes - should match hardware sector size.
btree_written
Sum of all btree writes, in (kilo/mega/giga) bytes
bucket_size
Size of buckets
cache_replacement_policy
One of either lru, fifo or random.
freelist_percent
Size of the freelist as a percentage of nbuckets. Can be written to
increase the number of buckets kept on the freelist, which lets you
artificially reduce the size of the cache at runtime. Mostly for testing
purposes (i.e. testing how different size caches affect your hit rate).
io_errors
Number of errors that have occurred, decayed by io_error_halflife.
metadata_written
Sum of all non data writes (btree writes and all other metadata).
nbuckets
Total buckets in this cache
priority_stats
Statistics about how recently data in the cache has been accessed.
This can reveal your working set size. Unused is the percentage of
the cache that doesn't contain any data. Metadata is bcache's
metadata overhead. Average is the average priority of cache buckets.
Next is a list of quantiles with the priority threshold of each.
written
Sum of all data that has been written to the cache; comparison with
btree_written gives the amount of write inflation in bcache.
linux-7.2.3/Documentation/admin-guide/binderfs.rst
+87
View File
@@ -0,0 +1,87 @@
.. SPDX-License-Identifier: GPL-2.0
The Android binderfs Filesystem
===============================
Android binderfs is a filesystem for the Android binder IPC mechanism. It
allows to dynamically add and remove binder devices at runtime. Binder devices
located in a new binderfs instance are independent of binder devices located in
other binderfs instances. Mounting a new binderfs instance makes it possible
to get a set of private binder devices.
Mounting binderfs
-----------------
Android binderfs can be mounted with::
mkdir /dev/binderfs
mount -t binder binder /dev/binderfs
at which point a new instance of binderfs will show up at ``/dev/binderfs``.
In a fresh instance of binderfs no binder devices will be present. There will
only be a ``binder-control`` device which serves as the request handler for
binderfs. Mounting another binderfs instance at a different location will
create a new and separate instance from all other binderfs mounts. This is
identical to the behavior of e.g. ``devpts`` and ``tmpfs``. The Android
binderfs filesystem can be mounted in user namespaces.
Options
-------
max
binderfs instances can be mounted with a limit on the number of binder
devices that can be allocated. The ``max=<count>`` mount option serves as
a per-instance limit. If ``max=<count>`` is set then only ``<count>`` number
of binder devices can be allocated in this binderfs instance.
stats
Using ``stats=global`` enables global binder statistics.
``stats=global`` is only available for a binderfs instance mounted in the
initial user namespace. An attempt to use the option to mount a binderfs
instance in another user namespace will return a permission error.
Allocating binder Devices
-------------------------
.. _ioctl: http://man7.org/linux/man-pages/man2/ioctl.2.html
To allocate a new binder device in a binderfs instance a request needs to be
sent through the ``binder-control`` device node. A request is sent in the form
of an `ioctl() <ioctl_>`_.
What a program needs to do is to open the ``binder-control`` device node and
send a ``BINDER_CTL_ADD`` request to the kernel. Users of binderfs need to
tell the kernel which name the new binder device should get. By default a name
can only contain up to ``BINDERFS_MAX_NAME`` chars including the terminating
zero byte.
Once the request is made via an `ioctl() <ioctl_>`_ passing a ``struct
binder_device`` with the name to the kernel it will allocate a new binder
device and return the major and minor number of the new device in the struct
(This is necessary because binderfs allocates a major device number
dynamically.). After the `ioctl() <ioctl_>`_ returns there will be a new
binder device located under /dev/binderfs with the chosen name.
Deleting binder Devices
-----------------------
.. _unlink: http://man7.org/linux/man-pages/man2/unlink.2.html
.. _rm: http://man7.org/linux/man-pages/man1/rm.1.html
Binderfs binder devices can be deleted via `unlink() <unlink_>`_. This means
that the `rm() <rm_>`_ tool can be used to delete them. Note that the
``binder-control`` device cannot be deleted since this would make the binderfs
instance unusable. The ``binder-control`` device will be deleted when the
binderfs instance is unmounted and all references to it have been dropped.
Binder features
---------------
Assuming an instance of binderfs has been mounted at ``/dev/binderfs``, the
features supported by the binder driver can be located under
``/dev/binderfs/features/``. The presence of individual files can be tested
to determine whether a particular feature is supported by the driver.
Example::
cat /dev/binderfs/features/oneway_spam_detection
1
linux-7.2.3/Documentation/admin-guide/binfmt-misc.rst
+151
View File
@@ -0,0 +1,151 @@
Kernel Support for miscellaneous Binary Formats (binfmt_misc)
=============================================================
This Kernel feature allows you to invoke almost (for restrictions see below)
every program by simply typing its name in the shell.
This includes for example compiled Java(TM), Python or Emacs programs.
To achieve this you must tell binfmt_misc which interpreter has to be invoked
with which binary. Binfmt_misc recognises the binary-type by matching some bytes
at the beginning of the file with a magic byte sequence (masking out specified
bits) you have supplied. Binfmt_misc can also recognise a filename extension
aka ``.com`` or ``.exe``.
First you must mount binfmt_misc::
mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
To actually register a new binary type, you have to set up a string looking like
``:name:type:offset:magic:mask:interpreter:flags`` (where you can choose the
``:`` upon your needs) and echo it to ``/proc/sys/fs/binfmt_misc/register``.
Here is what the fields mean:
- ``name``
is an identifier string. A new /proc file will be created with this
name below ``/proc/sys/fs/binfmt_misc``; cannot contain slashes ``/`` for
obvious reasons.
- ``type``
is the type of recognition. Give ``M`` for magic and ``E`` for extension.
- ``offset``
is the offset of the magic/mask in the file, counted in bytes. This
defaults to 0 if you omit it (i.e. you write ``:name:type::magic...``).
Ignored when using filename extension matching.
- ``magic``
is the byte sequence binfmt_misc is matching for. The magic string
may contain hex-encoded characters like ``\x0a`` or ``\xA4``. Note that you
must escape any NUL bytes; parsing halts at the first one. In a shell
environment you might have to write ``\\x0a`` to prevent the shell from
eating your ``\``.
If you chose filename extension matching, this is the extension to be
recognised (without the ``.``, the ``\x0a`` specials are not allowed).
Extension matching is case sensitive, and slashes ``/`` are not allowed!
- ``mask``
is an (optional, defaults to all 0xff) mask. You can mask out some
bits from matching by supplying a string like magic and as long as magic.
The mask is anded with the byte sequence of the file. Note that you must
escape any NUL bytes; parsing halts at the first one. Ignored when using
filename extension matching.
- ``interpreter``
is the program that should be invoked with the binary as first
argument (specify the full path)
- ``flags``
is an optional field that controls several aspects of the invocation
of the interpreter. It is a string of capital letters, each controls a
certain aspect. The following flags are supported:
``P`` - preserve-argv[0]
Legacy behavior of binfmt_misc is to overwrite
the original argv[0] with the full path to the binary. When this
flag is included, binfmt_misc will add an argument to the argument
vector for this purpose, thus preserving the original ``argv[0]``.
e.g. If your interp is set to ``/bin/foo`` and you run ``blah``
(which is in ``/usr/local/bin``), then the kernel will execute
``/bin/foo`` with ``argv[]`` set to ``["/bin/foo", "/usr/local/bin/blah", "blah"]``. The interp has to be aware of this so it can
execute ``/usr/local/bin/blah``
with ``argv[]`` set to ``["blah"]``.
``O`` - open-binary
Legacy behavior of binfmt_misc is to pass the full path
of the binary to the interpreter as an argument. When this flag is
included, binfmt_misc will open the file for reading and pass its
descriptor into the auxilary vector with the key "AT_EXECFD", thus
allowing the interpreter to execute non-readable binaries. This
feature should be used with care - the interpreter has to be trusted
not to emit the contents of the non-readable binary.
``C`` - credentials
Currently, the behavior of binfmt_misc is to calculate
the credentials and security token of the new process according to
the interpreter. When this flag is included, these attributes are
calculated according to the binary. It also implies the ``O`` flag.
This feature should be used with care as the interpreter
will run with root permissions when a setuid binary owned by root
is run with binfmt_misc.
``F`` - fix binary
The usual behaviour of binfmt_misc is to spawn the
binary lazily when the misc format file is invoked. However,
this doesn't work very well in the face of mount namespaces and
changeroots, so the ``F`` mode opens the binary as soon as the
emulation is installed and uses the opened image to spawn the
emulator, meaning it is always available once installed,
regardless of how the environment changes.
There are some restrictions:
- the whole register string may not exceed 1920 characters
- the magic must reside in the first 128 bytes of the file, i.e.
offset+size(magic) has to be less than 128
- the interpreter string may not exceed 127 characters
To use binfmt_misc you have to mount it first. You can mount it with
``mount -t binfmt_misc none /proc/sys/fs/binfmt_misc`` command, or you can add
a line ``none /proc/sys/fs/binfmt_misc binfmt_misc defaults 0 0`` to your
``/etc/fstab`` so it auto mounts on boot.
You may want to add the binary formats in one of your ``/etc/rc`` scripts during
boot-up. Read the manual of your init program to figure out how to do this
right.
Think about the order of adding entries! Later added entries are matched first!
A few examples (assumed you are in ``/proc/sys/fs/binfmt_misc``):
- enable support for em86 (like binfmt_em86, for Alpha AXP only)::
echo ':i386:M::\x7fELF\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff:/bin/em86:' > register
echo ':i486:M::\x7fELF\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x06:\xff\xff\xff\xff\xff\xfe\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfb\xff\xff:/bin/em86:' > register
- enable support for packed DOS applications (pre-configured dosemu hdimages)::
echo ':DEXE:M::\x0eDEX::/usr/bin/dosexec:' > register
- enable support for Windows executables using wine::
echo ':DOSWin:M::MZ::/usr/local/bin/wine:' > register
For java support see Documentation/admin-guide/java.rst
You can enable/disable binfmt_misc or one binary type by echoing 0 (to disable)
or 1 (to enable) to ``/proc/sys/fs/binfmt_misc/status`` or
``/proc/.../the_name``.
Catting the file tells you the current status of ``binfmt_misc/the_entry``.
You can remove one entry or all entries by echoing -1 to ``/proc/.../the_name``
or ``/proc/sys/fs/binfmt_misc/status``.
Hints
-----
If you want to pass special arguments to your interpreter, you can
write a wrapper script for it.
See :doc:`Documentation/admin-guide/java.rst <./java>` for an example.
Your interpreter should NOT look in the PATH for the filename; the kernel
passes it the full filename (or the file descriptor) to use. Using ``$PATH`` can
cause unexpected behaviour and can be a security hazard.
Richard Günther <rguenth@tat.physik.uni-tuebingen.de>
linux-7.2.3/Documentation/admin-guide/blockdev/drbd/DRBD-8.3-data-packets.svg
+588
View File
@@ -0,0 +1,588 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
version="1.0"
width="210mm"
height="297mm"
viewBox="0 0 21000 29700"
id="svg2"
style="fill-rule:evenodd">
<defs
id="defs4" />
<g
id="Default"
style="visibility:visible">
<desc
id="desc180">Master slide</desc>
</g>
<path
d="M 11999,8601 L 11899,8301 L 12099,8301 L 11999,8601 z"
id="path193"
style="fill:#008000;visibility:visible" />
<path
d="M 11999,7801 L 11999,8361"
id="path197"
style="fill:none;stroke:#008000;visibility:visible" />
<path
d="M 7999,10401 L 7899,10101 L 8099,10101 L 7999,10401 z"
id="path209"
style="fill:#008000;visibility:visible" />
<path
d="M 7999,9601 L 7999,10161"
id="path213"
style="fill:none;stroke:#008000;visibility:visible" />
<path
d="M 11999,7801 L 11685,7840 L 11724,7644 L 11999,7801 z"
id="path225"
style="fill:#008000;visibility:visible" />
<path
d="M 7999,7001 L 11764,7754"
id="path229"
style="fill:none;stroke:#008000;visibility:visible" />
<g
transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,-1244.4792,1416.5139)"
id="g245"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<text
id="text247">
<tspan
x="9139 9368 9579 9808 9986 10075 10252 10481 10659 10837 10909"
y="9284"
id="tspan249">RSDataReply</tspan>
</text>
</g>
<path
d="M 7999,9601 L 8281,9458 L 8311,9655 L 7999,9601 z"
id="path259"
style="fill:#008000;visibility:visible" />
<path
d="M 11999,9001 L 8236,9565"
id="path263"
style="fill:none;stroke:#008000;visibility:visible" />
<g
transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,1620.9382,-1639.4947)"
id="g279"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<text
id="text281">
<tspan
x="8743 8972 9132 9310 9573 9801 10013 10242 10419 10597 10775 10953 11114"
y="7023"
id="tspan283">CsumRSRequest</tspan>
</text>
</g>
<text
id="text297"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4034 4263 4440 4703 4881 5042 5219 5397 5503 5681 5842 6003 6180 6341 6519 6625 6803 6980 7158 7336 7497 7586 7692"
y="5707"
id="tspan299">w_make_resync_request()</tspan>
</text>
<text
id="text313"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12199 12305 12483 12644 12821 12893 13054 13232 13410 13638 13816 13905 14083 14311 14489 14667 14845 15023 15184 15272 15378"
y="7806"
id="tspan315">receive_DataRequest()</tspan>
</text>
<text
id="text329"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12199 12377 12483 12660 12838 13016 13194 13372 13549 13621 13799 13977 14083 14261 14438 14616 14794 14955 15133 15294 15399"
y="8606"
id="tspan331">drbd_endio_read_sec()</tspan>
</text>
<text
id="text345"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12191 12420 12597 12775 12953 13131 13309 13486 13664 13825 13986 14164 14426 14604 14710 14871 15049 15154 15332 15510 15616"
y="9007"
id="tspan347">w_e_end_csum_rs_req()</tspan>
</text>
<text
id="text361"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4444 4550 4728 4889 5066 5138 5299 5477 5655 5883 6095 6324 6501 6590 6768 6997 7175 7352 7424 7585 7691"
y="9507"
id="tspan363">receive_RSDataReply()</tspan>
</text>
<text
id="text377"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4457 4635 4741 4918 5096 5274 5452 5630 5807 5879 6057 6235 6464 6569 6641 6730 6908 7086 7247 7425 7585 7691"
y="10407"
id="tspan379">drbd_endio_write_sec()</tspan>
</text>
<text
id="text393"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4647 4825 5003 5180 5358 5536 5714 5820 5997 6158 6319 6497 6658 6836 7013 7085 7263 7424 7585 7691"
y="10907"
id="tspan395">e_end_resync_block()</tspan>
</text>
<path
d="M 11999,11601 L 11685,11640 L 11724,11444 L 11999,11601 z"
id="path405"
style="fill:#000080;visibility:visible" />
<path
d="M 7999,10801 L 11764,11554"
id="path409"
style="fill:none;stroke:#000080;visibility:visible" />
<g
transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,2434.7562,-1674.649)"
id="g425"
style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded">
<text
id="text427">
<tspan
x="9320 9621 9726 9798 9887 10065 10277 10438"
y="10943"
id="tspan429">WriteAck</tspan>
</text>
</g>
<text
id="text443"
style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12199 12377 12555 12644 12821 13033 13105 13283 13444 13604 13816 13977 14138 14244"
y="11559"
id="tspan445">got_BlockAck()</tspan>
</text>
<text
id="text459"
style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="7999 8304 8541 8778 8990 9201 9413 9650 10001 10120 10357 10594 10806 11043 11280 11398 11703 11940 12152 12364 12601 12812 12931 13049 13261 13498 13710 13947 14065 14302 14540 14658 14777 14870 15107 15225 15437 15649 15886"
y="4877"
id="tspan461">Checksum based Resync, case not in sync</tspan>
</text>
<text
id="text475"
style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="6961 7266 7571 7854 8159 8299 8536 8654 8891 9010 9247 9484 9603 9840 9958 10077 10170 10407"
y="2806"
id="tspan477">DRBD-8.3 data flow</tspan>
</text>
<text
id="text491"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="5190 5419 5596 5774 5952 6113 6291 6468 6646 6824 6985 7146 7324 7586 7692"
y="7005"
id="tspan493">w_e_send_csum()</tspan>
</text>
<path
d="M 11999,17601 L 11899,17301 L 12099,17301 L 11999,17601 z"
id="path503"
style="fill:#008000;visibility:visible" />
<path
d="M 11999,16801 L 11999,17361"
id="path507"
style="fill:none;stroke:#008000;visibility:visible" />
<path
d="M 11999,16801 L 11685,16840 L 11724,16644 L 11999,16801 z"
id="path519"
style="fill:#008000;visibility:visible" />
<path
d="M 7999,16001 L 11764,16754"
id="path523"
style="fill:none;stroke:#008000;visibility:visible" />
<g
transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,-2539.5806,1529.3491)"
id="g539"
style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded">
<text
id="text541">
<tspan
x="9269 9498 9709 9798 9959 10048 10226 10437 10598 10776"
y="18265"
id="tspan543">RSIsInSync</tspan>
</text>
</g>
<path
d="M 7999,18601 L 8281,18458 L 8311,18655 L 7999,18601 z"
id="path553"
style="fill:#000080;visibility:visible" />
<path
d="M 11999,18001 L 8236,18565"
id="path557"
style="fill:none;stroke:#000080;visibility:visible" />
<g
transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,3461.4027,-1449.3012)"
id="g573"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<text
id="text575">
<tspan
x="8743 8972 9132 9310 9573 9801 10013 10242 10419 10597 10775 10953 11114"
y="16023"
id="tspan577">CsumRSRequest</tspan>
</text>
</g>
<text
id="text591"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12199 12305 12483 12644 12821 12893 13054 13232 13410 13638 13816 13905 14083 14311 14489 14667 14845 15023 15184 15272 15378"
y="16806"
id="tspan593">receive_DataRequest()</tspan>
</text>
<text
id="text607"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12199 12377 12483 12660 12838 13016 13194 13372 13549 13621 13799 13977 14083 14261 14438 14616 14794 14955 15133 15294 15399"
y="17606"
id="tspan609">drbd_endio_read_sec()</tspan>
</text>
<text
id="text623"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12191 12420 12597 12775 12953 13131 13309 13486 13664 13825 13986 14164 14426 14604 14710 14871 15049 15154 15332 15510 15616"
y="18007"
id="tspan625">w_e_end_csum_rs_req()</tspan>
</text>
<text
id="text639"
style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded">
<tspan
x="5735 5913 6091 6180 6357 6446 6607 6696 6874 7085 7246 7424 7585 7691"
y="18507"
id="tspan641">got_IsInSync()</tspan>
</text>
<text
id="text655"
style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="7999 8304 8541 8778 8990 9201 9413 9650 10001 10120 10357 10594 10806 11043 11280 11398 11703 11940 12152 12364 12601 12812 12931 13049 13261 13498 13710 13947 14065 14159 14396 14514 14726 14937 15175"
y="13877"
id="tspan657">Checksum based Resync, case in sync</tspan>
</text>
<path
d="M 12000,24601 L 11900,24301 L 12100,24301 L 12000,24601 z"
id="path667"
style="fill:#008000;visibility:visible" />
<path
d="M 12000,23801 L 12000,24361"
id="path671"
style="fill:none;stroke:#008000;visibility:visible" />
<path
d="M 8000,26401 L 7900,26101 L 8100,26101 L 8000,26401 z"
id="path683"
style="fill:#008000;visibility:visible" />
<path
d="M 8000,25601 L 8000,26161"
id="path687"
style="fill:none;stroke:#008000;visibility:visible" />
<path
d="M 12000,23801 L 11686,23840 L 11725,23644 L 12000,23801 z"
id="path699"
style="fill:#008000;visibility:visible" />
<path
d="M 8000,23001 L 11765,23754"
id="path703"
style="fill:none;stroke:#008000;visibility:visible" />
<g
transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,-3543.8452,1630.5143)"
id="g719"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<text
id="text721">
<tspan
x="9464 9710 9921 10150 10328 10505 10577"
y="25236"
id="tspan723">OVReply</tspan>
</text>
</g>
<path
d="M 8000,25601 L 8282,25458 L 8312,25655 L 8000,25601 z"
id="path733"
style="fill:#008000;visibility:visible" />
<path
d="M 12000,25001 L 8237,25565"
id="path737"
style="fill:none;stroke:#008000;visibility:visible" />
<g
transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,4918.2801,-1381.2128)"
id="g753"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<text
id="text755">
<tspan
x="9142 9388 9599 9828 10006 10183 10361 10539 10700"
y="23106"
id="tspan757">OVRequest</tspan>
</text>
</g>
<text
id="text771"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12200 12306 12484 12645 12822 12894 13055 13233 13411 13656 13868 14097 14274 14452 14630 14808 14969 15058 15163"
y="23806"
id="tspan773">receive_OVRequest()</tspan>
</text>
<text
id="text787"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12200 12378 12484 12661 12839 13017 13195 13373 13550 13622 13800 13978 14084 14262 14439 14617 14795 14956 15134 15295 15400"
y="24606"
id="tspan789">drbd_endio_read_sec()</tspan>
</text>
<text
id="text803"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12192 12421 12598 12776 12954 13132 13310 13487 13665 13843 14004 14182 14288 14465 14643 14749"
y="25007"
id="tspan805">w_e_end_ov_req()</tspan>
</text>
<text
id="text819"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="5101 5207 5385 5546 5723 5795 5956 6134 6312 6557 6769 6998 7175 7353 7425 7586 7692"
y="25507"
id="tspan821">receive_OVReply()</tspan>
</text>
<text
id="text835"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4492 4670 4776 4953 5131 5309 5487 5665 5842 5914 6092 6270 6376 6554 6731 6909 7087 7248 7426 7587 7692"
y="26407"
id="tspan837">drbd_endio_read_sec()</tspan>
</text>
<text
id="text851"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4902 5131 5308 5486 5664 5842 6020 6197 6375 6553 6714 6892 6998 7175 7353 7425 7586 7692"
y="26907"
id="tspan853">w_e_end_ov_reply()</tspan>
</text>
<path
d="M 12000,27601 L 11686,27640 L 11725,27444 L 12000,27601 z"
id="path863"
style="fill:#000080;visibility:visible" />
<path
d="M 8000,26801 L 11765,27554"
id="path867"
style="fill:none;stroke:#000080;visibility:visible" />
<g
transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,5704.1907,-1328.312)"
id="g883"
style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded">
<text
id="text885">
<tspan
x="9279 9525 9736 9965 10143 10303 10481 10553"
y="26935"
id="tspan887">OVResult</tspan>
</text>
</g>
<text
id="text901"
style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12200 12378 12556 12645 12822 13068 13280 13508 13686 13847 14025 14097 14185 14291"
y="27559"
id="tspan903">got_OVResult()</tspan>
</text>
<text
id="text917"
style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="8000 8330 8567 8660 8754 8991 9228 9346 9558 9795 9935 10028 10146"
y="21877"
id="tspan919">Online verify</tspan>
</text>
<text
id="text933"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4641 4870 5047 5310 5488 5649 5826 6004 6182 6343 6521 6626 6804 6982 7160 7338 7499 7587 7693"
y="23005"
id="tspan935">w_make_ov_request()</tspan>
</text>
<path
d="M 8000,6500 L 7900,6200 L 8100,6200 L 8000,6500 z"
id="path945"
style="fill:#008000;visibility:visible" />
<path
d="M 8000,5700 L 8000,6260"
id="path949"
style="fill:none;stroke:#008000;visibility:visible" />
<path
d="M 3900,5500 L 3700,5500 L 3700,11000 L 3900,11000"
id="path961"
style="fill:none;stroke:#000000;visibility:visible" />
<path
d="M 3900,14500 L 3700,14500 L 3700,18600 L 3900,18600"
id="path973"
style="fill:none;stroke:#000000;visibility:visible" />
<path
d="M 3900,22800 L 3700,22800 L 3700,26900 L 3900,26900"
id="path985"
style="fill:none;stroke:#000000;visibility:visible" />
<text
id="text1001"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4492 4670 4776 4953 5131 5309 5487 5665 5842 5914 6092 6270 6376 6554 6731 6909 7087 7248 7426 7587 7692"
y="6506"
id="tspan1003">drbd_endio_read_sec()</tspan>
</text>
<text
id="text1017"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4034 4263 4440 4703 4881 5042 5219 5397 5503 5681 5842 6003 6180 6341 6519 6625 6803 6980 7158 7336 7497 7586 7692"
y="14708"
id="tspan1019">w_make_resync_request()</tspan>
</text>
<text
id="text1033"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="5190 5419 5596 5774 5952 6113 6291 6468 6646 6824 6985 7146 7324 7586 7692"
y="16006"
id="tspan1035">w_e_send_csum()</tspan>
</text>
<path
d="M 8000,15501 L 7900,15201 L 8100,15201 L 8000,15501 z"
id="path1045"
style="fill:#008000;visibility:visible" />
<path
d="M 8000,14701 L 8000,15261"
id="path1049"
style="fill:none;stroke:#008000;visibility:visible" />
<text
id="text1065"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4492 4670 4776 4953 5131 5309 5487 5665 5842 5914 6092 6270 6376 6554 6731 6909 7087 7248 7426 7587 7692"
y="15507"
id="tspan1067">drbd_endio_read_sec()</tspan>
</text>
<path
d="M 16100,9000 L 16300,9000 L 16300,7500 L 16100,7500"
id="path1077"
style="fill:none;stroke:#000000;visibility:visible" />
<path
d="M 16100,18000 L 16300,18000 L 16300,16500 L 16100,16500"
id="path1089"
style="fill:none;stroke:#000000;visibility:visible" />
<path
d="M 16100,25000 L 16300,25000 L 16300,23500 L 16100,23500"
id="path1101"
style="fill:none;stroke:#000000;visibility:visible" />
<text
id="text1117"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="2026 2132 2293 2471 2648 2826 3004 3076 3254 3431 3503 3681 3787"
y="5402"
id="tspan1119">rs_begin_io()</tspan>
</text>
<text
id="text1133"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="2027 2133 2294 2472 2649 2827 3005 3077 3255 3432 3504 3682 3788"
y="14402"
id="tspan1135">rs_begin_io()</tspan>
</text>
<text
id="text1149"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="2026 2132 2293 2471 2648 2826 3004 3076 3254 3431 3503 3681 3787"
y="22602"
id="tspan1151">rs_begin_io()</tspan>
</text>
<text
id="text1165"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="1426 1532 1693 1871 2031 2209 2472 2649 2721 2899 2988 3166 3344 3416 3593 3699"
y="11302"
id="tspan1167">rs_complete_io()</tspan>
</text>
<text
id="text1181"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="1526 1632 1793 1971 2131 2309 2572 2749 2821 2999 3088 3266 3444 3516 3693 3799"
y="18931"
id="tspan1183">rs_complete_io()</tspan>
</text>
<text
id="text1197"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="1526 1632 1793 1971 2131 2309 2572 2749 2821 2999 3088 3266 3444 3516 3693 3799"
y="27231"
id="tspan1199">rs_complete_io()</tspan>
</text>
<text
id="text1213"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="16126 16232 16393 16571 16748 16926 17104 17176 17354 17531 17603 17781 17887"
y="7402"
id="tspan1215">rs_begin_io()</tspan>
</text>
<text
id="text1229"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="16127 16233 16394 16572 16749 16927 17105 17177 17355 17532 17604 17782 17888"
y="16331"
id="tspan1231">rs_begin_io()</tspan>
</text>
<text
id="text1245"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="16127 16233 16394 16572 16749 16927 17105 17177 17355 17532 17604 17782 17888"
y="23302"
id="tspan1247">rs_begin_io()</tspan>
</text>
<text
id="text1261"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="16115 16221 16382 16560 16720 16898 17161 17338 17410 17588 17677 17855 18033 18105 18282 18388"
y="9302"
id="tspan1263">rs_complete_io()</tspan>
</text>
<text
id="text1277"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="16115 16221 16382 16560 16720 16898 17161 17338 17410 17588 17677 17855 18033 18105 18282 18388"
y="18331"
id="tspan1279">rs_complete_io()</tspan>
</text>
<text
id="text1293"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="16126 16232 16393 16571 16731 16909 17172 17349 17421 17599 17688 17866 18044 18116 18293 18399"
y="25302"
id="tspan1295">rs_complete_io()</tspan>
</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

linux-7.2.3/Documentation/admin-guide/blockdev/drbd/DRBD-data-packets.svg
+459
View File
@@ -0,0 +1,459 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
version="1.0"
width="210mm"
height="297mm"
viewBox="0 0 21000 29700"
id="svg2"
style="fill-rule:evenodd">
<defs
id="defs4" />
<g
id="Default"
style="visibility:visible">
<desc
id="desc176">Master slide</desc>
</g>
<path
d="M 11999,19601 L 11899,19301 L 12099,19301 L 11999,19601 z"
id="path189"
style="fill:#008000;visibility:visible" />
<path
d="M 11999,18801 L 11999,19361"
id="path193"
style="fill:none;stroke:#008000;visibility:visible" />
<path
d="M 7999,21401 L 7899,21101 L 8099,21101 L 7999,21401 z"
id="path205"
style="fill:#008000;visibility:visible" />
<path
d="M 7999,20601 L 7999,21161"
id="path209"
style="fill:none;stroke:#008000;visibility:visible" />
<path
d="M 11999,18801 L 11685,18840 L 11724,18644 L 11999,18801 z"
id="path221"
style="fill:#008000;visibility:visible" />
<path
d="M 7999,18001 L 11764,18754"
id="path225"
style="fill:none;stroke:#008000;visibility:visible" />
<text
x="-3023.845"
y="1106.8124"
transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,0,0)"
id="text243"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="6115.1553 6344.1553 6555.1553 6784.1553 6962.1553 7051.1553 7228.1553 7457.1553 7635.1553 7813.1553 7885.1553"
y="21390.812"
id="tspan245">RSDataReply</tspan>
</text>
<path
d="M 7999,20601 L 8281,20458 L 8311,20655 L 7999,20601 z"
id="path255"
style="fill:#008000;visibility:visible" />
<path
d="M 11999,20001 L 8236,20565"
id="path259"
style="fill:none;stroke:#008000;visibility:visible" />
<text
x="3502.5356"
y="-2184.6621"
transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,0,0)"
id="text277"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12321.536 12550.536 12761.536 12990.536 13168.536 13257.536 13434.536 13663.536 13841.536 14019.536 14196.536 14374.536 14535.536"
y="15854.338"
id="tspan279">RSDataRequest</tspan>
</text>
<text
id="text293"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4034 4263 4440 4703 4881 5042 5219 5397 5503 5681 5842 6003 6180 6341 6519 6625 6803 6980 7158 7336 7497 7586 7692"
y="17807"
id="tspan295">w_make_resync_request()</tspan>
</text>
<text
id="text309"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12199 12305 12483 12644 12821 12893 13054 13232 13410 13638 13816 13905 14083 14311 14489 14667 14845 15023 15184 15272 15378"
y="18806"
id="tspan311">receive_DataRequest()</tspan>
</text>
<text
id="text325"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12199 12377 12483 12660 12838 13016 13194 13372 13549 13621 13799 13977 14083 14261 14438 14616 14794 14955 15133 15294 15399"
y="19606"
id="tspan327">drbd_endio_read_sec()</tspan>
</text>
<text
id="text341"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12191 12420 12597 12775 12953 13131 13309 13486 13664 13770 13931 14109 14287 14375 14553 14731 14837 15015 15192 15298"
y="20007"
id="tspan343">w_e_end_rsdata_req()</tspan>
</text>
<text
id="text357"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4444 4550 4728 4889 5066 5138 5299 5477 5655 5883 6095 6324 6501 6590 6768 6997 7175 7352 7424 7585 7691"
y="20507"
id="tspan359">receive_RSDataReply()</tspan>
</text>
<text
id="text373"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4457 4635 4741 4918 5096 5274 5452 5630 5807 5879 6057 6235 6464 6569 6641 6730 6908 7086 7247 7425 7585 7691"
y="21407"
id="tspan375">drbd_endio_write_sec()</tspan>
</text>
<text
id="text389"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4647 4825 5003 5180 5358 5536 5714 5820 5997 6158 6319 6497 6658 6836 7013 7085 7263 7424 7585 7691"
y="21907"
id="tspan391">e_end_resync_block()</tspan>
</text>
<path
d="M 11999,22601 L 11685,22640 L 11724,22444 L 11999,22601 z"
id="path401"
style="fill:#000080;visibility:visible" />
<path
d="M 7999,21801 L 11764,22554"
id="path405"
style="fill:none;stroke:#000080;visibility:visible" />
<text
x="4290.3008"
y="-2369.6162"
transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,0,0)"
id="text423"
style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded">
<tspan
x="13610.301 13911.301 14016.301 14088.301 14177.301 14355.301 14567.301 14728.301"
y="19573.385"
id="tspan425">WriteAck</tspan>
</text>
<text
id="text439"
style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12199 12377 12555 12644 12821 13033 13105 13283 13444 13604 13816 13977 14138 14244"
y="22559"
id="tspan441">got_BlockAck()</tspan>
</text>
<text
id="text455"
style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="7999 8304 8541 8753 8964 9201 9413 9531 9769 9862 10099 10310 10522 10734 10852 10971 11208 11348 11585 11822"
y="16877"
id="tspan457">Resync blocks, 4-32K</tspan>
</text>
<path
d="M 12000,7601 L 11900,7301 L 12100,7301 L 12000,7601 z"
id="path467"
style="fill:#008000;visibility:visible" />
<path
d="M 12000,6801 L 12000,7361"
id="path471"
style="fill:none;stroke:#008000;visibility:visible" />
<path
d="M 12000,6801 L 11686,6840 L 11725,6644 L 12000,6801 z"
id="path483"
style="fill:#008000;visibility:visible" />
<path
d="M 8000,6001 L 11765,6754"
id="path487"
style="fill:none;stroke:#008000;visibility:visible" />
<text
x="-1288.1796"
y="1279.7666"
transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,0,0)"
id="text505"
style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded">
<tspan
x="8174.8208 8475.8203 8580.8203 8652.8203 8741.8203 8919.8203 9131.8203 9292.8203"
y="9516.7666"
id="tspan507">WriteAck</tspan>
</text>
<path
d="M 8000,8601 L 8282,8458 L 8312,8655 L 8000,8601 z"
id="path517"
style="fill:#000080;visibility:visible" />
<path
d="M 12000,8001 L 8237,8565"
id="path521"
style="fill:none;stroke:#000080;visibility:visible" />
<text
x="1065.6655"
y="-2097.7664"
transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,0,0)"
id="text539"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="10682.666 10911.666 11088.666 11177.666"
y="4107.2339"
id="tspan541">Data</tspan>
</text>
<text
id="text555"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4746 4924 5030 5207 5385 5563 5826 6003 6164 6342 6520 6626 6803 6981 7159 7337 7498 7587 7692"
y="5505"
id="tspan557">drbd_make_request()</tspan>
</text>
<text
id="text571"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12200 12306 12484 12645 12822 12894 13055 13233 13411 13639 13817 13906 14084 14190"
y="6806"
id="tspan573">receive_Data()</tspan>
</text>
<text
id="text587"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12200 12378 12484 12661 12839 13017 13195 13373 13550 13622 13800 13978 14207 14312 14384 14473 14651 14829 14990 15168 15328 15434"
y="7606"
id="tspan589">drbd_endio_write_sec()</tspan>
</text>
<text
id="text603"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12192 12370 12548 12725 12903 13081 13259 13437 13509 13686 13847 14008 14114"
y="8007"
id="tspan605">e_end_block()</tspan>
</text>
<text
id="text619"
style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded">
<tspan
x="5647 5825 6003 6092 6269 6481 6553 6731 6892 7052 7264 7425 7586 7692"
y="8606"
id="tspan621">got_BlockAck()</tspan>
</text>
<text
id="text635"
style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="8000 8305 8542 8779 9016 9109 9346 9486 9604 9956 10049 10189 10328 10565 10705 10942 11179 11298 11603 11742 11835 11954 12191 12310 12428 12665 12902 13139 13279 13516 13753"
y="4877"
id="tspan637">Regular mirrored write, 512-32K</tspan>
</text>
<text
id="text651"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="5381 5610 5787 5948 6126 6304 6482 6659 6837 7015 7087 7265 7426 7587 7692"
y="6003"
id="tspan653">w_send_dblock()</tspan>
</text>
<path
d="M 8000,6800 L 7900,6500 L 8100,6500 L 8000,6800 z"
id="path663"
style="fill:#008000;visibility:visible" />
<path
d="M 8000,6000 L 8000,6560"
id="path667"
style="fill:none;stroke:#008000;visibility:visible" />
<text
id="text683"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4602 4780 4886 5063 5241 5419 5597 5775 5952 6024 6202 6380 6609 6714 6786 6875 7053 7231 7409 7515 7587 7692"
y="6905"
id="tspan685">drbd_endio_write_pri()</tspan>
</text>
<path
d="M 12000,13602 L 11900,13302 L 12100,13302 L 12000,13602 z"
id="path695"
style="fill:#008000;visibility:visible" />
<path
d="M 12000,12802 L 12000,13362"
id="path699"
style="fill:none;stroke:#008000;visibility:visible" />
<path
d="M 12000,12802 L 11686,12841 L 11725,12645 L 12000,12802 z"
id="path711"
style="fill:#008000;visibility:visible" />
<path
d="M 8000,12002 L 11765,12755"
id="path715"
style="fill:none;stroke:#008000;visibility:visible" />
<text
x="-2155.5266"
y="1201.5964"
transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,0,0)"
id="text733"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="7202.4736 7431.4736 7608.4736 7697.4736 7875.4736 8104.4736 8282.4736 8459.4736 8531.4736"
y="15454.597"
id="tspan735">DataReply</tspan>
</text>
<path
d="M 8000,14602 L 8282,14459 L 8312,14656 L 8000,14602 z"
id="path745"
style="fill:#008000;visibility:visible" />
<path
d="M 12000,14002 L 8237,14566"
id="path749"
style="fill:none;stroke:#008000;visibility:visible" />
<text
x="2280.3804"
y="-2103.2141"
transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,0,0)"
id="text767"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="11316.381 11545.381 11722.381 11811.381 11989.381 12218.381 12396.381 12573.381 12751.381 12929.381 13090.381"
y="9981.7861"
id="tspan769">DataRequest</tspan>
</text>
<text
id="text783"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="4746 4924 5030 5207 5385 5563 5826 6003 6164 6342 6520 6626 6803 6981 7159 7337 7498 7587 7692"
y="11506"
id="tspan785">drbd_make_request()</tspan>
</text>
<text
id="text799"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12200 12306 12484 12645 12822 12894 13055 13233 13411 13639 13817 13906 14084 14312 14490 14668 14846 15024 15185 15273 15379"
y="12807"
id="tspan801">receive_DataRequest()</tspan>
</text>
<text
id="text815"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12200 12378 12484 12661 12839 13017 13195 13373 13550 13622 13800 13978 14084 14262 14439 14617 14795 14956 15134 15295 15400"
y="13607"
id="tspan817">drbd_endio_read_sec()</tspan>
</text>
<text
id="text831"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="12192 12421 12598 12776 12954 13132 13310 13487 13665 13843 14021 14110 14288 14465 14571 14749 14927 15033"
y="14008"
id="tspan833">w_e_end_data_req()</tspan>
</text>
<g
id="g835"
style="visibility:visible">
<desc
id="desc837">Drawing</desc>
<text
id="text847"
style="font-size:318px;font-weight:400;fill:#008000;font-family:Helvetica embedded">
<tspan
x="4885 4991 5169 5330 5507 5579 5740 5918 6096 6324 6502 6591 6769 6997 7175 7353 7425 7586 7692"
y="14607"
id="tspan849">receive_DataReply()</tspan>
</text>
</g>
<text
id="text863"
style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="8000 8305 8398 8610 8821 8914 9151 9363 9575 9693 9833 10070 10307 10544 10663 10781 11018 11255 11493 11632 11869 12106"
y="10878"
id="tspan865">Diskless read, 512-32K</tspan>
</text>
<text
id="text879"
style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="5029 5258 5435 5596 5774 5952 6130 6307 6413 6591 6769 6947 7125 7230 7408 7586 7692"
y="12004"
id="tspan881">w_send_read_req()</tspan>
</text>
<text
id="text895"
style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="6961 7266 7571 7854 8159 8278 8515 8633 8870 9107 9226 9463 9581 9700 9793 10030"
y="2806"
id="tspan897">DRBD 8 data flow</tspan>
</text>
<path
d="M 3900,5300 L 3700,5300 L 3700,7000 L 3900,7000"
id="path907"
style="fill:none;stroke:#000000;visibility:visible" />
<path
d="M 3900,17600 L 3700,17600 L 3700,22000 L 3900,22000"
id="path919"
style="fill:none;stroke:#000000;visibility:visible" />
<path
d="M 16100,20000 L 16300,20000 L 16300,18500 L 16100,18500"
id="path931"
style="fill:none;stroke:#000000;visibility:visible" />
<text
id="text947"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="2126 2304 2376 2554 2731 2909 3087 3159 3337 3515 3587 3764 3870"
y="5202"
id="tspan949">al_begin_io()</tspan>
</text>
<text
id="text963"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="1632 1810 1882 2060 2220 2398 2661 2839 2910 3088 3177 3355 3533 3605 3783 3888"
y="7331"
id="tspan965">al_complete_io()</tspan>
</text>
<text
id="text979"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="2126 2232 2393 2571 2748 2926 3104 3176 3354 3531 3603 3781 3887"
y="17431"
id="tspan981">rs_begin_io()</tspan>
</text>
<text
id="text995"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="1626 1732 1893 2071 2231 2409 2672 2849 2921 3099 3188 3366 3544 3616 3793 3899"
y="22331"
id="tspan997">rs_complete_io()</tspan>
</text>
<text
id="text1011"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="16027 16133 16294 16472 16649 16827 17005 17077 17255 17432 17504 17682 17788"
y="18402"
id="tspan1013">rs_begin_io()</tspan>
</text>
<text
id="text1027"
style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded">
<tspan
x="16115 16221 16382 16560 16720 16898 17161 17338 17410 17588 17677 17855 18033 18105 18282 18388"
y="20331"
id="tspan1029">rs_complete_io()</tspan>
</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

linux-7.2.3/Documentation/admin-guide/blockdev/drbd/conn-states-8.dot
+18
View File
@@ -0,0 +1,18 @@
digraph conn_states {
StandAllone -> WFConnection [ label = "ioctl_set_net()" ]
WFConnection -> Unconnected [ label = "unable to bind()" ]
WFConnection -> WFReportParams [ label = "in connect() after accept" ]
WFReportParams -> StandAllone [ label = "checks in receive_param()" ]
WFReportParams -> Connected [ label = "in receive_param()" ]
WFReportParams -> WFBitMapS [ label = "sync_handshake()" ]
WFReportParams -> WFBitMapT [ label = "sync_handshake()" ]
WFBitMapS -> SyncSource [ label = "receive_bitmap()" ]
WFBitMapT -> SyncTarget [ label = "receive_bitmap()" ]
SyncSource -> Connected
SyncTarget -> Connected
SyncSource -> PausedSyncS
SyncTarget -> PausedSyncT
PausedSyncS -> SyncSource
PausedSyncT -> SyncTarget
Connected -> WFConnection [ label = "* on network error" ]
}
linux-7.2.3/Documentation/admin-guide/blockdev/drbd/data-structure-v9.rst
+42
View File
@@ -0,0 +1,42 @@
================================
kernel data structure for DRBD-9
================================
This describes the in kernel data structure for DRBD-9. Starting with
Linux v3.14 we are reorganizing DRBD to use this data structure.
Basic Data Structure
====================
A node has a number of DRBD resources. Each such resource has a number of
devices (aka volumes) and connections to other nodes ("peer nodes"). Each DRBD
device is represented by a block device locally.
The DRBD objects are interconnected to form a matrix as depicted below; a
drbd_peer_device object sits at each intersection between a drbd_device and a
drbd_connection::
/--------------+---------------+.....+---------------\
| resource | device | | device |
+--------------+---------------+.....+---------------+
| connection | peer_device | | peer_device |
+--------------+---------------+.....+---------------+
: : : : :
: : : : :
+--------------+---------------+.....+---------------+
| connection | peer_device | | peer_device |
\--------------+---------------+.....+---------------/
In this table, horizontally, devices can be accessed from resources by their
volume number. Likewise, peer_devices can be accessed from connections by
their volume number. Objects in the vertical direction are connected by double
linked lists. There are back pointers from peer_devices to their connections a
devices, and from connections and devices to their resource.
All resources are in the drbd_resources double-linked list. In addition, all
devices can be accessed by their minor device number via the drbd_devices idr.
The drbd_resource, drbd_connection, and drbd_device objects are reference
counted. The peer_device objects only serve to establish the links between
devices and connections; their lifetime is determined by the lifetime of the
device and connection which they reference.
linux-7.2.3/Documentation/admin-guide/blockdev/drbd/disk-states-8.dot
+16
View File
@@ -0,0 +1,16 @@
digraph disk_states {
Diskless -> Inconsistent [ label = "ioctl_set_disk()" ]
Diskless -> Consistent [ label = "ioctl_set_disk()" ]
Diskless -> Outdated [ label = "ioctl_set_disk()" ]
Consistent -> Outdated [ label = "receive_param()" ]
Consistent -> UpToDate [ label = "receive_param()" ]
Consistent -> Inconsistent [ label = "start resync" ]
Outdated -> Inconsistent [ label = "start resync" ]
UpToDate -> Inconsistent [ label = "ioctl_replicate" ]
Inconsistent -> UpToDate [ label = "resync completed" ]
Consistent -> Failed [ label = "io completion error" ]
Outdated -> Failed [ label = "io completion error" ]
UpToDate -> Failed [ label = "io completion error" ]
Inconsistent -> Failed [ label = "io completion error" ]
Failed -> Diskless [ label = "sending notify to peer" ]
}
linux-7.2.3/Documentation/admin-guide/blockdev/drbd/drbd-connection-state-overview.dot
+85
View File
@@ -0,0 +1,85 @@
// vim: set sw=2 sts=2 :
digraph {
rankdir=BT
bgcolor=white
node [shape=plaintext]
node [fontcolor=black]
StandAlone [ style=filled,fillcolor=gray,label=StandAlone ]
node [fontcolor=lightgray]
Unconnected [ label=Unconnected ]
CommTrouble [ shape=record,
label="{communication loss|{Timeout|BrokenPipe|NetworkFailure}}" ]
node [fontcolor=gray]
subgraph cluster_try_connect {
label="try to connect, handshake"
rank=max
WFConnection [ label=WFConnection ]
WFReportParams [ label=WFReportParams ]
}
TearDown [ label=TearDown ]
Connected [ label=Connected,style=filled,fillcolor=green,fontcolor=black ]
node [fontcolor=lightblue]
StartingSyncS [ label=StartingSyncS ]
StartingSyncT [ label=StartingSyncT ]
subgraph cluster_bitmap_exchange {
node [fontcolor=red]
fontcolor=red
label="new application (WRITE?) requests blocked\lwhile bitmap is exchanged"
WFBitMapT [ label=WFBitMapT ]
WFSyncUUID [ label=WFSyncUUID ]
WFBitMapS [ label=WFBitMapS ]
}
node [fontcolor=blue]
cluster_resync [ shape=record,label="{<any>resynchronisation process running\l'concurrent' application requests allowed|{{<T>PausedSyncT\nSyncTarget}|{<S>PausedSyncS\nSyncSource}}}" ]
node [shape=box,fontcolor=black]
// drbdadm [label="drbdadm connect"]
// handshake [label="drbd_connect()\ndrbd_do_handshake\ndrbd_sync_handshake() etc."]
// comm_error [label="communication trouble"]
//
// edges
// --------------------------------------
StandAlone -> Unconnected [ label="drbdadm connect" ]
Unconnected -> StandAlone [ label="drbdadm disconnect\lor serious communication trouble" ]
Unconnected -> WFConnection [ label="receiver thread is started" ]
WFConnection -> WFReportParams [ headlabel="accept()\land/or \lconnect()\l" ]
WFReportParams -> StandAlone [ label="during handshake\lpeers do not agree\labout something essential" ]
WFReportParams -> Connected [ label="data identical\lno sync needed",color=green,fontcolor=green ]
WFReportParams -> WFBitMapS
WFReportParams -> WFBitMapT
WFBitMapT -> WFSyncUUID [minlen=0.1,constraint=false]
WFBitMapS -> cluster_resync:S
WFSyncUUID -> cluster_resync:T
edge [color=green]
cluster_resync:any -> Connected [ label="resnyc done",fontcolor=green ]
edge [color=red]
WFReportParams -> CommTrouble
Connected -> CommTrouble
cluster_resync:any -> CommTrouble
edge [color=black]
CommTrouble -> Unconnected [label="receiver thread is stopped" ]
}
linux-7.2.3/Documentation/admin-guide/blockdev/drbd/figures.rst
+30
View File
@@ -0,0 +1,30 @@
.. SPDX-License-Identifier: GPL-2.0
.. The here included files are intended to help understand the implementation
Data flows that Relate some functions, and write packets
========================================================
.. kernel-figure:: DRBD-8.3-data-packets.svg
:alt: DRBD-8.3-data-packets.svg
:align: center
.. kernel-figure:: DRBD-data-packets.svg
:alt: DRBD-data-packets.svg
:align: center
Sub graphs of DRBD's state transitions
======================================
.. kernel-figure:: conn-states-8.dot
:alt: conn-states-8.dot
:align: center
.. kernel-figure:: disk-states-8.dot
:alt: disk-states-8.dot
:align: center
.. kernel-figure:: peer-states-8.dot
:alt: peer-states-8.dot
:align: center
linux-7.2.3/Documentation/admin-guide/blockdev/drbd/index.rst
+19
View File
@@ -0,0 +1,19 @@
==========================================
Distributed Replicated Block Device - DRBD
==========================================
Description
===========
DRBD is a shared-nothing, synchronously replicated block device. It
is designed to serve as a building block for high availability
clusters and in this context, is a "drop-in" replacement for shared
storage. Simplistically, you could see it as a network RAID 1.
Please visit https://www.drbd.org to find out more.
.. toctree::
:maxdepth: 1
data-structure-v9
figures
linux-7.2.3/Documentation/admin-guide/blockdev/drbd/peer-states-8.dot
+8
View File
@@ -0,0 +1,8 @@
digraph peer_states {
Secondary -> Primary [ label = "recv state packet" ]
Primary -> Secondary [ label = "recv state packet" ]
Primary -> Unknown [ label = "connection lost" ]
Secondary -> Unknown [ label = "connection lost" ]
Unknown -> Primary [ label = "connected" ]
Unknown -> Secondary [ label = "connected" ]
}
linux-7.2.3/Documentation/admin-guide/blockdev/floppy.rst
+255
View File
@@ -0,0 +1,255 @@
=============
Floppy Driver
=============
FAQ list:
=========
A FAQ list may be found in the fdutils package (see below), and also
at <https://fdutils.linux.lu/faq.html>.
LILO configuration options (Thinkpad users, read this)
======================================================
The floppy driver is configured using the 'floppy=' option in
lilo. This option can be typed at the boot prompt, or entered in the
lilo configuration file.
Example: If your kernel is called linux-2.6.9, type the following line
at the lilo boot prompt (if you have a thinkpad)::
linux-2.6.9 floppy=thinkpad
You may also enter the following line in /etc/lilo.conf, in the description
of linux-2.6.9::
append = "floppy=thinkpad"
Several floppy related options may be given, example::
linux-2.6.9 floppy=daring floppy=two_fdc
append = "floppy=daring floppy=two_fdc"
If you give options both in the lilo config file and on the boot
prompt, the option strings of both places are concatenated, the boot
prompt options coming last. That's why there are also options to
restore the default behavior.
Module configuration options
============================
If you use the floppy driver as a module, use the following syntax::
modprobe floppy floppy="<options>"
Example::
modprobe floppy floppy="omnibook messages"
If you need certain options enabled every time you load the floppy driver,
you can put::
options floppy floppy="omnibook messages"
in a configuration file in /etc/modprobe.d/.
The floppy driver related options are:
floppy=asus_pci
Sets the bit mask to allow only units 0 and 1. (default)
floppy=daring
Tells the floppy driver that you have a well behaved floppy controller.
This allows more efficient and smoother operation, but may fail on
certain controllers. This may speed up certain operations.
floppy=0,daring
Tells the floppy driver that your floppy controller should be used
with caution.
floppy=one_fdc
Tells the floppy driver that you have only one floppy controller.
(default)
floppy=two_fdc / floppy=<address>,two_fdc
Tells the floppy driver that you have two floppy controllers.
The second floppy controller is assumed to be at <address>.
This option is not needed if the second controller is at address
0x370, and if you use the 'cmos' option.
floppy=thinkpad
Tells the floppy driver that you have a Thinkpad. Thinkpads use an
inverted convention for the disk change line.
floppy=0,thinkpad
Tells the floppy driver that you don't have a Thinkpad.
floppy=omnibook / floppy=nodma
Tells the floppy driver not to use Dma for data transfers.
This is needed on HP Omnibooks, which don't have a workable
DMA channel for the floppy driver. This option is also useful
if you frequently get "Unable to allocate DMA memory" messages.
Indeed, dma memory needs to be continuous in physical memory,
and is thus harder to find, whereas non-dma buffers may be
allocated in virtual memory. However, I advise against this if
you have an FDC without a FIFO (8272A or 82072). 82072A and
later are OK. You also need at least a 486 to use nodma.
If you use nodma mode, I suggest you also set the FIFO
threshold to 10 or lower, in order to limit the number of data
transfer interrupts.
If you have a FIFO-able FDC, the floppy driver automatically
falls back on non DMA mode if no DMA-able memory can be found.
If you want to avoid this, explicitly ask for 'yesdma'.
floppy=yesdma
Tells the floppy driver that a workable DMA channel is available.
(default)
floppy=nofifo
Disables the FIFO entirely. This is needed if you get "Bus
master arbitration error" messages from your Ethernet card (or
from other devices) while accessing the floppy.
floppy=usefifo
Enables the FIFO. (default)
floppy=<threshold>,fifo_depth
Sets the FIFO threshold. This is mostly relevant in DMA
mode. If this is higher, the floppy driver tolerates more
interrupt latency, but it triggers more interrupts (i.e. it
imposes more load on the rest of the system). If this is
lower, the interrupt latency should be lower too (faster
processor). The benefit of a lower threshold is less
interrupts.
To tune the fifo threshold, switch on over/underrun messages
using 'floppycontrol --messages'. Then access a floppy
disk. If you get a huge amount of "Over/Underrun - retrying"
messages, then the fifo threshold is too low. Try with a
higher value, until you only get an occasional Over/Underrun.
It is a good idea to compile the floppy driver as a module
when doing this tuning. Indeed, it allows to try different
fifo values without rebooting the machine for each test. Note
that you need to do 'floppycontrol --messages' every time you
re-insert the module.
Usually, tuning the fifo threshold should not be needed, as
the default (0xa) is reasonable.
floppy=<drive>,<type>,cmos
Sets the CMOS type of <drive> to <type>. This is mandatory if
you have more than two floppy drives (only two can be
described in the physical CMOS), or if your BIOS uses
non-standard CMOS types. The CMOS types are:
== ==================================
0 Use the value of the physical CMOS
1 5 1/4 DD
2 5 1/4 HD
3 3 1/2 DD
4 3 1/2 HD
5 3 1/2 ED
6 3 1/2 ED
16 unknown or not installed
== ==================================
(Note: there are two valid types for ED drives. This is because 5 was
initially chosen to represent floppy *tapes*, and 6 for ED drives.
AMI ignored this, and used 5 for ED drives. That's why the floppy
driver handles both.)
floppy=unexpected_interrupts
Print a warning message when an unexpected interrupt is received.
(default)
floppy=no_unexpected_interrupts / floppy=L40SX
Don't print a message when an unexpected interrupt is received. This
is needed on IBM L40SX laptops in certain video modes. (There seems
to be an interaction between video and floppy. The unexpected
interrupts affect only performance, and can be safely ignored.)
floppy=broken_dcl
Don't use the disk change line, but assume that the disk was
changed whenever the device node is reopened. Needed on some
boxes where the disk change line is broken or unsupported.
This should be regarded as a stopgap measure, indeed it makes
floppy operation less efficient due to unneeded cache
flushings, and slightly more unreliable. Please verify your
cable, connection and jumper settings if you have any DCL
problems. However, some older drives, and also some laptops
are known not to have a DCL.
floppy=debug
Print debugging messages.
floppy=messages
Print informational messages for some operations (disk change
notifications, warnings about over and underruns, and about
autodetection).
floppy=silent_dcl_clear
Uses a less noisy way to clear the disk change line (which
doesn't involve seeks). Implied by 'daring' option.
floppy=<nr>,irq
Sets the floppy IRQ to <nr> instead of 6.
floppy=<nr>,dma
Sets the floppy DMA channel to <nr> instead of 2.
floppy=slow
Use PS/2 stepping rate::
PS/2 floppies have much slower step rates than regular floppies.
It's been recommended that take about 1/4 of the default speed
in some more extreme cases.
Supporting utilities and additional documentation:
==================================================
Additional parameters of the floppy driver can be configured at
runtime. Utilities which do this can be found in the fdutils package.
This package also contains a new version of mtools which allows to
access high capacity disks (up to 1992K on a high density 3 1/2 disk!).
It also contains additional documentation about the floppy driver.
The latest version can be found at fdutils homepage:
https://fdutils.linux.lu
The fdutils releases can be found at:
https://fdutils.linux.lu/download.html
http://www.tux.org/pub/knaff/fdutils/
ftp://metalab.unc.edu/pub/Linux/utils/disk-management/
Reporting problems about the floppy driver
==========================================
If you have a question or a bug report about the floppy driver, mail
me at Alain.Knaff@poboxes.com . If you post to Usenet, preferably use
comp.os.linux.hardware. As the volume in these groups is rather high,
be sure to include the word "floppy" (or "FLOPPY") in the subject
line. If the reported problem happens when mounting floppy disks, be
sure to mention also the type of the filesystem in the subject line.
Be sure to read the FAQ before mailing/posting any bug reports!
Alain
Changelog
=========
10-30-2004 :
Cleanup, updating, add reference to module configuration.
James Nelson <james4765@gmail.com>
6-3-2000 :
Original Document
linux-7.2.3/Documentation/admin-guide/blockdev/index.rst
+17
View File
@@ -0,0 +1,17 @@
.. SPDX-License-Identifier: GPL-2.0
=============
Block Devices
=============
.. toctree::
:maxdepth: 1
floppy
nbd
paride
ramdisk
zoned_loop
zram
drbd/index
linux-7.2.3/Documentation/admin-guide/blockdev/nbd.rst
+31
View File
@@ -0,0 +1,31 @@
==================================
Network Block Device (TCP version)
==================================
1) Overview
-----------
What is it: With this compiled in the kernel (or as a module), Linux
can use a remote server as one of its block devices. So every time
the client computer wants to read, e.g., /dev/nb0, it sends a
request over TCP to the server, which will reply with the data read.
This can be used for stations with low disk space (or even diskless)
to borrow disk space from another computer.
Unlike NFS, it is possible to put any filesystem on it, etc.
For more information, or to download the nbd-client and nbd-server
tools, go to https://github.com/NetworkBlockDevice/nbd.
The nbd kernel module need only be installed on the client
system, as the nbd-server is completely in userspace. In fact,
the nbd-server has been successfully ported to other operating
systems, including Windows.
A) NBD parameters
-----------------
max_part
Number of partitions per device (default: 0).
nbds_max
Number of block devices that should be initialized (default: 16).
linux-7.2.3/Documentation/admin-guide/blockdev/paride.rst
+207
View File
@@ -0,0 +1,207 @@
===================================
Linux and parallel port IDE devices
===================================
PARIDE v1.03 (c) 1997-8 Grant Guenther <grant@torque.net>
PATA_PARPORT (c) 2023 Ondrej Zary
1. Introduction
===============
Owing to the simplicity and near universality of the parallel port interface
to personal computers, many external devices such as portable hard-disk,
CD-ROM, LS-120 and tape drives use the parallel port to connect to their
host computer. While some devices (notably scanners) use ad-hoc methods
to pass commands and data through the parallel port interface, most
external devices are actually identical to an internal model, but with
a parallel-port adapter chip added in. Some of the original parallel port
adapters were little more than mechanisms for multiplexing a SCSI bus.
(The Iomega PPA-3 adapter used in the ZIP drives is an example of this
approach). Most current designs, however, take a different approach.
The adapter chip reproduces a small ISA or IDE bus in the external device
and the communication protocol provides operations for reading and writing
device registers, as well as data block transfer functions. Sometimes,
the device being addressed via the parallel cable is a standard SCSI
controller like an NCR 5380. The "ditto" family of external tape
drives use the ISA replicator to interface a floppy disk controller,
which is then connected to a floppy-tape mechanism. The vast majority
of external parallel port devices, however, are now based on standard
IDE type devices, which require no intermediate controller. If one
were to open up a parallel port CD-ROM drive, for instance, one would
find a standard ATAPI CD-ROM drive, a power supply, and a single adapter
that interconnected a standard PC parallel port cable and a standard
IDE cable. It is usually possible to exchange the CD-ROM device with
any other device using the IDE interface.
The document describes the support in Linux for parallel port IDE
devices. It does not cover parallel port SCSI devices, "ditto" tape
drives or scanners. Many different devices are supported by the
parallel port IDE subsystem, including:
- MicroSolutions backpack CD-ROM
- MicroSolutions backpack PD/CD
- MicroSolutions backpack hard-drives
- MicroSolutions backpack 8000t tape drive
- SyQuest EZ-135, EZ-230 & SparQ drives
- Avatar Shark
- Imation Superdisk LS-120
- Maxell Superdisk LS-120
- FreeCom Power CD
- Hewlett-Packard 5GB and 8GB tape drives
- Hewlett-Packard 7100 and 7200 CD-RW drives
as well as most of the clone and no-name products on the market.
To support such a wide range of devices, pata_parport is actually structured
in two parts. There is a base pata_parport module which provides an interface
to kernel libata subsystem, registry and some common methods for accessing
the parallel ports.
The second component is a set of low-level protocol drivers for each of the
parallel port IDE adapter chips. Thanks to the interest and encouragement of
Linux users from many parts of the world, support is available for almost all
known adapter protocols:
==== ====================================== ====
aten ATEN EH-100 (HK)
bpck Microsolutions backpack (US)
comm DataStor (old-type) "commuter" adapter (TW)
dstr DataStor EP-2000 (TW)
epat Shuttle EPAT (UK)
epia Shuttle EPIA (UK)
fit2 FIT TD-2000 (US)
fit3 FIT TD-3000 (US)
friq Freecom IQ cable (DE)
frpw Freecom Power (DE)
kbic KingByte KBIC-951A and KBIC-971A (TW)
ktti KT Technology PHd adapter (SG)
on20 OnSpec 90c20 (US)
on26 OnSpec 90c26 (US)
==== ====================================== ====
2. Using pata_parport subsystem
===============================
While configuring the Linux kernel, you may choose either to build
the pata_parport drivers into your kernel, or to build them as modules.
In either case, you will need to select "Parallel port IDE device support"
and at least one of the parallel port communication protocols.
If you do not know what kind of parallel port adapter is used in your drive,
you could begin by checking the file names and any text files on your DOS
installation floppy. Alternatively, you can look at the markings on
the adapter chip itself. That's usually sufficient to identify the
correct device.
You can actually select all the protocol modules, and allow the pata_parport
subsystem to try them all for you.
For the "brand-name" products listed above, here are the protocol
and high-level drivers that you would use:
================ ============ ========
Manufacturer Model Protocol
================ ============ ========
MicroSolutions CD-ROM bpck
MicroSolutions PD drive bpck
MicroSolutions hard-drive bpck
MicroSolutions 8000t tape bpck
SyQuest EZ, SparQ epat
Imation Superdisk epat
Maxell Superdisk friq
Avatar Shark epat
FreeCom CD-ROM frpw
Hewlett-Packard 5GB Tape epat
Hewlett-Packard 7200e (CD) epat
Hewlett-Packard 7200e (CD-R) epat
================ ============ ========
All parports and all protocol drivers are probed automatically unless probe=0
parameter is used. So just "modprobe epat" is enough for an Imation SuperDisk
drive to work.
Manual device creation::
# echo "port protocol mode unit delay" >/sys/bus/pata_parport/new_device
where:
======== ================================================
port parport name (or "auto" for all parports)
protocol protocol name (or "auto" for all protocols)
mode mode number (protocol-specific) or -1 for probe
unit unit number (for backpack only, see below)
delay I/O delay (see troubleshooting section below)
======== ================================================
If you happen to be using a MicroSolutions backpack device, you will
also need to know the unit ID number for each drive. This is usually
the last two digits of the drive's serial number (but read MicroSolutions'
documentation about this).
If you omit the parameters from the end, defaults will be used, e.g.:
Probe all parports with all protocols::
# echo auto >/sys/bus/pata_parport/new_device
Probe parport0 using protocol epat and mode 4 (EPP-16)::
# echo "parport0 epat 4" >/sys/bus/pata_parport/new_device
Probe parport0 using all protocols::
# echo "parport0 auto" >/sys/bus/pata_parport/new_device
Probe all parports using protoocol epat::
# echo "auto epat" >/sys/bus/pata_parport/new_device
Deleting devices::
# echo pata_parport.0 >/sys/bus/pata_parport/delete_device
3. Troubleshooting
==================
3.1 Use EPP mode if you can
----------------------------
The most common problems that people report with the pata_parport drivers
concern the parallel port CMOS settings. At this time, none of the
protocol modules support ECP mode, or any ECP combination modes.
If you are able to do so, please set your parallel port into EPP mode
using your CMOS setup procedure.
3.2 Check the port delay
-------------------------
Some parallel ports cannot reliably transfer data at full speed. To
offset the errors, the protocol modules introduce a "port
delay" between each access to the i/o ports. Each protocol sets
a default value for this delay. In most cases, the user can override
the default and set it to 0 - resulting in somewhat higher transfer
rates. In some rare cases (especially with older 486 systems) the
default delays are not long enough. if you experience corrupt data
transfers, or unexpected failures, you may wish to increase the
port delay.
3.3 Some drives need a printer reset
-------------------------------------
There appear to be a number of "noname" external drives on the market
that do not always power up correctly. We have noticed this with some
drives based on OnSpec and older Freecom adapters. In these rare cases,
the adapter can often be reinitialised by issuing a "printer reset" on
the parallel port. As the reset operation is potentially disruptive in
multiple device environments, the pata_parport drivers will not do it
automatically. You can however, force a printer reset by doing::
insmod lp reset=1
rmmod lp
If you have one of these marginal cases, you should probably build
your pata_parport drivers as modules, and arrange to do the printer reset
before loading the pata_parport drivers.
linux-7.2.3/Documentation/admin-guide/blockdev/ramdisk.rst
+153
View File
@@ -0,0 +1,153 @@
==========================================
Using the RAM disk block device with Linux
==========================================
.. Contents:
1) Overview
2) Kernel Command Line Parameters
3) Using "rdev"
4) An Example of Creating a Compressed RAM Disk
1) Overview
-----------
The RAM disk driver is a way to use main system memory as a block device. It
is required for initrd, an initial filesystem used if you need to load modules
in order to access the root filesystem (see Documentation/admin-guide/initrd.rst). It can
also be used for a temporary filesystem for crypto work, since the contents
are erased on reboot.
The RAM disk dynamically grows as more space is required. It does this by using
RAM from the buffer cache. The driver marks the buffers it is using as dirty
so that the VM subsystem does not try to reclaim them later.
The RAM disk supports up to 16 RAM disks by default, and can be reconfigured
to support an unlimited number of RAM disks (at your own risk). Just change
the configuration symbol BLK_DEV_RAM_COUNT in the Block drivers config menu
and (re)build the kernel.
To use RAM disk support with your system, run './MAKEDEV ram' from the /dev
directory. RAM disks are all major number 1, and start with minor number 0
for /dev/ram0, etc. If used, modern kernels use /dev/ram0 for an initrd.
The new RAM disk also has the ability to load compressed RAM disk images,
allowing one to squeeze more programs onto an average installation or
rescue floppy disk.
2) Parameters
---------------------------------
2a) Kernel Command Line Parameters
ramdisk_size=N
Size of the ramdisk.
This parameter tells the RAM disk driver to set up RAM disks of N k size. The
default is 4096 (4 MB).
2b) Module parameters
rd_nr
/dev/ramX devices created.
max_part
Maximum partition number.
rd_size
See ramdisk_size.
3) Using "rdev"
---------------
"rdev" is an obsolete, deprecated, antiquated utility that could be used
to set the boot device in a Linux kernel image.
Instead of using rdev, just place the boot device information on the
kernel command line and pass it to the kernel from the bootloader.
You can also pass arguments to the kernel by setting FDARGS in
arch/x86/boot/Makefile and specify in initrd image by setting FDINITRD in
arch/x86/boot/Makefile.
Some of the kernel command line boot options that may apply here are::
ramdisk_start=N
ramdisk_size=M
If you make a boot disk that has LILO, then for the above, you would use::
append = "ramdisk_start=N ramdisk_size=M"
4) An Example of Creating a Compressed RAM Disk
-----------------------------------------------
To create a RAM disk image, you will need a spare block device to
construct it on. This can be the RAM disk device itself, or an
unused disk partition (such as an unmounted swap partition). For this
example, we will use the RAM disk device, "/dev/ram0".
Note: This technique should not be done on a machine with less than 8 MB
of RAM. If using a spare disk partition instead of /dev/ram0, then this
restriction does not apply.
a) Decide on the RAM disk size that you want. Say 2 MB for this example.
Create it by writing to the RAM disk device. (This step is not currently
required, but may be in the future.) It is wise to zero out the
area (esp. for disks) so that maximal compression is achieved for
the unused blocks of the image that you are about to create::
dd if=/dev/zero of=/dev/ram0 bs=1k count=2048
b) Make a filesystem on it. Say ext2fs for this example::
mke2fs -vm0 /dev/ram0 2048
c) Mount it, copy the files you want to it (eg: /etc/* /dev/* ...)
and unmount it again.
d) Compress the contents of the RAM disk. The level of compression
will be approximately 50% of the space used by the files. Unused
space on the RAM disk will compress to almost nothing::
dd if=/dev/ram0 bs=1k count=2048 | gzip -v9 > /tmp/ram_image.gz
e) Put the kernel onto the floppy::
dd if=zImage of=/dev/fd0 bs=1k
f) Put the RAM disk image onto the floppy, after the kernel. Use an offset
that is slightly larger than the kernel, so that you can put another
(possibly larger) kernel onto the same floppy later without overlapping
the RAM disk image. An offset of 400 kB for kernels about 350 kB in
size would be reasonable. Make sure offset+size of ram_image.gz is
not larger than the total space on your floppy (usually 1440 kB)::
dd if=/tmp/ram_image.gz of=/dev/fd0 bs=1k seek=400
g) Make sure that you have already specified the boot information in
FDARGS and FDINITRD or that you use a bootloader to pass kernel
command line boot options to the kernel.
That is it. You now have your boot/root compressed RAM disk floppy. Some
users may wish to combine steps (d) and (f) by using a pipe.
Paul Gortmaker 12/95
Changelog:
----------
SEPT-2020 :
Removed usage of "rdev"
10-22-04 :
Updated to reflect changes in command line options, remove
obsolete references, general cleanup.
James Nelson (james4765@gmail.com)
12-95 :
Original Document
linux-7.2.3/Documentation/admin-guide/blockdev/zoned_loop.rst
+190
View File
@@ -0,0 +1,190 @@
.. SPDX-License-Identifier: GPL-2.0
=======================
Zoned Loop Block Device
=======================
.. Contents:
1) Overview
2) Creating a Zoned Device
3) Deleting a Zoned Device
4) Example
1) Overview
-----------
The zoned loop block device driver (zloop) allows a user to create a zoned block
device using one regular file per zone as backing storage. This driver does not
directly control any hardware and uses read, write and truncate operations to
regular files of a file system to emulate a zoned block device.
Using zloop, zoned block devices with a configurable capacity, zone size and
number of conventional zones can be created. The storage for each zone of the
device is implemented using a regular file with a maximum size equal to the zone
size. The size of a file backing a conventional zone is always equal to the zone
size. The size of a file backing a sequential zone indicates the amount of data
sequentially written to the file, that is, the size of the file directly
indicates the position of the write pointer of the zone.
When resetting a sequential zone, its backing file size is truncated to zero.
Conversely, for a zone finish operation, the backing file is truncated to the
zone size. With this, the maximum capacity of a zloop zoned block device created
can be larger configured to be larger than the storage space available on the
backing file system. Of course, for such configuration, writing more data than
the storage space available on the backing file system will result in write
errors.
The zoned loop block device driver implements a complete zone transition state
machine. That is, zones can be empty, implicitly opened, explicitly opened,
closed or full. The current implementation does not support any limits on the
maximum number of open and active zones.
No user tools are necessary to create and delete zloop devices.
2) Creating a Zoned Device
--------------------------
Once the zloop module is loaded (or if zloop is compiled in the kernel), the
character device file /dev/zloop-control can be used to add a zloop device.
This is done by writing an "add" command directly to the /dev/zloop-control
device::
$ modprobe zloop
$ ls -l /dev/zloop*
crw-------. 1 root root 10, 123 Jan 6 19:18 /dev/zloop-control
$ mkdir -p <base directory/<device ID>
$ echo "add [options]" > /dev/zloop-control
The options available for the add command can be listed by reading the
/dev/zloop-control device::
$ cat /dev/zloop-control
add id=%d,capacity_mb=%u,zone_size_mb=%u,zone_capacity_mb=%u,conv_zones=%u,max_open_zones=%u,base_dir=%s,nr_queues=%u,queue_depth=%u,buffered_io,zone_append=%u,ordered_zone_append,discard_write_cache
remove id=%d
In more details, the options that can be used with the "add" command are as
follows.
=================== =========================================================
id Device number (the X in /dev/zloopX).
Default: automatically assigned.
capacity_mb Device total capacity in MiB. This is always rounded up
to the nearest higher multiple of the zone size.
Default: 16384 MiB (16 GiB).
zone_size_mb Device zone size in MiB. Default: 256 MiB.
zone_capacity_mb Device zone capacity (must always be equal to or lower
than the zone size. Default: zone size.
conv_zones Total number of conventioanl zones starting from
sector 0
Default: 8
max_open_zones Maximum number of open sequential write required zones
(0 for no limit).
Default: 0
base_dir Path to the base directory where to create the directory
containing the zone files of the device.
Default=/var/local/zloop.
The device directory containing the zone files is always
named with the device ID. E.g. the default zone file
directory for /dev/zloop0 is /var/local/zloop/0.
nr_queues Number of I/O queues of the zoned block device. This
value is always capped by the number of online CPUs
Default: 1
queue_depth Maximum I/O queue depth per I/O queue.
Default: 64
buffered_io Do buffered IOs instead of direct IOs (default: false)
zone_append Enable or disable a zloop device native zone append
support.
Default: 1 (enabled).
If native zone append support is disabled, the block layer
will emulate this operation using regular write
operations.
ordered_zone_append Enable zloop mitigation of zone append reordering.
Default: disabled.
This is useful for testing file systems file data mapping
(extents), as when enabled, this can significantly reduce
the number of data extents needed to for a file data
mapping.
discard_write_cache Discard all data that was not explicitly persisted using a
flush operation when the device is removed by truncating
each zone file to the size recorded during the last flush
operation. This simulates power fail events where
uncommitted data is lost.
=================== =========================================================
3) Deleting a Zoned Device
--------------------------
Deleting an unused zoned loop block device is done by issuing the "remove"
command to /dev/zloop-control, specifying the ID of the device to remove::
$ echo "remove id=X" > /dev/zloop-control
The remove command does not have any option.
A zoned device that was removed can be re-added again without any change to the
state of the device zones: the device zones are restored to their last state
before the device was removed. Adding again a zoned device after it was removed
must always be done using the same configuration as when the device was first
added. If a zone configuration change is detected, an error will be returned and
the zoned device will not be created.
To fully delete a zoned device, after executing the remove operation, the device
base directory containing the backing files of the device zones must be deleted.
4) Example
----------
The following sequence of commands creates a 2GB zoned device with zones of 64
MB and a zone capacity of 63 MB::
$ modprobe zloop
$ mkdir -p /var/local/zloop/0
$ echo "add capacity_mb=2048,zone_size_mb=64,zone_capacity_mb=63" > /dev/zloop-control
For the device created (/dev/zloop0), the zone backing files are all created
under the default base directory (/var/local/zloop)::
$ ls -l /var/local/zloop/0
total 0
-rw-------. 1 root root 67108864 Jan 6 22:23 cnv-000000
-rw-------. 1 root root 67108864 Jan 6 22:23 cnv-000001
-rw-------. 1 root root 67108864 Jan 6 22:23 cnv-000002
-rw-------. 1 root root 67108864 Jan 6 22:23 cnv-000003
-rw-------. 1 root root 67108864 Jan 6 22:23 cnv-000004
-rw-------. 1 root root 67108864 Jan 6 22:23 cnv-000005
-rw-------. 1 root root 67108864 Jan 6 22:23 cnv-000006
-rw-------. 1 root root 67108864 Jan 6 22:23 cnv-000007
-rw-------. 1 root root 0 Jan 6 22:23 seq-000008
-rw-------. 1 root root 0 Jan 6 22:23 seq-000009
...
The zoned device created (/dev/zloop0) can then be used normally::
$ lsblk -z
NAME ZONED ZONE-SZ ZONE-NR ZONE-AMAX ZONE-OMAX ZONE-APP ZONE-WGRAN
zloop0 host-managed 64M 32 0 0 1M 4K
$ blkzone report /dev/zloop0
start: 0x000000000, len 0x020000, cap 0x020000, wptr 0x000000 reset:0 non-seq:0, zcond: 0(nw) [type: 1(CONVENTIONAL)]
start: 0x000020000, len 0x020000, cap 0x020000, wptr 0x000000 reset:0 non-seq:0, zcond: 0(nw) [type: 1(CONVENTIONAL)]
start: 0x000040000, len 0x020000, cap 0x020000, wptr 0x000000 reset:0 non-seq:0, zcond: 0(nw) [type: 1(CONVENTIONAL)]
start: 0x000060000, len 0x020000, cap 0x020000, wptr 0x000000 reset:0 non-seq:0, zcond: 0(nw) [type: 1(CONVENTIONAL)]
start: 0x000080000, len 0x020000, cap 0x020000, wptr 0x000000 reset:0 non-seq:0, zcond: 0(nw) [type: 1(CONVENTIONAL)]
start: 0x0000a0000, len 0x020000, cap 0x020000, wptr 0x000000 reset:0 non-seq:0, zcond: 0(nw) [type: 1(CONVENTIONAL)]
start: 0x0000c0000, len 0x020000, cap 0x020000, wptr 0x000000 reset:0 non-seq:0, zcond: 0(nw) [type: 1(CONVENTIONAL)]
start: 0x0000e0000, len 0x020000, cap 0x020000, wptr 0x000000 reset:0 non-seq:0, zcond: 0(nw) [type: 1(CONVENTIONAL)]
start: 0x000100000, len 0x020000, cap 0x01f800, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
start: 0x000120000, len 0x020000, cap 0x01f800, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
...
Deleting this device is done using the command::
$ echo "remove id=0" > /dev/zloop-control
The removed device can be re-added again using the same "add" command as when
the device was first created. To fully delete a zoned device, its backing files
should also be deleted after executing the remove command::
$ rm -r /var/local/zloop/0
linux-7.2.3/Documentation/admin-guide/blockdev/zram.rst
+578
View File
@@ -0,0 +1,578 @@
========================================
zram: Compressed RAM-based block devices
========================================
Introduction
============
The zram module creates RAM-based block devices named /dev/zram<id>
(<id> = 0, 1, ...). Pages written to these disks are compressed and stored
in memory itself. These disks allow very fast I/O and compression provides
good amounts of memory savings. Some of the use cases include /tmp storage,
use as swap disks, various caches under /var and maybe many more. :)
Statistics for individual zram devices are exported through sysfs nodes at
/sys/block/zram<id>/
Usage
=====
There are several ways to configure and manage zram device(-s):
a) using zram and zram_control sysfs attributes
b) using zramctl utility, provided by util-linux (util-linux@vger.kernel.org).
In this document we will describe only 'manual' zram configuration steps,
IOW, zram and zram_control sysfs attributes.
In order to get a better idea about zramctl please consult util-linux
documentation, zramctl man-page or `zramctl --help`. Please be informed
that zram maintainers do not develop/maintain util-linux or zramctl, should
you have any questions please contact util-linux@vger.kernel.org
Following shows a typical sequence of steps for using zram.
WARNING
=======
For the sake of simplicity we skip error checking parts in most of the
examples below. However, it is your sole responsibility to handle errors.
zram sysfs attributes always return negative values in case of errors.
The list of possible return codes:
======== =============================================================
-EBUSY an attempt to modify an attribute that cannot be changed once
the device has been initialised. Please reset device first.
-ENOMEM zram was not able to allocate enough memory to fulfil your
needs.
-EINVAL invalid input has been provided.
-EAGAIN re-try operation later (e.g. when attempting to run recompress
and writeback simultaneously).
======== =============================================================
If you use 'echo', the returned value is set by the 'echo' utility,
and, in general case, something like::
echo foo > /sys/block/zram0/comp_algorithm
if [ $? -ne 0 ]; then
handle_error
fi
should suffice.
1) Load Module
==============
::
modprobe zram num_devices=4
This creates 4 devices: /dev/zram{0,1,2,3}
num_devices parameter is optional and tells zram how many devices should be
pre-created. Default: 1.
2) Select compression algorithm
===============================
Using comp_algorithm device attribute one can see available and
currently selected (shown in square brackets) compression algorithms,
or change the selected compression algorithm (once the device is initialised
there is no way to change compression algorithm).
Examples::
#show supported compression algorithms
cat /sys/block/zram0/comp_algorithm
lzo [lz4]
#select lzo compression algorithm
echo lzo > /sys/block/zram0/comp_algorithm
For the time being, the `comp_algorithm` content shows only compression
algorithms that are supported by zram.
3) Set compression algorithm parameters: Optional
=================================================
Compression algorithms may support specific parameters which can be
tweaked for particular dataset. ZRAM has an `algorithm_params` device
attribute which provides a per-algorithm params configuration.
For example, several compression algorithms support `level` parameter.
In addition, certain compression algorithms support pre-trained dictionaries,
which significantly change algorithms' characteristics. In order to configure
compression algorithm to use external pre-trained dictionary, pass full
path to the `dict` along with other parameters::
#pass path to pre-trained zstd dictionary
echo "algo=zstd dict=/etc/dictionary" > /sys/block/zram0/algorithm_params
#same, but using algorithm priority
echo "priority=1 dict=/etc/dictionary" > \
/sys/block/zram0/algorithm_params
#pass path to pre-trained zstd dictionary and compression level
echo "algo=zstd level=8 dict=/etc/dictionary" > \
/sys/block/zram0/algorithm_params
Parameters are algorithm specific: not all algorithms support pre-trained
dictionaries, not all algorithms support `level`. Furthermore, for certain
algorithms `level` controls the compression level (the higher the value the
better the compression ratio, it even can take negatives values for some
algorithms), for other algorithms `level` is acceleration level (the higher
the value the lower the compression ratio).
4) Set Disksize
===============
Set disk size by writing the value to sysfs node 'disksize'.
The value can be either in bytes or you can use mem suffixes.
Examples::
# Initialize /dev/zram0 with 50MB disksize
echo $((50*1024*1024)) > /sys/block/zram0/disksize
# Using mem suffixes
echo 256K > /sys/block/zram0/disksize
echo 512M > /sys/block/zram0/disksize
echo 1G > /sys/block/zram0/disksize
Note:
There is little point creating a zram of greater than twice the size of memory
since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the
size of the disk when not in use so a huge zram is wasteful.
5) Set memory limit: Optional
=============================
Set memory limit by writing the value to sysfs node 'mem_limit'.
The value can be either in bytes or you can use mem suffixes.
In addition, you could change the value in runtime.
Examples::
# limit /dev/zram0 with 50MB memory
echo $((50*1024*1024)) > /sys/block/zram0/mem_limit
# Using mem suffixes
echo 256K > /sys/block/zram0/mem_limit
echo 512M > /sys/block/zram0/mem_limit
echo 1G > /sys/block/zram0/mem_limit
# To disable memory limit
echo 0 > /sys/block/zram0/mem_limit
6) Activate
===========
::
mkswap /dev/zram0
swapon /dev/zram0
mkfs.ext4 /dev/zram1
mount /dev/zram1 /tmp
7) Add/remove zram devices
==========================
zram provides a control interface, which enables dynamic (on-demand) device
addition and removal.
In order to add a new /dev/zramX device, perform a read operation on the hot_add
attribute. This will return either the new device's device id (meaning that you
can use /dev/zram<id>) or an error code.
Example::
cat /sys/class/zram-control/hot_add
1
To remove the existing /dev/zramX device (where X is a device id)
execute::
echo X > /sys/class/zram-control/hot_remove
8) Stats
========
Per-device statistics are exported as various nodes under /sys/block/zram<id>/
A brief description of exported device attributes follows. For more details
please read Documentation/ABI/testing/sysfs-block-zram.
====================== ====== ===============================================
Name access description
====================== ====== ===============================================
disksize RW show and set the device's disk size
initstate RO shows the initialization state of the device
reset WO trigger device reset
mem_used_max WO reset the `mem_used_max` counter (see later)
mem_limit WO specifies the maximum amount of memory ZRAM can
use to store the compressed data
writeback_limit WO specifies the maximum amount of write IO zram
can write out to backing device as 4KB unit
writeback_limit_enable RW show and set writeback_limit feature
writeback_batch_size RW show and set maximum number of in-flight
writeback operations
compressed_writeback RW show and set compressed writeback feature
comp_algorithm RW show and change the compression algorithm
algorithm_params WO setup compression algorithm parameters
compact WO trigger memory compaction
debug_stat RO this file is used for zram debugging purposes
backing_dev RW set up backend storage for zram to write out
idle WO mark allocated slot as idle
====================== ====== ===============================================
User space is advised to use the following files to read the device statistics.
File /sys/block/zram<id>/stat
Represents block layer statistics. Read Documentation/block/stat.rst for
details.
File /sys/block/zram<id>/io_stat
The stat file represents device's I/O statistics not accounted by block
layer and, thus, not available in zram<id>/stat file. It consists of a
single line of text and contains the following stats separated by
whitespace:
============= =============================================================
failed_reads The number of failed reads
failed_writes The number of failed writes
invalid_io The number of non-page-size-aligned I/O requests
notify_free Depending on device usage scenario it may account
a) the number of pages freed because of swap slot free
notifications
b) the number of pages freed because of
REQ_OP_DISCARD requests sent by bio. The former ones are
sent to a swap block device when a swap slot is freed,
which implies that this disk is being used as a swap disk.
The latter ones are sent by filesystem mounted with
discard option, whenever some data blocks are getting
discarded.
============= =============================================================
File /sys/block/zram<id>/mm_stat
The mm_stat file represents the device's mm statistics. It consists of a single
line of text and contains the following stats separated by whitespace:
================ =============================================================
orig_data_size uncompressed size of data stored in this disk.
Unit: bytes
compr_data_size compressed size of data stored in this disk
mem_used_total the amount of memory allocated for this disk. This
includes allocator fragmentation and metadata overhead,
allocated for this disk. So, allocator space efficiency
can be calculated using compr_data_size and this statistic.
Unit: bytes
mem_limit the maximum amount of memory ZRAM can use to store
the compressed data
mem_used_max the maximum amount of memory zram has consumed to
store the data
same_pages the number of same element filled pages written to this disk.
No memory is allocated for such pages.
pages_compacted the number of pages freed during compaction
huge_pages the number of incompressible pages
huge_pages_since the number of incompressible pages since zram set up
================ =============================================================
File /sys/block/zram<id>/bd_stat
The bd_stat file represents a device's backing device statistics. It consists of
a single line of text and contains the following stats separated by whitespace:
============== =============================================================
bd_count size of data written in backing device.
Unit: 4K bytes
bd_reads the number of reads from backing device
Unit: 4K bytes
bd_writes the number of writes to backing device
Unit: 4K bytes
============== =============================================================
9) Deactivate
==============
::
swapoff /dev/zram0
umount /dev/zram1
10) Reset
=========
Write any positive value to 'reset' sysfs node::
echo 1 > /sys/block/zram0/reset
echo 1 > /sys/block/zram1/reset
This frees all the memory allocated for the given device and
resets the disksize to zero. You must set the disksize again
before reusing the device.
Optional Feature
================
IDLE pages tracking
-------------------
zram has built-in support for idle pages tracking (that is, allocated but
not used pages). This feature is useful for e.g. zram writeback and
recompression. In order to mark pages as idle, execute the following command::
echo all > /sys/block/zramX/idle
This will mark all allocated zram pages as idle. The idle mark will be
removed only when the page (block) is accessed (e.g. overwritten or freed).
Additionally, when CONFIG_ZRAM_TRACK_ENTRY_ACTIME is enabled, pages can be
marked as idle based on how many seconds have passed since the last access to
a particular zram page::
echo 86400 > /sys/block/zramX/idle
In this example, all pages which haven't been accessed in more than 86400
seconds (one day) will be marked idle.
writeback
---------
With CONFIG_ZRAM_WRITEBACK, zram can write idle/incompressible page
to backing storage rather than keeping it in memory.
To use the feature, admin should set up backing device via::
echo /dev/sda5 > /sys/block/zramX/backing_dev
before disksize setting. It supports only partitions at this moment.
If admin wants to use incompressible page writeback, they could do it via::
echo huge > /sys/block/zramX/writeback
Admin can request writeback of idle pages at right timing via::
echo idle > /sys/block/zramX/writeback
With the command, zram will writeback idle pages from memory to the storage.
Additionally, if a user choose to writeback only huge and idle pages
this can be accomplished with::
echo huge_idle > /sys/block/zramX/writeback
If a user chooses to writeback only incompressible pages (pages that none of
algorithms can compress) this can be accomplished with::
echo incompressible > /sys/block/zramX/writeback
If an admin wants to write a specific page in zram device to the backing device,
they could write a page index into the interface::
echo "page_index=1251" > /sys/block/zramX/writeback
In Linux 6.16 this interface underwent some rework. First, the interface
now supports `key=value` format for all of its parameters (`type=huge_idle`,
etc.) Second, the support for `page_indexes` was introduced, which specify
`LOW-HIGH` range (or ranges) of pages to be written-back. This reduces the
number of syscalls, but more importantly this enables optimal post-processing
target selection strategy. Usage example::
echo "type=idle" > /sys/block/zramX/writeback
echo "page_indexes=1-100 page_indexes=200-300" > \
/sys/block/zramX/writeback
We also now permit multiple page_index params per call and a mix of
single pages and page ranges::
echo page_index=42 page_index=99 page_indexes=100-200 \
page_indexes=500-700 > /sys/block/zramX/writeback
If there are lots of write IO with flash device, potentially, it has
flash wearout problem so that admin needs to design write limitation
to guarantee storage health for entire product life.
To overcome the concern, zram supports "writeback_limit" feature.
The "writeback_limit_enable"'s default value is 0 so that it doesn't limit
any writeback. IOW, if admin wants to apply writeback budget, they should
enable writeback_limit_enable via::
$ echo 1 > /sys/block/zramX/writeback_limit_enable
Once writeback_limit_enable is set, zram doesn't allow any writeback
until admin sets the budget via /sys/block/zramX/writeback_limit.
(If admin doesn't enable writeback_limit_enable, writeback_limit's value
assigned via /sys/block/zramX/writeback_limit is meaningless.)
If admin wants to limit writeback as per-day 400M, they could do it
like below::
$ MB_SHIFT=20
$ 4K_SHIFT=12
$ echo $((400<<MB_SHIFT>>4K_SHIFT)) > \
/sys/block/zram0/writeback_limit.
$ echo 1 > /sys/block/zram0/writeback_limit_enable
If admins want to allow further write again once the budget is exhausted,
they could do it like below::
$ echo $((400<<MB_SHIFT>>4K_SHIFT)) > \
/sys/block/zram0/writeback_limit
If an admin wants to see the remaining writeback budget since last set::
$ cat /sys/block/zramX/writeback_limit
If an admin wants to disable writeback limit, they could do::
$ echo 0 > /sys/block/zramX/writeback_limit_enable
The writeback_limit count will reset whenever you reset zram (e.g.,
system reboot, echo 1 > /sys/block/zramX/reset) so keeping how many of
writeback happened until you reset the zram to allocate extra writeback
budget in next setting is user's job.
By default zram stores written back pages in decompressed (raw) form, which
means that writeback operation involves decompression of the page before
writing it to the backing device. This behavior can be changed by enabling
`compressed_writeback` feature, which causes zram to write compressed pages
to the backing device, thus avoiding decompression overhead. To enable
this feature, execute::
$ echo yes > /sys/block/zramX/compressed_writeback
Note that this feature should be configured before the `zramX` device is
initialized.
Depending on backing device storage type, writeback operation may benefit
from a higher number of in-flight write requests (batched writes). The
number of maximum in-flight writeback operations can be configured via
`writeback_batch_size` attribute. To change the default value (which is 32),
execute::
$ echo 64 > /sys/block/zramX/writeback_batch_size
If admin wants to measure writeback count in a certain period, they could
know it via /sys/block/zram0/bd_stat's 3rd column.
recompression
-------------
With `CONFIG_ZRAM_MULTI_COMP`, zram can recompress pages using alternative
(secondary) compression algorithms. The basic idea is that alternative
compression algorithm can provide better compression ratio at a price of
(potentially) slower compression/decompression speeds. Alternative compression
algorithm can, for example, be more successful compressing huge pages (those
that default algorithm failed to compress). Another application is idle pages
recompression - pages that are cold and sit in the memory can be recompressed
using more effective algorithm and, hence, reduce zsmalloc memory usage.
With `CONFIG_ZRAM_MULTI_COMP`, zram supports up to 4 compression algorithms:
one primary and up to 3 secondary ones. Primary zram compressor is explained
in "3) Select compression algorithm", secondary algorithms are configured
using recomp_algorithm device attribute.
Example:::
#show supported recompression algorithms
cat /sys/block/zramX/recomp_algorithm
#1: lzo lzo-rle lz4 lz4hc [zstd]
#2: lzo lzo-rle lz4 [lz4hc] zstd
Alternative compression algorithms are sorted by priority. In the example
above, zstd is used as the first alternative algorithm, which has priority
of 1, while lz4hc is configured as a compression algorithm with priority 2.
Alternative compression algorithm's priority is provided during algorithms
configuration:::
#select zstd recompression algorithm, priority 1
echo "algo=zstd priority=1" > /sys/block/zramX/recomp_algorithm
#select deflate recompression algorithm, priority 2
echo "algo=deflate priority=2" > /sys/block/zramX/recomp_algorithm
Another device attribute that `CONFIG_ZRAM_MULTI_COMP` enables is `recompress`,
which controls recompression.
Examples:::
#IDLE pages recompression is activated by `idle` mode
echo "type=idle priority=1" > /sys/block/zramX/recompress
#HUGE pages recompression is activated by `huge` mode
echo "type=huge priority=2" > /sys/block/zram0/recompress
#HUGE_IDLE pages recompression is activated by `huge_idle` mode
echo "type=huge_idle priority=1" > /sys/block/zramX/recompress
The number of idle pages can be significant, so user-space can pass a size
threshold (in bytes) to the recompress knob: zram will recompress only pages
of equal or greater size:::
#recompress all pages larger than 3000 bytes
echo "threshold=3000 priority=1" > /sys/block/zramX/recompress
#recompress idle pages larger than 2000 bytes
echo "type=idle threshold=2000 priority=1" > \
/sys/block/zramX/recompress
It is also possible to limit the number of pages zram re-compression will
attempt to recompress:::
echo "type=huge_idle priority=1 max_pages=42" > \
/sys/block/zramX/recompress
It is advised to always specify `priority` parameter. While it is also
possible to specify `algo` parameter, so that `zram` will use algorithm's
name to determine the priority, it is not recommended, since it can lead to
unexpected results when the same algorithm is configured with different
priorities (e.g. different parameters). `priority` is the only way to
guarantee that the expected algorithm will be used.
memory tracking
===============
With CONFIG_ZRAM_MEMORY_TRACKING, user can know information of the
zram block. It could be useful to catch cold or incompressible
pages of the process with*pagemap.
If you enable the feature, you could see block state via
/sys/kernel/debug/zram/zram0/block_state". The output is as follows::
300 75.033841 .wh...
301 63.806904 s.....
302 63.806919 ..hi..
303 62.801919 ....r.
304 146.781902 ..hi.n
First column
zram's block index.
Second column
access time since the system was booted
Third column
state of the block:
s:
same page
w:
written page to backing store
h:
huge page
i:
idle page
r:
recompressed page (secondary compression algorithm)
n:
none (including secondary) of algorithms could compress it
First line of above example says 300th block is accessed at 75.033841sec
and the block's state is huge so it is written back to the backing
storage. It's a debugging feature so anyone shouldn't rely on it to work
properly.
Nitin Gupta
ngupta@vflare.org
linux-7.2.3/Documentation/admin-guide/bootconfig.rst
+335
View File
@@ -0,0 +1,335 @@
.. SPDX-License-Identifier: GPL-2.0
.. _bootconfig:
==================
Boot Configuration
==================
:Author: Masami Hiramatsu <mhiramat@kernel.org>
Overview
========
The boot configuration expands the current kernel command line to support
additional key-value data when booting the kernel in an efficient way.
This allows administrators to pass a structured-Key config file.
Config File Syntax
==================
The boot config syntax is a simple structured key-value. Each key consists
of dot-connected-words, and key and value are connected by ``=``. The value
string has to be terminated by the following delimiters described below.
Each key word must contain only alphabets, numbers, dash (``-``) or underscore
(``_``). And each value only contains printable characters or spaces except
for delimiters such as semi-colon (``;``), new-line (``\n``), comma (``,``),
hash (``#``) and closing brace (``}``).
If the ``=`` is followed by whitespace up to one of these delimiters, the
key is assigned an empty value.
For arrays, the array values are comma (``,``) separated, and comments and
line breaks with newline (``\n``) are allowed between array values for
readability. Thus the first entry of the array must be on the same line as
the key.::
KEY[.WORD[...]] = VALUE[, VALUE2[...]][;]
Unlike the kernel command line syntax, white spaces (including tabs) are
ignored around the comma and ``=``.
If you want to use those delimiters in a value, you can use either double-
quotes (``"VALUE"``) or single-quotes (``'VALUE'``) to quote it. Note that
you can not escape these quotes.
There can be a key which doesn't have value or has an empty value. Those keys
are used for checking if the key exists or not (like a boolean).
Key-Value Syntax
----------------
The boot config file syntax allows user to merge partially same word keys
by brace. For example::
foo.bar.baz = value1
foo.bar.qux.quux = value2
These can be written also in::
foo.bar {
baz = value1
qux.quux = value2
}
Or more shorter, written as following::
foo.bar { baz = value1; qux.quux = value2 }
In both styles, same key words are automatically merged when parsing it
at boot time. So you can append similar trees or key-values.
Same-key Values
---------------
It is prohibited that two or more values or arrays share a same-key.
For example,::
foo = bar, baz
foo = qux # !ERROR! we can not re-define same key
If you want to update the value, you must use the override operator
``:=`` explicitly. For example::
foo = bar, baz
foo := qux
then, the ``qux`` is assigned to ``foo`` key. This is useful for
overriding the default value by adding (partial) custom bootconfigs
without parsing the default bootconfig.
If you want to append the value to existing key as an array member,
you can use ``+=`` operator. For example::
foo = bar, baz
foo += qux
In this case, the key ``foo`` has ``bar``, ``baz`` and ``qux``.
Moreover, sub-keys and a value can coexist under a parent key.
For example, following config is allowed.::
foo = value1
foo.bar = value2
foo := value3 # This will update foo's value.
Note, since there is no syntax to put a raw value directly under a
structured key, you have to define it outside of the brace. For example::
foo {
bar = value1
bar {
baz = value2
qux = value3
}
}
Also, the order of the value node under a key is fixed. If there
are a value and subkeys, the value is always the first child node
of the key. Thus if user specifies subkeys first, e.g.::
foo.bar = value1
foo = value2
In the program (and /proc/bootconfig), it will be shown as below::
foo = value2
foo.bar = value1
Comments
--------
The config syntax accepts shell-script style comments. The comments starting
with hash ("#") until newline ("\n") will be ignored.
::
# comment line
foo = value # value is set to foo.
bar = 1, # 1st element
2, # 2nd element
3 # 3rd element
This is parsed as below::
foo = value
bar = 1, 2, 3
Note that you can NOT put a comment or a newline between value and delimiter
(``,`` or ``;``). This means following config has a syntax error ::
key = 1 # comment
,2
/proc/bootconfig
================
/proc/bootconfig is a user-space interface of the boot config.
Unlike /proc/cmdline, this file shows the key-value style list.
Each key-value pair is shown in each line with following style::
KEY[.WORDS...] = "[VALUE]"[,"VALUE2"...]
Boot Kernel With a Boot Config
==============================
There are two options to boot the kernel with bootconfig: attaching the
bootconfig to the initrd image or embedding it in the kernel itself.
Attaching a Boot Config to Initrd
---------------------------------
Since the boot configuration file is loaded with initrd by default,
it will be added to the end of the initrd (initramfs) image file with
padding, size, checksum and 12-byte magic word as below.
[initrd][bootconfig][padding][size(le32)][checksum(le32)][#BOOTCONFIG\n]
The size and checksum fields are unsigned 32bit little endian value.
When the boot configuration is added to the initrd image, the total
file size is aligned to 4 bytes. To fill the gap, null characters
(``\0``) will be added. Thus the ``size`` is the length of the bootconfig
file + padding bytes.
The Linux kernel decodes the last part of the initrd image in memory to
get the boot configuration data.
Because of this "piggyback" method, there is no need to change or
update the boot loader and the kernel image itself as long as the boot
loader passes the correct initrd file size. If by any chance, the boot
loader passes a longer size, the kernel fails to find the bootconfig data.
To do this operation, Linux kernel provides ``bootconfig`` command under
tools/bootconfig, which allows admin to apply or delete the config file
to/from initrd image. You can build it by the following command::
# make -C tools/bootconfig
To add your boot config file to initrd image, run bootconfig as below
(Old data is removed automatically if exists)::
# tools/bootconfig/bootconfig -a your-config /boot/initrd.img-X.Y.Z
To remove the config from the image, you can use -d option as below::
# tools/bootconfig/bootconfig -d /boot/initrd.img-X.Y.Z
Then add "bootconfig" on the normal kernel command line to tell the
kernel to look for the bootconfig at the end of the initrd file.
Alternatively, build your kernel with the ``CONFIG_BOOT_CONFIG_FORCE``
Kconfig option selected.
Embedding a Boot Config into Kernel
-----------------------------------
If you can not use initrd, you can also embed the bootconfig file in the
kernel by Kconfig options. In this case, you need to recompile the kernel
with the following configs::
CONFIG_BOOT_CONFIG_EMBED=y
CONFIG_BOOT_CONFIG_EMBED_FILE="/PATH/TO/BOOTCONFIG/FILE"
``CONFIG_BOOT_CONFIG_EMBED_FILE`` requires an absolute path or a relative
path to the bootconfig file from source tree or object tree.
The kernel will embed it as the default bootconfig.
Just as when attaching the bootconfig to the initrd, you need ``bootconfig``
option on the kernel command line to enable the embedded bootconfig, or,
alternatively, build your kernel with the ``CONFIG_BOOT_CONFIG_FORCE``
Kconfig option selected.
Note that even if you set this option, you can override the embedded
bootconfig by another bootconfig which attached to the initrd.
Kernel parameters via Boot Config
=================================
In addition to the kernel command line, the boot config can be used for
passing the kernel parameters. All the key-value pairs under ``kernel``
key will be passed to kernel cmdline directly. Moreover, the key-value
pairs under ``init`` will be passed to init process via the cmdline.
The parameters are concatenated with user-given kernel cmdline string
as the following order, so that the command line parameter can override
bootconfig parameters (this depends on how the subsystem handles parameters
but in general, earlier parameter will be overwritten by later one.)::
[bootconfig params][cmdline params] -- [bootconfig init params][cmdline init params]
Here is an example of the bootconfig file for kernel/init parameters.::
kernel {
root = 01234567-89ab-cdef-0123-456789abcd
}
init {
splash
}
This will be copied into the kernel cmdline string as the following::
root="01234567-89ab-cdef-0123-456789abcd" -- splash
If user gives some other command line like,::
ro bootconfig -- quiet
The final kernel cmdline will be the following::
root="01234567-89ab-cdef-0123-456789abcd" ro bootconfig -- splash quiet
Config File Limitation
======================
Currently the maximum config size is 32KB and the total key-words (not
key-value entries) must be under 1024 nodes.
Note: this is not the number of entries but nodes, an entry must consume
more than 2 nodes (a key-word and a value). So theoretically, it will be
up to 512 key-value pairs. If keys contains 3 words in average, it can
contain 256 key-value pairs. In most cases, the number of config items
will be under 100 entries and smaller than 8KB, so it would be enough.
If the node number exceeds 1024, parser returns an error even if the file
size is smaller than 32KB. (Note that this maximum size is not including
the padding null characters.)
Anyway, since bootconfig command verifies it when appending a boot config
to initrd image, user can notice it before boot.
Bootconfig APIs
===============
User can query or loop on key-value pairs, also it is possible to find
a root (prefix) key node and find key-values under that node.
If you have a key string, you can query the value directly with the key
using xbc_find_value(). If you want to know what keys exist in the boot
config, you can use xbc_for_each_key_value() to iterate key-value pairs.
Note that you need to use xbc_array_for_each_value() for accessing
each array's value, e.g.::
vnode = NULL;
xbc_find_value("key.word", &vnode);
if (vnode && xbc_node_is_array(vnode))
xbc_array_for_each_value(vnode, value) {
printk("%s ", value);
}
If you want to focus on keys which have a prefix string, you can use
xbc_find_node() to find a node by the prefix string, and iterate
keys under the prefix node with xbc_node_for_each_key_value().
But the most typical usage is to get the named value under prefix
or get the named array under prefix as below::
root = xbc_find_node("key.prefix");
value = xbc_node_find_value(root, "option", &vnode);
...
xbc_node_for_each_array_value(root, "array-option", value, anode) {
...
}
This accesses a value of "key.prefix.option" and an array of
"key.prefix.array-option".
Locking is not needed, since after initialization, the config becomes
read-only. All data and keys must be copied if you need to modify it.
Functions and structures
========================
.. kernel-doc:: include/linux/bootconfig.h
.. kernel-doc:: lib/bootconfig.c
linux-7.2.3/Documentation/admin-guide/braille-console.rst
+38
View File
@@ -0,0 +1,38 @@
Linux Braille Console
=====================
To get early boot messages on a braille device (before userspace screen
readers can start), you first need to compile the support for the usual serial
console (see :ref:`Documentation/admin-guide/serial-console.rst <serial_console>`), and
for braille device
(in :menuselection:`Device Drivers --> Accessibility support --> Console on braille device`).
Then you need to specify a ``console=brl``, option on the kernel command line, the
format is::
console=brl,serial_options...
where ``serial_options...`` are the same as described in
:ref:`Documentation/admin-guide/serial-console.rst <serial_console>`.
So for instance you can use ``console=brl,ttyS0`` if the braille device is connected to the first serial port, and ``console=brl,ttyS0,115200`` to
override the baud rate to 115200, etc.
By default, the braille device will just show the last kernel message (console
mode). To review previous messages, press the Insert key to switch to the VT
review mode. In review mode, the arrow keys permit to browse in the VT content,
`PAGE-UP`/`PAGE-DOWN` keys go at the top/bottom of the screen, and
the `HOME` key goes back
to the cursor, hence providing very basic screen reviewing facility.
Sound feedback can be obtained by adding the ``braille_console.sound=1`` kernel
parameter.
For simplicity, only one braille console can be enabled, other uses of
``console=brl,...`` will be discarded. Also note that it does not interfere with
the console selection mechanism described in
:ref:`Documentation/admin-guide/serial-console.rst <serial_console>`.
For now, only the VisioBraille device is supported.
Samuel Thibault <samuel.thibault@ens-lyon.org>
linux-7.2.3/Documentation/admin-guide/btmrvl.rst
+124
View File
@@ -0,0 +1,124 @@
=============
btmrvl driver
=============
All commands are used via debugfs interface.
Set/get driver configurations
=============================
Path: /debug/btmrvl/config/
gpiogap=[n], hscfgcmd
These commands are used to configure the host sleep parameters::
bit 8:0 -- Gap
bit 16:8 -- GPIO
where GPIO is the pin number of GPIO used to wake up the host.
It could be any valid GPIO pin# (e.g. 0-7) or 0xff (SDIO interface
wakeup will be used instead).
where Gap is the gap in milli seconds between wakeup signal and
wakeup event, or 0xff for special host sleep setting.
Usage::
# Use SDIO interface to wake up the host and set GAP to 0x80:
echo 0xff80 > /debug/btmrvl/config/gpiogap
echo 1 > /debug/btmrvl/config/hscfgcmd
# Use GPIO pin #3 to wake up the host and set GAP to 0xff:
echo 0x03ff > /debug/btmrvl/config/gpiogap
echo 1 > /debug/btmrvl/config/hscfgcmd
psmode=[n], pscmd
These commands are used to enable/disable auto sleep mode
where the option is::
1 -- Enable auto sleep mode
0 -- Disable auto sleep mode
Usage::
# Enable auto sleep mode
echo 1 > /debug/btmrvl/config/psmode
echo 1 > /debug/btmrvl/config/pscmd
# Disable auto sleep mode
echo 0 > /debug/btmrvl/config/psmode
echo 1 > /debug/btmrvl/config/pscmd
hsmode=[n], hscmd
These commands are used to enable host sleep or wake up firmware
where the option is::
1 -- Enable host sleep
0 -- Wake up firmware
Usage::
# Enable host sleep
echo 1 > /debug/btmrvl/config/hsmode
echo 1 > /debug/btmrvl/config/hscmd
# Wake up firmware
echo 0 > /debug/btmrvl/config/hsmode
echo 1 > /debug/btmrvl/config/hscmd
Get driver status
=================
Path: /debug/btmrvl/status/
Usage::
cat /debug/btmrvl/status/<args>
where the args are:
curpsmode
This command displays current auto sleep status.
psstate
This command display the power save state.
hsstate
This command display the host sleep state.
txdnldrdy
This command displays the value of Tx download ready flag.
Issuing a raw hci command
=========================
Use hcitool to issue raw hci command, refer to hcitool manual
Usage::
Hcitool cmd <ogf> <ocf> [Parameters]
Interface Control Command::
hcitool cmd 0x3f 0x5b 0xf5 0x01 0x00 --Enable All interface
hcitool cmd 0x3f 0x5b 0xf5 0x01 0x01 --Enable Wlan interface
hcitool cmd 0x3f 0x5b 0xf5 0x01 0x02 --Enable BT interface
hcitool cmd 0x3f 0x5b 0xf5 0x00 0x00 --Disable All interface
hcitool cmd 0x3f 0x5b 0xf5 0x00 0x01 --Disable Wlan interface
hcitool cmd 0x3f 0x5b 0xf5 0x00 0x02 --Disable BT interface
SD8688 firmware
===============
Images:
- /lib/firmware/sd8688_helper.bin
- /lib/firmware/sd8688.bin
The images can be downloaded from:
git.infradead.org/users/dwmw2/linux-firmware.git/libertas/
linux-7.2.3/Documentation/admin-guide/bug-bisect.rst
+165
View File
@@ -0,0 +1,165 @@
.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
.. [see the bottom of this file for redistribution information]
======================
Bisecting a regression
======================
This document describes how to use a ``git bisect`` to find the source code
change that broke something -- for example when some functionality stopped
working after upgrading from Linux 6.0 to 6.1.
The text focuses on the gist of the process. If you are new to bisecting the
kernel, better follow Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
instead: it depicts everything from start to finish while covering multiple
aspects even kernel developers occasionally forget. This includes detecting
situations early where a bisection would be a waste of time, as nobody would
care about the result -- for example, because the problem happens after the
kernel marked itself as 'tainted', occurs in an abandoned version, was already
fixed, or is caused by a .config change you or your Linux distributor performed.
Finding the change causing a kernel issue using a bisection
===========================================================
*Note: the following process assumes you prepared everything for a bisection.
This includes having a Git clone with the appropriate sources, installing the
software required to build and install kernels, as well as a .config file stored
in a safe place (the following example assumes '~/prepared_kernel_.config') to
use as pristine base at each bisection step; ideally, you have also worked out
a fully reliable and straight-forward way to reproduce the regression, too.*
* Preparation: start the bisection and tell Git about the points in the history
you consider to be working and broken, which Git calls 'good' and 'bad'::
git bisect start
git bisect good v6.0
git bisect bad v6.1
Instead of Git tags like 'v6.0' and 'v6.1' you can specify commit-ids, too.
1. Copy your prepared .config into the build directory and adjust it to the
needs of the codebase Git checked out for testing::
cp ~/prepared_kernel_.config .config
make olddefconfig
2. Now build, install, and boot a kernel. This might fail for unrelated reasons,
for example, when a compile error happens at the current stage of the
bisection a later change resolves. In such cases run ``git bisect skip`` and
go back to step 1.
3. Check if the functionality that regressed works in the kernel you just built.
If it works, execute::
git bisect good
If it is broken, run::
git bisect bad
Note, getting this wrong just once will send the rest of the bisection
totally off course. To prevent having to start anew later you thus want to
ensure what you tell Git is correct; it is thus often wise to spend a few
minutes more on testing in case your reproducer is unreliable.
After issuing one of these two commands, Git will usually check out another
bisection point and print something like 'Bisecting: 675 revisions left to
test after this (roughly 10 steps)'. In that case go back to step 1.
If Git instead prints something like 'cafecaca0c0dacafecaca0c0dacafecaca0c0da
is the first bad commit', then you have finished the bisection. In that case
move to the next point below. Note, right after displaying that line Git will
show some details about the culprit including its patch description; this can
easily fill your terminal, so you might need to scroll up to see the message
mentioning the culprit's commit-id.
In case you missed Git's output, you can always run ``git bisect log`` to
print the status: it will show how many steps remain or mention the result of
the bisection.
* Recommended complementary task: put the bisection log and the current .config
file aside for the bug report; furthermore tell Git to reset the sources to
the state before the bisection::
git bisect log > ~/bisection-log
cp .config ~/bisection-config-culprit
git bisect reset
* Recommended optional task: try reverting the culprit on top of the latest
codebase and check if that fixes your bug; if that is the case, it validates
the bisection and enables developers to resolve the regression through a
revert.
To try this, update your clone and check out latest mainline. Then tell Git
to revert the change by specifying its commit-id::
git revert --no-edit cafec0cacaca0
Git might reject this, for example when the bisection landed on a merge
commit. In that case, abandon the attempt. Do the same, if Git fails to revert
the culprit on its own because later changes depend on it -- at least unless
you bisected a stable or longterm kernel series, in which case you want to
check out its latest codebase and try a revert there.
If a revert succeeds, build and test another kernel to check if reverting
resolved your regression.
With that the process is complete. Now report the regression as described by
Documentation/admin-guide/reporting-issues.rst.
Bisecting linux-next
--------------------
If you face a problem only happening in linux-next, bisect between the
linux-next branches 'stable' and 'master'. The following commands will start
the process for a linux-next tree you added as a remote called 'next'::
git bisect start
git bisect good next/stable
git bisect bad next/master
The 'stable' branch refers to the state of linux-mainline that the current
linux-next release (found in the 'master' branch) is based on -- the former
thus should be free of any problems that show up in -next, but not in Linus'
tree.
This will bisect across a wide range of changes, some of which you might have
used in earlier linux-next releases without problems. Sadly there is no simple
way to avoid checking them: bisecting from one linux-next release to a later
one (say between 'next-20241020' and 'next-20241021') is impossible, as they
share no common history.
Additional reading material
---------------------------
* The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and
`fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_
in the Git documentation.
* `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_
from kernel developer Nathan Chancellor.
* `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_.
* `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_.
..
end-of-content
..
This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If
you spot a typo or small mistake, feel free to let him know directly and
he'll fix it. You are free to do the same in a mostly informal way if you
want to contribute changes to the text -- but for copyright reasons please CC
linux-doc@vger.kernel.org and 'sign-off' your contribution as
Documentation/process/submitting-patches.rst explains in the section 'Sign
your work - the Developer's Certificate of Origin'.
..
This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
of the file. If you want to distribute this text under CC-BY-4.0 only,
please use 'The Linux kernel development community' for author attribution
and link this as source:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/bug-bisect.rst
..
Note: Only the content of this RST file as found in the Linux kernel sources
is available under CC-BY-4.0, as versions of this text that were processed
(for example by the kernel's build system) might contain content taken from
files which use a more restrictive license.
linux-7.2.3/Documentation/admin-guide/bug-hunting.rst
+370
View File
@@ -0,0 +1,370 @@
Bug hunting
===========
Kernel bug reports often come with a stack dump like the one below::
------------[ cut here ]------------
WARNING: CPU: 1 PID: 28102 at kernel/module.c:1108 module_put+0x57/0x70
Modules linked in: dvb_usb_gp8psk(-) dvb_usb dvb_core nvidia_drm(PO) nvidia_modeset(PO) snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core snd_pcm snd_timer snd soundcore nvidia(PO) [last unloaded: rc_core]
CPU: 1 PID: 28102 Comm: rmmod Tainted: P WC O 4.8.4-build.1 #1
Hardware name: MSI MS-7309/MS-7309, BIOS V1.12 02/23/2009
00000000 c12ba080 00000000 00000000 c103ed6a c1616014 00000001 00006dc6
c1615862 00000454 c109e8a7 c109e8a7 00000009 ffffffff 00000000 f13f6a10
f5f5a600 c103ee33 00000009 00000000 00000000 c109e8a7 f80ca4d0 c109f617
Call Trace:
[<c12ba080>] ? dump_stack+0x44/0x64
[<c103ed6a>] ? __warn+0xfa/0x120
[<c109e8a7>] ? module_put+0x57/0x70
[<c109e8a7>] ? module_put+0x57/0x70
[<c103ee33>] ? warn_slowpath_null+0x23/0x30
[<c109e8a7>] ? module_put+0x57/0x70
[<f80ca4d0>] ? gp8psk_fe_set_frontend+0x460/0x460 [dvb_usb_gp8psk]
[<c109f617>] ? symbol_put_addr+0x27/0x50
[<f80bc9ca>] ? dvb_usb_adapter_frontend_exit+0x3a/0x70 [dvb_usb]
[<f80bb3bf>] ? dvb_usb_exit+0x2f/0xd0 [dvb_usb]
[<c13d03bc>] ? usb_disable_endpoint+0x7c/0xb0
[<f80bb48a>] ? dvb_usb_device_exit+0x2a/0x50 [dvb_usb]
[<c13d2882>] ? usb_unbind_interface+0x62/0x250
[<c136b514>] ? __pm_runtime_idle+0x44/0x70
[<c13620d8>] ? __device_release_driver+0x78/0x120
[<c1362907>] ? driver_detach+0x87/0x90
[<c1361c48>] ? bus_remove_driver+0x38/0x90
[<c13d1c18>] ? usb_deregister+0x58/0xb0
[<c109fbb0>] ? SyS_delete_module+0x130/0x1f0
[<c1055654>] ? task_work_run+0x64/0x80
[<c1000fa5>] ? exit_to_usermode_loop+0x85/0x90
[<c10013f0>] ? do_fast_syscall_32+0x80/0x130
[<c1549f43>] ? sysenter_past_esp+0x40/0x6a
---[ end trace 6ebc60ef3981792f ]---
Such stack traces provide enough information to identify the line inside the
Kernel's source code where the bug happened. Depending on the severity of
the issue, it may also contain the word **Oops**, as on this one::
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<c06969d4>] iret_exc+0x7d0/0xa59
*pdpt = 000000002258a001 *pde = 0000000000000000
Oops: 0002 [#1] PREEMPT SMP
...
Despite being an **Oops** or some other sort of stack trace, the offended
line is usually required to identify and handle the bug. Along this chapter,
we'll refer to "Oops" for all kinds of stack traces that need to be analyzed.
If the kernel is compiled with ``CONFIG_DEBUG_INFO``, you can enhance the
quality of the stack trace by using ``scripts/decode_stacktrace.sh``.
Modules linked in
-----------------
Modules that are tainted or are being loaded or unloaded are marked with
"(...)", where the taint flags are described in
Documentation/admin-guide/tainted-kernels.rst, "being loaded" is
annotated with "+", and "being unloaded" is annotated with "-".
Where is the Oops message located?
----------------------------------
Normally the Oops text is read from the kernel buffers by klogd and
handed to ``syslogd`` which writes it to a syslog file, typically
``/var/log/messages`` (depends on ``/etc/syslog.conf``). On systems with
systemd, it may also be stored by the ``journald`` daemon, and accessed
by running ``journalctl`` command.
Sometimes ``klogd`` dies, in which case you can run ``dmesg > file`` to
read the data from the kernel buffers and save it. Or you can
``cat /proc/kmsg > file``, however you have to break in to stop the transfer,
since ``kmsg`` is a "never ending file".
If the machine has crashed so badly that you cannot enter commands or
the disk is not available then you have three options:
(1) Hand copy the text from the screen and type it in after the machine
has restarted. Messy but it is the only option if you have not
planned for a crash. Alternatively, you can take a picture of
the screen with a digital camera - not nice, but better than
nothing. If the messages scroll off the top of the console, you
may find that booting with a higher resolution (e.g., ``vga=791``)
will allow you to read more of the text. (Caveat: This needs ``vesafb``,
so won't help for 'early' oopses.)
(2) Boot with a serial console (see
:ref:`Documentation/admin-guide/serial-console.rst <serial_console>`),
run a null modem to a second machine and capture the output there
using your favourite communication program. Minicom works well.
(3) Use Kdump (see Documentation/admin-guide/kdump/kdump.rst),
extract the kernel ring buffer from old memory with using dmesg
gdbmacro in Documentation/admin-guide/kdump/gdbmacros.txt.
Finding the bug's location
--------------------------
Reporting a bug works best if you point the location of the bug at the
Kernel source file. There are two methods for doing that. Usually, using
``gdb`` is easier, but the Kernel should be pre-compiled with debug info.
gdb
^^^
The GNU debugger (``gdb``) is the best way to figure out the exact file and line
number of the OOPS from the ``vmlinux`` file.
The usage of gdb works best on a kernel compiled with ``CONFIG_DEBUG_INFO``.
This can be set by running::
$ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO
On a kernel compiled with ``CONFIG_DEBUG_INFO``, you can simply copy the
EIP value from the OOPS::
EIP: 0060:[<c021e50e>] Not tainted VLI
And use GDB to translate that to human-readable form::
$ gdb vmlinux
(gdb) l *0xc021e50e
If you don't have ``CONFIG_DEBUG_INFO`` enabled, you use the function
offset from the OOPS::
EIP is at vt_ioctl+0xda8/0x1482
And recompile the kernel with ``CONFIG_DEBUG_INFO`` enabled::
$ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO
$ make vmlinux
$ gdb vmlinux
(gdb) l *vt_ioctl+0xda8
0x1888 is in vt_ioctl (drivers/tty/vt/vt_ioctl.c:293).
288 {
289 struct vc_data *vc = NULL;
290 int ret = 0;
291
292 console_lock();
293 if (VT_BUSY(vc_num))
294 ret = -EBUSY;
295 else if (vc_num)
296 vc = vc_deallocate(vc_num);
297 console_unlock();
or, if you want to be more verbose::
(gdb) p vt_ioctl
$1 = {int (struct tty_struct *, unsigned int, unsigned long)} 0xae0 <vt_ioctl>
(gdb) l *0xae0+0xda8
You could, instead, use the object file::
$ make drivers/tty/
$ gdb drivers/tty/vt/vt_ioctl.o
(gdb) l *vt_ioctl+0xda8
If you have a call trace, such as::
Call Trace:
[<ffffffff8802c8e9>] :jbd:log_wait_commit+0xa3/0xf5
[<ffffffff810482d9>] autoremove_wake_function+0x0/0x2e
[<ffffffff8802770b>] :jbd:journal_stop+0x1be/0x1ee
...
this shows the problem likely is in the :jbd: module. You can load that module
in gdb and list the relevant code::
$ gdb fs/jbd/jbd.ko
(gdb) l *log_wait_commit+0xa3
.. note::
You can also do the same for any function call at the stack trace,
like this one::
[<f80bc9ca>] ? dvb_usb_adapter_frontend_exit+0x3a/0x70 [dvb_usb]
The position where the above call happened can be seen with::
$ gdb drivers/media/usb/dvb-usb/dvb-usb.o
(gdb) l *dvb_usb_adapter_frontend_exit+0x3a
objdump
^^^^^^^
To debug a kernel, use objdump and look for the hex offset from the crash
output to find the valid line of code/assembler. Without debug symbols, you
will see the assembler code for the routine shown, but if your kernel has
debug symbols the C code will also be available. (Debug symbols can be enabled
in the kernel hacking menu of the menu configuration.) For example::
$ objdump -r -S -l --disassemble net/ipv4/tcp.o
.. note::
You need to be at the top level of the kernel tree for this to pick up
your C files.
If you don't have access to the source code you can still debug some crash
dumps using the following method (example crash dump output as shown by
Dave Miller)::
EIP is at +0x14/0x4c0
...
Code: 44 24 04 e8 6f 05 00 00 e9 e8 fe ff ff 8d 76 00 8d bc 27 00 00
00 00 55 57 56 53 81 ec bc 00 00 00 8b ac 24 d0 00 00 00 8b 5d 08
<8b> 83 3c 01 00 00 89 44 24 14 8b 45 28 85 c0 89 44 24 18 0f 85
Put the bytes into a "foo.s" file like this:
.text
.globl foo
foo:
.byte .... /* bytes from Code: part of OOPS dump */
Compile it with "gcc -c -o foo.o foo.s" then look at the output of
"objdump --disassemble foo.o".
Output:
ip_queue_xmit:
push %ebp
push %edi
push %esi
push %ebx
sub $0xbc, %esp
mov 0xd0(%esp), %ebp ! %ebp = arg0 (skb)
mov 0x8(%ebp), %ebx ! %ebx = skb->sk
mov 0x13c(%ebx), %eax ! %eax = inet_sk(sk)->opt
``scripts/decodecode`` can be used to automate most of this, depending
on what CPU architecture is being debugged.
Reporting the bug
-----------------
Once you find where the bug happened, by inspecting its location,
you could either try to fix it yourself or report it upstream.
In order to report it upstream, you should identify the bug tracker, if any, or
mailing list used for the development of the affected code. This can be done by
using the ``get_maintainer.pl`` script.
For example, if you find a bug at the gspca's sonixj.c file, you can get
its maintainers with::
$ ./scripts/get_maintainer.pl --bug -f drivers/media/usb/gspca/sonixj.c
Hans Verkuil <hverkuil@kernel.org> (odd fixer:GSPCA USB WEBCAM DRIVER,commit_signer:1/1=100%)
Mauro Carvalho Chehab <mchehab@kernel.org> (maintainer:MEDIA INPUT INFRASTRUCTURE (V4L/DVB),commit_signer:1/1=100%)
Tejun Heo <tj@kernel.org> (commit_signer:1/1=100%)
Bhaktipriya Shridhar <bhaktipriya96@gmail.com> (commit_signer:1/1=100%,authored:1/1=100%,added_lines:4/4=100%,removed_lines:9/9=100%)
linux-media@vger.kernel.org (open list:GSPCA USB WEBCAM DRIVER)
linux-kernel@vger.kernel.org (open list)
Please notice that it will point to:
- The last developers that touched the source code (if this is done inside
a git tree). On the above example, Tejun and Bhaktipriya (in this
specific case, none really involved on the development of this file);
- The driver maintainer (Hans Verkuil);
- The subsystem maintainer (Mauro Carvalho Chehab);
- The driver and/or subsystem mailing list (linux-media@vger.kernel.org);
- The Linux Kernel mailing list (linux-kernel@vger.kernel.org);
- The bug reporting URIs for the driver/subsystem (none in the above example).
If the listing contains bug reporting URIs at the end, please prefer them over
email. Otherwise, please report bugs to the mailing list used for the
development of the code (linux-media ML) copying the driver maintainer (Hans).
If you are totally stumped as to whom to send the report, and
``get_maintainer.pl`` didn't provide you anything useful, send it to
linux-kernel@vger.kernel.org.
Thanks for your help in making Linux as stable as humanly possible.
Fixing the bug
--------------
If you know programming, you could help us by not only reporting the bug,
but also providing us with a solution. After all, open source is about
sharing what you do and don't you want to be recognised for your genius?
If you decide to take this way, once you have worked out a fix please submit
it upstream.
Please do read
:ref:`Documentation/process/submitting-patches.rst <submittingpatches>` though
to help your code get accepted.
---------------------------------------------------------------------------
Notes on Oops tracing with ``klogd``
------------------------------------
In order to help Linus and the other kernel developers there has been
substantial support incorporated into ``klogd`` for processing protection
faults. In order to have full support for address resolution at least
version 1.3-pl3 of the ``sysklogd`` package should be used.
When a protection fault occurs the ``klogd`` daemon automatically
translates important addresses in the kernel log messages to their
symbolic equivalents. This translated kernel message is then
forwarded through whatever reporting mechanism ``klogd`` is using. The
protection fault message can be simply cut out of the message files
and forwarded to the kernel developers.
Two types of address resolution are performed by ``klogd``. The first is
static translation and the second is dynamic translation.
Static translation uses the System.map file.
In order to do static translation the ``klogd`` daemon
must be able to find a system map file at daemon initialization time.
See the klogd man page for information on how ``klogd`` searches for map
files.
Dynamic address translation is important when kernel loadable modules
are being used. Since memory for kernel modules is allocated from the
kernel's dynamic memory pools there are no fixed locations for either
the start of the module or for functions and symbols in the module.
The kernel supports system calls which allow a program to determine
which modules are loaded and their location in memory. Using these
system calls the klogd daemon builds a symbol table which can be used
to debug a protection fault which occurs in a loadable kernel module.
At the very minimum klogd will provide the name of the module which
generated the protection fault. There may be additional symbolic
information available if the developer of the loadable module chose to
export symbol information from the module.
Since the kernel module environment can be dynamic there must be a
mechanism for notifying the ``klogd`` daemon when a change in module
environment occurs. There are command line options available which
allow klogd to signal the currently executing daemon that symbol
information should be refreshed. See the ``klogd`` manual page for more
information.
A patch is included with the sysklogd distribution which modifies the
``modules-2.0.0`` package to automatically signal klogd whenever a module
is loaded or unloaded. Applying this patch provides essentially
seamless support for debugging protection faults which occur with
kernel loadable modules.
The following is an example of a protection fault in a loadable module
processed by ``klogd``::
Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc
Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000
Aug 29 09:51:01 blizard kernel: *pde = 00000000
Aug 29 09:51:01 blizard kernel: Oops: 0002
Aug 29 09:51:01 blizard kernel: CPU: 0
Aug 29 09:51:01 blizard kernel: EIP: 0010:[oops:_oops+16/3868]
Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212
Aug 29 09:51:01 blizard kernel: eax: 315e97cc ebx: 003a6f80 ecx: 001be77b edx: 00237c0c
Aug 29 09:51:01 blizard kernel: esi: 00000000 edi: bffffdb3 ebp: 00589f90 esp: 00589f8c
Aug 29 09:51:01 blizard kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000)
Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001
Aug 29 09:51:01 blizard kernel: 00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00
Aug 29 09:51:01 blizard kernel: bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036
Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128]
Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3
linux-7.2.3/Documentation/admin-guide/cgroup-v1/blkio-controller.rst
+301
View File
@@ -0,0 +1,301 @@
===================
Block IO Controller
===================
Overview
========
cgroup subsys "blkio" implements the block io controller. There seems to be
a need of various kinds of IO control policies (like proportional BW, max BW)
both at leaf nodes as well as at intermediate nodes in a storage hierarchy.
Plan is to use the same cgroup based management interface for blkio controller
and based on user options switch IO policies in the background.
One IO control policy is throttling policy which can be used to
specify upper IO rate limits on devices. This policy is implemented in
generic block layer and can be used on leaf nodes as well as higher
level logical devices like device mapper.
HOWTO
=====
Throttling/Upper Limit policy
-----------------------------
Enable Block IO controller::
CONFIG_BLK_CGROUP=y
Enable throttling in block layer::
CONFIG_BLK_DEV_THROTTLING=y
Mount blkio controller (see cgroups.txt, Why are cgroups needed?)::
mount -t cgroup -o blkio none /sys/fs/cgroup/blkio
Specify a bandwidth rate on particular device for root group. The format
for policy is "<major>:<minor> <bytes_per_second>"::
echo "8:16 1048576" > /sys/fs/cgroup/blkio/blkio.throttle.read_bps_device
This will put a limit of 1MB/second on reads happening for root group
on device having major/minor number 8:16.
Run dd to read a file and see if rate is throttled to 1MB/s or not::
# dd iflag=direct if=/mnt/common/zerofile of=/dev/null bs=4K count=1024
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 4.0001 s, 1.0 MB/s
Limits for writes can be put using blkio.throttle.write_bps_device file.
Hierarchical Cgroups
====================
Throttling implements hierarchy support; however,
throttling's hierarchy support is enabled iff "sane_behavior" is
enabled from cgroup side, which currently is a development option and
not publicly available.
If somebody created a hierarchy like as follows::
root
/ \
test1 test2
|
test3
Throttling with "sane_behavior" will handle the
hierarchy correctly. For throttling, all limits apply
to the whole subtree while all statistics are local to the IOs
directly generated by tasks in that cgroup.
Throttling without "sane_behavior" enabled from cgroup side will
practically treat all groups at same level as if it looks like the
following::
pivot
/ / \ \
root test1 test2 test3
Various user visible config options
===================================
CONFIG_BLK_CGROUP
Block IO controller.
CONFIG_BFQ_CGROUP_DEBUG
Debug help. Right now some additional stats file show up in cgroup
if this option is enabled.
CONFIG_BLK_DEV_THROTTLING
Enable block device throttling support in block layer.
Details of cgroup files
=======================
Proportional weight policy files
--------------------------------
blkio.bfq.weight
Specifies per cgroup weight. This is default weight of the group
on all the devices until and unless overridden by per device rule
(see `blkio.bfq.weight_device` below).
Currently allowed range of weights is from 1 to 1000. For more details,
see Documentation/block/bfq-iosched.rst.
blkio.bfq.weight_device
Specifies per cgroup per device weights, overriding the default group
weight. For more details, see Documentation/block/bfq-iosched.rst.
Following is the format::
# echo dev_maj:dev_minor weight > blkio.bfq.weight_device
Configure weight=300 on /dev/sdb (8:16) in this cgroup::
# echo 8:16 300 > blkio.bfq.weight_device
# cat blkio.bfq.weight_device
dev weight
8:16 300
Configure weight=500 on /dev/sda (8:0) in this cgroup::
# echo 8:0 500 > blkio.bfq.weight_device
# cat blkio.bfq.weight_device
dev weight
8:0 500
8:16 300
Remove specific weight for /dev/sda in this cgroup::
# echo 8:0 0 > blkio.bfq.weight_device
# cat blkio.bfq.weight_device
dev weight
8:16 300
blkio.time
Disk time allocated to cgroup per device in milliseconds. First
two fields specify the major and minor number of the device and
third field specifies the disk time allocated to group in
milliseconds.
blkio.sectors
Number of sectors transferred to/from disk by the group. First
two fields specify the major and minor number of the device and
third field specifies the number of sectors transferred by the
group to/from the device.
blkio.io_service_bytes
Number of bytes transferred to/from the disk by the group. These
are further divided by the type of operation - read or write, sync
or async. First two fields specify the major and minor number of the
device, third field specifies the operation type and the fourth field
specifies the number of bytes.
blkio.io_serviced
Number of IOs (bio) issued to the disk by the group. These
are further divided by the type of operation - read or write, sync
or async. First two fields specify the major and minor number of the
device, third field specifies the operation type and the fourth field
specifies the number of IOs.
blkio.io_service_time
Total amount of time between request dispatch and request completion
for the IOs done by this cgroup. This is in nanoseconds to make it
meaningful for flash devices too. For devices with queue depth of 1,
this time represents the actual service time. When queue_depth > 1,
that is no longer true as requests may be served out of order. This
may cause the service time for a given IO to include the service time
of multiple IOs when served out of order which may result in total
io_service_time > actual time elapsed. This time is further divided by
the type of operation - read or write, sync or async. First two fields
specify the major and minor number of the device, third field
specifies the operation type and the fourth field specifies the
io_service_time in ns.
blkio.io_wait_time
Total amount of time the IOs for this cgroup spent waiting in the
scheduler queues for service. This can be greater than the total time
elapsed since it is cumulative io_wait_time for all IOs. It is not a
measure of total time the cgroup spent waiting but rather a measure of
the wait_time for its individual IOs. For devices with queue_depth > 1
this metric does not include the time spent waiting for service once
the IO is dispatched to the device but till it actually gets serviced
(there might be a time lag here due to re-ordering of requests by the
device). This is in nanoseconds to make it meaningful for flash
devices too. This time is further divided by the type of operation -
read or write, sync or async. First two fields specify the major and
minor number of the device, third field specifies the operation type
and the fourth field specifies the io_wait_time in ns.
blkio.io_merged
Total number of bios/requests merged into requests belonging to this
cgroup. This is further divided by the type of operation - read or
write, sync or async.
blkio.io_queued
Total number of requests queued up at any given instant for this
cgroup. This is further divided by the type of operation - read or
write, sync or async.
blkio.avg_queue_size
Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
The average queue size for this cgroup over the entire time of this
cgroup's existence. Queue size samples are taken each time one of the
queues of this cgroup gets a timeslice.
blkio.group_wait_time
Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
This is the amount of time the cgroup had to wait since it became busy
(i.e., went from 0 to 1 request queued) to get a timeslice for one of
its queues. This is different from the io_wait_time which is the
cumulative total of the amount of time spent by each IO in that cgroup
waiting in the scheduler queue. This is in nanoseconds. If this is
read when the cgroup is in a waiting (for timeslice) state, the stat
will only report the group_wait_time accumulated till the last time it
got a timeslice and will not include the current delta.
blkio.empty_time
Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
This is the amount of time a cgroup spends without any pending
requests when not being served, i.e., it does not include any time
spent idling for one of the queues of the cgroup. This is in
nanoseconds. If this is read when the cgroup is in an empty state,
the stat will only report the empty_time accumulated till the last
time it had a pending request and will not include the current delta.
blkio.idle_time
Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y.
This is the amount of time spent by the IO scheduler idling for a
given cgroup in anticipation of a better request than the existing ones
from other queues/cgroups. This is in nanoseconds. If this is read
when the cgroup is in an idling state, the stat will only report the
idle_time accumulated till the last idle period and will not include
the current delta.
blkio.dequeue
Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. This
gives the statistics about how many a times a group was dequeued
from service tree of the device. First two fields specify the major
and minor number of the device and third field specifies the number
of times a group was dequeued from a particular device.
blkio.*_recursive
Recursive version of various stats. These files show the
same information as their non-recursive counterparts but
include stats from all the descendant cgroups.
Throttling/Upper limit policy files
-----------------------------------
blkio.throttle.read_bps_device
Specifies upper limit on READ rate from the device. IO rate is
specified in bytes per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.throttle.read_bps_device
blkio.throttle.write_bps_device
Specifies upper limit on WRITE rate to the device. IO rate is
specified in bytes per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.throttle.write_bps_device
blkio.throttle.read_iops_device
Specifies upper limit on READ rate from the device. IO rate is
specified in IO per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.throttle.read_iops_device
blkio.throttle.write_iops_device
Specifies upper limit on WRITE rate to the device. IO rate is
specified in io per second. Rules are per device. Following is
the format::
echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.throttle.write_iops_device
Note: If both BW and IOPS rules are specified for a device, then IO is
subjected to both the constraints.
blkio.throttle.io_serviced
Number of IOs (bio) issued to the disk by the group. These
are further divided by the type of operation - read or write, sync
or async. First two fields specify the major and minor number of the
device, third field specifies the operation type and the fourth field
specifies the number of IOs.
blkio.throttle.io_service_bytes
Number of bytes transferred to/from the disk by the group. These
are further divided by the type of operation - read or write, sync
or async. First two fields specify the major and minor number of the
device, third field specifies the operation type and the fourth field
specifies the number of bytes.
Common files among various policies
-----------------------------------
blkio.reset_stats
Writing an int to this file will result in resetting all the stats
for that cgroup.
linux-7.2.3/Documentation/admin-guide/cgroup-v1/cgroups.rst
+697
View File
@@ -0,0 +1,697 @@
==============
Control Groups
==============
Written by Paul Menage <menage@google.com> based on
Documentation/admin-guide/cgroup-v1/cpusets.rst
Original copyright statements from cpusets.txt:
Portions Copyright (C) 2004 BULL SA.
Portions Copyright (c) 2004-2006 Silicon Graphics, Inc.
Modified by Paul Jackson <pj@sgi.com>
Modified by Christoph Lameter <cl@gentwo.org>
.. CONTENTS:
1. Control Groups
1.1 What are cgroups ?
1.2 Why are cgroups needed ?
1.3 How are cgroups implemented ?
1.4 What does notify_on_release do ?
1.5 What does clone_children do ?
1.6 How do I use cgroups ?
2. Usage Examples and Syntax
2.1 Basic Usage
2.2 Attaching processes
2.3 Mounting hierarchies by name
3. Kernel API
3.1 Overview
3.2 Synchronization
3.3 Subsystem API
4. Extended attributes usage
5. Questions
1. Control Groups
=================
1.1 What are cgroups ?
----------------------
Control Groups provide a mechanism for aggregating/partitioning sets of
tasks, and all their future children, into hierarchical groups with
specialized behaviour.
Definitions:
A *cgroup* associates a set of tasks with a set of parameters for one
or more subsystems.
A *subsystem* is a module that makes use of the task grouping
facilities provided by cgroups to treat groups of tasks in
particular ways. A subsystem is typically a "resource controller" that
schedules a resource or applies per-cgroup limits, but it may be
anything that wants to act on a group of processes, e.g. a
virtualization subsystem.
A *hierarchy* is a set of cgroups arranged in a tree, such that
every task in the system is in exactly one of the cgroups in the
hierarchy, and a set of subsystems; each subsystem has system-specific
state attached to each cgroup in the hierarchy. Each hierarchy has
an instance of the cgroup virtual filesystem associated with it.
At any one time there may be multiple active hierarchies of task
cgroups. Each hierarchy is a partition of all tasks in the system.
User-level code may create and destroy cgroups by name in an
instance of the cgroup virtual file system, specify and query to
which cgroup a task is assigned, and list the task PIDs assigned to
a cgroup. Those creations and assignments only affect the hierarchy
associated with that instance of the cgroup file system.
On their own, the only use for cgroups is for simple job
tracking. The intention is that other subsystems hook into the generic
cgroup support to provide new attributes for cgroups, such as
accounting/limiting the resources which processes in a cgroup can
access. For example, cpusets (see Documentation/admin-guide/cgroup-v1/cpusets.rst) allow
you to associate a set of CPUs and a set of memory nodes with the
tasks in each cgroup.
.. _cgroups-why-needed:
1.2 Why are cgroups needed ?
----------------------------
There are multiple efforts to provide process aggregations in the
Linux kernel, mainly for resource-tracking purposes. Such efforts
include cpusets, CKRM/ResGroups, UserBeanCounters, and virtual server
namespaces. These all require the basic notion of a
grouping/partitioning of processes, with newly forked processes ending
up in the same group (cgroup) as their parent process.
The kernel cgroup patch provides the minimum essential kernel
mechanisms required to efficiently implement such groups. It has
minimal impact on the system fast paths, and provides hooks for
specific subsystems such as cpusets to provide additional behaviour as
desired.
Multiple hierarchy support is provided to allow for situations where
the division of tasks into cgroups is distinctly different for
different subsystems - having parallel hierarchies allows each
hierarchy to be a natural division of tasks, without having to handle
complex combinations of tasks that would be present if several
unrelated subsystems needed to be forced into the same tree of
cgroups.
At one extreme, each resource controller or subsystem could be in a
separate hierarchy; at the other extreme, all subsystems
would be attached to the same hierarchy.
As an example of a scenario (originally proposed by vatsa@in.ibm.com)
that can benefit from multiple hierarchies, consider a large
university server with various users - students, professors, system
tasks etc. The resource planning for this server could be along the
following lines::
CPU : "Top cpuset"
/ \
CPUSet1 CPUSet2
| |
(Professors) (Students)
In addition (system tasks) are attached to topcpuset (so
that they can run anywhere) with a limit of 20%
Memory : Professors (50%), Students (30%), system (20%)
Disk : Professors (50%), Students (30%), system (20%)
Network : WWW browsing (20%), Network File System (60%), others (20%)
/ \
Professors (15%) students (5%)
Browsers like Firefox/Lynx go into the WWW network class, while (k)nfsd goes
into the NFS network class.
At the same time Firefox/Lynx will share an appropriate CPU/Memory class
depending on who launched it (prof/student).
With the ability to classify tasks differently for different resources
(by putting those resource subsystems in different hierarchies),
the admin can easily set up a script which receives exec notifications
and depending on who is launching the browser he can::
# echo browser_pid > /sys/fs/cgroup/<restype>/<userclass>/tasks
With only a single hierarchy, he now would potentially have to create
a separate cgroup for every browser launched and associate it with
appropriate network and other resource class. This may lead to
proliferation of such cgroups.
Also let's say that the administrator would like to give enhanced network
access temporarily to a student's browser (since it is night and the user
wants to do online gaming :)) OR give one of the student's simulation
apps enhanced CPU power.
With ability to write PIDs directly to resource classes, it's just a
matter of::
# echo pid > /sys/fs/cgroup/network/<new_class>/tasks
(after some time)
# echo pid > /sys/fs/cgroup/network/<orig_class>/tasks
Without this ability, the administrator would have to split the cgroup into
multiple separate ones and then associate the new cgroups with the
new resource classes.
1.3 How are cgroups implemented ?
---------------------------------
Control Groups extends the kernel as follows:
- Each task in the system has a reference-counted pointer to a
css_set.
- A css_set contains a set of reference-counted pointers to
cgroup_subsys_state objects, one for each cgroup subsystem
registered in the system. There is no direct link from a task to
the cgroup of which it's a member in each hierarchy, but this
can be determined by following pointers through the
cgroup_subsys_state objects. This is because accessing the
subsystem state is something that's expected to happen frequently
and in performance-critical code, whereas operations that require a
task's actual cgroup assignments (in particular, moving between
cgroups) are less common. A linked list runs through the cg_list
field of each task_struct using the css_set, anchored at
css_set->tasks.
- A cgroup hierarchy filesystem can be mounted for browsing and
manipulation from user space.
- You can list all the tasks (by PID) attached to any cgroup.
The implementation of cgroups requires a few, simple hooks
into the rest of the kernel, none in performance-critical paths:
- in init/main.c, to initialize the root cgroups and initial
css_set at system boot.
- in fork and exit, to attach and detach a task from its css_set.
In addition, a new file system of type "cgroup" may be mounted, to
enable browsing and modifying the cgroups presently known to the
kernel. When mounting a cgroup hierarchy, you may specify a
comma-separated list of subsystems to mount as the filesystem mount
options. By default, mounting the cgroup filesystem attempts to
mount a hierarchy containing all registered subsystems.
If an active hierarchy with exactly the same set of subsystems already
exists, it will be reused for the new mount. If no existing hierarchy
matches, and any of the requested subsystems are in use in an existing
hierarchy, the mount will fail with -EBUSY. Otherwise, a new hierarchy
is activated, associated with the requested subsystems.
It's not currently possible to bind a new subsystem to an active
cgroup hierarchy, or to unbind a subsystem from an active cgroup
hierarchy. This may be possible in future, but is fraught with nasty
error-recovery issues.
When a cgroup filesystem is unmounted, if there are any
child cgroups created below the top-level cgroup, that hierarchy
will remain active even though unmounted; if there are no
child cgroups then the hierarchy will be deactivated.
No new system calls are added for cgroups - all support for
querying and modifying cgroups is via this cgroup file system.
Each task under /proc has an added file named 'cgroup' displaying,
for each active hierarchy, the subsystem names and the cgroup name
as the path relative to the root of the cgroup file system.
Each cgroup is represented by a directory in the cgroup file system
containing the following files describing that cgroup:
- tasks: list of tasks (by PID) attached to that cgroup. This list
is not guaranteed to be sorted. Writing a thread ID into this file
moves the thread into this cgroup.
- cgroup.procs: list of thread group IDs in the cgroup. This list is
not guaranteed to be sorted or free of duplicate TGIDs, and userspace
should sort/uniquify the list if this property is required.
Writing a thread group ID into this file moves all threads in that
group into this cgroup.
- notify_on_release flag: run the release agent on exit?
- release_agent: the path to use for release notifications (this file
exists in the top cgroup only)
Other subsystems such as cpusets may add additional files in each
cgroup dir.
New cgroups are created using the mkdir system call or shell
command. The properties of a cgroup, such as its flags, are
modified by writing to the appropriate file in that cgroups
directory, as listed above.
The named hierarchical structure of nested cgroups allows partitioning
a large system into nested, dynamically changeable, "soft-partitions".
The attachment of each task, automatically inherited at fork by any
children of that task, to a cgroup allows organizing the work load
on a system into related sets of tasks. A task may be re-attached to
any other cgroup, if allowed by the permissions on the necessary
cgroup file system directories.
When a task is moved from one cgroup to another, it gets a new
css_set pointer - if there's an already existing css_set with the
desired collection of cgroups then that group is reused, otherwise a new
css_set is allocated. The appropriate existing css_set is located by
looking into a hash table.
To allow access from a cgroup to the css_sets (and hence tasks)
that comprise it, a set of cg_cgroup_link objects form a lattice;
each cg_cgroup_link is linked into a list of cg_cgroup_links for
a single cgroup on its cgrp_link_list field, and a list of
cg_cgroup_links for a single css_set on its cg_link_list.
Thus the set of tasks in a cgroup can be listed by iterating over
each css_set that references the cgroup, and sub-iterating over
each css_set's task set.
The use of a Linux virtual file system (vfs) to represent the
cgroup hierarchy provides for a familiar permission and name space
for cgroups, with a minimum of additional kernel code.
1.4 What does notify_on_release do ?
------------------------------------
If the notify_on_release flag is enabled (1) in a cgroup, then
whenever the last task in the cgroup leaves (exits or attaches to
some other cgroup) and the last child cgroup of that cgroup
is removed, then the kernel runs the command specified by the contents
of the "release_agent" file in that hierarchy's root directory,
supplying the pathname (relative to the mount point of the cgroup
file system) of the abandoned cgroup. This enables automatic
removal of abandoned cgroups. The default value of
notify_on_release in the root cgroup at system boot is disabled
(0). The default value of other cgroups at creation is the current
value of their parents' notify_on_release settings. The default value of
a cgroup hierarchy's release_agent path is empty.
1.5 What does clone_children do ?
---------------------------------
This flag only affects the cpuset controller. If the clone_children
flag is enabled (1) in a cgroup, a new cpuset cgroup will copy its
configuration from the parent during initialization.
1.6 How do I use cgroups ?
--------------------------
To start a new job that is to be contained within a cgroup, using
the "cpuset" cgroup subsystem, the steps are something like::
1) mount -t tmpfs cgroup_root /sys/fs/cgroup
2) mkdir /sys/fs/cgroup/cpuset
3) mount -t cgroup -ocpuset cpuset /sys/fs/cgroup/cpuset
4) Create the new cgroup by doing mkdir's and write's (or echo's) in
the /sys/fs/cgroup/cpuset virtual file system.
5) Start a task that will be the "founding father" of the new job.
6) Attach that task to the new cgroup by writing its PID to the
/sys/fs/cgroup/cpuset tasks file for that cgroup.
7) fork, exec or clone the job tasks from this founding father task.
For example, the following sequence of commands will setup a cgroup
named "Charlie", containing just CPUs 2 and 3, and Memory Node 1,
and then start a subshell 'sh' in that cgroup::
mount -t tmpfs cgroup_root /sys/fs/cgroup
mkdir /sys/fs/cgroup/cpuset
mount -t cgroup cpuset -ocpuset /sys/fs/cgroup/cpuset
cd /sys/fs/cgroup/cpuset
mkdir Charlie
cd Charlie
/bin/echo 2-3 > cpuset.cpus
/bin/echo 1 > cpuset.mems
/bin/echo $$ > tasks
sh
# The subshell 'sh' is now running in cgroup Charlie
# The next line should display '/Charlie'
cat /proc/self/cgroup
2. Usage Examples and Syntax
============================
2.1 Basic Usage
---------------
Creating, modifying, using cgroups can be done through the cgroup
virtual filesystem.
To mount a cgroup hierarchy with all available subsystems, type::
# mount -t cgroup xxx /sys/fs/cgroup
The "xxx" is not interpreted by the cgroup code, but will appear in
/proc/mounts so may be any useful identifying string that you like.
Note: Some subsystems do not work without some user input first. For instance,
if cpusets are enabled the user will have to populate the cpus and mems files
for each new cgroup created before that group can be used.
As explained in section `1.2 Why are cgroups needed?` you should create
different hierarchies of cgroups for each single resource or group of
resources you want to control. Therefore, you should mount a tmpfs on
/sys/fs/cgroup and create directories for each cgroup resource or resource
group::
# mount -t tmpfs cgroup_root /sys/fs/cgroup
# mkdir /sys/fs/cgroup/rg1
To mount a cgroup hierarchy with just the cpuset and memory
subsystems, type::
# mount -t cgroup -o cpuset,memory hier1 /sys/fs/cgroup/rg1
While remounting cgroups is currently supported, it is not recommend
to use it. Remounting allows changing bound subsystems and
release_agent. Rebinding is hardly useful as it only works when the
hierarchy is empty and release_agent itself should be replaced with
conventional fsnotify. The support for remounting will be removed in
the future.
To Specify a hierarchy's release_agent::
# mount -t cgroup -o cpuset,release_agent="/sbin/cpuset_release_agent" \
xxx /sys/fs/cgroup/rg1
Note that specifying 'release_agent' more than once will return failure.
Note that changing the set of subsystems is currently only supported
when the hierarchy consists of a single (root) cgroup. Supporting
the ability to arbitrarily bind/unbind subsystems from an existing
cgroup hierarchy is intended to be implemented in the future.
Then under /sys/fs/cgroup/rg1 you can find a tree that corresponds to the
tree of the cgroups in the system. For instance, /sys/fs/cgroup/rg1
is the cgroup that holds the whole system.
If you want to change the value of release_agent::
# echo "/sbin/new_release_agent" > /sys/fs/cgroup/rg1/release_agent
It can also be changed via remount.
If you want to create a new cgroup under /sys/fs/cgroup/rg1::
# cd /sys/fs/cgroup/rg1
# mkdir my_cgroup
Now you want to do something with this cgroup:
# cd my_cgroup
In this directory you can find several files::
# ls
cgroup.procs notify_on_release tasks
(plus whatever files added by the attached subsystems)
Now attach your shell to this cgroup::
# /bin/echo $$ > tasks
You can also create cgroups inside your cgroup by using mkdir in this
directory::
# mkdir my_sub_cs
To remove a cgroup, just use rmdir::
# rmdir my_sub_cs
This will fail if the cgroup is in use (has cgroups inside, or
has processes attached, or is held alive by other subsystem-specific
reference).
2.2 Attaching processes
-----------------------
::
# /bin/echo PID > tasks
Note that it is PID, not PIDs. You can only attach ONE task at a time.
If you have several tasks to attach, you have to do it one after another::
# /bin/echo PID1 > tasks
# /bin/echo PID2 > tasks
...
# /bin/echo PIDn > tasks
You can attach the current shell task by echoing 0::
# echo 0 > tasks
You can use the cgroup.procs file instead of the tasks file to move all
threads in a threadgroup at once. Echoing the PID of any task in a
threadgroup to cgroup.procs causes all tasks in that threadgroup to be
attached to the cgroup. Writing 0 to cgroup.procs moves all tasks
in the writing task's threadgroup.
Note: Since every task is always a member of exactly one cgroup in each
mounted hierarchy, to remove a task from its current cgroup you must
move it into a new cgroup (possibly the root cgroup) by writing to the
new cgroup's tasks file.
Note: Due to some restrictions enforced by some cgroup subsystems, moving
a process to another cgroup can fail.
2.3 Mounting hierarchies by name
--------------------------------
Passing the name=<x> option when mounting a cgroups hierarchy
associates the given name with the hierarchy. This can be used when
mounting a pre-existing hierarchy, in order to refer to it by name
rather than by its set of active subsystems. Each hierarchy is either
nameless, or has a unique name.
The name should match [\w.-]+
When passing a name=<x> option for a new hierarchy, you need to
specify subsystems manually; the legacy behaviour of mounting all
subsystems when none are explicitly specified is not supported when
you give a subsystem a name.
The name of the subsystem appears as part of the hierarchy description
in /proc/mounts and /proc/<pid>/cgroups.
3. Kernel API
=============
3.1 Overview
------------
Each kernel subsystem that wants to hook into the generic cgroup
system needs to create a cgroup_subsys object. This contains
various methods, which are callbacks from the cgroup system, along
with a subsystem ID which will be assigned by the cgroup system.
Other fields in the cgroup_subsys object include:
- subsys_id: a unique array index for the subsystem, indicating which
entry in cgroup->subsys[] this subsystem should be managing.
- name: should be initialized to a unique subsystem name. Should be
no longer than MAX_CGROUP_TYPE_NAMELEN.
- early_init: indicate if the subsystem needs early initialization
at system boot.
Each cgroup object created by the system has an array of pointers,
indexed by subsystem ID; this pointer is entirely managed by the
subsystem; the generic cgroup code will never touch this pointer.
3.2 Synchronization
-------------------
There is a global mutex, cgroup_mutex, used by the cgroup
system. This should be taken by anything that wants to modify a
cgroup. It may also be taken to prevent cgroups from being
modified, but more specific locks may be more appropriate in that
situation.
See kernel/cgroup/cgroup.c for more details.
Subsystems can take/release the cgroup_mutex via the functions
cgroup_lock()/cgroup_unlock().
Accessing a task's cgroup pointer may be done in the following ways:
- while holding cgroup_mutex
- while holding the task's alloc_lock (via task_lock())
- inside an rcu_read_lock() section via rcu_dereference()
3.3 Subsystem API
-----------------
Each subsystem should:
- add an entry in linux/cgroup_subsys.h
- define a cgroup_subsys object called <name>_cgrp_subsys
Each subsystem may export the following methods. The only mandatory
methods are css_alloc/free. Any others that are null are presumed to
be successful no-ops.
``struct cgroup_subsys_state *css_alloc(struct cgroup *cgrp)``
(cgroup_mutex held by caller)
Called to allocate a subsystem state object for a cgroup. The
subsystem should allocate its subsystem state object for the passed
cgroup, returning a pointer to the new object on success or a
ERR_PTR() value. On success, the subsystem pointer should point to
a structure of type cgroup_subsys_state (typically embedded in a
larger subsystem-specific object), which will be initialized by the
cgroup system. Note that this will be called at initialization to
create the root subsystem state for this subsystem; this case can be
identified by the passed cgroup object having a NULL parent (since
it's the root of the hierarchy) and may be an appropriate place for
initialization code.
``int css_online(struct cgroup *cgrp)``
(cgroup_mutex held by caller)
Called after @cgrp successfully completed all allocations and made
visible to cgroup_for_each_child/descendant_*() iterators. The
subsystem may choose to fail creation by returning -errno. This
callback can be used to implement reliable state sharing and
propagation along the hierarchy. See the comment on
cgroup_for_each_live_descendant_pre() for details.
``void css_offline(struct cgroup *cgrp);``
(cgroup_mutex held by caller)
This is the counterpart of css_online() and called iff css_online()
has succeeded on @cgrp. This signifies the beginning of the end of
@cgrp. @cgrp is being removed and the subsystem should start dropping
all references it's holding on @cgrp. When all references are dropped,
cgroup removal will proceed to the next step - css_free(). After this
callback, @cgrp should be considered dead to the subsystem.
``void css_free(struct cgroup *cgrp)``
(cgroup_mutex held by caller)
The cgroup system is about to free @cgrp; the subsystem should free
its subsystem state object. By the time this method is called, @cgrp
is completely unused; @cgrp->parent is still valid. (Note - can also
be called for a newly-created cgroup if an error occurs after this
subsystem's create() method has been called for the new cgroup).
``int can_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
(cgroup_mutex held by caller)
Called prior to moving one or more tasks into a cgroup; if the
subsystem returns an error, this will abort the attach operation.
@tset contains the tasks to be attached and is guaranteed to have at
least one task in it.
If there are multiple tasks in the taskset, then:
- it's guaranteed that all are from the same thread group
- @tset contains all tasks from the thread group whether or not
they're switching cgroups
- the first task is the leader
Each @tset entry also contains the task's old cgroup and tasks which
aren't switching cgroup can be skipped easily using the
cgroup_taskset_for_each() iterator. Note that this isn't called on a
fork. If this method returns 0 (success) then this should remain valid
while the caller holds cgroup_mutex and it is ensured that either
attach() or cancel_attach() will be called in future.
``void css_reset(struct cgroup_subsys_state *css)``
(cgroup_mutex held by caller)
An optional operation which should restore @css's configuration to the
initial state. This is currently only used on the unified hierarchy
when a subsystem is disabled on a cgroup through
"cgroup.subtree_control" but should remain enabled because other
subsystems depend on it. cgroup core makes such a css invisible by
removing the associated interface files and invokes this callback so
that the hidden subsystem can return to the initial neutral state.
This prevents unexpected resource control from a hidden css and
ensures that the configuration is in the initial state when it is made
visible again later.
``void cancel_attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
(cgroup_mutex held by caller)
Called when a task attach operation has failed after can_attach() has succeeded.
A subsystem whose can_attach() has some side-effects should provide this
function, so that the subsystem can implement a rollback. If not, not necessary.
This will be called only about subsystems whose can_attach() operation have
succeeded. The parameters are identical to can_attach().
``void attach(struct cgroup *cgrp, struct cgroup_taskset *tset)``
(cgroup_mutex held by caller)
Called after the task has been attached to the cgroup, to allow any
post-attachment activity that requires memory allocations or blocking.
The parameters are identical to can_attach().
``void fork(struct task_struct *task)``
Called when a task is forked into a cgroup.
``void exit(struct task_struct *task)``
Called during task exit.
``void free(struct task_struct *task)``
Called when the task_struct is freed.
``void bind(struct cgroup *root)``
(cgroup_mutex held by caller)
Called when a cgroup subsystem is rebound to a different hierarchy
and root cgroup. Currently this will only involve movement between
the default hierarchy (which never has sub-cgroups) and a hierarchy
that is being created/destroyed (and hence has no sub-cgroups).
4. Extended attribute usage
===========================
cgroup filesystem supports certain types of extended attributes in its
directories and files. The current supported types are:
- Trusted (XATTR_TRUSTED)
- Security (XATTR_SECURITY)
Both require CAP_SYS_ADMIN capability to set.
Like in tmpfs, the extended attributes in cgroup filesystem are stored
using kernel memory and it's advised to keep the usage at minimum. This
is the reason why user defined extended attributes are not supported, since
any user can do it and there's no limit in the value size.
The current known users for this feature are SELinux to limit cgroup usage
in containers and systemd for assorted meta data like main PID in a cgroup
(systemd creates a cgroup per service).
5. Questions
============
::
Q: what's up with this '/bin/echo' ?
A: bash's builtin 'echo' command does not check calls to write() against
errors. If you use it in the cgroup file system, you won't be
able to tell whether a command succeeded or failed.
Q: When I attach processes, only the first of the line gets really attached !
A: We can only return one error code per call to write(). So you should also
put only ONE PID.
linux-7.2.3/Documentation/admin-guide/cgroup-v1/cpuacct.rst
+50
View File
@@ -0,0 +1,50 @@
=========================
CPU Accounting Controller
=========================
The CPU accounting controller is used to group tasks using cgroups and
account the CPU usage of these groups of tasks.
The CPU accounting controller supports multi-hierarchy groups. An accounting
group accumulates the CPU usage of all of its child groups and the tasks
directly present in its group.
Accounting groups can be created by first mounting the cgroup filesystem::
# mount -t cgroup -ocpuacct none /sys/fs/cgroup
With the above step, the initial or the parent accounting group becomes
visible at /sys/fs/cgroup. At bootup, this group includes all the tasks in
the system. /sys/fs/cgroup/tasks lists the tasks in this cgroup.
/sys/fs/cgroup/cpuacct.usage gives the CPU time (in nanoseconds) obtained
by this group which is essentially the CPU time obtained by all the tasks
in the system.
New accounting groups can be created under the parent group /sys/fs/cgroup::
# cd /sys/fs/cgroup
# mkdir g1
# echo $$ > g1/tasks
The above steps create a new group g1 and move the current shell
process (bash) into it. CPU time consumed by this bash and its children
can be obtained from g1/cpuacct.usage and the same is accumulated in
/sys/fs/cgroup/cpuacct.usage also.
cpuacct.stat file lists a few statistics which further divide the
CPU time obtained by the cgroup into user and system times. Currently
the following statistics are supported:
user: Time spent by tasks of the cgroup in user mode.
system: Time spent by tasks of the cgroup in kernel mode.
user and system are in USER_HZ unit.
cpuacct controller uses percpu_counter interface to collect user and
system times. This has two side effects:
- It is theoretically possible to see wrong values for user and system times.
This is because percpu_counter_read() on 32bit systems isn't safe
against concurrent writes.
- It is possible to see slightly outdated values for user and system times
due to the batch processing nature of percpu_counter.
linux-7.2.3/Documentation/admin-guide/cgroup-v1/cpusets.rst
+884
View File
@@ -0,0 +1,884 @@
.. _cpusets:
=======
CPUSETS
=======
Copyright (C) 2004 BULL SA.
Written by Simon.Derr@bull.net
- Portions Copyright (c) 2004-2006 Silicon Graphics, Inc.
- Modified by Paul Jackson <pj@sgi.com>
- Modified by Christoph Lameter <cl@gentwo.org>
- Modified by Paul Menage <menage@google.com>
- Modified by Hidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
.. CONTENTS:
1. Cpusets
1.1 What are cpusets ?
1.2 Why are cpusets needed ?
1.3 How are cpusets implemented ?
1.4 What are exclusive cpusets ?
1.5 What is memory_pressure ?
1.6 What is memory spread ?
1.7 What is sched_load_balance ?
1.8 What is sched_relax_domain_level ?
1.9 How do I use cpusets ?
2. Usage Examples and Syntax
2.1 Basic Usage
2.2 Adding/removing cpus
2.3 Setting flags
2.4 Attaching processes
3. Questions
4. Contact
1. Cpusets
==========
1.1 What are cpusets ?
----------------------
Cpusets provide a mechanism for assigning a set of CPUs and Memory
Nodes to a set of tasks. In this document "Memory Node" refers to
an on-line node that contains memory.
Cpusets constrain the CPU and Memory placement of tasks to only
the resources within a task's current cpuset. They form a nested
hierarchy visible in a virtual file system. These are the essential
hooks, beyond what is already present, required to manage dynamic
job placement on large systems.
Cpusets use the generic cgroup subsystem described in
Documentation/admin-guide/cgroup-v1/cgroups.rst.
Requests by a task, using the sched_setaffinity(2) system call to
include CPUs in its CPU affinity mask, and using the mbind(2) and
set_mempolicy(2) system calls to include Memory Nodes in its memory
policy, are both filtered through that task's cpuset, filtering out any
CPUs or Memory Nodes not in that cpuset. The scheduler will not
schedule a task on a CPU that is not allowed in its cpus_allowed
vector, and the kernel page allocator will not allocate a page on a
node that is not allowed in the requesting task's mems_allowed vector.
User level code may create and destroy cpusets by name in the cgroup
virtual file system, manage the attributes and permissions of these
cpusets and which CPUs and Memory Nodes are assigned to each cpuset,
specify and query to which cpuset a task is assigned, and list the
task pids assigned to a cpuset.
1.2 Why are cpusets needed ?
----------------------------
The management of large computer systems, with many processors (CPUs),
complex memory cache hierarchies and multiple Memory Nodes having
non-uniform access times (NUMA) presents additional challenges for
the efficient scheduling and memory placement of processes.
Frequently more modest sized systems can be operated with adequate
efficiency just by letting the operating system automatically share
the available CPU and Memory resources amongst the requesting tasks.
But larger systems, which benefit more from careful processor and
memory placement to reduce memory access times and contention,
and which typically represent a larger investment for the customer,
can benefit from explicitly placing jobs on properly sized subsets of
the system.
This can be especially valuable on:
* Web Servers running multiple instances of the same web application,
* Servers running different applications (for instance, a web server
and a database), or
* NUMA systems running large HPC applications with demanding
performance characteristics.
These subsets, or "soft partitions" must be able to be dynamically
adjusted, as the job mix changes, without impacting other concurrently
executing jobs. The location of the running jobs pages may also be moved
when the memory locations are changed.
The kernel cpuset patch provides the minimum essential kernel
mechanisms required to efficiently implement such subsets. It
leverages existing CPU and Memory Placement facilities in the Linux
kernel to avoid any additional impact on the critical scheduler or
memory allocator code.
1.3 How are cpusets implemented ?
---------------------------------
Cpusets provide a Linux kernel mechanism to constrain which CPUs and
Memory Nodes are used by a process or set of processes.
The Linux kernel already has a pair of mechanisms to specify on which
CPUs a task may be scheduled (sched_setaffinity) and on which Memory
Nodes it may obtain memory (mbind, set_mempolicy).
Cpusets extends these two mechanisms as follows:
- Cpusets are sets of allowed CPUs and Memory Nodes, known to the
kernel.
- Each task in the system is attached to a cpuset, via a pointer
in the task structure to a reference counted cgroup structure.
- Calls to sched_setaffinity are filtered to just those CPUs
allowed in that task's cpuset.
- Calls to mbind and set_mempolicy are filtered to just
those Memory Nodes allowed in that task's cpuset.
- The root cpuset contains all the systems CPUs and Memory
Nodes.
- For any cpuset, one can define child cpusets containing a subset
of the parents CPU and Memory Node resources.
- The hierarchy of cpusets can be mounted at /dev/cpuset, for
browsing and manipulation from user space.
- A cpuset may be marked exclusive, which ensures that no other
cpuset (except direct ancestors and descendants) may contain
any overlapping CPUs or Memory Nodes.
- You can list all the tasks (by pid) attached to any cpuset.
The implementation of cpusets requires a few, simple hooks
into the rest of the kernel, none in performance critical paths:
- in init/main.c, to initialize the root cpuset at system boot.
- in fork and exit, to attach and detach a task from its cpuset.
- in sched_setaffinity, to mask the requested CPUs by what's
allowed in that task's cpuset.
- in sched.c migrate_live_tasks(), to keep migrating tasks within
the CPUs allowed by their cpuset, if possible.
- in the mbind and set_mempolicy system calls, to mask the requested
Memory Nodes by what's allowed in that task's cpuset.
- in page_alloc.c, to restrict memory to allowed nodes.
- in vmscan.c, to restrict page recovery to the current cpuset.
You should mount the "cgroup" filesystem type in order to enable
browsing and modifying the cpusets presently known to the kernel. No
new system calls are added for cpusets - all support for querying and
modifying cpusets is via this cpuset file system.
The /proc/<pid>/status file for each task has four added lines,
displaying the task's cpus_allowed (on which CPUs it may be scheduled)
and mems_allowed (on which Memory Nodes it may obtain memory),
in the two formats seen in the following example::
Cpus_allowed: ffffffff,ffffffff,ffffffff,ffffffff
Cpus_allowed_list: 0-127
Mems_allowed: ffffffff,ffffffff
Mems_allowed_list: 0-63
Each cpuset is represented by a directory in the cgroup file system
containing (on top of the standard cgroup files) the following
files describing that cpuset:
- cpuset.cpus: list of CPUs in that cpuset
- cpuset.mems: list of Memory Nodes in that cpuset
- cpuset.memory_migrate flag: if set, move pages to cpusets nodes
- cpuset.cpu_exclusive flag: is cpu placement exclusive?
- cpuset.mem_exclusive flag: is memory placement exclusive?
- cpuset.mem_hardwall flag: is memory allocation hardwalled
- cpuset.memory_pressure: measure of how much paging pressure in cpuset
- cpuset.memory_spread_page flag: if set, spread page cache evenly on allowed nodes
- cpuset.memory_spread_slab flag: OBSOLETE. Doesn't have any function.
- cpuset.sched_load_balance flag: if set, load balance within CPUs on that cpuset
- cpuset.sched_relax_domain_level: the searching range when migrating tasks
In addition, only the root cpuset has the following file:
- cpuset.memory_pressure_enabled flag: compute memory_pressure?
New cpusets are created using the mkdir system call or shell
command. The properties of a cpuset, such as its flags, allowed
CPUs and Memory Nodes, and attached tasks, are modified by writing
to the appropriate file in that cpusets directory, as listed above.
The named hierarchical structure of nested cpusets allows partitioning
a large system into nested, dynamically changeable, "soft-partitions".
The attachment of each task, automatically inherited at fork by any
children of that task, to a cpuset allows organizing the work load
on a system into related sets of tasks such that each set is constrained
to using the CPUs and Memory Nodes of a particular cpuset. A task
may be re-attached to any other cpuset, if allowed by the permissions
on the necessary cpuset file system directories.
Such management of a system "in the large" integrates smoothly with
the detailed placement done on individual tasks and memory regions
using the sched_setaffinity, mbind and set_mempolicy system calls.
The following rules apply to each cpuset:
- Its CPUs and Memory Nodes must be a subset of its parents.
- It can't be marked exclusive unless its parent is.
- If its cpu or memory is exclusive, they may not overlap any sibling.
These rules, and the natural hierarchy of cpusets, enable efficient
enforcement of the exclusive guarantee, without having to scan all
cpusets every time any of them change to ensure nothing overlaps a
exclusive cpuset. Also, the use of a Linux virtual file system (vfs)
to represent the cpuset hierarchy provides for a familiar permission
and name space for cpusets, with a minimum of additional kernel code.
The cpus and mems files in the root (top_cpuset) cpuset are
read-only. The cpus file automatically tracks the value of
cpu_online_mask using a CPU hotplug notifier, and the mems file
automatically tracks the value of node_states[N_MEMORY]--i.e.,
nodes with memory--using the cpuset_track_online_nodes() hook.
The cpuset.effective_cpus and cpuset.effective_mems files are
normally read-only copies of cpuset.cpus and cpuset.mems files
respectively. If the cpuset cgroup filesystem is mounted with the
special "cpuset_v2_mode" option, the behavior of these files will become
similar to the corresponding files in cpuset v2. In other words, hotplug
events will not change cpuset.cpus and cpuset.mems. Those events will
only affect cpuset.effective_cpus and cpuset.effective_mems which show
the actual cpus and memory nodes that are currently used by this cpuset.
See Documentation/admin-guide/cgroup-v2.rst for more information about
cpuset v2 behavior.
1.4 What are exclusive cpusets ?
--------------------------------
If a cpuset is cpu or mem exclusive, no other cpuset, other than
a direct ancestor or descendant, may share any of the same CPUs or
Memory Nodes.
A cpuset that is cpuset.mem_exclusive *or* cpuset.mem_hardwall is "hardwalled",
i.e. it restricts kernel allocations for page, buffer and other data
commonly shared by the kernel across multiple users. All cpusets,
whether hardwalled or not, restrict allocations of memory for user
space. This enables configuring a system so that several independent
jobs can share common kernel data, such as file system pages, while
isolating each job's user allocation in its own cpuset. To do this,
construct a large mem_exclusive cpuset to hold all the jobs, and
construct child, non-mem_exclusive cpusets for each individual job.
Only a small amount of typical kernel memory, such as requests from
interrupt handlers, is allowed to be taken outside even a
mem_exclusive cpuset.
1.5 What is memory_pressure ?
-----------------------------
The memory_pressure of a cpuset provides a simple per-cpuset metric
of the rate that the tasks in a cpuset are attempting to free up in
use memory on the nodes of the cpuset to satisfy additional memory
requests.
This enables batch managers monitoring jobs running in dedicated
cpusets to efficiently detect what level of memory pressure that job
is causing.
This is useful both on tightly managed systems running a wide mix of
submitted jobs, which may choose to terminate or re-prioritize jobs that
are trying to use more memory than allowed on the nodes assigned to them,
and with tightly coupled, long running, massively parallel scientific
computing jobs that will dramatically fail to meet required performance
goals if they start to use more memory than allowed to them.
This mechanism provides a very economical way for the batch manager
to monitor a cpuset for signs of memory pressure. It's up to the
batch manager or other user code to decide what to do about it and
take action.
==>
Unless this feature is enabled by writing "1" to the special file
/dev/cpuset/memory_pressure_enabled, the hook in the rebalance
code of __alloc_pages() for this metric reduces to simply noticing
that the cpuset_memory_pressure_enabled flag is zero. So only
systems that enable this feature will compute the metric.
Why a per-cpuset, running average:
Because this meter is per-cpuset, rather than per-task or mm,
the system load imposed by a batch scheduler monitoring this
metric is sharply reduced on large systems, because a scan of
the tasklist can be avoided on each set of queries.
Because this meter is a running average, instead of an accumulating
counter, a batch scheduler can detect memory pressure with a
single read, instead of having to read and accumulate results
for a period of time.
Because this meter is per-cpuset rather than per-task or mm,
the batch scheduler can obtain the key information, memory
pressure in a cpuset, with a single read, rather than having to
query and accumulate results over all the (dynamically changing)
set of tasks in the cpuset.
A per-cpuset simple digital filter (requires a spinlock and 3 words
of data per-cpuset) is kept, and updated by any task attached to that
cpuset, if it enters the synchronous (direct) page reclaim code.
A per-cpuset file provides an integer number representing the recent
(half-life of 10 seconds) rate of direct page reclaims caused by
the tasks in the cpuset, in units of reclaims attempted per second,
times 1000.
1.6 What is memory spread ?
---------------------------
There are two boolean flag files per cpuset that control where the
kernel allocates pages for the file system buffers and related in
kernel data structures. They are called 'cpuset.memory_spread_page' and
'cpuset.memory_spread_slab'.
If the per-cpuset boolean flag file 'cpuset.memory_spread_page' is set, then
the kernel will spread the file system buffers (page cache) evenly
over all the nodes that the faulting task is allowed to use, instead
of preferring to put those pages on the node where the task is running.
If the per-cpuset boolean flag file 'cpuset.memory_spread_slab' is set,
then the kernel will spread some file system related slab caches,
such as for inodes and dentries evenly over all the nodes that the
faulting task is allowed to use, instead of preferring to put those
pages on the node where the task is running.
The setting of these flags does not affect anonymous data segment or
stack segment pages of a task.
By default, both kinds of memory spreading are off, and memory
pages are allocated on the node local to where the task is running,
except perhaps as modified by the task's NUMA mempolicy or cpuset
configuration, so long as sufficient free memory pages are available.
When new cpusets are created, they inherit the memory spread settings
of their parent.
Setting memory spreading causes allocations for the affected page
or slab caches to ignore the task's NUMA mempolicy and be spread
instead. Tasks using mbind() or set_mempolicy() calls to set NUMA
mempolicies will not notice any change in these calls as a result of
their containing task's memory spread settings. If memory spreading
is turned off, then the currently specified NUMA mempolicy once again
applies to memory page allocations.
Both 'cpuset.memory_spread_page' and 'cpuset.memory_spread_slab' are boolean flag
files. By default they contain "0", meaning that the feature is off
for that cpuset. If a "1" is written to that file, then that turns
the named feature on.
The implementation is simple.
Setting the flag 'cpuset.memory_spread_page' turns on a per-process flag
PFA_SPREAD_PAGE for each task that is in that cpuset or subsequently
joins that cpuset. The page allocation calls for the page cache
is modified to perform an inline check for this PFA_SPREAD_PAGE task
flag, and if set, a call to a new routine cpuset_mem_spread_node()
returns the node to prefer for the allocation.
Similarly, setting 'cpuset.memory_spread_slab' turns on the flag
PFA_SPREAD_SLAB, and appropriately marked slab caches will allocate
pages from the node returned by cpuset_mem_spread_node().
The cpuset_mem_spread_node() routine is also simple. It uses the
value of a per-task rotor cpuset_mem_spread_rotor to select the next
node in the current task's mems_allowed to prefer for the allocation.
This memory placement policy is also known (in other contexts) as
round-robin or interleave.
This policy can provide substantial improvements for jobs that need
to place thread local data on the corresponding node, but that need
to access large file system data sets that need to be spread across
the several nodes in the jobs cpuset in order to fit. Without this
policy, especially for jobs that might have one thread reading in the
data set, the memory allocation across the nodes in the jobs cpuset
can become very uneven.
1.7 What is sched_load_balance ?
--------------------------------
The kernel scheduler (kernel/sched/core.c) automatically load balances
tasks. If one CPU is underutilized, kernel code running on that
CPU will look for tasks on other more overloaded CPUs and move those
tasks to itself, within the constraints of such placement mechanisms
as cpusets and sched_setaffinity.
The algorithmic cost of load balancing and its impact on key shared
kernel data structures such as the task list increases more than
linearly with the number of CPUs being balanced. So the scheduler
has support to partition the systems CPUs into a number of sched
domains such that it only load balances within each sched domain.
Each sched domain covers some subset of the CPUs in the system;
no two sched domains overlap; some CPUs might not be in any sched
domain and hence won't be load balanced.
Put simply, it costs less to balance between two smaller sched domains
than one big one, but doing so means that overloads in one of the
two domains won't be load balanced to the other one.
By default, there is one sched domain covering all CPUs, including those
marked isolated using the kernel boot time "isolcpus=" argument. However,
the isolated CPUs will not participate in load balancing, and will not
have tasks running on them unless explicitly assigned.
This default load balancing across all CPUs is not well suited for
the following two situations:
1) On large systems, load balancing across many CPUs is expensive.
If the system is managed using cpusets to place independent jobs
on separate sets of CPUs, full load balancing is unnecessary.
2) Systems supporting realtime on some CPUs need to minimize
system overhead on those CPUs, including avoiding task load
balancing if that is not needed.
When the per-cpuset flag "cpuset.sched_load_balance" is enabled (the default
setting), it requests that all the CPUs in that cpusets allowed 'cpuset.cpus'
be contained in a single sched domain, ensuring that load balancing
can move a task (not otherwised pinned, as by sched_setaffinity)
from any CPU in that cpuset to any other.
When the per-cpuset flag "cpuset.sched_load_balance" is disabled, then the
scheduler will avoid load balancing across the CPUs in that cpuset,
--except-- in so far as is necessary because some overlapping cpuset
has "sched_load_balance" enabled.
So, for example, if the top cpuset has the flag "cpuset.sched_load_balance"
enabled, then the scheduler will have one sched domain covering all
CPUs, and the setting of the "cpuset.sched_load_balance" flag in any other
cpusets won't matter, as we're already fully load balancing.
Therefore in the above two situations, the top cpuset flag
"cpuset.sched_load_balance" should be disabled, and only some of the smaller,
child cpusets have this flag enabled.
When doing this, you don't usually want to leave any unpinned tasks in
the top cpuset that might use non-trivial amounts of CPU, as such tasks
may be artificially constrained to some subset of CPUs, depending on
the particulars of this flag setting in descendant cpusets. Even if
such a task could use spare CPU cycles in some other CPUs, the kernel
scheduler might not consider the possibility of load balancing that
task to that underused CPU.
Of course, tasks pinned to a particular CPU can be left in a cpuset
that disables "cpuset.sched_load_balance" as those tasks aren't going anywhere
else anyway.
There is an impedance mismatch here, between cpusets and sched domains.
Cpusets are hierarchical and nest. Sched domains are flat; they don't
overlap and each CPU is in at most one sched domain.
It is necessary for sched domains to be flat because load balancing
across partially overlapping sets of CPUs would risk unstable dynamics
that would be beyond our understanding. So if each of two partially
overlapping cpusets enables the flag 'cpuset.sched_load_balance', then we
form a single sched domain that is a superset of both. We won't move
a task to a CPU outside its cpuset, but the scheduler load balancing
code might waste some compute cycles considering that possibility.
This mismatch is why there is not a simple one-to-one relation
between which cpusets have the flag "cpuset.sched_load_balance" enabled,
and the sched domain configuration. If a cpuset enables the flag, it
will get balancing across all its CPUs, but if it disables the flag,
it will only be assured of no load balancing if no other overlapping
cpuset enables the flag.
If two cpusets have partially overlapping 'cpuset.cpus' allowed, and only
one of them has this flag enabled, then the other may find its
tasks only partially load balanced, just on the overlapping CPUs.
This is just the general case of the top_cpuset example given a few
paragraphs above. In the general case, as in the top cpuset case,
don't leave tasks that might use non-trivial amounts of CPU in
such partially load balanced cpusets, as they may be artificially
constrained to some subset of the CPUs allowed to them, for lack of
load balancing to the other CPUs.
CPUs in "cpuset.isolcpus" were excluded from load balancing by the
isolcpus= kernel boot option, and will never be load balanced regardless
of the value of "cpuset.sched_load_balance" in any cpuset.
1.7.1 sched_load_balance implementation details.
------------------------------------------------
The per-cpuset flag 'cpuset.sched_load_balance' defaults to enabled (contrary
to most cpuset flags.) When enabled for a cpuset, the kernel will
ensure that it can load balance across all the CPUs in that cpuset
(makes sure that all the CPUs in the cpus_allowed of that cpuset are
in the same sched domain.)
If two overlapping cpusets both have 'cpuset.sched_load_balance' enabled,
then they will be (must be) both in the same sched domain.
If, as is the default, the top cpuset has 'cpuset.sched_load_balance' enabled,
then by the above that means there is a single sched domain covering
the whole system, regardless of any other cpuset settings.
The kernel commits to user space that it will avoid load balancing
where it can. It will pick as fine a granularity partition of sched
domains as it can while still providing load balancing for any set
of CPUs allowed to a cpuset having 'cpuset.sched_load_balance' enabled.
The internal kernel cpuset to scheduler interface passes from the
cpuset code to the scheduler code a partition of the load balanced
CPUs in the system. This partition is a set of subsets (represented
as an array of struct cpumask) of CPUs, pairwise disjoint, that cover
all the CPUs that must be load balanced.
The cpuset code builds a new such partition and passes it to the
scheduler sched domain setup code, to have the sched domains rebuilt
as necessary, whenever:
- the 'cpuset.sched_load_balance' flag of a cpuset with non-empty CPUs changes,
- or CPUs come or go from a cpuset with this flag enabled,
- or 'cpuset.sched_relax_domain_level' value of a cpuset with non-empty CPUs
and with this flag enabled changes,
- or a cpuset with non-empty CPUs and with this flag enabled is removed,
- or a cpu is offlined/onlined.
This partition exactly defines what sched domains the scheduler should
setup - one sched domain for each element (struct cpumask) in the
partition.
The scheduler remembers the currently active sched domain partitions.
When the scheduler routine partition_sched_domains() is invoked from
the cpuset code to update these sched domains, it compares the new
partition requested with the current, and updates its sched domains,
removing the old and adding the new, for each change.
1.8 What is sched_relax_domain_level ?
--------------------------------------
In sched domain, the scheduler migrates tasks in 2 ways; periodic load
balance on tick, and at time of some schedule events.
When a task is woken up, scheduler try to move the task on idle CPU.
For example, if a task A running on CPU X activates another task B
on the same CPU X, and if CPU Y is X's sibling and performing idle,
then scheduler migrate task B to CPU Y so that task B can start on
CPU Y without waiting task A on CPU X.
And if a CPU run out of tasks in its runqueue, the CPU try to pull
extra tasks from other busy CPUs to help them before it is going to
be idle.
Of course it takes some searching cost to find movable tasks and/or
idle CPUs, the scheduler might not search all CPUs in the domain
every time. In fact, in some architectures, the searching ranges on
events are limited in the same socket or node where the CPU locates,
while the load balance on tick searches all.
For example, assume CPU Z is relatively far from CPU X. Even if CPU Z
is idle while CPU X and the siblings are busy, scheduler can't migrate
woken task B from X to Z since it is out of its searching range.
As the result, task B on CPU X need to wait task A or wait load balance
on the next tick. For some applications in special situation, waiting
1 tick may be too long.
The 'cpuset.sched_relax_domain_level' file allows you to request changing
this searching range as you like. This file takes int value which
indicates size of searching range in levels approximately as follows,
otherwise initial value -1 that indicates the cpuset has no request.
====== ===========================================================
-1 no request. use system default or follow request of others.
0 no search.
1 search siblings (hyperthreads in a core).
2 search cores in a package.
3 search cpus in a node [= system wide on non-NUMA system]
4 search nodes in a chunk of node [on NUMA system]
5 search system wide [on NUMA system]
====== ===========================================================
Not all levels can be present and values can change depending on the
system architecture and kernel configuration. Check
/sys/kernel/debug/sched/domains/cpu*/domain*/ for system-specific
details.
The system default is architecture dependent. The system default
can be changed using the relax_domain_level= boot parameter.
This file is per-cpuset and affect the sched domain where the cpuset
belongs to. Therefore if the flag 'cpuset.sched_load_balance' of a cpuset
is disabled, then 'cpuset.sched_relax_domain_level' have no effect since
there is no sched domain belonging the cpuset.
If multiple cpusets are overlapping and hence they form a single sched
domain, the largest value among those is used. Be careful, if one
requests 0 and others are -1 then 0 is used.
Note that modifying this file will have both good and bad effects,
and whether it is acceptable or not depends on your situation.
Don't modify this file if you are not sure.
If your situation is:
- The migration costs between each cpu can be assumed considerably
small(for you) due to your special application's behavior or
special hardware support for CPU cache etc.
- The searching cost doesn't have impact(for you) or you can make
the searching cost enough small by managing cpuset to compact etc.
- The latency is required even it sacrifices cache hit rate etc.
then increasing 'sched_relax_domain_level' would benefit you.
1.9 How do I use cpusets ?
--------------------------
In order to minimize the impact of cpusets on critical kernel
code, such as the scheduler, and due to the fact that the kernel
does not support one task updating the memory placement of another
task directly, the impact on a task of changing its cpuset CPU
or Memory Node placement, or of changing to which cpuset a task
is attached, is subtle.
If a cpuset has its Memory Nodes modified, then for each task attached
to that cpuset, the next time that the kernel attempts to allocate
a page of memory for that task, the kernel will notice the change
in the task's cpuset, and update its per-task memory placement to
remain within the new cpusets memory placement. If the task was using
mempolicy MPOL_BIND, and the nodes to which it was bound overlap with
its new cpuset, then the task will continue to use whatever subset
of MPOL_BIND nodes are still allowed in the new cpuset. If the task
was using MPOL_BIND and now none of its MPOL_BIND nodes are allowed
in the new cpuset, then the task will be essentially treated as if it
was MPOL_BIND bound to the new cpuset (even though its NUMA placement,
as queried by get_mempolicy(), doesn't change). If a task is moved
from one cpuset to another, then the kernel will adjust the task's
memory placement, as above, the next time that the kernel attempts
to allocate a page of memory for that task.
If a cpuset has its 'cpuset.cpus' modified, then each task in that cpuset
will have its allowed CPU placement changed immediately. Similarly,
if a task's pid is written to another cpuset's 'tasks' file, then its
allowed CPU placement is changed immediately. If such a task had been
bound to some subset of its cpuset using the sched_setaffinity() call,
the task will be allowed to run on any CPU allowed in its new cpuset,
negating the effect of the prior sched_setaffinity() call.
In summary, the memory placement of a task whose cpuset is changed is
updated by the kernel, on the next allocation of a page for that task,
and the processor placement is updated immediately.
Normally, once a page is allocated (given a physical page
of main memory) then that page stays on whatever node it
was allocated, so long as it remains allocated, even if the
cpusets memory placement policy 'cpuset.mems' subsequently changes.
If the cpuset flag file 'cpuset.memory_migrate' is set true, then when
tasks are attached to that cpuset, any pages that task had
allocated to it on nodes in its previous cpuset are migrated
to the task's new cpuset. The relative placement of the page within
the cpuset is preserved during these migration operations if possible.
For example if the page was on the second valid node of the prior cpuset
then the page will be placed on the second valid node of the new cpuset.
Also if 'cpuset.memory_migrate' is set true, then if that cpuset's
'cpuset.mems' file is modified, pages allocated to tasks in that
cpuset, that were on nodes in the previous setting of 'cpuset.mems',
will be moved to nodes in the new setting of 'mems.'
Pages that were not in the task's prior cpuset, or in the cpuset's
prior 'cpuset.mems' setting, will not be moved.
There is an exception to the above. If hotplug functionality is used
to remove all the CPUs that are currently assigned to a cpuset,
then all the tasks in that cpuset will be moved to the nearest ancestor
with non-empty cpus. But the moving of some (or all) tasks might fail if
cpuset is bound with another cgroup subsystem which has some restrictions
on task attaching. In this failing case, those tasks will stay
in the original cpuset, and the kernel will automatically update
their cpus_allowed to allow all online CPUs. When memory hotplug
functionality for removing Memory Nodes is available, a similar exception
is expected to apply there as well. In general, the kernel prefers to
violate cpuset placement, over starving a task that has had all
its allowed CPUs or Memory Nodes taken offline.
There is a second exception to the above. GFP_ATOMIC requests are
kernel internal allocations that must be satisfied, immediately.
The kernel may drop some request, in rare cases even panic, if a
GFP_ATOMIC alloc fails. If the request cannot be satisfied within
the current task's cpuset, then we relax the cpuset, and look for
memory anywhere we can find it. It's better to violate the cpuset
than stress the kernel.
To start a new job that is to be contained within a cpuset, the steps are:
1) mkdir /sys/fs/cgroup/cpuset
2) mount -t cgroup -ocpuset cpuset /sys/fs/cgroup/cpuset
3) Create the new cpuset by doing mkdir's and write's (or echo's) in
the /sys/fs/cgroup/cpuset virtual file system.
4) Start a task that will be the "founding father" of the new job.
5) Attach that task to the new cpuset by writing its pid to the
/sys/fs/cgroup/cpuset tasks file for that cpuset.
6) fork, exec or clone the job tasks from this founding father task.
For example, the following sequence of commands will setup a cpuset
named "Charlie", containing just CPUs 2 and 3, and Memory Node 1,
and then start a subshell 'sh' in that cpuset::
mount -t cgroup -ocpuset cpuset /sys/fs/cgroup/cpuset
cd /sys/fs/cgroup/cpuset
mkdir Charlie
cd Charlie
/bin/echo 2-3 > cpuset.cpus
/bin/echo 1 > cpuset.mems
/bin/echo $$ > tasks
sh
# The subshell 'sh' is now running in cpuset Charlie
# The next line should display '/Charlie'
cat /proc/self/cpuset
There are ways to query or modify cpusets:
- via the cpuset file system directly, using the various cd, mkdir, echo,
cat, rmdir commands from the shell, or their equivalent from C.
- via the C library libcpuset.
- via the C library libcgroup.
(https://github.com/libcgroup/libcgroup/)
- via the python application cset.
(http://code.google.com/p/cpuset/)
The sched_setaffinity calls can also be done at the shell prompt using
SGI's runon or Robert Love's taskset. The mbind and set_mempolicy
calls can be done at the shell prompt using the numactl command
(part of Andi Kleen's numa package).
2. Usage Examples and Syntax
============================
2.1 Basic Usage
---------------
Creating, modifying, using the cpusets can be done through the cpuset
virtual filesystem.
To mount it, type:
# mount -t cgroup -o cpuset cpuset /sys/fs/cgroup/cpuset
Then under /sys/fs/cgroup/cpuset you can find a tree that corresponds to the
tree of the cpusets in the system. For instance, /sys/fs/cgroup/cpuset
is the cpuset that holds the whole system.
If you want to create a new cpuset under /sys/fs/cgroup/cpuset::
# cd /sys/fs/cgroup/cpuset
# mkdir my_cpuset
Now you want to do something with this cpuset::
# cd my_cpuset
In this directory you can find several files::
# ls
cgroup.clone_children cpuset.memory_pressure
cgroup.event_control cpuset.memory_spread_page
cgroup.procs cpuset.memory_spread_slab
cpuset.cpu_exclusive cpuset.mems
cpuset.cpus cpuset.sched_load_balance
cpuset.mem_exclusive cpuset.sched_relax_domain_level
cpuset.mem_hardwall notify_on_release
cpuset.memory_migrate tasks
Reading them will give you information about the state of this cpuset:
the CPUs and Memory Nodes it can use, the processes that are using
it, its properties. By writing to these files you can manipulate
the cpuset.
Set some flags::
# /bin/echo 1 > cpuset.cpu_exclusive
Add some cpus::
# /bin/echo 0-7 > cpuset.cpus
Add some mems::
# /bin/echo 0-7 > cpuset.mems
Now attach your shell to this cpuset::
# /bin/echo $$ > tasks
You can also create cpusets inside your cpuset by using mkdir in this
directory::
# mkdir my_sub_cs
To remove a cpuset, just use rmdir::
# rmdir my_sub_cs
This will fail if the cpuset is in use (has cpusets inside, or has
processes attached).
Note that for legacy reasons, the "cpuset" filesystem exists as a
wrapper around the cgroup filesystem.
The command::
mount -t cpuset X /sys/fs/cgroup/cpuset
is equivalent to::
mount -t cgroup -ocpuset,noprefix X /sys/fs/cgroup/cpuset
echo "/sbin/cpuset_release_agent" > /sys/fs/cgroup/cpuset/release_agent
2.2 Adding/removing cpus
------------------------
This is the syntax to use when writing in the cpus or mems files
in cpuset directories::
# /bin/echo 1-4 > cpuset.cpus -> set cpus list to cpus 1,2,3,4
# /bin/echo 1,2,3,4 > cpuset.cpus -> set cpus list to cpus 1,2,3,4
To add a CPU to a cpuset, write the new list of CPUs including the
CPU to be added. To add 6 to the above cpuset::
# /bin/echo 1-4,6 > cpuset.cpus -> set cpus list to cpus 1,2,3,4,6
Similarly to remove a CPU from a cpuset, write the new list of CPUs
without the CPU to be removed.
To remove all the CPUs::
# /bin/echo "" > cpuset.cpus -> clear cpus list
2.3 Setting flags
-----------------
The syntax is very simple::
# /bin/echo 1 > cpuset.cpu_exclusive -> set flag 'cpuset.cpu_exclusive'
# /bin/echo 0 > cpuset.cpu_exclusive -> unset flag 'cpuset.cpu_exclusive'
2.4 Attaching processes
-----------------------
::
# /bin/echo PID > tasks
Note that it is PID, not PIDs. You can only attach ONE task at a time.
If you have several tasks to attach, you have to do it one after another::
# /bin/echo PID1 > tasks
# /bin/echo PID2 > tasks
...
# /bin/echo PIDn > tasks
3. Questions
============
Q:
what's up with this '/bin/echo' ?
A:
bash's builtin 'echo' command does not check calls to write() against
errors. If you use it in the cpuset file system, you won't be
able to tell whether a command succeeded or failed.
Q:
When I attach processes, only the first of the line gets really attached !
A:
We can only return one error code per call to write(). So you should also
put only ONE pid.
4. Contact
==========
Web: http://www.bullopensource.org/cpuset
linux-7.2.3/Documentation/admin-guide/cgroup-v1/devices.rst
+132
View File
@@ -0,0 +1,132 @@
===========================
Device Whitelist Controller
===========================
1. Description
==============
Implement a cgroup to track and enforce open and mknod restrictions
on device files. A device cgroup associates a device access
whitelist with each cgroup. A whitelist entry has 4 fields.
'type' is a (all), c (char), or b (block). 'all' means it applies
to all types and all major and minor numbers. Major and minor are
either an integer or * for all. Access is a composition of r
(read), w (write), and m (mknod).
The root device cgroup starts with rwm to 'all'. A child device
cgroup gets a copy of the parent. Administrators can then remove
devices from the whitelist or add new entries. A child cgroup can
never receive a device access which is denied by its parent.
2. User Interface
=================
An entry is added using devices.allow, and removed using
devices.deny. For instance::
echo 'c 1:3 mr' > /sys/fs/cgroup/1/devices.allow
allows cgroup 1 to read and mknod the device usually known as
/dev/null. Doing::
echo a > /sys/fs/cgroup/1/devices.deny
will remove the default 'a *:* rwm' entry. Doing::
echo a > /sys/fs/cgroup/1/devices.allow
will add the 'a *:* rwm' entry to the whitelist.
3. Security
===========
Any task can move itself between cgroups. This clearly won't
suffice, but we can decide the best way to adequately restrict
movement as people get some experience with this. We may just want
to require CAP_SYS_ADMIN, which at least is a separate bit from
CAP_MKNOD. We may want to just refuse moving to a cgroup which
isn't a descendant of the current one. Or we may want to use
CAP_MAC_ADMIN, since we really are trying to lock down root.
CAP_SYS_ADMIN is needed to modify the whitelist or move another
task to a new cgroup. (Again we'll probably want to change that).
A cgroup may not be granted more permissions than the cgroup's
parent has.
4. Hierarchy
============
device cgroups maintain hierarchy by making sure a cgroup never has more
access permissions than its parent. Every time an entry is written to
a cgroup's devices.deny file, all its children will have that entry removed
from their whitelist and all the locally set whitelist entries will be
re-evaluated. In case one of the locally set whitelist entries would provide
more access than the cgroup's parent, it'll be removed from the whitelist.
Example::
A
/ \
B
group behavior exceptions
A allow "b 8:* rwm", "c 116:1 rw"
B deny "c 1:3 rwm", "c 116:2 rwm", "b 3:* rwm"
If a device is denied in group A::
# echo "c 116:* r" > A/devices.deny
it'll propagate down and after revalidating B's entries, the whitelist entry
"c 116:2 rwm" will be removed::
group whitelist entries denied devices
A all "b 8:* rwm", "c 116:* rw"
B "c 1:3 rwm", "b 3:* rwm" all the rest
In case parent's exceptions change and local exceptions are not allowed
anymore, they'll be deleted.
Notice that new whitelist entries will not be propagated::
A
/ \
B
group whitelist entries denied devices
A "c 1:3 rwm", "c 1:5 r" all the rest
B "c 1:3 rwm", "c 1:5 r" all the rest
when adding ``c *:3 rwm``::
# echo "c *:3 rwm" >A/devices.allow
the result::
group whitelist entries denied devices
A "c *:3 rwm", "c 1:5 r" all the rest
B "c 1:3 rwm", "c 1:5 r" all the rest
but now it'll be possible to add new entries to B::
# echo "c 2:3 rwm" >B/devices.allow
# echo "c 50:3 r" >B/devices.allow
or even::
# echo "c *:3 rwm" >B/devices.allow
Allowing or denying all by writing 'a' to devices.allow or devices.deny will
not be possible once the device cgroups has children.
4.1 Hierarchy (internal implementation)
---------------------------------------
device cgroups is implemented internally using a behavior (ALLOW, DENY) and a
list of exceptions. The internal state is controlled using the same user
interface to preserve compatibility with the previous whitelist-only
implementation. Removal or addition of exceptions that will reduce the access
to devices will be propagated down the hierarchy.
For every propagated exception, the effective rules will be re-evaluated based
on current parent's access rules.
linux-7.2.3/Documentation/admin-guide/cgroup-v1/freezer-subsystem.rst
+131
View File
@@ -0,0 +1,131 @@
==============
Cgroup Freezer
==============
The cgroup freezer is useful to batch job management system which start
and stop sets of tasks in order to schedule the resources of a machine
according to the desires of a system administrator. This sort of program
is often used on HPC clusters to schedule access to the cluster as a
whole. The cgroup freezer uses cgroups to describe the set of tasks to
be started/stopped by the batch job management system. It also provides
a means to start and stop the tasks composing the job.
The cgroup freezer will also be useful for checkpointing running groups
of tasks. The freezer allows the checkpoint code to obtain a consistent
image of the tasks by attempting to force the tasks in a cgroup into a
quiescent state. Once the tasks are quiescent another task can
walk /proc or invoke a kernel interface to gather information about the
quiesced tasks. Checkpointed tasks can be restarted later should a
recoverable error occur. This also allows the checkpointed tasks to be
migrated between nodes in a cluster by copying the gathered information
to another node and restarting the tasks there.
Sequences of SIGSTOP and SIGCONT are not always sufficient for stopping
and resuming tasks in userspace. Both of these signals are observable
from within the tasks we wish to freeze. While SIGSTOP cannot be caught,
blocked, or ignored it can be seen by waiting or ptracing parent tasks.
SIGCONT is especially unsuitable since it can be caught by the task. Any
programs designed to watch for SIGSTOP and SIGCONT could be broken by
attempting to use SIGSTOP and SIGCONT to stop and resume tasks. We can
demonstrate this problem using nested bash shells::
$ echo $$
16644
$ bash
$ echo $$
16690
From a second, unrelated bash shell:
$ kill -SIGSTOP 16690
$ kill -SIGCONT 16690
<at this point 16690 exits and causes 16644 to exit too>
This happens because bash can observe both signals and choose how it
responds to them.
Another example of a program which catches and responds to these
signals is gdb. In fact any program designed to use ptrace is likely to
have a problem with this method of stopping and resuming tasks.
In contrast, the cgroup freezer uses the kernel freezer code to
prevent the freeze/unfreeze cycle from becoming visible to the tasks
being frozen. This allows the bash example above and gdb to run as
expected.
The cgroup freezer is hierarchical. Freezing a cgroup freezes all
tasks belonging to the cgroup and all its descendant cgroups. Each
cgroup has its own state (self-state) and the state inherited from the
parent (parent-state). Iff both states are THAWED, the cgroup is
THAWED.
The following cgroupfs files are created by cgroup freezer.
* freezer.state: Read-write.
When read, returns the effective state of the cgroup - "THAWED",
"FREEZING" or "FROZEN". This is the combined self and parent-states.
If any is freezing, the cgroup is freezing (FREEZING or FROZEN).
FREEZING cgroup transitions into FROZEN state when all tasks
belonging to the cgroup and its descendants become frozen. Note that
a cgroup reverts to FREEZING from FROZEN after a new task is added
to the cgroup or one of its descendant cgroups until the new task is
frozen.
When written, sets the self-state of the cgroup. Two values are
allowed - "FROZEN" and "THAWED". If FROZEN is written, the cgroup,
if not already freezing, enters FREEZING state along with all its
descendant cgroups.
If THAWED is written, the self-state of the cgroup is changed to
THAWED. Note that the effective state may not change to THAWED if
the parent-state is still freezing. If a cgroup's effective state
becomes THAWED, all its descendants which are freezing because of
the cgroup also leave the freezing state.
* freezer.self_freezing: Read only.
Shows the self-state. 0 if the self-state is THAWED; otherwise, 1.
This value is 1 iff the last write to freezer.state was "FROZEN".
* freezer.parent_freezing: Read only.
Shows the parent-state. 0 if none of the cgroup's ancestors is
frozen; otherwise, 1.
The root cgroup is non-freezable and the above interface files don't
exist.
* Examples of usage::
# mkdir /sys/fs/cgroup/freezer
# mount -t cgroup -ofreezer freezer /sys/fs/cgroup/freezer
# mkdir /sys/fs/cgroup/freezer/0
# echo $some_pid > /sys/fs/cgroup/freezer/0/tasks
to get status of the freezer subsystem::
# cat /sys/fs/cgroup/freezer/0/freezer.state
THAWED
to freeze all tasks in the container::
# echo FROZEN > /sys/fs/cgroup/freezer/0/freezer.state
# cat /sys/fs/cgroup/freezer/0/freezer.state
FREEZING
# cat /sys/fs/cgroup/freezer/0/freezer.state
FROZEN
to unfreeze all tasks in the container::
# echo THAWED > /sys/fs/cgroup/freezer/0/freezer.state
# cat /sys/fs/cgroup/freezer/0/freezer.state
THAWED
This is the basic mechanism which should do the right thing for user space task
in a simple scenario.
This freezer implementation is affected by shortcomings (see commit
76f969e8948d8 ("cgroup: cgroup v2 freezer")) and cgroup v2 freezer is
recommended.
linux-7.2.3/Documentation/admin-guide/cgroup-v1/hugetlb.rst
+139
View File
@@ -0,0 +1,139 @@
==================
HugeTLB Controller
==================
HugeTLB controller can be created by first mounting the cgroup filesystem.
# mount -t cgroup -o hugetlb none /sys/fs/cgroup
With the above step, the initial or the parent HugeTLB group becomes
visible at /sys/fs/cgroup. At bootup, this group includes all the tasks in
the system. /sys/fs/cgroup/tasks lists the tasks in this cgroup.
New groups can be created under the parent group /sys/fs/cgroup::
# cd /sys/fs/cgroup
# mkdir g1
# echo $$ > g1/tasks
The above steps create a new group g1 and move the current shell
process (bash) into it.
Brief summary of control files::
hugetlb.<hugepagesize>.rsvd.limit_in_bytes # set/show limit of "hugepagesize" hugetlb reservations
hugetlb.<hugepagesize>.rsvd.max_usage_in_bytes # show max "hugepagesize" hugetlb reservations and no-reserve faults
hugetlb.<hugepagesize>.rsvd.usage_in_bytes # show current reservations and no-reserve faults for "hugepagesize" hugetlb
hugetlb.<hugepagesize>.rsvd.failcnt # show the number of allocation failure due to HugeTLB reservation limit
hugetlb.<hugepagesize>.limit_in_bytes # set/show limit of "hugepagesize" hugetlb faults
hugetlb.<hugepagesize>.max_usage_in_bytes # show max "hugepagesize" hugetlb usage recorded
hugetlb.<hugepagesize>.usage_in_bytes # show current usage for "hugepagesize" hugetlb
hugetlb.<hugepagesize>.failcnt # show the number of allocation failure due to HugeTLB usage limit
hugetlb.<hugepagesize>.numa_stat # show the numa information of the hugetlb memory charged to this cgroup
For a system supporting three hugepage sizes (64k, 32M and 1G), the control
files include::
hugetlb.1GB.limit_in_bytes
hugetlb.1GB.max_usage_in_bytes
hugetlb.1GB.numa_stat
hugetlb.1GB.usage_in_bytes
hugetlb.1GB.failcnt
hugetlb.1GB.rsvd.limit_in_bytes
hugetlb.1GB.rsvd.max_usage_in_bytes
hugetlb.1GB.rsvd.usage_in_bytes
hugetlb.1GB.rsvd.failcnt
hugetlb.64KB.limit_in_bytes
hugetlb.64KB.max_usage_in_bytes
hugetlb.64KB.numa_stat
hugetlb.64KB.usage_in_bytes
hugetlb.64KB.failcnt
hugetlb.64KB.rsvd.limit_in_bytes
hugetlb.64KB.rsvd.max_usage_in_bytes
hugetlb.64KB.rsvd.usage_in_bytes
hugetlb.64KB.rsvd.failcnt
hugetlb.32MB.limit_in_bytes
hugetlb.32MB.max_usage_in_bytes
hugetlb.32MB.numa_stat
hugetlb.32MB.usage_in_bytes
hugetlb.32MB.failcnt
hugetlb.32MB.rsvd.limit_in_bytes
hugetlb.32MB.rsvd.max_usage_in_bytes
hugetlb.32MB.rsvd.usage_in_bytes
hugetlb.32MB.rsvd.failcnt
1. Page fault accounting
::
hugetlb.<hugepagesize>.limit_in_bytes
hugetlb.<hugepagesize>.max_usage_in_bytes
hugetlb.<hugepagesize>.usage_in_bytes
hugetlb.<hugepagesize>.failcnt
The HugeTLB controller allows users to limit the HugeTLB usage (page fault) per
control group and enforces the limit during page fault. Since HugeTLB
doesn't support page reclaim, enforcing the limit at page fault time implies
that, the application will get SIGBUS signal if it tries to fault in HugeTLB
pages beyond its limit. Therefore the application needs to know exactly how many
HugeTLB pages it uses beforehand, and the sysadmin needs to make sure that
there are enough available on the machine for all the users to avoid processes
getting SIGBUS.
2. Reservation accounting
::
hugetlb.<hugepagesize>.rsvd.limit_in_bytes
hugetlb.<hugepagesize>.rsvd.max_usage_in_bytes
hugetlb.<hugepagesize>.rsvd.usage_in_bytes
hugetlb.<hugepagesize>.rsvd.failcnt
The HugeTLB controller allows limiting the HugeTLB reservations per control
group and enforces the controller limit at reservation time and at the fault of
HugeTLB memory for which no reservation exists. Since reservation limits are
enforced at reservation time (on mmap or shget), reservation limits never cause
the application to get SIGBUS signal if the memory was reserved beforehand. For
MAP_NORESERVE allocations, the reservation limit behaves the same as the fault
limit, enforcing memory usage at fault time and causing the application to
receive a SIGBUS if it's crossing its limit.
Reservation limits are superior to page fault limits described above, since
reservation limits are enforced at reservation time (on mmap or shget), and
never cause the application to get SIGBUS signal if the memory was reserved
beforehand. This allows for easier fallback to alternatives such as
non-HugeTLB memory for example. In the case of page fault accounting, it's very
hard to avoid processes getting SIGBUS since the sysadmin needs to precisely know
the HugeTLB usage of all the tasks in the system and make sure there are enough
pages to satisfy all requests. Avoiding tasks getting SIGBUS on overcommitted
systems is practically impossible with page fault accounting.
3. Caveats with shared memory
For shared HugeTLB memory, both HugeTLB reservation and page faults are charged
to the first task that causes the memory to be reserved or faulted, and all
subsequent uses of this reserved or faulted memory is done without charging.
Shared HugeTLB memory is only uncharged when it is unreserved or deallocated.
This is usually when the HugeTLB file is deleted, and not when the task that
caused the reservation or fault has exited.
4. Caveats with HugeTLB cgroup offline.
When a HugeTLB cgroup goes offline with some reservations or faults still
charged to it, the behavior is as follows:
- The fault charges are charged to the parent HugeTLB cgroup (reparented),
- the reservation charges remain on the offline HugeTLB cgroup.
This means that if a HugeTLB cgroup gets offlined while there is still HugeTLB
reservations charged to it, that cgroup persists as a zombie until all HugeTLB
reservations are uncharged. HugeTLB reservations behave in this manner to match
the memory controller whose cgroups also persist as zombie until all charged
memory is uncharged. Also, the tracking of HugeTLB reservations is a bit more
complex compared to the tracking of HugeTLB faults, so it is significantly
harder to reparent reservations at offline time.
linux-7.2.3/Documentation/admin-guide/cgroup-v1/index.rst
+24
View File
@@ -0,0 +1,24 @@
.. _cgroup-v1:
========================
Control Groups version 1
========================
.. toctree::
:maxdepth: 1
cgroups
blkio-controller
cpuacct
cpusets
devices
freezer-subsystem
hugetlb
memcg_test
memory
misc
net_cls
net_prio
pids
rdma
linux-7.2.3/Documentation/admin-guide/cgroup-v1/memcg_test.rst
+342
View File
@@ -0,0 +1,342 @@
=====================================================
Memory Resource Controller(Memcg) Implementation Memo
=====================================================
Last Updated: 2010/2
Base Kernel Version: based on 2.6.33-rc7-mm(candidate for 34).
Because VM is getting complex (one of reasons is memcg...), memcg's behavior
is complex. This is a document for memcg's internal behavior.
Please note that implementation details can be changed.
(*) Topics on API should be in Documentation/admin-guide/cgroup-v1/memory.rst)
0. How to record usage ?
========================
2 objects are used.
page_cgroup ....an object per page.
Allocated at boot or memory hotplug. Freed at memory hot removal.
swap_cgroup ... an entry per swp_entry.
Allocated at swapon(). Freed at swapoff().
The page_cgroup has USED bit and double count against a page_cgroup never
occurs. swap_cgroup is used only when a charged page is swapped-out.
1. Charge
=========
a page/swp_entry may be charged (usage += PAGE_SIZE) at
mem_cgroup_try_charge()
2. Uncharge
===========
a page/swp_entry may be uncharged (usage -= PAGE_SIZE) by
mem_cgroup_uncharge()
Called when a page's refcount goes down to 0.
mem_cgroup_uncharge_swap()
Called when swp_entry's refcnt goes down to 0. A charge against swap
disappears.
3. charge-commit
=======================
Memcg pages are charged in two steps:
- mem_cgroup_try_charge()
- commit_charge()
At try_charge(), there are no flags to say "this page is charged".
at this point, usage += PAGE_SIZE.
At commit(), the page is associated with the memcg.
Under below explanation, we assume CONFIG_SWAP=y.
4. Anonymous
============
Anonymous page is newly allocated at
- page fault into MAP_ANONYMOUS mapping.
- Copy-On-Write.
4.1 Swap-in.
At swap-in, the page is taken from swap-cache. There are 2 cases.
(a) If the SwapCache is newly allocated and read, it has no charges.
(b) If the SwapCache has been mapped by processes, it has been
charged already.
4.2 Swap-out.
At swap-out, typical state transition is below.
(a) add to swap cache. (marked as SwapCache)
swp_entry's refcnt += 1.
(b) fully unmapped.
swp_entry's refcnt += # of ptes.
(c) write back to swap.
(d) delete from swap cache. (remove from SwapCache)
swp_entry's refcnt -= 1.
Finally, at task exit,
(e) zap_pte() is called and swp_entry's refcnt -=1 -> 0.
5. Page Cache
=============
Page Cache is charged at
- filemap_add_folio().
The logic is very clear. (About migration, see below)
Note:
__filemap_remove_folio() is called by filemap_remove_folio()
and __remove_mapping().
6. Shmem(tmpfs) Page Cache
===========================
The best way to understand shmem's page state transition is to read
mm/shmem.c.
But brief explanation of the behavior of memcg around shmem will be
helpful to understand the logic.
Shmem's page (just leaf page, not direct/indirect block) can be on
- radix-tree of shmem's inode.
- SwapCache.
- Both on radix-tree and SwapCache. This happens at swap-in
and swap-out,
It's charged when...
- A new page is added to shmem's radix-tree.
- A swp page is read. (move a charge from swap_cgroup to page_cgroup)
7. Page Migration
=================
mem_cgroup_migrate()
8. LRU
======
Each memcg has its own vector of LRUs (inactive anon, active anon,
inactive file, active file, unevictable) of pages from each node,
each LRU handled under a single lru_lock for that memcg and node.
9. Typical Tests.
=================
Tests for racy cases.
9.1 Small limit to memcg.
-------------------------
When you do test to do racy case, it's good test to set memcg's limit
to be very small rather than GB. Many races found in the test under
xKB or xxMB limits.
(Memory behavior under GB and Memory behavior under MB shows very
different situation.)
9.2 Shmem
---------
Historically, memcg's shmem handling was poor and we saw some amount
of troubles here. This is because shmem is page-cache but can be
SwapCache. Test with shmem/tmpfs is always good test.
9.3 Migration
-------------
For NUMA, migration is an another special case. To do easy test, cpuset
is useful. Following is a sample script to do migration::
mount -t cgroup -o cpuset none /opt/cpuset
mkdir /opt/cpuset/01
echo 1 > /opt/cpuset/01/cpuset.cpus
echo 0 > /opt/cpuset/01/cpuset.mems
echo 1 > /opt/cpuset/01/cpuset.memory_migrate
mkdir /opt/cpuset/02
echo 1 > /opt/cpuset/02/cpuset.cpus
echo 1 > /opt/cpuset/02/cpuset.mems
echo 1 > /opt/cpuset/02/cpuset.memory_migrate
In above set, when you moves a task from 01 to 02, page migration to
node 0 to node 1 will occur. Following is a script to migrate all
under cpuset.::
--
move_task()
{
for pid in $1
do
/bin/echo $pid >$2/tasks 2>/dev/null
echo -n $pid
echo -n " "
done
echo END
}
G1_TASK=`cat ${G1}/tasks`
G2_TASK=`cat ${G2}/tasks`
move_task "${G1_TASK}" ${G2} &
--
9.4 Memory hotplug
------------------
memory hotplug test is one of good test.
to offline memory, do following::
# echo offline > /sys/devices/system/memory/memoryXXX/state
(XXX is the place of memory)
This is an easy way to test page migration, too.
9.5 nested cgroups
------------------
Use tests like the following for testing nested cgroups::
mkdir /opt/cgroup/01/child_a
mkdir /opt/cgroup/01/child_b
set limit to 01.
add limit to 01/child_b
run jobs under child_a and child_b
create/delete following groups at random while jobs are running::
/opt/cgroup/01/child_a/child_aa
/opt/cgroup/01/child_b/child_bb
/opt/cgroup/01/child_c
running new jobs in new group is also good.
9.6 Mount with other subsystems
-------------------------------
Mounting with other subsystems is a good test because there is a
race and lock dependency with other cgroup subsystems.
example::
# mount -t cgroup none /cgroup -o cpuset,memory,cpu,devices
and do task move, mkdir, rmdir etc...under this.
9.7 swapoff
-----------
Besides management of swap is one of complicated parts of memcg,
call path of swap-in at swapoff is not same as usual swap-in path..
It's worth to be tested explicitly.
For example, test like following is good:
(Shell-A)::
# mount -t cgroup none /cgroup -o memory
# mkdir /cgroup/test
# echo 40M > /cgroup/test/memory.limit_in_bytes
# echo 0 > /cgroup/test/tasks
Run malloc(100M) program under this. You'll see 60M of swaps.
(Shell-B)::
# move all tasks in /cgroup/test to /cgroup
# /sbin/swapoff -a
# rmdir /cgroup/test
# kill malloc task.
Of course, tmpfs v.s. swapoff test should be tested, too.
9.8 OOM-Killer
--------------
Out-of-memory caused by memcg's limit will kill tasks under
the memcg. When hierarchy is used, a task under hierarchy
will be killed by the kernel.
In this case, panic_on_oom shouldn't be invoked and tasks
in other groups shouldn't be killed.
It's not difficult to cause OOM under memcg as following.
Case A) when you can swapoff::
#swapoff -a
#echo 50M > /memory.limit_in_bytes
run 51M of malloc
Case B) when you use mem+swap limitation::
#echo 50M > memory.limit_in_bytes
#echo 50M > memory.memsw.limit_in_bytes
run 51M of malloc
9.9 Move charges at task migration
----------------------------------
Charges associated with a task can be moved along with task migration.
(Shell-A)::
#mkdir /cgroup/A
#echo $$ >/cgroup/A/tasks
run some programs which uses some amount of memory in /cgroup/A.
(Shell-B)::
#mkdir /cgroup/B
#echo 1 >/cgroup/B/memory.move_charge_at_immigrate
#echo "pid of the program running in group A" >/cgroup/B/tasks
You can see charges have been moved by reading ``*.usage_in_bytes`` or
memory.stat of both A and B.
See 8.2 of Documentation/admin-guide/cgroup-v1/memory.rst to see what value should
be written to move_charge_at_immigrate.
9.10 Memory thresholds
----------------------
Memory controller implements memory thresholds using cgroups notification
API. You can use samples/cgroup/cgroup_event_listener.c to test it.
(Shell-A) Create cgroup and run event listener::
# mkdir /cgroup/A
# ./cgroup_event_listener /cgroup/A/memory.usage_in_bytes 5M
(Shell-B) Add task to cgroup and try to allocate and free memory::
# echo $$ >/cgroup/A/tasks
# a="$(dd if=/dev/zero bs=1M count=10)"
# a=
You will see message from cgroup_event_listener every time you cross
the thresholds.
Use /cgroup/A/memory.memsw.usage_in_bytes to test memsw thresholds.
It's good idea to test root cgroup as well.
linux-7.2.3/Documentation/admin-guide/cgroup-v1/memory.rst
+963
View File
@@ -0,0 +1,963 @@
==========================
Memory Resource Controller
==========================
.. caution::
This document is hopelessly outdated and it asks for a complete
rewrite. It still contains a useful information so we are keeping it
here but make sure to check the current code if you need a deeper
understanding.
.. note::
The Memory Resource Controller has generically been referred to as the
memory controller in this document. Do not confuse memory controller
used here with the memory controller that is used in hardware.
.. hint::
When we mention a cgroup (cgroupfs's directory) with memory controller,
we call it "memory cgroup". When you see git-log and source code, you'll
see patch's title and function names tend to use "memcg".
In this document, we avoid using it.
Benefits and Purpose of the memory controller
=============================================
The memory controller isolates the memory behaviour of a group of tasks
from the rest of the system. The article on LWN [12]_ mentions some probable
uses of the memory controller. The memory controller can be used to
a. Isolate an application or a group of applications
Memory-hungry applications can be isolated and limited to a smaller
amount of memory.
b. Create a cgroup with a limited amount of memory; this can be used
as a good alternative to booting with mem=XXXX.
c. Virtualization solutions can control the amount of memory they want
to assign to a virtual machine instance.
d. A CD/DVD burner could control the amount of memory used by the
rest of the system to ensure that burning does not fail due to lack
of available memory.
e. There are several other use cases; find one or use the controller just
for fun (to learn and hack on the VM subsystem).
Current Status: linux-2.6.34-mmotm(development version of 2010/April)
Features:
- accounting anonymous pages, file caches, swap caches usage and limiting them.
- pages are linked to per-memcg LRU exclusively, and there is no global LRU.
- optionally, memory+swap usage can be accounted and limited.
- hierarchical accounting
- soft limit
- moving (recharging) account at moving a task is selectable.
- usage threshold notifier
- memory pressure notifier
- oom-killer disable knob and oom-notifier
- Root cgroup has no limit controls.
Kernel memory support is a work in progress, and the current version provides
basically functionality. (See :ref:`section 2.7
<cgroup-v1-memory-kernel-extension>`)
Brief summary of control files.
==================================== ==========================================
tasks attach a task(thread) and show list of
threads
cgroup.procs show list of processes
cgroup.event_control an interface for event_fd()
This knob is not available on CONFIG_PREEMPT_RT systems.
memory.usage_in_bytes show current usage for memory
(See 5.5 for details)
memory.memsw.usage_in_bytes show current usage for memory+Swap
(See 5.5 for details)
memory.limit_in_bytes set/show limit of memory usage
memory.memsw.limit_in_bytes set/show limit of memory+Swap usage
memory.failcnt show the number of memory usage hits limits
memory.memsw.failcnt show the number of memory+Swap hits limits
memory.max_usage_in_bytes show max memory usage recorded
memory.memsw.max_usage_in_bytes show max memory+Swap usage recorded
memory.soft_limit_in_bytes set/show soft limit of memory usage
This knob is not available on CONFIG_PREEMPT_RT systems.
This knob is deprecated and shouldn't be
used.
memory.stat show various statistics
memory.use_hierarchy set/show hierarchical account enabled
This knob is deprecated and shouldn't be
used.
memory.force_empty trigger forced page reclaim
memory.pressure_level set memory pressure notifications
This knob is deprecated and shouldn't be
used.
memory.swappiness set/show swappiness parameter of vmscan
(See sysctl's vm.swappiness)
Per memcg knob does not exist in cgroup v2.
memory.move_charge_at_immigrate This knob is deprecated.
memory.oom_control set/show oom controls.
This knob is deprecated and shouldn't be
used.
memory.numa_stat show the number of memory usage per numa
node
memory.kmem.limit_in_bytes Deprecated knob to set and read the kernel
memory hard limit. Kernel hard limit is not
supported since 5.16. Writing any value to
do file will not have any effect same as if
nokmem kernel parameter was specified.
Kernel memory is still charged and reported
by memory.kmem.usage_in_bytes.
memory.kmem.usage_in_bytes show current kernel memory allocation
memory.kmem.failcnt show the number of kernel memory usage
hits limits
memory.kmem.max_usage_in_bytes show max kernel memory usage recorded
memory.kmem.tcp.limit_in_bytes set/show hard limit for tcp buf memory
This knob is deprecated and shouldn't be
used.
memory.kmem.tcp.usage_in_bytes show current tcp buf memory allocation
This knob is deprecated and shouldn't be
used.
memory.kmem.tcp.failcnt show the number of tcp buf memory usage
hits limits
This knob is deprecated and shouldn't be
used.
memory.kmem.tcp.max_usage_in_bytes show max tcp buf memory usage recorded
This knob is deprecated and shouldn't be
used.
==================================== ==========================================
1. History
==========
The memory controller has a long history. A request for comments for the memory
controller was posted by Balbir Singh [1]_. At the time the RFC was posted
there were several implementations for memory control. The goal of the
RFC was to build consensus and agreement for the minimal features required
for memory control. The first RSS controller was posted by Balbir Singh [2]_
in Feb 2007. Pavel Emelianov [3]_ [4]_ [5]_ has since posted three versions
of the RSS controller. At OLS, at the resource management BoF, everyone
suggested that we handle both page cache and RSS together. Another request was
raised to allow user space handling of OOM. The current memory controller is
at version 6; it combines both mapped (RSS) and unmapped Page
Cache Control [11]_.
2. Memory Control
=================
Memory is a unique resource in the sense that it is present in a limited
amount. If a task requires a lot of CPU processing, the task can spread
its processing over a period of hours, days, months or years, but with
memory, the same physical memory needs to be reused to accomplish the task.
The memory controller implementation has been divided into phases. These
are:
1. Memory controller
2. mlock(2) controller
3. Kernel user memory accounting and slab control
4. user mappings length controller
The memory controller is the first controller developed.
2.1. Design
-----------
The core of the design is a counter called the page_counter. The
page_counter tracks the current memory usage and limit of the group of
processes associated with the controller. Each cgroup has a memory controller
specific data structure (mem_cgroup) associated with it.
2.2. Accounting
---------------
.. code-block::
:caption: Figure 1: Hierarchy of Accounting
+--------------------+
| mem_cgroup |
| (page_counter) |
+--------------------+
/ ^ \
/ | \
+---------------+ | +---------------+
| mm_struct | |.... | mm_struct |
| | | | |
+---------------+ | +---------------+
|
+ --------------+
|
+---------------+ +------+--------+
| page +----------> page_cgroup|
| | | |
+---------------+ +---------------+
Figure 1 shows the important aspects of the controller
1. Accounting happens per cgroup
2. Each mm_struct knows about which cgroup it belongs to
3. Each page has a pointer to the page_cgroup, which in turn knows the
cgroup it belongs to
The accounting is done as follows: mem_cgroup_charge_common() is invoked to
set up the necessary data structures and check if the cgroup that is being
charged is over its limit. If it is, then reclaim is invoked on the cgroup.
More details can be found in the reclaim section of this document.
If everything goes well, a page meta-data-structure called page_cgroup is
updated. page_cgroup has its own LRU on cgroup.
(*) page_cgroup structure is allocated at boot/memory-hotplug time.
2.2.1 Accounting details
------------------------
All mapped anon pages (RSS) and cache pages (Page Cache) are accounted.
Some pages which are never reclaimable and will not be on the LRU
are not accounted. We just account pages under usual VM management.
RSS pages are accounted at page_fault unless they've already been accounted
for earlier. A file page will be accounted for as Page Cache when it's
inserted into inode (xarray). While it's mapped into the page tables of
processes, duplicate accounting is carefully avoided.
An RSS page is unaccounted when it's fully unmapped. A PageCache page is
unaccounted when it's removed from xarray. Even if RSS pages are fully
unmapped (by kswapd), they may exist as SwapCache in the system until they
are really freed. Such SwapCaches are also accounted.
A swapped-in page is accounted after adding into swapcache.
Note: The kernel does swapin-readahead and reads multiple swaps at once.
Since page's memcg recorded into swap whatever memsw enabled, the page will
be accounted after swapin.
At page migration, accounting information is kept.
Note: we just account pages-on-LRU because our purpose is to control amount
of used pages; not-on-LRU pages tend to be out-of-control from VM view.
2.3 Shared Page Accounting
--------------------------
Shared pages are accounted on the basis of the first touch approach. The
cgroup that first touches a page is accounted for the page. The principle
behind this approach is that a cgroup that aggressively uses a shared
page will eventually get charged for it (once it is uncharged from
the cgroup that brought it in -- this will happen on memory pressure).
2.4 Swap Extension
--------------------------------------
Swap usage is always recorded for each of cgroup. Swap Extension allows you to
read and limit it.
When CONFIG_SWAP is enabled, following files are added.
- memory.memsw.usage_in_bytes.
- memory.memsw.limit_in_bytes.
memsw means memory+swap. Usage of memory+swap is limited by
memsw.limit_in_bytes.
Example: Assume a system with 4G of swap. A task which allocates 6G of memory
(by mistake) under 2G memory limitation will use all swap.
In this case, setting memsw.limit_in_bytes=3G will prevent bad use of swap.
By using the memsw limit, you can avoid system OOM which can be caused by swap
shortage.
2.4.1 why 'memory+swap' rather than swap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The global LRU(kswapd) can swap out arbitrary pages. Swap-out means
to move account from memory to swap...there is no change in usage of
memory+swap. In other words, when we want to limit the usage of swap without
affecting global LRU, memory+swap limit is better than just limiting swap from
an OS point of view.
2.4.2. What happens when a cgroup hits memory.memsw.limit_in_bytes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When a cgroup hits memory.memsw.limit_in_bytes, it's useless to do swap-out
in this cgroup. Then, swap-out will not be done by cgroup routine and file
caches are dropped. But as mentioned above, global LRU can do swapout memory
from it for sanity of the system's memory management state. You can't forbid
it by cgroup.
2.5 Reclaim
-----------
Each cgroup maintains a per cgroup LRU which has the same structure as
global VM. When a cgroup goes over its limit, we first try
to reclaim memory from the cgroup so as to make space for the new
pages that the cgroup has touched. If the reclaim is unsuccessful,
an OOM routine is invoked to select and kill the bulkiest task in the
cgroup. (See :ref:`10. OOM Control <cgroup-v1-memory-oom-control>` below.)
The reclaim algorithm has not been modified for cgroups, except that
pages that are selected for reclaiming come from the per-cgroup LRU
list.
.. note::
Reclaim does not work for the root cgroup, since we cannot set any
limits on the root cgroup.
.. note::
When panic_on_oom is set to "2", the whole system will panic.
When oom event notifier is registered, event will be delivered.
(See :ref:`oom_control <cgroup-v1-memory-oom-control>` section)
2.6 Locking
-----------
Lock order is as follows::
folio_lock
mm->page_table_lock or split pte_lock
mapping->i_pages lock
lruvec->lru_lock.
Per-node-per-memcgroup LRU (cgroup's private LRU) is guarded by
lruvec->lru_lock; the folio LRU flag is cleared before
isolating a page from its LRU under lruvec->lru_lock.
.. _cgroup-v1-memory-kernel-extension:
2.7 Kernel Memory Extension
-----------------------------------------------
With the Kernel memory extension, the Memory Controller is able to limit
the amount of kernel memory used by the system. Kernel memory is fundamentally
different than user memory, since it can't be swapped out, which makes it
possible to DoS the system by consuming too much of this precious resource.
Kernel memory accounting is enabled for all memory cgroups by default. But
it can be disabled system-wide by passing cgroup.memory=nokmem to the kernel
at boot time. In this case, kernel memory will not be accounted at all.
Kernel memory limits are not imposed for the root cgroup. Usage for the root
cgroup may or may not be accounted. The memory used is accumulated into
memory.kmem.usage_in_bytes, or in a separate counter when it makes sense.
(currently only for tcp).
The main "kmem" counter is fed into the main counter, so kmem charges will
also be visible from the user counter.
Currently no soft limit is implemented for kernel memory. It is future work
to trigger slab reclaim when those limits are reached.
2.7.1 Current Kernel Memory resources accounted
-----------------------------------------------
stack pages:
every process consumes some stack pages. By accounting into
kernel memory, we prevent new processes from being created when the kernel
memory usage is too high.
slab pages:
pages allocated by the SLAB or SLUB allocator are tracked. A copy
of each kmem_cache is created every time the cache is touched by the first time
from inside the memcg. The creation is done lazily, so some objects can still be
skipped while the cache is being created. All objects in a slab page should
belong to the same memcg. This only fails to hold when a task is migrated to a
different memcg during the page allocation by the cache.
sockets memory pressure:
some sockets protocols have memory pressure
thresholds. The Memory Controller allows them to be controlled individually
per cgroup, instead of globally.
tcp memory pressure:
sockets memory pressure for the tcp protocol.
2.7.2 Common use cases
----------------------
Because the "kmem" counter is fed to the main user counter, kernel memory can
never be limited completely independently of user memory. Say "U" is the user
limit, and "K" the kernel limit. There are three possible ways limits can be
set:
U != 0, K = unlimited:
This is the standard memcg limitation mechanism already present before kmem
accounting. Kernel memory is completely ignored.
U != 0, K < U:
Kernel memory is a subset of the user memory. This setup is useful in
deployments where the total amount of memory per-cgroup is overcommitted.
Overcommitting kernel memory limits is definitely not recommended, since the
box can still run out of non-reclaimable memory.
In this case, the admin could set up K so that the sum of all groups is
never greater than the total memory, and freely set U at the cost of his
QoS.
.. warning::
In the current implementation, memory reclaim will NOT be triggered for
a cgroup when it hits K while staying below U, which makes this setup
impractical.
U != 0, K >= U:
Since kmem charges will also be fed to the user counter and reclaim will be
triggered for the cgroup for both kinds of memory. This setup gives the
admin a unified view of memory, and it is also useful for people who just
want to track kernel memory usage.
3. User Interface
=================
To use the user interface:
1. Enable CONFIG_CGROUPS and CONFIG_MEMCG options
2. Prepare the cgroups (see :ref:`Why are cgroups needed?
<cgroups-why-needed>` for the background information)::
# mount -t tmpfs none /sys/fs/cgroup
# mkdir /sys/fs/cgroup/memory
# mount -t cgroup none /sys/fs/cgroup/memory -o memory
3. Make the new group and move bash into it::
# mkdir /sys/fs/cgroup/memory/0
# echo $$ > /sys/fs/cgroup/memory/0/tasks
4. Since now we're in the 0 cgroup, we can alter the memory limit::
# echo 4M > /sys/fs/cgroup/memory/0/memory.limit_in_bytes
The limit can now be queried::
# cat /sys/fs/cgroup/memory/0/memory.limit_in_bytes
4194304
.. note::
We can use a suffix (k, K, m, M, g or G) to indicate values in kilo,
mega or gigabytes. (Here, Kilo, Mega, Giga are Kibibytes, Mebibytes,
Gibibytes.)
.. note::
We can write "-1" to reset the ``*.limit_in_bytes(unlimited)``.
.. note::
We cannot set limits on the root cgroup any more.
We can check the usage::
# cat /sys/fs/cgroup/memory/0/memory.usage_in_bytes
1216512
A successful write to this file does not guarantee a successful setting of
this limit to the value written into the file. This can be due to a
number of factors, such as rounding up to page boundaries or the total
availability of memory on the system. The user is required to re-read
this file after a write to guarantee the value committed by the kernel::
# echo 1 > memory.limit_in_bytes
# cat memory.limit_in_bytes
4096
The memory.failcnt field gives the number of times that the cgroup limit was
exceeded.
The memory.stat file gives accounting information. Now, the number of
caches, RSS and Active pages/Inactive pages are shown.
4. Testing
==========
For testing features and implementation, see memcg_test.txt.
Performance test is also important. To see pure memory controller's overhead,
testing on tmpfs will give you good numbers of small overheads.
Example: do kernel make on tmpfs.
Page-fault scalability is also important. At measuring parallel
page fault test, multi-process test may be better than multi-thread
test because it has noise of shared objects/status.
But the above two are testing extreme situations.
Trying usual test under memory controller is always helpful.
.. _cgroup-v1-memory-test-troubleshoot:
4.1 Troubleshooting
-------------------
Sometimes a user might find that the application under a cgroup is
terminated by the OOM killer. There are several causes for this:
1. The cgroup limit is too low (just too low to do anything useful)
2. The user is using anonymous memory and swap is turned off or too low
A sync followed by echo 1 > /proc/sys/vm/drop_caches will help get rid of
some of the pages cached in the cgroup (page cache pages).
To know what happens, disabling OOM_Kill as per :ref:`"10. OOM Control"
<cgroup-v1-memory-oom-control>` (below) and seeing what happens will be
helpful.
.. _cgroup-v1-memory-test-task-migration:
4.2 Task migration
------------------
When a task migrates from one cgroup to another, its charge is not
carried forward by default. The pages allocated from the original cgroup still
remain charged to it, the charge is dropped when the page is freed or
reclaimed.
You can move charges of a task along with task migration.
See :ref:`8. "Move charges at task migration" <cgroup-v1-memory-move-charges>`
4.3 Removing a cgroup
---------------------
A cgroup can be removed by rmdir, but as discussed in :ref:`sections 4.1
<cgroup-v1-memory-test-troubleshoot>` and :ref:`4.2
<cgroup-v1-memory-test-task-migration>`, a cgroup might have some charge
associated with it, even though all tasks have migrated away from it. (because
we charge against pages, not against tasks.)
We move the stats to parent, and no change on the charge except uncharging
from the child.
Charges recorded in swap information is not updated at removal of cgroup.
Recorded information is discarded and a cgroup which uses swap (swapcache)
will be charged as a new owner of it.
5. Misc. interfaces
===================
5.1 force_empty
---------------
memory.force_empty interface is provided to make cgroup's memory usage empty.
When writing anything to this::
# echo 0 > memory.force_empty
the cgroup will be reclaimed and as many pages reclaimed as possible.
The typical use case for this interface is before calling rmdir().
Though rmdir() offlines memcg, but the memcg may still stay there due to
charged file caches. Some out-of-use page caches may keep charged until
memory pressure happens. If you want to avoid that, force_empty will be useful.
5.2 stat file
-------------
memory.stat file includes following statistics:
* per-memory cgroup local status
=============== ===============================================================
cache # of bytes of page cache memory.
rss # of bytes of anonymous and swap cache memory (includes
transparent hugepages).
rss_huge # of bytes of anonymous transparent hugepages.
mapped_file # of bytes of mapped file (includes tmpfs/shmem)
pgpgin # of charging events to the memory cgroup. The charging
event happens each time a page is accounted as either mapped
anon page(RSS) or cache page(Page Cache) to the cgroup.
pgpgout # of uncharging events to the memory cgroup. The uncharging
event happens each time a page is unaccounted from the
cgroup.
swap # of bytes of swap usage
swapcached # of bytes of swap cached in memory
dirty # of bytes that are waiting to get written back to the disk.
writeback # of bytes of file/anon cache that are queued for syncing to
disk.
inactive_anon # of bytes of anonymous and swap cache memory on inactive
LRU list.
active_anon # of bytes of anonymous and swap cache memory on active
LRU list.
inactive_file # of bytes of file-backed memory and MADV_FREE anonymous
memory (LazyFree pages) on inactive LRU list.
active_file # of bytes of file-backed memory on active LRU list.
unevictable # of bytes of memory that cannot be reclaimed (mlocked etc).
=============== ===============================================================
* status considering hierarchy (see memory.use_hierarchy settings):
========================= ===================================================
hierarchical_memory_limit # of bytes of memory limit with regard to
hierarchy
under which the memory cgroup is
hierarchical_memsw_limit # of bytes of memory+swap limit with regard to
hierarchy under which memory cgroup is.
total_<counter> # hierarchical version of <counter>, which in
addition to the cgroup's own value includes the
sum of all hierarchical children's values of
<counter>, i.e. total_cache
========================= ===================================================
* additional vm parameters (depends on CONFIG_DEBUG_VM):
========================= ========================================
recent_rotated_anon VM internal parameter. (see mm/vmscan.c)
recent_rotated_file VM internal parameter. (see mm/vmscan.c)
recent_scanned_anon VM internal parameter. (see mm/vmscan.c)
recent_scanned_file VM internal parameter. (see mm/vmscan.c)
========================= ========================================
.. hint::
recent_rotated means recent frequency of LRU rotation.
recent_scanned means recent # of scans to LRU.
showing for better debug please see the code for meanings.
.. note::
Only anonymous and swap cache memory is listed as part of 'rss' stat.
This should not be confused with the true 'resident set size' or the
amount of physical memory used by the cgroup.
'rss + mapped_file" will give you resident set size of cgroup.
Note that some kernel configurations might account complete larger
allocations (e.g., THP) towards 'rss' and 'mapped_file', even if
only some, but not all that memory is mapped.
(Note: file and shmem may be shared among other cgroups. In that case,
mapped_file is accounted only when the memory cgroup is owner of page
cache.)
5.3 swappiness
--------------
Overrides /proc/sys/vm/swappiness for the particular group. The tunable
in the root cgroup corresponds to the global swappiness setting.
Please note that unlike during the global reclaim, limit reclaim
enforces that 0 swappiness really prevents from any swapping even if
there is a swap storage available. This might lead to memcg OOM killer
if there are no file pages to reclaim.
5.4 failcnt
-----------
A memory cgroup provides memory.failcnt and memory.memsw.failcnt files.
This failcnt(== failure count) shows the number of times that a usage counter
hit its limit. When a memory cgroup hits a limit, failcnt increases and
memory under it will be reclaimed.
You can reset failcnt by writing 0 to failcnt file::
# echo 0 > .../memory.failcnt
5.5 usage_in_bytes
------------------
For efficiency, as other kernel components, memory cgroup uses some optimization
to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz
value for efficient access. (Of course, when necessary, it's synchronized.)
If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
value in memory.stat(see 5.2).
5.6 numa_stat
-------------
This is similar to numa_maps but operates on a per-memcg basis. This is
useful for providing visibility into the numa locality information within
an memcg since the pages are allowed to be allocated from any physical
node. One of the use cases is evaluating application performance by
combining this information with the application's CPU allocation.
Each memcg's numa_stat file includes "total", "file", "anon" and "unevictable"
per-node page counts including "hierarchical_<counter>" which sums up all
hierarchical children's values in addition to the memcg's own value.
The output format of memory.numa_stat is::
total=<total pages> N0=<node 0 pages> N1=<node 1 pages> ...
file=<total file pages> N0=<node 0 pages> N1=<node 1 pages> ...
anon=<total anon pages> N0=<node 0 pages> N1=<node 1 pages> ...
unevictable=<total anon pages> N0=<node 0 pages> N1=<node 1 pages> ...
hierarchical_<counter>=<counter pages> N0=<node 0 pages> N1=<node 1 pages> ...
The "total" count is sum of file + anon + unevictable.
6. Hierarchy support
====================
The memory controller supports a deep hierarchy and hierarchical accounting.
The hierarchy is created by creating the appropriate cgroups in the
cgroup filesystem. Consider for example, the following cgroup filesystem
hierarchy::
root
/ | \
/ | \
a b c
| \
| \
d e
In the diagram above, with hierarchical accounting enabled, all memory
usage of e, is accounted to its ancestors up until the root (i.e, c and root).
If one of the ancestors goes over its limit, the reclaim algorithm reclaims
from the tasks in the ancestor and the children of the ancestor.
6.1 Hierarchical accounting and reclaim
---------------------------------------
Hierarchical accounting is enabled by default. Disabling the hierarchical
accounting is deprecated. An attempt to do it will result in a failure
and a warning printed to dmesg.
For compatibility reasons writing 1 to memory.use_hierarchy will always pass::
# echo 1 > memory.use_hierarchy
7. Soft limits (DEPRECATED)
===========================
THIS IS DEPRECATED!
Soft limits allow for greater sharing of memory. The idea behind soft limits
is to allow control groups to use as much of the memory as needed, provided
a. There is no memory contention
b. They do not exceed their hard limit
When the system detects memory contention or low memory, control groups
are pushed back to their soft limits. If the soft limit of each control
group is very high, they are pushed back as much as possible to make
sure that one control group does not starve the others of memory.
Please note that soft limits is a best-effort feature; it comes with
no guarantees, but it does its best to make sure that when memory is
heavily contended for, memory is allocated based on the soft limit
hints/setup. Currently soft limit based reclaim is set up such that
it gets invoked from balance_pgdat (kswapd).
7.1 Interface
-------------
Soft limits can be setup by using the following commands (in this example we
assume a soft limit of 256 MiB)::
# echo 256M > memory.soft_limit_in_bytes
If we want to change this to 1G, we can at any time use::
# echo 1G > memory.soft_limit_in_bytes
.. note::
Soft limits take effect over a long period of time, since they involve
reclaiming memory for balancing between memory cgroups
.. note::
It is recommended to set the soft limit always below the hard limit,
otherwise the hard limit will take precedence.
.. _cgroup-v1-memory-move-charges:
8. Move charges at task migration (DEPRECATED!)
===============================================
THIS IS DEPRECATED!
Reading memory.move_charge_at_immigrate will always return 0 and writing
to it will always return -EINVAL.
9. Memory thresholds
====================
Memory cgroup implements memory thresholds using the cgroups notification
API (see cgroups.txt). It allows to register multiple memory and memsw
thresholds and gets notifications when it crosses.
To register a threshold, an application must:
- create an eventfd using eventfd(2);
- open memory.usage_in_bytes or memory.memsw.usage_in_bytes;
- write string like "<event_fd> <fd of memory.usage_in_bytes> <threshold>" to
cgroup.event_control.
Application will be notified through eventfd when memory usage crosses
threshold in any direction.
It's applicable for root and non-root cgroup.
.. _cgroup-v1-memory-oom-control:
10. OOM Control (DEPRECATED)
============================
THIS IS DEPRECATED!
memory.oom_control file is for OOM notification and other controls.
Memory cgroup implements OOM notifier using the cgroup notification
API (See cgroups.txt). It allows to register multiple OOM notification
delivery and gets notification when OOM happens.
To register a notifier, an application must:
- create an eventfd using eventfd(2)
- open memory.oom_control file
- write string like "<event_fd> <fd of memory.oom_control>" to
cgroup.event_control
The application will be notified through eventfd when OOM happens.
OOM notification doesn't work for the root cgroup.
You can disable the OOM-killer by writing "1" to memory.oom_control file, as:
#echo 1 > memory.oom_control
If OOM-killer is disabled, tasks under cgroup will hang/sleep
in memory cgroup's OOM-waitqueue when they request accountable memory.
For running them, you have to relax the memory cgroup's OOM status by
* enlarge limit or reduce usage.
To reduce usage,
* kill some tasks.
* move some tasks to other group with account migration.
* remove some files (on tmpfs?)
Then, stopped tasks will work again.
At reading, current status of OOM is shown.
- oom_kill_disable 0 or 1
(if 1, oom-killer is disabled)
- under_oom 0 or 1
(if 1, the memory cgroup is under OOM, tasks may be stopped.)
- oom_kill integer counter
The number of processes belonging to this cgroup killed by any
kind of OOM killer.
11. Memory Pressure (DEPRECATED)
================================
THIS IS DEPRECATED!
The pressure level notifications can be used to monitor the memory
allocation cost; based on the pressure, applications can implement
different strategies of managing their memory resources. The pressure
levels are defined as following:
The "low" level means that the system is reclaiming memory for new
allocations. Monitoring this reclaiming activity might be useful for
maintaining cache level. Upon notification, the program (typically
"Activity Manager") might analyze vmstat and act in advance (i.e.
prematurely shutdown unimportant services).
The "medium" level means that the system is experiencing medium memory
pressure, the system might be making swap, paging out active file caches,
etc. Upon this event applications may decide to further analyze
vmstat/zoneinfo/memcg or internal memory usage statistics and free any
resources that can be easily reconstructed or re-read from a disk.
The "critical" level means that the system is actively thrashing, it is
about to out of memory (OOM) or even the in-kernel OOM killer is on its
way to trigger. Applications should do whatever they can to help the
system. It might be too late to consult with vmstat or any other
statistics, so it's advisable to take an immediate action.
By default, events are propagated upward until the event is handled, i.e. the
events are not pass-through. For example, you have three cgroups: A->B->C. Now
you set up an event listener on cgroups A, B and C, and suppose group C
experiences some pressure. In this situation, only group C will receive the
notification, i.e. groups A and B will not receive it. This is done to avoid
excessive "broadcasting" of messages, which disturbs the system and which is
especially bad if we are low on memory or thrashing. Group B, will receive
notification only if there are no event listeners for group C.
There are three optional modes that specify different propagation behavior:
- "default": this is the default behavior specified above. This mode is the
same as omitting the optional mode parameter, preserved by backwards
compatibility.
- "hierarchy": events always propagate up to the root, similar to the default
behavior, except that propagation continues regardless of whether there are
event listeners at each level, with the "hierarchy" mode. In the above
example, groups A, B, and C will receive notification of memory pressure.
- "local": events are pass-through, i.e. they only receive notifications when
memory pressure is experienced in the memcg for which the notification is
registered. In the above example, group C will receive notification if
registered for "local" notification and the group experiences memory
pressure. However, group B will never receive notification, regardless if
there is an event listener for group C or not, if group B is registered for
local notification.
The level and event notification mode ("hierarchy" or "local", if necessary) are
specified by a comma-delimited string, i.e. "low,hierarchy" specifies
hierarchical, pass-through, notification for all ancestor memcgs. Notification
that is the default, non pass-through behavior, does not specify a mode.
"medium,local" specifies pass-through notification for the medium level.
The file memory.pressure_level is only used to setup an eventfd. To
register a notification, an application must:
- create an eventfd using eventfd(2);
- open memory.pressure_level;
- write string as "<event_fd> <fd of memory.pressure_level> <level[,mode]>"
to cgroup.event_control.
Application will be notified through eventfd when memory pressure is at
the specific level (or higher). Read/write operations to
memory.pressure_level are no implemented.
Test:
Here is a small script example that makes a new cgroup, sets up a
memory limit, sets up a notification in the cgroup and then makes child
cgroup experience a critical pressure::
# cd /sys/fs/cgroup/memory/
# mkdir foo
# cd foo
# cgroup_event_listener memory.pressure_level low,hierarchy &
# echo 8000000 > memory.limit_in_bytes
# echo 8000000 > memory.memsw.limit_in_bytes
# echo $$ > tasks
# dd if=/dev/zero | read x
(Expect a bunch of notifications, and eventually, the oom-killer will
trigger.)
12. TODO
========
1. Make per-cgroup scanner reclaim not-shared pages first
2. Teach controller to account for shared-pages
3. Start reclamation in the background when the limit is
not yet hit but the usage is getting closer
Summary
=======
Overall, the memory controller has been a stable controller and has been
commented and discussed quite extensively in the community.
References
==========
.. [1] Singh, Balbir. RFC: Memory Controller, http://lwn.net/Articles/206697/
.. [2] Singh, Balbir. Memory Controller (RSS Control),
http://lwn.net/Articles/222762/
.. [3] Emelianov, Pavel. Resource controllers based on process cgroups
https://lore.kernel.org/r/45ED7DEC.7010403@sw.ru
.. [4] Emelianov, Pavel. RSS controller based on process cgroups (v2)
https://lore.kernel.org/r/461A3010.90403@sw.ru
.. [5] Emelianov, Pavel. RSS controller based on process cgroups (v3)
https://lore.kernel.org/r/465D9739.8070209@openvz.org
6. Menage, Paul. Control Groups v10, http://lwn.net/Articles/236032/
7. Vaidyanathan, Srinivasan, Control Groups: Pagecache accounting and control
subsystem (v3), http://lwn.net/Articles/235534/
8. Singh, Balbir. RSS controller v2 test results (lmbench),
https://lore.kernel.org/r/464C95D4.7070806@linux.vnet.ibm.com
9. Singh, Balbir. RSS controller v2 AIM9 results
https://lore.kernel.org/r/464D267A.50107@linux.vnet.ibm.com
10. Singh, Balbir. Memory controller v6 test results,
https://lore.kernel.org/r/20070819094658.654.84837.sendpatchset@balbir-laptop
.. [11] Singh, Balbir. Memory controller introduction (v6),
https://lore.kernel.org/r/20070817084228.26003.12568.sendpatchset@balbir-laptop
.. [12] Corbet, Jonathan, Controlling memory use in cgroups,
http://lwn.net/Articles/243795/
linux-7.2.3/Documentation/admin-guide/cgroup-v1/misc.rst
+4
View File
@@ -0,0 +1,4 @@
===============
Misc controller
===============
Please refer "Misc" documentation in Documentation/admin-guide/cgroup-v2.rst
linux-7.2.3/Documentation/admin-guide/cgroup-v1/net_cls.rst
+44
View File
@@ -0,0 +1,44 @@
=========================
Network classifier cgroup
=========================
The Network classifier cgroup provides an interface to
tag network packets with a class identifier (classid).
The Traffic Controller (tc) can be used to assign
different priorities to packets from different cgroups.
Also, Netfilter (iptables) can use this tag to perform
actions on such packets.
Creating a net_cls cgroups instance creates a net_cls.classid file.
This net_cls.classid value is initialized to 0.
You can write hexadecimal values to net_cls.classid; the format for these
values is 0xAAAABBBB; AAAA is the major handle number and BBBB
is the minor handle number.
Reading net_cls.classid yields a decimal result.
Example::
mkdir /sys/fs/cgroup/net_cls
mount -t cgroup -onet_cls net_cls /sys/fs/cgroup/net_cls
mkdir /sys/fs/cgroup/net_cls/0
echo 0x100001 > /sys/fs/cgroup/net_cls/0/net_cls.classid
- setting a 10:1 handle::
cat /sys/fs/cgroup/net_cls/0/net_cls.classid
1048577
- configuring tc::
tc qdisc add dev eth0 root handle 10: htb
tc class add dev eth0 parent 10: classid 10:1 htb rate 40mbit
- creating traffic class 10:1::
tc filter add dev eth0 parent 10: protocol ip prio 10 handle 1: cgroup
configuring iptables, basic example::
iptables -A OUTPUT -m cgroup ! --cgroup 0x100001 -j DROP
linux-7.2.3/Documentation/admin-guide/cgroup-v1/net_prio.rst
+57
View File
@@ -0,0 +1,57 @@
=======================
Network priority cgroup
=======================
The Network priority cgroup provides an interface to allow an administrator to
dynamically set the priority of network traffic generated by various
applications
Nominally, an application would set the priority of its traffic via the
SO_PRIORITY socket option. This however, is not always possible because:
1) The application may not have been coded to set this value
2) The priority of application traffic is often a site-specific administrative
decision rather than an application defined one.
This cgroup allows an administrator to assign a process to a group which defines
the priority of egress traffic on a given interface. Network priority groups can
be created by first mounting the cgroup filesystem::
# mount -t cgroup -onet_prio none /sys/fs/cgroup/net_prio
With the above step, the initial group acting as the parent accounting group
becomes visible at '/sys/fs/cgroup/net_prio'. This group includes all tasks in
the system. '/sys/fs/cgroup/net_prio/tasks' lists the tasks in this cgroup.
Each net_prio cgroup contains two files that are subsystem specific
net_prio.prioidx
This file is read-only, and is simply informative. It contains a unique
integer value that the kernel uses as an internal representation of this
cgroup.
net_prio.ifpriomap
This file contains a map of the priorities assigned to traffic originating
from processes in this group and egressing the system on various interfaces.
It contains a list of tuples in the form <ifname priority>. Contents of this
file can be modified by echoing a string into the file using the same tuple
format. For example::
echo "eth0 5" > /sys/fs/cgroups/net_prio/iscsi/net_prio.ifpriomap
This command would force any traffic originating from processes belonging to the
iscsi net_prio cgroup and egressing on interface eth0 to have the priority of
said traffic set to the value 5. The parent accounting group also has a
writeable 'net_prio.ifpriomap' file that can be used to set a system default
priority.
Priorities are set immediately prior to queueing a frame to the device
queueing discipline (qdisc) so priorities will be assigned prior to the hardware
queue selection being made.
One usage for the net_prio cgroup is with mqprio qdisc allowing application
traffic to be steered to hardware/driver based traffic classes. These mappings
can then be managed by administrators or other networking protocols such as
DCBX.
A new net_prio cgroup inherits the parent's configuration.
linux-7.2.3/Documentation/admin-guide/cgroup-v1/pids.rst
+93
View File
@@ -0,0 +1,93 @@
=========================
Process Number Controller
=========================
Abstract
--------
The process number controller is used to allow a cgroup hierarchy to stop any
new tasks from being fork()'d or clone()'d after a certain limit is reached.
Since it is trivial to hit the task limit without hitting any kmemcg limits in
place, PIDs are a fundamental resource. As such, PID exhaustion must be
preventable in the scope of a cgroup hierarchy by allowing resource limiting of
the number of tasks in a cgroup.
Usage
-----
In order to use the `pids` controller, set the maximum number of tasks in
pids.max (this is not available in the root cgroup for obvious reasons). The
number of processes currently in the cgroup is given by pids.current.
Organisational operations are not blocked by cgroup policies, so it is possible
to have pids.current > pids.max. This can be done by either setting the limit to
be smaller than pids.current, or attaching enough processes to the cgroup such
that pids.current > pids.max. However, it is not possible to violate a cgroup
policy through fork() or clone(). fork() and clone() will return -EAGAIN if the
creation of a new process would cause a cgroup policy to be violated.
To set a cgroup to have no limit, set pids.max to "max". This is the default for
all new cgroups (N.B. that PID limits are hierarchical, so the most stringent
limit in the hierarchy is followed).
pids.current tracks all child cgroup hierarchies, so parent/pids.current is a
superset of parent/child/pids.current.
The pids.events file contains event counters:
- max: Number of times fork failed in the cgroup because limit was hit in
self or ancestors.
Example
-------
First, we mount the pids controller::
# mkdir -p /sys/fs/cgroup/pids
# mount -t cgroup -o pids none /sys/fs/cgroup/pids
Then we create a hierarchy, set limits and attach processes to it::
# mkdir -p /sys/fs/cgroup/pids/parent/child
# echo 2 > /sys/fs/cgroup/pids/parent/pids.max
# echo $$ > /sys/fs/cgroup/pids/parent/cgroup.procs
# cat /sys/fs/cgroup/pids/parent/pids.current
2
#
It should be noted that attempts to overcome the set limit (2 in this case) will
fail::
# cat /sys/fs/cgroup/pids/parent/pids.current
2
# ( /bin/echo "Here's some processes for you." | cat )
sh: fork: Resource temporary unavailable
#
Even if we migrate to a child cgroup (which doesn't have a set limit), we will
not be able to overcome the most stringent limit in the hierarchy (in this case,
parent's)::
# echo $$ > /sys/fs/cgroup/pids/parent/child/cgroup.procs
# cat /sys/fs/cgroup/pids/parent/pids.current
2
# cat /sys/fs/cgroup/pids/parent/child/pids.current
2
# cat /sys/fs/cgroup/pids/parent/child/pids.max
max
# ( /bin/echo "Here's some processes for you." | cat )
sh: fork: Resource temporary unavailable
#
We can set a limit that is smaller than pids.current, which will stop any new
processes from being forked at all (note that the shell itself counts towards
pids.current)::
# echo 1 > /sys/fs/cgroup/pids/parent/pids.max
# /bin/echo "We can't even spawn a single process now."
sh: fork: Resource temporary unavailable
# echo 0 > /sys/fs/cgroup/pids/parent/pids.max
# /bin/echo "We can't even spawn a single process now."
sh: fork: Resource temporary unavailable
#
linux-7.2.3/Documentation/admin-guide/cgroup-v1/rdma.rst
+183
View File
@@ -0,0 +1,183 @@
===============
RDMA Controller
===============
.. Contents
1. Overview
1-1. What is RDMA controller?
1-2. Why RDMA controller needed?
1-3. How is RDMA controller implemented?
2. Usage Examples
3. RDMA Interface Files
1. Overview
===========
1-1. What is RDMA controller?
-----------------------------
RDMA controller allows user to limit RDMA/IB specific resources that a given
set of processes can use. These processes are grouped using RDMA controller.
RDMA controller defines two resources which can be limited for processes of a
cgroup.
1-2. Why RDMA controller needed?
--------------------------------
Currently user space applications can easily take away all the rdma verb
specific resources such as AH, CQ, QP, MR etc. Due to which other applications
in other cgroup or kernel space ULPs may not even get chance to allocate any
rdma resources. This can lead to service unavailability.
Therefore RDMA controller is needed through which resource consumption
of processes can be limited. Through this controller different rdma
resources can be accounted.
1-3. How is RDMA controller implemented?
----------------------------------------
RDMA cgroup allows limit configuration of resources. Rdma cgroup maintains
resource accounting per cgroup, per device using resource pool structure.
Each such resource pool is limited up to 64 resources in given resource pool
by rdma cgroup, which can be extended later if required.
This resource pool object is linked to the cgroup css. Typically there
are 0 to 4 resource pool instances per cgroup, per device in most use cases.
But nothing limits to have it more. At present hundreds of RDMA devices per
single cgroup may not be handled optimally, however there is no
known use case or requirement for such configuration either.
Since RDMA resources can be allocated from any process and can be freed by any
of the child processes which shares the address space, rdma resources are
always owned by the creator cgroup css. This allows process migration from one
to other cgroup without major complexity of transferring resource ownership;
because such ownership is not really present due to shared nature of
rdma resources. Linking resources around css also ensures that cgroups can be
deleted after processes migrated. This allow progress migration as well with
active resources, even though that is not a primary use case.
Whenever RDMA resource charging occurs, owner rdma cgroup is returned to
the caller. Same rdma cgroup should be passed while uncharging the resource.
This also allows process migrated with active RDMA resource to charge
to new owner cgroup for new resource. It also allows to uncharge resource of
a process from previously charged cgroup which is migrated to new cgroup,
even though that is not a primary use case.
Resource pool object is created in following situations.
(a) User sets the limit and no previous resource pool exist for the device
of interest for the cgroup.
(b) No resource limits were configured, but IB/RDMA stack tries to
charge the resource. So that it correctly uncharge them when applications are
running without limits and later on when limits are enforced during uncharging,
otherwise usage count will drop to negative.
Resource pool is destroyed if all the resource limits are set to max and
it is the last resource getting deallocated.
User should set all the limit to max value if it intents to remove/unconfigure
the resource pool for a particular device.
IB stack honors limits enforced by the rdma controller. When application
query about maximum resource limits of IB device, it returns minimum of
what is configured by user for a given cgroup and what is supported by
IB device.
Following resources can be accounted by rdma controller.
========== =============================
hca_handle Maximum number of HCA Handles
hca_object Maximum number of HCA Objects
========== =============================
2. Usage Examples
=================
(a) Configure resource limit::
echo mlx4_0 hca_handle=2 hca_object=2000 > /sys/fs/cgroup/rdma/1/rdma.max
echo ocrdma1 hca_handle=3 > /sys/fs/cgroup/rdma/2/rdma.max
(b) Query resource limit::
cat /sys/fs/cgroup/rdma/2/rdma.max
#Output:
mlx4_0 hca_handle=2 hca_object=2000
ocrdma1 hca_handle=3 hca_object=max
(c) Query current usage::
cat /sys/fs/cgroup/rdma/2/rdma.current
#Output:
mlx4_0 hca_handle=1 hca_object=20
ocrdma1 hca_handle=1 hca_object=23
(d) Delete resource limit::
echo mlx4_0 hca_handle=max hca_object=max > /sys/fs/cgroup/rdma/1/rdma.max
3. RDMA Interface Files
========================
The following interface files are available in each non-root RDMA cgroup.
rdma.max
A read-write file which describes the configured resource limit
for an RDMA/IB device. See the Usage Examples above.
rdma.current
A read-only file which describes the current resource usage.
rdma.peak
A read-only nested-keyed file which shows the historical high
watermark of resource usage per device since the cgroup was created.
An example for mlx4 and ocrdma device follows::
mlx4_0 hca_handle=1 hca_object=20
ocrdma1 hca_handle=0 hca_object=23
rdma.events
A read-only nested-keyed file which exists on non-root cgroups
and contains the following keys:
max
The number of times a process in this cgroup or its
descendants attempted an RDMA resource allocation that
was rejected because a rdma.max limit in the subtree
was reached. This is a hierarchical counter propagated
upward to all ancestor cgroups. A value change in this
file generates a file modified event.
alloc_fail
The number of RDMA resource allocation attempts that
originated in this cgroup or its descendants and failed
due to a rdma.max limit being reached. This is a
hierarchical counter propagated upward.
An example for mlx4 device follows::
mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=3 hca_object.max=0 hca_object.alloc_fail=0
rdma.events.local
Similar to rdma.events but the fields are local to the cgroup,
i.e. not hierarchical. The file modified event generated on this
file reflects only the local events.
The following nested keys are defined.
max
The number of times a process in this cgroup or its
descendants attempted an RDMA resource allocation that
was rejected because this cgroup's own rdma.max limit
was reached.
alloc_fail
The number of RDMA resource allocation attempts
originating from this cgroup that failed due to this
cgroup's or an ancestor's rdma.max limit.
An example for mlx4 device follows::
mlx4_0 hca_handle.max=5 hca_handle.alloc_fail=0 hca_object.max=0 hca_object.alloc_fail=0
linux-7.2.3/Documentation/admin-guide/cgroup-v2.rst
+3524
View File
File diff suppressed because it is too large Load Diff
linux-7.2.3/Documentation/admin-guide/cifs/authors.rst
+69
View File
@@ -0,0 +1,69 @@
=======
Authors
=======
Original Author
---------------
Steve French (smfrench@gmail.com, sfrench@samba.org)
The author wishes to express his appreciation and thanks to:
Andrew Tridgell (Samba team) for his early suggestions about SMB/CIFS VFS
improvements. Thanks to IBM for allowing me time and test resources to pursue
this project, to Jim McDonough from IBM (and the Samba Team) for his help, to
the IBM Linux JFS team for explaining many esoteric Linux filesystem features.
Jeremy Allison of the Samba team has done invaluable work in adding the server
side of the original CIFS Unix extensions and reviewing and implementing
portions of the newer CIFS POSIX extensions into the Samba 3 file server. Thank
Dave Boutcher of IBM Rochester (author of the OS/400 smb/cifs filesystem client)
for proving years ago that very good smb/cifs clients could be done on Unix-like
operating systems. Volker Lendecke, Andrew Tridgell, Urban Widmark, John
Newbigin and others for their work on the Linux smbfs module. Thanks to
the other members of the Storage Network Industry Association CIFS Technical
Workgroup for their work specifying this highly complex protocol and finally
thanks to the Samba team for their technical advice and encouragement.
Patch Contributors
------------------
- Zwane Mwaikambo
- Andi Kleen
- Amrut Joshi
- Shobhit Dayal
- Sergey Vlasov
- Richard Hughes
- Yury Umanets
- Mark Hamzy (for some of the early cifs IPv6 work)
- Domen Puncer
- Jesper Juhl (in particular for lots of whitespace/formatting cleanup)
- Vince Negri and Dave Stahl (for finding an important caching bug)
- Adrian Bunk (kcalloc cleanups)
- Miklos Szeredi
- Kazeon team for various fixes especially for 2.4 version.
- Asser Ferno (Change Notify support)
- Shaggy (Dave Kleikamp) for innumerable small fs suggestions and some good cleanup
- Gunter Kukkukk (testing and suggestions for support of old servers)
- Igor Mammedov (DFS support)
- Jeff Layton (many, many fixes, as well as great work on the cifs Kerberos code)
- Scott Lovenberg
- Pavel Shilovsky (for great work adding SMB2 support, and various SMB3 features)
- Aurelien Aptel (for DFS SMB3 work and some key bug fixes)
- Ronnie Sahlberg (for SMB3 xattr work, bug fixes, and lots of great work on compounding)
- Shirish Pargaonkar (for many ACL patches over the years)
- Sachin Prabhu (many bug fixes, including for reconnect, copy offload and security)
- Paulo Alcantara (for some excellent work in DFS, and in booting from SMB3)
- Long Li (some great work on RDMA, SMB Direct)
Test case and Bug Report contributors
-------------------------------------
Thanks to those in the community who have submitted detailed bug reports
and debug of problems they have found: Jochen Dolze, David Blaine,
Rene Scharfe, Martin Josefsson, Alexander Wild, Anthony Liguori,
Lars Muller, Urban Widmark, Massimiliano Ferrero, Howard Owen,
Olaf Kirch, Kieron Briggs, Nick Millington and others. Also special
mention to the Stanford Checker (SWAT) which pointed out many minor
bugs in error paths. Valuable suggestions also have come from Al Viro
and Dave Miller.
And thanks to the IBM LTC and Power test teams and SuSE and Citrix and RedHat testers for finding multiple bugs during excellent stress test runs.
linux-7.2.3/Documentation/admin-guide/cifs/changes.rst
+9
View File
@@ -0,0 +1,9 @@
=======
Changes
=======
See https://wiki.samba.org/index.php/LinuxCIFSKernel for summary
information about fixes/improvements to CIFS/SMB2/SMB3 support (changes
to cifs.ko module) by kernel version (and cifs internal module version).
This may be easier to read than parsing the output of
"git log fs/smb/client" by release.
linux-7.2.3/Documentation/admin-guide/cifs/index.rst
+14
View File
@@ -0,0 +1,14 @@
.. SPDX-License-Identifier: GPL-2.0
====
CIFS
====
.. toctree::
:maxdepth: 2
introduction
usage
todo
changes
authors
linux-7.2.3/Documentation/admin-guide/cifs/introduction.rst
+53
View File
@@ -0,0 +1,53 @@
============
Introduction
============
This is the client VFS module for the SMB3 NAS protocol as well
as for older dialects such as the Common Internet File System (CIFS)
protocol which was the successor to the Server Message Block
(SMB) protocol, the native file sharing mechanism for most early
PC operating systems. New and improved versions of CIFS are now
called SMB2 and SMB3. Use of SMB3 (and later, including SMB3.1.1
the most current dialect) is strongly preferred over using older
dialects like CIFS due to security reasons. All modern dialects,
including the most recent, SMB3.1.1, are supported by the CIFS VFS
module. The SMB3 protocol is implemented and supported by all major
file servers such as Windows (including Windows 2019 Server), as
well as by Samba (which provides excellent CIFS/SMB2/SMB3 server
support and tools for Linux and many other operating systems).
Apple systems also support SMB3 well, as do most Network Attached
Storage vendors, so this network filesystem client can mount to a
wide variety of systems. It also supports mounting to the cloud
(for example Microsoft Azure), including the necessary security
features.
The intent of this module is to provide the most advanced network
file system function for SMB3 compliant servers, including advanced
security features, excellent parallelized high performance i/o, better
POSIX compliance, secure per-user session establishment, encryption,
high performance safe distributed caching (leases/oplocks), optional packet
signing, large files, Unicode support and other internationalization
improvements. Since both Samba server and this filesystem client support the
CIFS Unix extensions, and the Linux client also supports SMB3 POSIX extensions,
the combination can provide a reasonable alternative to other network and
cluster file systems for fileserving in some Linux to Linux environments,
not just in Linux to Windows (or Linux to Mac) environments.
This filesystem has a mount utility (mount.cifs) and various user space
tools (including smbinfo and setcifsacl) that can be obtained from
https://git.samba.org/?p=cifs-utils.git
or
git://git.samba.org/cifs-utils.git
mount.cifs should be installed in the directory with the other mount helpers.
For more information on the module see the project wiki page at
https://wiki.samba.org/index.php/LinuxCIFS
and
https://wiki.samba.org/index.php/LinuxCIFS_utils
linux-7.2.3/Documentation/admin-guide/cifs/todo.rst
+133
View File
@@ -0,0 +1,133 @@
====
TODO
====
As of 6.7 kernel. See https://wiki.samba.org/index.php/LinuxCIFSKernel
for list of features added by release
A Partial List of Missing Features
==================================
Contributions are welcome. There are plenty of opportunities
for visible, important contributions to this module. Here
is a partial list of the known problems and missing features:
a) SMB3 (and SMB3.1.1) missing optional features:
multichannel performance optimizations, algorithmic channel selection,
directory leases optimizations,
support for faster packet signing (GMAC),
support for compression over the network,
T10 copy offload ie "ODX" (copy chunk, and "Duplicate Extents" ioctl
are currently the only two server side copy mechanisms supported)
b) Better optimized compounding and error handling for sparse file support,
perhaps addition of new optional SMB3.1.1 fsctls to make collapse range
and insert range more atomic
c) Support for SMB3.1.1 over QUIC (and perhaps other socket based protocols
like SCTP)
d) quota support (needs minor kernel change since quota calls otherwise
won't make it to network filesystems or deviceless filesystems).
e) Additional use cases can be optimized to use "compounding" (e.g.
open/query/close and open/setinfo/close) to reduce the number of
roundtrips to the server and improve performance. Various cases
(stat, statfs, create, unlink, mkdir, xattrs) already have been improved by
using compounding but more can be done. In addition we could
significantly reduce redundant opens by using deferred close (with
handle caching leases) and better using reference counters on file
handles.
f) Finish inotify support so kde and gnome file list windows
will autorefresh (partially complete by Asser). Needs minor kernel
vfs change to support removing D_NOTIFY on a file.
g) Add GUI tool to configure /proc/fs/cifs settings and for display of
the CIFS statistics (started)
h) implement support for security and trusted categories of xattrs
(requires minor protocol extension) to enable better support for SELINUX
i) Add support for tree connect contexts (see MS-SMB2) a new SMB3.1.1 protocol
feature (may be especially useful for virtualization).
j) Create UID mapping facility so server UIDs can be mapped on a per
mount or a per server basis to client UIDs or nobody if no mapping
exists. Also better integration with winbind for resolving SID owners
k) Add tools to take advantage of more smb3 specific ioctls and features
(passthrough ioctl/fsctl is now implemented in cifs.ko to allow
sending various SMB3 fsctls and query info and set info calls
directly from user space) Add tools to make setting various non-POSIX
metadata attributes easier from tools (e.g. extending what was done
in smb-info tool).
l) encrypted file support (currently the attribute showing the file is
encrypted on the server is reported, but changing the attribute is not
supported).
m) improved stats gathering tools (perhaps integration with nfsometer?)
to extend and make easier to use what is currently in /proc/fs/cifs/Stats
n) Add support for claims based ACLs ("DAC")
o) mount helper GUI (to simplify the various configuration options on mount)
p) Expand support for witness protocol to allow for notification of share
move, and server network adapter changes. Currently only notifications by
the witness protocol for server move is supported by the Linux client.
q) Allow mount.cifs to be more verbose in reporting errors with dialect
or unsupported feature errors. This would now be easier due to the
implementation of the new mount API.
r) updating cifs documentation, and user guide.
s) Addressing bugs found by running a broader set of xfstests in standard
file system xfstest suite.
t) split cifs and smb3 support into separate modules so legacy (and less
secure) CIFS dialect can be disabled in environments that don't need it
and simplify the code.
v) Additional testing of POSIX Extensions for SMB3.1.1
w) Support for the Mac SMB3.1.1 extensions to improve interop with Apple servers
x) Support for additional authentication options (e.g. IAKERB, peer-to-peer
Kerberos, SCRAM and others supported by existing servers)
y) Improved tracing, more eBPF trace points, better scripts for performance
analysis
Known Bugs
==========
See https://bugzilla.samba.org - search on product "CifsVFS" for
current bug list. Also check http://bugzilla.kernel.org (Product = File System, Component = CIFS)
and xfstest results e.g. https://wiki.samba.org/index.php/Xfstest-results-smb3
Misc testing to do
==================
1) check out max path names and max path name components against various server
types. Try nested symlinks (8 deep). Return max path name in stat -f information
2) Improve xfstest's cifs/smb3 enablement and adapt xfstests where needed to test
cifs/smb3 better
3) Additional performance testing and optimization using iozone and similar -
there are some easy changes that can be done to parallelize sequential writes,
and when signing is disabled to request larger read sizes (larger than
negotiated size) and send larger write sizes to modern servers.
4) More exhaustively test against less common servers
5) Continue to extend the smb3 "buildbot" which does automated xfstesting
against Windows, Samba and Azure currently - to add additional tests and
to allow the buildbot to execute the tests faster. The URL for the
buildbot is: http://smb3-test-rhel-75.southcentralus.cloudapp.azure.com
6) Address various coverity warnings (most are not bugs per-se, but
the more warnings are addressed, the easier it is to spot real
problems that static analyzers will point out in the future).
linux-7.2.3/Documentation/admin-guide/cifs/usage.rst
+864
View File
@@ -0,0 +1,864 @@
=====
Usage
=====
This module supports the SMB3 family of advanced network protocols (as well
as older dialects, originally called "CIFS" or SMB1).
The CIFS VFS module for Linux supports many advanced network filesystem
features such as hierarchical DFS like namespace, hardlinks, locking and more.
It was designed to comply with the SNIA CIFS Technical Reference (which
supersedes the 1992 X/Open SMB Standard) as well as to perform best practice
practical interoperability with Windows 2000, Windows XP, Samba and equivalent
servers. This code was developed in participation with the Protocol Freedom
Information Foundation. CIFS and now SMB3 has now become a defacto
standard for interoperating between Macs and Windows and major NAS appliances.
Please see
MS-SMB2 (for detailed SMB2/SMB3/SMB3.1.1 protocol specification)
or https://samba.org/samba/PFIF/
for more details.
For questions or bug reports please contact:
smfrench@gmail.com
See the project page at: https://wiki.samba.org/index.php/LinuxCIFS_utils
Build instructions
==================
For Linux:
1) Download the kernel (e.g. from https://www.kernel.org)
and change directory into the top of the kernel directory tree
(e.g. /usr/src/linux-2.5.73)
2) make menuconfig (or make xconfig)
3) select cifs from within the network filesystem choices
4) save and exit
5) make
Installation instructions
=========================
If you have built the CIFS vfs as module (successfully) simply
type ``make modules_install`` (or if you prefer, manually copy the file to
the modules directory e.g. /lib/modules/6.3.0-060300-generic/kernel/fs/smb/client/cifs.ko).
If you have built the CIFS vfs into the kernel itself, follow the instructions
for your distribution on how to install a new kernel (usually you
would simply type ``make install``).
If you do not have the utility mount.cifs (in the Samba 4.x source tree and on
the CIFS VFS web site) copy it to the same directory in which mount helpers
reside (usually /sbin). Although the helper software is not
required, mount.cifs is recommended. Most distros include a ``cifs-utils``
package that includes this utility so it is recommended to install this.
Note that running the Winbind pam/nss module (logon service) on all of your
Linux clients is useful in mapping Uids and Gids consistently across the
domain to the proper network user. The mount.cifs mount helper can be
found at cifs-utils.git on git.samba.org
If cifs is built as a module, then the size and number of network buffers
and maximum number of simultaneous requests to one server can be configured.
Changing these from their defaults is not recommended. By executing modinfo::
modinfo <path to cifs.ko>
on kernel/fs/smb/client/cifs.ko the list of configuration changes that can be made
at module initialization time (by running insmod cifs.ko) can be seen.
Recommendations
===============
To improve security the SMB2.1 dialect or later (usually will get SMB3.1.1) is now
the new default. To use old dialects (e.g. to mount Windows XP) use "vers=1.0"
on mount (or vers=2.0 for Windows Vista). Note that the CIFS (vers=1.0) is
much older and less secure than the default dialect SMB3 which includes
many advanced security features such as downgrade attack detection
and encrypted shares and stronger signing and authentication algorithms.
There are additional mount options that may be helpful for SMB3 to get
improved POSIX behavior (NB: can use vers=3 to force SMB3 or later, never 2.1):
``mfsymlinks`` and either ``cifsacl`` or ``modefromsid`` (usually with ``idsfromsid``)
Allowing User Mounts
====================
To permit users to mount and unmount over directories they own is possible
with the cifs vfs. A way to enable such mounting is to mark the mount.cifs
utility as suid (e.g. ``chmod +s /sbin/mount.cifs``). To enable users to
umount shares they mount requires
1) mount.cifs version 1.4 or later
2) an entry for the share in /etc/fstab indicating that a user may
unmount it e.g.::
//server/usersharename /mnt/username cifs user 0 0
Note that when the mount.cifs utility is run suid (allowing user mounts),
in order to reduce risks, the ``nosuid`` mount flag is passed in on mount to
disallow execution of an suid program mounted on the remote target.
When mount is executed as root, nosuid is not passed in by default,
and execution of suid programs on the remote target would be enabled
by default. This can be changed, as with nfs and other filesystems,
by simply specifying ``nosuid`` among the mount options. For user mounts
though to be able to pass the suid flag to mount requires rebuilding
mount.cifs with the following flag: CIFS_ALLOW_USR_SUID
There is a corresponding manual page for cifs mounting in the Samba 3.0 and
later source tree in docs/manpages/mount.cifs.8
Allowing User Unmounts
======================
To permit users to unmount directories that they have user mounted (see above),
the utility umount.cifs may be used. It may be invoked directly, or if
umount.cifs is placed in /sbin, umount can invoke the cifs umount helper
(at least for most versions of the umount utility) for umount of cifs
mounts, unless umount is invoked with -i (which will avoid invoking a umount
helper). As with mount.cifs, to enable user unmounts umount.cifs must be marked
as suid (e.g. ``chmod +s /sbin/umount.cifs``) or equivalent (some distributions
allow adding entries to a file to the /etc/permissions file to achieve the
equivalent suid effect). For this utility to succeed the target path
must be a cifs mount, and the uid of the current user must match the uid
of the user who mounted the resource.
Also note that the customary way of allowing user mounts and unmounts is
(instead of using mount.cifs and unmount.cifs as suid) to add a line
to the file /etc/fstab for each //server/share you wish to mount, but
this can become unwieldy when potential mount targets include many
or unpredictable UNC names.
Samba Considerations
====================
Most current servers support SMB2.1 and SMB3 which are more secure,
but there are useful protocol extensions for the older less secure CIFS
dialect, so to get the maximum benefit if mounting using the older dialect
(CIFS/SMB1), we recommend using a server that supports the SNIA CIFS
Unix Extensions standard (e.g. almost any version of Samba ie version
2.2.5 or later) but the CIFS vfs works fine with a wide variety of CIFS servers.
Note that uid, gid and file permissions will display default values if you do
not have a server that supports the Unix extensions for CIFS (such as Samba
2.2.5 or later). To enable the Unix CIFS Extensions in the Samba server, add
the line::
unix extensions = yes
to your smb.conf file on the server. Note that the following smb.conf settings
are also useful (on the Samba server) when the majority of clients are Unix or
Linux::
case sensitive = yes
delete readonly = yes
ea support = yes
Note that server ea support is required for supporting xattrs from the Linux
cifs client, and that EA support is present in later versions of Samba (e.g.
3.0.6 and later (also EA support works in all versions of Windows, at least to
shares on NTFS filesystems). Extended Attribute (xattr) support is an optional
feature of most Linux filesystems which may require enabling via
make menuconfig. Client support for extended attributes (user xattr) can be
disabled on a per-mount basis by specifying ``nouser_xattr`` on mount.
The CIFS client can get and set POSIX ACLs (getfacl, setfacl) to Samba servers
version 3.10 and later. Setting POSIX ACLs requires enabling both XATTR and
then POSIX support in the CIFS configuration options when building the cifs
module. POSIX ACL support can be disabled on a per mount basic by specifying
``noacl`` on mount.
Some administrators may want to change Samba's smb.conf ``map archive`` and
``create mask`` parameters from the default. Unless the create mask is changed
newly created files can end up with an unnecessarily restrictive default mode,
which may not be what you want, although if the CIFS Unix extensions are
enabled on the server and client, subsequent setattr calls (e.g. chmod) can
fix the mode. Note that creating special devices (mknod) remotely
may require specifying a mkdev function to Samba if you are not using
Samba 3.0.6 or later. For more information on these see the manual pages
(``man smb.conf``) on the Samba server system. Note that the cifs vfs,
unlike the smbfs vfs, does not read the smb.conf on the client system
(the few optional settings are passed in on mount via -o parameters instead).
Note that Samba 2.2.7 or later includes a fix that allows the CIFS VFS to delete
open files (required for strict POSIX compliance). Windows Servers already
supported this feature. Samba server does not allow symlinks that refer to files
outside of the share, so in Samba versions prior to 3.0.6, most symlinks to
files with absolute paths (ie beginning with slash) such as::
ln -s /mnt/foo bar
would be forbidden. Samba 3.0.6 server or later includes the ability to create
such symlinks safely by converting unsafe symlinks (ie symlinks to server
files that are outside of the share) to a samba specific format on the server
that is ignored by local server applications and non-cifs clients and that will
not be traversed by the Samba server). This is opaque to the Linux client
application using the cifs vfs. Absolute symlinks will work to Samba 3.0.5 or
later, but only for remote clients using the CIFS Unix extensions, and will
be invisible to Windows clients and typically will not affect local
applications running on the same server as Samba.
Use instructions
================
Once the CIFS VFS support is built into the kernel or installed as a module
(cifs.ko), you can use mount syntax like the following to access Samba or
Mac or Windows servers::
mount -t cifs //9.53.216.11/e$ /mnt -o username=myname,password=mypassword
Before -o the option -v may be specified to make the mount.cifs
mount helper display the mount steps more verbosely.
After -o the following commonly used cifs vfs specific options
are supported::
username=<username>
password=<password>
domain=<domain name>
Other cifs mount options are described below. Use of TCP names (in addition to
ip addresses) is available if the mount helper (mount.cifs) is installed. If
you do not trust the server to which are mounted, or if you do not have
cifs signing enabled (and the physical network is insecure), consider use
of the standard mount options ``noexec`` and ``nosuid`` to reduce the risk of
running an altered binary on your local system (downloaded from a hostile server
or altered by a hostile router).
Although mounting using format corresponding to the CIFS URL specification is
not possible in mount.cifs yet, it is possible to use an alternate format
for the server and sharename (which is somewhat similar to NFS style mount
syntax) instead of the more widely used UNC format (i.e. \\server\share)::
mount -t cifs tcp_name_of_server:share_name /mnt -o user=myname,pass=mypasswd
When using the mount helper mount.cifs, passwords may be specified via alternate
mechanisms, instead of specifying it after -o using the normal ``pass=`` syntax
on the command line:
1) By including it in a credential file. Specify credentials=filename as one
of the mount options. Credential files contain two lines::
username=someuser
password=your_password
2) By specifying the password in the PASSWD environment variable (similarly
the user name can be taken from the USER environment variable).
3) By specifying the password in a file by name via PASSWD_FILE
4) By specifying the password in a file by file descriptor via PASSWD_FD
If no password is provided, mount.cifs will prompt for password entry
Restrictions
============
Servers must support either "pure-TCP" (port 445 TCP/IP CIFS connections) or RFC
1001/1002 support for "Netbios-Over-TCP/IP." This is not likely to be a
problem as most servers support this.
Valid filenames differ between Windows and Linux. Windows typically restricts
filenames which contain certain reserved characters (e.g.the character :
which is used to delimit the beginning of a stream name by Windows), while
Linux allows a slightly wider set of valid characters in filenames. Windows
servers can remap such characters when an explicit mapping is specified in
the Server's registry. Samba starting with version 3.10 will allow such
filenames (ie those which contain valid Linux characters, which normally
would be forbidden for Windows/CIFS semantics) as long as the server is
configured for Unix Extensions (and the client has not disabled
/proc/fs/cifs/LinuxExtensionsEnabled). In addition the mount option
``mapposix`` can be used on CIFS (vers=1.0) to force the mapping of
illegal Windows/NTFS/SMB characters to a remap range (this mount parameter
is the default for SMB3). This remap (``mapposix``) range is also
compatible with Mac (and "Services for Mac" on some older Windows).
When POSIX Extensions for SMB 3.1.1 are negotiated, remapping is automatically
disabled.
CIFS VFS Mount Options
======================
A partial list of the supported mount options follows:
username
The user name to use when trying to establish
the CIFS session.
password
The user password. If the mount helper is
installed, the user will be prompted for password
if not supplied.
ip
The ip address of the target server
unc
The target server Universal Network Name (export) to
mount.
domain
Set the SMB/CIFS workgroup name prepended to the
username during CIFS session establishment
forceuid
Set the default uid for inodes to the uid
passed in on mount. For mounts to servers
which do support the CIFS Unix extensions, such as a
properly configured Samba server, the server provides
the uid, gid and mode so this parameter should not be
specified unless the server and clients uid and gid
numbering differ. If the server and client are in the
same domain (e.g. running winbind or nss_ldap) and
the server supports the Unix Extensions then the uid
and gid can be retrieved from the server (and uid
and gid would not have to be specified on the mount.
For servers which do not support the CIFS Unix
extensions, the default uid (and gid) returned on lookup
of existing files will be the uid (gid) of the person
who executed the mount (root, except when mount.cifs
is configured setuid for user mounts) unless the ``uid=``
(gid) mount option is specified. Also note that permission
checks (authorization checks) on accesses to a file occur
at the server, but there are cases in which an administrator
may want to restrict at the client as well. For those
servers which do not report a uid/gid owner
(such as Windows), permissions can also be checked at the
client, and a crude form of client side permission checking
can be enabled by specifying file_mode and dir_mode on
the client. (default)
forcegid
(similar to above but for the groupid instead of uid) (default)
noforceuid
Fill in file owner information (uid) by requesting it from
the server if possible. With this option, the value given in
the uid= option (on mount) will only be used if the server
can not support returning uids on inodes.
noforcegid
(similar to above but for the group owner, gid, instead of uid)
uid
Set the default uid for inodes, and indicate to the
cifs kernel driver which local user mounted. If the server
supports the unix extensions the default uid is
not used to fill in the owner fields of inodes (files)
unless the ``forceuid`` parameter is specified.
gid
Set the default gid for inodes (similar to above).
file_mode
If CIFS Unix extensions are not supported by the server
this overrides the default mode for file inodes.
fsc
Enable local disk caching using FS-Cache (off by default). This
option could be useful to improve performance on a slow link,
heavily loaded server and/or network where reading from the
disk is faster than reading from the server (over the network).
This could also impact scalability positively as the
number of calls to the server are reduced. However, local
caching is not suitable for all workloads for e.g. read-once
type workloads. So, you need to consider carefully your
workload/scenario before using this option. Currently, local
disk caching is functional for CIFS files opened as read-only.
dir_mode
If CIFS Unix extensions are not supported by the server
this overrides the default mode for directory inodes.
port
attempt to contact the server on this tcp port, before
trying the usual ports (port 445, then 139).
iocharset
Codepage used to convert local path names to and from
Unicode. Unicode is used by default for network path
names if the server supports it. If iocharset is
not specified then the nls_default specified
during the local client kernel build will be used.
If server does not support Unicode, this parameter is
unused.
rsize
default read size (usually 16K). The client currently
can not use rsize larger than CIFSMaxBufSize. CIFSMaxBufSize
defaults to 16K and may be changed (from 8K to the maximum
kmalloc size allowed by your kernel) at module install time
for cifs.ko. Setting CIFSMaxBufSize to a very large value
will cause cifs to use more memory and may reduce performance
in some cases. To use rsize greater than 127K (the original
cifs protocol maximum) also requires that the server support
a new Unix Capability flag (for very large read) which some
newer servers (e.g. Samba 3.0.26 or later) do. rsize can be
set from a minimum of 2048 to a maximum of 130048 (127K or
CIFSMaxBufSize, whichever is smaller)
wsize
default write size (default 57344)
maximum wsize currently allowed by CIFS is 57344 (fourteen
4096 byte pages)
actimeo=n
attribute cache timeout in seconds (default 1 second).
After this timeout, the cifs client requests fresh attribute
information from the server. This option allows to tune the
attribute cache timeout to suit the workload needs. Shorter
timeouts mean better the cache coherency, but increased number
of calls to the server. Longer timeouts mean reduced number
of calls to the server at the expense of less stricter cache
coherency checks (i.e. incorrect attribute cache for a short
period of time).
rw
mount the network share read-write (note that the
server may still consider the share read-only)
ro
mount network share read-only
version
used to distinguish different versions of the
mount helper utility (not typically needed)
sep
if first mount option (after the -o), overrides
the comma as the separator between the mount
parameters. e.g.::
-o user=myname,password=mypassword,domain=mydom
could be passed instead with period as the separator by::
-o sep=.user=myname.password=mypassword.domain=mydom
this might be useful when comma is contained within username
or password or domain. This option is less important
when the cifs mount helper cifs.mount (version 1.1 or later)
is used.
nosuid
Do not allow remote executables with the suid bit
program to be executed. This is only meaningful for mounts
to servers such as Samba which support the CIFS Unix Extensions.
If you do not trust the servers in your network (your mount
targets) it is recommended that you specify this option for
greater security.
exec
Permit execution of binaries on the mount.
noexec
Do not permit execution of binaries on the mount.
dev
Recognize block devices on the remote mount.
nodev
Do not recognize devices on the remote mount.
suid
Allow remote files on this mountpoint with suid enabled to
be executed (default for mounts when executed as root,
nosuid is default for user mounts).
credentials
Although ignored by the cifs kernel component, it is used by
the mount helper, mount.cifs. When mount.cifs is installed it
opens and reads the credential file specified in order
to obtain the userid and password arguments which are passed to
the cifs vfs.
guest
Although ignored by the kernel component, the mount.cifs
mount helper will not prompt the user for a password
if guest is specified on the mount options. If no
password is specified a null password will be used.
perm
Client does permission checks (vfs_permission check of uid
and gid of the file against the mode and desired operation),
Note that this is in addition to the normal ACL check on the
target machine done by the server software.
Client permission checking is enabled by default.
noperm
Client does not do permission checks. This can expose
files on this mount to access by other users on the local
client system. It is typically only needed when the server
supports the CIFS Unix Extensions but the UIDs/GIDs on the
client and server system do not match closely enough to allow
access by the user doing the mount, but it may be useful with
non CIFS Unix Extension mounts for cases in which the default
mode is specified on the mount but is not to be enforced on the
client (e.g. perhaps when MultiUserMount is enabled)
Note that this does not affect the normal ACL check on the
target machine done by the server software (of the server
ACL against the user name provided at mount time).
serverino
Use server's inode numbers instead of generating automatically
incrementing inode numbers on the client. Although this will
make it easier to spot hardlinked files (as they will have
the same inode numbers) and inode numbers may be persistent,
note that the server does not guarantee that the inode numbers
are unique if multiple server side mounts are exported under a
single share (since inode numbers on the servers might not
be unique if multiple filesystems are mounted under the same
shared higher level directory). Note that some older
(e.g. pre-Windows 2000) do not support returning UniqueIDs
or the CIFS Unix Extensions equivalent and for those
this mount option will have no effect. Exporting cifs mounts
under nfsd requires this mount option on the cifs mount.
This is now the default if server supports the
required network operation.
noserverino
Client generates inode numbers (rather than using the actual one
from the server). These inode numbers will vary after
unmount or reboot which can confuse some applications,
but not all server filesystems support unique inode
numbers.
setuids
If the CIFS Unix extensions are negotiated with the server
the client will attempt to set the effective uid and gid of
the local process on newly created files, directories, and
devices (create, mkdir, mknod). If the CIFS Unix Extensions
are not negotiated, for newly created files and directories
instead of using the default uid and gid specified on
the mount, cache the new file's uid and gid locally which means
that the uid for the file can change when the inode is
reloaded (or the user remounts the share).
nosetuids
The client will not attempt to set the uid and gid on
on newly created files, directories, and devices (create,
mkdir, mknod) which will result in the server setting the
uid and gid to the default (usually the server uid of the
user who mounted the share). Letting the server (rather than
the client) set the uid and gid is the default. If the CIFS
Unix Extensions are not negotiated then the uid and gid for
new files will appear to be the uid (gid) of the mounter or the
uid (gid) parameter specified on the mount.
netbiosname
When mounting to servers via port 139, specifies the RFC1001
source name to use to represent the client netbios machine
name when doing the RFC1001 netbios session initialize.
direct
Do not do inode data caching on files opened on this mount.
This precludes mmapping files on this mount. In some cases
with fast networks and little or no caching benefits on the
client (e.g. when the application is doing large sequential
reads bigger than page size without rereading the same data)
this can provide better performance than the default
behavior which caches reads (readahead) and writes
(writebehind) through the local Linux client pagecache
if oplock (caching token) is granted and held. Note that
direct allows write operations larger than page size
to be sent to the server.
strictcache
Use for switching on strict cache mode. In this mode the
client read from the cache all the time it has Oplock Level II,
otherwise - read from the server. All written data are stored
in the cache, but if the client doesn't have Exclusive Oplock,
it writes the data to the server.
rwpidforward
Forward pid of a process who opened a file to any read or write
operation on that file. This prevent applications like WINE
from failing on read and write if we use mandatory brlock style.
acl
Allow setfacl and getfacl to manage posix ACLs if server
supports them. (default)
noacl
Do not allow setfacl and getfacl calls on this mount
user_xattr
Allow getting and setting user xattrs (those attributes whose
name begins with ``user.`` or ``os2.``) as OS/2 EAs (extended
attributes) to the server. This allows support of the
setfattr and getfattr utilities. (default)
nouser_xattr
Do not allow getfattr/setfattr to get/set/list xattrs
mapchars
Translate six of the seven reserved characters (not backslash)::
*?<>|:
to the remap range (above 0xF000), which also
allows the CIFS client to recognize files created with
such characters by Windows's POSIX emulation. This can
also be useful when mounting to most versions of Samba
(which also forbids creating and opening files
whose names contain any of these seven characters).
This has no effect if the server does not support
Unicode on the wire.
nomapchars
Do not translate any of these seven characters (default).
nocase
Request case insensitive path name matching (case
sensitive is the default if the server supports it).
(mount option ``ignorecase`` is identical to ``nocase``)
posixpaths
If CIFS Unix extensions are supported, attempt to
negotiate posix path name support which allows certain
characters forbidden in typical CIFS filenames, without
requiring remapping. (default)
noposixpaths
If CIFS Unix extensions are supported, do not request
posix path name support (this may cause servers to
reject creatingfile with certain reserved characters).
nounix
Disable the CIFS Unix Extensions for this mount (tree
connection). This is rarely needed, but it may be useful
in order to turn off multiple settings all at once (ie
posix acls, posix locks, posix paths, symlink support
and retrieving uids/gids/mode from the server) or to
work around a bug in server which implement the Unix
Extensions.
nobrl
Do not send byte range lock requests to the server.
This is necessary for certain applications that break
with cifs style mandatory byte range locks (and most
cifs servers do not yet support requesting advisory
byte range locks).
forcemandatorylock
Even if the server supports posix (advisory) byte range
locking, send only mandatory lock requests. For some
(presumably rare) applications, originally coded for
DOS/Windows, which require Windows style mandatory byte range
locking, they may be able to take advantage of this option,
forcing the cifs client to only send mandatory locks
even if the cifs server would support posix advisory locks.
``forcemand`` is accepted as a shorter form of this mount
option.
nostrictsync
If this mount option is set, when an application does an
fsync call then the cifs client does not send an SMB Flush
to the server (to force the server to write all dirty data
for this file immediately to disk), although cifs still sends
all dirty (cached) file data to the server and waits for the
server to respond to the write. Since SMB Flush can be
very slow, and some servers may be reliable enough (to risk
delaying slightly flushing the data to disk on the server),
turning on this option may be useful to improve performance for
applications that fsync too much, at a small risk of server
crash. If this mount option is not set, by default cifs will
send an SMB flush request (and wait for a response) on every
fsync call.
nodfs
Disable DFS (global name space support) even if the
server claims to support it. This can help work around
a problem with parsing of DFS paths with Samba server
versions 3.0.24 and 3.0.25.
remount
remount the share (often used to change from ro to rw mounts
or vice versa)
cifsacl
Report mode bits (e.g. on stat) based on the Windows ACL for
the file. (EXPERIMENTAL)
servern
Specify the server 's netbios name (RFC1001 name) to use
when attempting to setup a session to the server.
This is needed for mounting to some older servers (such
as OS/2 or Windows 98 and Windows ME) since they do not
support a default server name. A server name can be up
to 15 characters long and is usually uppercased.
sfu
When the CIFS Unix Extensions are not negotiated, attempt to
create device files and fifos in a format compatible with
Services for Unix (SFU). In addition retrieve bits 10-12
of the mode via the SETFILEBITS extended attribute (as
SFU does). In the future the bottom 9 bits of the
mode also will be emulated using queries of the security
descriptor (ACL).
mfsymlinks
Enable support for Minshall+French symlinks
(see http://wiki.samba.org/index.php/UNIX_Extensions#Minshall.2BFrench_symlinks)
This option is ignored when specified together with the
'sfu' option. Minshall+French symlinks are used even if
the server supports the CIFS Unix Extensions.
sign
Must use packet signing (helps avoid unwanted data modification
by intermediate systems in the route). Note that signing
does not work with lanman or plaintext authentication.
seal
Must seal (encrypt) all data on this mounted share before
sending on the network. Requires support for Unix Extensions.
Note that this differs from the sign mount option in that it
causes encryption of data sent over this mounted share but other
shares mounted to the same server are unaffected.
locallease
This option is rarely needed. Fcntl F_SETLEASE is
used by some applications such as Samba and NFSv4 server to
check to see whether a file is cacheable. CIFS has no way
to explicitly request a lease, but can check whether a file
is cacheable (oplocked). Unfortunately, even if a file
is not oplocked, it could still be cacheable (ie cifs client
could grant fcntl leases if no other local processes are using
the file) for cases for example such as when the server does not
support oplocks and the user is sure that the only updates to
the file will be from this client. Specifying this mount option
will allow the cifs client to check for leases (only) locally
for files which are not oplocked instead of denying leases
in that case. (EXPERIMENTAL)
sec
Security mode. Allowed values are:
none
attempt to connection as a null user (no name)
krb5
Use Kerberos version 5 authentication
krb5i
Use Kerberos authentication and packet signing
ntlm
Use NTLM password hashing (default)
ntlmi
Use NTLM password hashing with signing (if
/proc/fs/cifs/PacketSigningEnabled on or if
server requires signing also can be the default)
ntlmv2
Use NTLMv2 password hashing
ntlmv2i
Use NTLMv2 password hashing with packet signing
lanman
(if configured in kernel config) use older
lanman hash
hard
Retry file operations if server is not responding
soft
Limit retries to unresponsive servers (usually only
one retry) before returning an error. (default)
The mount.cifs mount helper also accepts a few mount options before -o
including:
=============== ===============================================================
-S take password from stdin (equivalent to setting the environment
variable ``PASSWD_FD=0``
-V print mount.cifs version
-? display simple usage information
=============== ===============================================================
With most 2.6 kernel versions of modutils, the version of the cifs kernel
module can be displayed via modinfo.
Misc /proc/fs/cifs Flags and Debug Info
=======================================
Informational pseudo-files:
======================= =======================================================
DebugData Displays information about active CIFS sessions and
shares, features enabled as well as the cifs.ko
version.
Stats Lists summary resource usage information as well as per
share statistics.
open_files List all the open file handles on all active SMB sessions.
mount_params List of all mount parameters available for the module
======================= =======================================================
Configuration pseudo-files:
======================= =======================================================
SecurityFlags Flags which control security negotiation and
also packet signing. Authentication (may/must)
flags (e.g. for NTLMv2) may be combined with
the signing flags. Specifying two different password
hashing mechanisms (as "must use") on the other hand
does not make much sense. Default flags are::
0x00C5
(NTLMv2 and packet signing allowed). Some SecurityFlags
may require enabling a corresponding menuconfig option.
may use packet signing 0x00001
must use packet signing 0x01001
may use NTLMv2 0x00004
must use NTLMv2 0x04004
may use Kerberos security (krb5) 0x00008
must use Kerberos 0x08008
may use NTLMSSP 0x00080
must use NTLMSSP 0x80080
seal (packet encryption) 0x00040
must seal 0x40040
cifsFYI If set to non-zero value, additional debug information
will be logged to the system error log. This field
contains three flags controlling different classes of
debugging entries. The maximum value it can be set
to is 7 which enables all debugging points (default 0).
Some debugging statements are not compiled into the
cifs kernel unless CONFIG_CIFS_DEBUG2 is enabled in the
kernel configuration. cifsFYI may be set to one or
more of the following flags (7 sets them all)::
+-----------------------------------------------+------+
| log cifs informational messages | 0x01 |
+-----------------------------------------------+------+
| log return codes from cifs entry points | 0x02 |
+-----------------------------------------------+------+
| log slow responses | 0x04 |
| (ie which take longer than 1 second) | |
| | |
| CONFIG_CIFS_STATS2 must be enabled in .config | |
+-----------------------------------------------+------+
traceSMB If set to one, debug information is logged to the
system error log with the start of smb requests
and responses (default 0)
LookupCacheEnable If set to one, inode information is kept cached
for one second improving performance of lookups
(default 1)
LinuxExtensionsEnabled If set to one then the client will attempt to
use the CIFS "UNIX" extensions which are optional
protocol enhancements that allow CIFS servers
to return accurate UID/GID information as well
as support symbolic links. If you use servers
such as Samba that support the CIFS Unix
extensions but do not want to use symbolic link
support and want to map the uid and gid fields
to values supplied at mount (rather than the
actual values, then set this to zero. (default 1)
dfscache List the content of the DFS cache.
If set to 0, the client will clear the cache.
======================= =======================================================
These experimental features and tracing can be enabled by changing flags in
/proc/fs/cifs (after the cifs module has been installed or built into the
kernel, e.g. insmod cifs). To enable a feature set it to 1 e.g. to enable
tracing to the kernel message log type::
echo 7 > /proc/fs/cifs/cifsFYI
cifsFYI functions as a bit mask. Setting it to 1 enables additional kernel
logging of various informational messages. 2 enables logging of non-zero
SMB return codes while 4 enables logging of requests that take longer
than one second to complete (except for byte range lock requests).
Setting it to 4 requires CONFIG_CIFS_STATS2 to be set in kernel configuration
(.config). Setting it to seven enables all three. Finally, tracing
the start of smb requests and responses can be enabled via::
echo 1 > /proc/fs/cifs/traceSMB
Per share (per client mount) statistics are available in /proc/fs/cifs/Stats.
Additional information is available if CONFIG_CIFS_STATS2 is enabled in the
kernel configuration (.config). The statistics returned include counters which
represent the number of attempted and failed (ie non-zero return code from the
server) SMB3 (or cifs) requests grouped by request type (read, write, close etc.).
Also recorded is the total bytes read and bytes written to the server for
that share. Note that due to client caching effects this can be less than the
number of bytes read and written by the application running on the client.
Statistics can be reset to zero by ``echo 0 > /proc/fs/cifs/Stats`` which may be
useful if comparing performance of two different scenarios.
Also note that ``cat /proc/fs/cifs/DebugData`` will display information about
the active sessions and the shares that are mounted.
Enabling Kerberos (extended security) works but requires version 1.2 or later
of the helper program cifs.upcall to be present and to be configured in the
/etc/request-key.conf file. The cifs.upcall helper program is from the Samba
project(https://www.samba.org). NTLM and NTLMv2 and LANMAN support do not
require this helper. Note that NTLMv2 security (which does not require the
cifs.upcall helper program), instead of using Kerberos, is sufficient for
some use cases.
DFS support allows transparent redirection to shares in an MS-DFS name space.
In addition, DFS support for target shares which are specified as UNC
names which begin with host names (rather than IP addresses) requires
a user space helper (such as cifs.upcall) to be present in order to
translate host names to ip address, and the user space helper must also
be configured in the file /etc/request-key.conf. Samba, Windows servers and
many NAS appliances support DFS as a way of constructing a global name
space to ease network configuration and improve reliability.
To use cifs Kerberos and DFS support, the Linux keyutils package should be
installed and something like the following lines should be added to the
/etc/request-key.conf file::
create cifs.spnego * * /usr/local/sbin/cifs.upcall %k
create dns_resolver * * /usr/local/sbin/cifs.upcall %k
CIFS kernel module parameters
=============================
These module parameters can be specified or modified either during the time of
module loading or during the runtime by using the interface::
/sys/module/cifs/parameters/<param>
i.e.::
echo "value" > /sys/module/cifs/parameters/<param>
More detailed descriptions of the available module parameters and their values
can be seen by doing:
modinfo cifs (or modinfo smb3)
================= ==========================================================
1. enable_oplocks Enable or disable oplocks. Oplocks are enabled by default.
[Y/y/1]. To disable use any of [N/n/0].
================= ==========================================================
linux-7.2.3/Documentation/admin-guide/cifs/winucase_convert.pl
Executable
+62
View File
@@ -0,0 +1,62 @@
#!/usr/bin/perl -w
#
# winucase_convert.pl -- convert "Windows 8 Upper Case Mapping Table.txt" to
# a two-level set of C arrays.
#
# Copyright 2013: Jeff Layton <jlayton@redhat.com>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
#
while(<>) {
next if (!/^0x(..)(..)\t0x(....)\t/);
$firstchar = hex($1);
$secondchar = hex($2);
$uppercase = hex($3);
$top[$firstchar][$secondchar] = $uppercase;
}
for ($i = 0; $i < 256; $i++) {
next if (!$top[$i]);
printf("static const wchar_t t2_%2.2x[256] = {", $i);
for ($j = 0; $j < 256; $j++) {
if (($j % 8) == 0) {
print "\n\t";
} else {
print " ";
}
printf("0x%4.4x,", $top[$i][$j] ? $top[$i][$j] : 0);
}
print "\n};\n\n";
}
printf("static const wchar_t *const toplevel[256] = {", $i);
for ($i = 0; $i < 256; $i++) {
if (($i % 8) == 0) {
print "\n\t";
} elsif ($top[$i]) {
print " ";
} else {
print " ";
}
if ($top[$i]) {
printf("t2_%2.2x,", $i);
} else {
print "NULL,";
}
}
print "\n};\n\n";
linux-7.2.3/Documentation/admin-guide/clearing-warn-once.rst
+9
View File
@@ -0,0 +1,9 @@
Clearing WARN_ONCE
------------------
WARN_ONCE / WARN_ON_ONCE / printk_once only emit a message once.
echo 1 > /sys/kernel/debug/clear_warn_once
clears the state and allows the warnings to print once again.
This can be useful after test suite runs to reproduce problems.
linux-7.2.3/Documentation/admin-guide/cpu-isolation.rst
+357
View File
@@ -0,0 +1,357 @@
.. SPDX-License-Identifier: GPL-2.0
=============
CPU Isolation
=============
Introduction
============
"CPU Isolation" means leaving a CPU exclusive to a given workload
without any undesired code interference from the kernel.
Those interferences, commonly pointed out as "noise", can be triggered
by asynchronous events (interrupts, timers, scheduler preemption by
workqueues and kthreads, ...) or synchronous events (syscalls and page
faults).
Such noise usually goes unnoticed. After all, synchronous events are a
component of the requested kernel service. And asynchronous events are
either sufficiently well-distributed by the scheduler when executed
as tasks or reasonably fast when executed as interrupt. The timer
interrupt can even execute 1024 times per seconds without a significant
and measurable impact most of the time.
However some rare and extreme workloads can be quite sensitive to
those kinds of noise. This is the case, for example, with high
bandwidth network processing that can't afford losing a single packet
or very low latency network processing. Typically those use cases
involve DPDK, bypassing the kernel networking stack and performing
direct access to the networking device from userspace.
In order to run a CPU without or with limited kernel noise, the
related housekeeping work needs to be either shut down, migrated or
offloaded.
Housekeeping
============
In the CPU isolation terminology, housekeeping is the work, often
asynchronous, that the kernel needs to process in order to maintain
all its services. It matches the noises and disturbances enumerated
above except when at least one CPU is isolated. Then housekeeping may
make use of further coping mechanisms if CPU-tied work must be
offloaded.
Housekeeping CPUs are the non-isolated CPUs where the kernel noise
is moved away from isolated CPUs.
The isolation can be implemented in several ways depending on the
nature of the noise:
- Unbound work, where "unbound" means not tied to any CPU, can be
simply migrated away from isolated CPUs to housekeeping CPUs.
This is the case of unbound workqueues, kthreads and timers.
- Bound work, where "bound" means tied to a specific CPU, usually
can't be moved away as-is by nature. Either:
- The work must switch to a locked implementation. E.g.:
This is the case of RCU with CONFIG_RCU_NOCB_CPU.
- The related feature must be shut down and considered
incompatible with isolated CPUs. E.g.: Lockup watchdog,
unreliable clocksources, etc...
- An elaborate and heavyweight coping mechanism stands as a
replacement. E.g.: the timer tick is shut down on nohz_full
CPUs but with the constraint of running a single task on
them. A significant cost penalty is added on kernel entry/exit
and a residual 1Hz scheduler tick is offloaded to housekeeping
CPUs.
In any case, housekeeping work has to be handled, which is why there
must be at least one housekeeping CPU in the system, preferably more
if the machine runs a lot of CPUs. For example one per node on NUMA
systems.
Also CPU isolation often means a tradeoff between noise-free isolated
CPUs and added overhead on housekeeping CPUs, sometimes even on
isolated CPUs entering the kernel.
Isolation features
==================
Different levels of isolation can be configured in the kernel, each of
which has its own drawbacks and tradeoffs.
Scheduler domain isolation
--------------------------
This feature isolates a CPU from the scheduler topology. As a result,
the target isn't part of the load balancing. Tasks won't migrate
either from or to it unless affined explicitly.
As a side effect the CPU is also isolated from unbound workqueues and
unbound kthreads.
Requirements
~~~~~~~~~~~~
- CONFIG_CPUSETS=y for the cpusets-based interface
Tradeoffs
~~~~~~~~~
By nature, the system load is overall less distributed since some CPUs
are extracted from the global load balancing.
Interfaces
~~~~~~~~~~
- Documentation/admin-guide/cgroup-v2.rst cpuset isolated partitions are recommended
because they are tunable at runtime.
- The 'isolcpus=' kernel boot parameter with the 'domain' flag is a
less flexible alternative that doesn't allow for runtime
reconfiguration.
IRQs isolation
--------------
Isolate the IRQs whenever possible, so that they don't fire on the
target CPUs.
Interfaces
~~~~~~~~~~
- The file /proc/irq/\*/smp_affinity as explained in detail in
Documentation/core-api/irq/irq-affinity.rst page.
- The "irqaffinity=" kernel boot parameter for a default setting.
- The "managed_irq" flag in the "isolcpus=" kernel boot parameter
tries a best effort affinity override for managed IRQs.
Full Dynticks (aka nohz_full)
-----------------------------
Full dynticks extends the dynticks idle mode, which stops the tick when
the CPU is idle, to CPUs running a single task in userspace. That is,
the timer tick is stopped if the environment allows it.
Global timer callbacks are also isolated from the nohz_full CPUs.
Requirements
~~~~~~~~~~~~
- CONFIG_NO_HZ_FULL=y
Constraints
~~~~~~~~~~~
- The isolated CPUs must run a single task only. Multitask requires
the tick to maintain preemption. This is usually fine since the
workload usually can't stand the latency of random context switches.
- No call to the kernel from isolated CPUs, at the risk of triggering
random noise.
- No use of POSIX CPU timers on isolated CPUs.
- Architecture must have a stable and reliable clocksource (no
unreliable TSC that requires the watchdog).
Tradeoffs
~~~~~~~~~
In terms of cost, this is the most invasive isolation feature. It is
assumed to be used when the workload spends most of its time in
userspace and doesn't rely on the kernel except for preparatory
work because:
- RCU adds more overhead due to the locked, offloaded and threaded
callbacks processing (the same that would be obtained with "rcu_nocbs"
boot parameter).
- Kernel entry/exit through syscalls, exceptions and IRQs are more
costly due to fully ordered RmW operations that maintain userspace
as RCU extended quiescent state. Also the CPU time is accounted on
kernel boundaries instead of periodically from the tick.
- Housekeeping CPUs must run a 1Hz residual remote scheduler tick
on behalf of the isolated CPUs.
Checklist
=========
You have set up each of the above isolation features but you still
observe jitters that trash your workload? Make sure to check a few
elements before proceeding.
Some of these checklist items are similar to those of real-time
workloads:
- Use mlock() to prevent your pages from being swapped away. Page
faults are usually not compatible with jitter sensitive workloads.
- Avoid SMT to prevent your hardware thread from being "preempted"
by another one.
- CPU frequency changes may induce subtle sorts of jitter in a
workload. Cpufreq should be used and tuned with caution.
- Deep C-states may result in latency issues upon wake-up. If this
happens to be a problem, C-states can be limited via kernel boot
parameters such as processor.max_cstate or intel_idle.max_cstate.
More finegrained tunings are described in
Documentation/admin-guide/pm/cpuidle.rst page
- Your system may be subject to firmware-originating interrupts - x86 has
System Management Interrupts (SMIs) for example. Check your system BIOS
to disable such interference, and with some luck your vendor will have
a BIOS tuning guidance for low-latency operations.
Full isolation example
======================
In this example, the system has 8 CPUs and the 8th is to be fully
isolated. Since CPUs start from 0, the 8th CPU is CPU 7.
Kernel parameters
-----------------
Set the following kernel boot parameters to disable SMT and setup tick
and IRQ isolation:
- Full dynticks: nohz_full=7
- IRQs isolation: irqaffinity=0-6
- Managed IRQs isolation: isolcpus=managed_irq,7
- Prevent SMT: nosmt
The full command line is then:
nohz_full=7 irqaffinity=0-6 isolcpus=managed_irq,7 nosmt
CPUSET configuration (cgroup v2)
--------------------------------
Assuming cgroup v2 is mounted to /sys/fs/cgroup, the following script
isolates CPU 7 from scheduler domains.
::
cd /sys/fs/cgroup
# Activate the cpuset subsystem
echo +cpuset > cgroup.subtree_control
# Create partition to be isolated
mkdir test
cd test
echo +cpuset > cgroup.subtree_control
# Isolate CPU 7
echo 7 > cpuset.cpus
echo "isolated" > cpuset.cpus.partition
The userspace workload
----------------------
Fake a pure userspace workload, the program below runs a dummy
userspace loop on the isolated CPU 7.
::
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
int main(void)
{
// Move the current task to the isolated cpuset (bind to CPU 7)
int fd = open("/sys/fs/cgroup/test/cgroup.procs", O_WRONLY);
if (fd < 0) {
perror("Can't open cpuset file...\n");
return 0;
}
write(fd, "0\n", 2);
close(fd);
// Run an endless dummy loop until the launcher kills us
while (1)
;
return 0;
}
Build it and save for later step:
::
# gcc user_loop.c -o user_loop
The launcher
------------
The below launcher runs the above program for 10 seconds and traces
the noise resulting from preempting tasks and IRQs.
::
TRACING=/sys/kernel/tracing/
# Make sure tracing is off for now
echo 0 > $TRACING/tracing_on
# Flush previous traces
echo > $TRACING/trace
# Record disturbance from other tasks
echo 1 > $TRACING/events/sched/sched_switch/enable
# Record disturbance from interrupts
echo 1 > $TRACING/events/irq_vectors/enable
# Now we can start tracing
echo 1 > $TRACING/tracing_on
# Run the dummy user_loop for 10 seconds on CPU 7
./user_loop &
USER_LOOP_PID=$!
sleep 10
kill $USER_LOOP_PID
# Disable tracing and save traces from CPU 7 in a file
echo 0 > $TRACING/tracing_on
cat $TRACING/per_cpu/cpu7/trace > trace.7
If no specific problem arose, the output of trace.7 should look like
the following:
::
<idle>-0 [007] d..2. 1980.976624: sched_switch: prev_comm=swapper/7 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=user_loop next_pid=1553 next_prio=120
user_loop-1553 [007] d.h.. 1990.946593: reschedule_entry: vector=253
user_loop-1553 [007] d.h.. 1990.946593: reschedule_exit: vector=253
That is, no specific noise triggered between the first trace and the
second during 10 seconds when user_loop was running.
Debugging
=========
Of course things are never so easy, especially on this matter.
Chances are that actual noise will be observed in the aforementioned
trace.7 file.
The best way to investigate further is to enable finer grained
tracepoints such as those of subsystems producing asynchronous
events: workqueue, timer, irq_vector, etc... It also can be
interesting to enable the tick_stop event to diagnose why the tick is
retained when that happens.
Some tools may also be useful for higher level analysis:
- Documentation/tools/rtla/rtla.rst provides a suite of tools to analyze
latency and noise in the system. For example Documentation/tools/rtla/rtla-osnoise.rst
runs a kernel tracer that analyzes and output a summary of the noises.
- dynticks-testing does something similar to rtla-osnoise but in userspace. It is available
at git://git.kernel.org/pub/scm/linux/kernel/git/frederic/dynticks-testing.git
linux-7.2.3/Documentation/admin-guide/cpu-load.rst
+117
View File
@@ -0,0 +1,117 @@
========
CPU load
========
Linux exports various bits of information via ``/proc/stat`` and
``/proc/uptime`` that userland tools, such as top(1), use to calculate
the average time system spent in a particular state, for example::
$ iostat
Linux 2.6.18.3-exp (linmac) 02/20/2007
avg-cpu: %user %nice %system %iowait %steal %idle
10.01 0.00 2.92 5.44 0.00 81.63
...
Here the system thinks that over the default sampling period the
system spent 10.01% of the time doing work in user space, 2.92% in the
kernel, and was overall 81.63% of the time idle.
In most cases the ``/proc/stat`` information reflects the reality quite
closely, however due to the nature of how/when the kernel collects
this data sometimes it can not be trusted at all.
So how is this information collected? Whenever timer interrupt is
signalled the kernel looks what kind of task was running at this
moment and increments the counter that corresponds to this tasks
kind/state. The problem with this is that the system could have
switched between various states multiple times between two timer
interrupts yet the counter is incremented only for the last state.
Example
-------
If we imagine the system with one task that periodically burns cycles
in the following manner::
time line between two timer interrupts
|--------------------------------------|
^ ^
|_ something begins working |
|_ something goes to sleep
(only to be awaken quite soon)
In the above situation the system will be 0% loaded according to the
``/proc/stat`` (since the timer interrupt will always happen when the
system is executing the idle handler), but in reality the load is
closer to 99%.
One can imagine many more situations where this behavior of the kernel
will lead to quite erratic information inside ``/proc/stat``::
/* gcc -o hog smallhog.c */
#include <time.h>
#include <limits.h>
#include <signal.h>
#include <sys/time.h>
#define HIST 10
static volatile sig_atomic_t stop;
static void sighandler(int signr)
{
(void) signr;
stop = 1;
}
static unsigned long hog (unsigned long niters)
{
stop = 0;
while (!stop && --niters);
return niters;
}
int main (void)
{
int i;
struct itimerval it = {
.it_interval = { .tv_sec = 0, .tv_usec = 1 },
.it_value = { .tv_sec = 0, .tv_usec = 1 } };
sigset_t set;
unsigned long v[HIST];
double tmp = 0.0;
unsigned long n;
signal(SIGALRM, &sighandler);
setitimer(ITIMER_REAL, &it, NULL);
hog (ULONG_MAX);
for (i = 0; i < HIST; ++i) v[i] = ULONG_MAX - hog(ULONG_MAX);
for (i = 0; i < HIST; ++i) tmp += v[i];
tmp /= HIST;
n = tmp - (tmp / 3.0);
sigemptyset(&set);
sigaddset(&set, SIGALRM);
for (;;) {
hog(n);
sigwait(&set, &i);
}
return 0;
}
References
----------
- https://lore.kernel.org/r/loom.20070212T063225-663@post.gmane.org
- Documentation/filesystems/proc.rst (1.8)
Thanks
------
Con Kolivas, Pavel Machek
linux-7.2.3/Documentation/admin-guide/cputopology.rst
+101
View File
@@ -0,0 +1,101 @@
===========================================
How CPU topology info is exported via sysfs
===========================================
CPU topology info is exported via sysfs. Items (attributes) are similar
to /proc/cpuinfo output of some architectures. They reside in
/sys/devices/system/cpu/cpuX/topology/. Please refer to the ABI file:
Documentation/ABI/stable/sysfs-devices-system-cpu.
Architecture-neutral, drivers/base/topology.c, exports these attributes.
However the die, cluster, book, and drawer hierarchy related sysfs files will
only be created if an architecture provides the related macros as described
below.
For an architecture to support this feature, it must define some of
these macros in include/asm-XXX/topology.h::
#define topology_physical_package_id(cpu)
#define topology_die_id(cpu)
#define topology_cluster_id(cpu)
#define topology_core_id(cpu)
#define topology_book_id(cpu)
#define topology_drawer_id(cpu)
#define topology_sibling_cpumask(cpu)
#define topology_core_cpumask(cpu)
#define topology_cluster_cpumask(cpu)
#define topology_die_cpumask(cpu)
#define topology_book_cpumask(cpu)
#define topology_drawer_cpumask(cpu)
The type of ``**_id macros`` is int.
The type of ``**_cpumask macros`` is ``(const) struct cpumask *``. The latter
correspond with appropriate ``**_siblings`` sysfs attributes (except for
topology_sibling_cpumask() which corresponds with thread_siblings).
To be consistent on all architectures, include/linux/topology.h
provides default definitions for any of the above macros that are
not defined by include/asm-XXX/topology.h:
1) topology_physical_package_id: -1
2) topology_die_id: -1
3) topology_cluster_id: -1
4) topology_core_id: 0
5) topology_book_id: -1
6) topology_drawer_id: -1
7) topology_sibling_cpumask: just the given CPU
8) topology_core_cpumask: just the given CPU
9) topology_cluster_cpumask: just the given CPU
10) topology_die_cpumask: just the given CPU
11) topology_book_cpumask: just the given CPU
12) topology_drawer_cpumask: just the given CPU
Additionally, CPU topology information is provided under
/sys/devices/system/cpu and includes these files. The internal
source for the output is in brackets ("[]").
=========== ==========================================================
kernel_max: the maximum CPU index allowed by the kernel configuration.
[NR_CPUS-1]
offline: CPUs that are not online because they have been
HOTPLUGGED off or exceed the limit of CPUs allowed by the
kernel configuration (kernel_max above).
[~cpu_online_mask + cpus >= NR_CPUS]
online: CPUs that are online and being scheduled [cpu_online_mask]
possible: CPUs that have been allocated resources and can be
brought online if they are present. [cpu_possible_mask]
present: CPUs that have been identified as being present in the
system. [cpu_present_mask]
=========== ==========================================================
The format for the above output is compatible with cpulist_parse()
[see <linux/cpumask.h>]. Some examples follow.
In this example, there are 64 CPUs in the system but cpus 32-63 exceed
the kernel max which is limited to 0..31 by the NR_CPUS config option
being 32. Note also that CPUs 2 and 4-31 are not online but could be
brought online as they are both present and possible::
kernel_max: 31
offline: 2,4-31,32-63
online: 0-1,3
possible: 0-31
present: 0-31
In this example, the NR_CPUS config option is 128, but the kernel was
started with possible_cpus=144. There are 4 CPUs in the system and cpu2
was manually taken offline (and is the only CPU that can be brought
online.)::
kernel_max: 127
offline: 2,4-127,128-143
online: 0-1,3
possible: 0-127
present: 0-3
See Documentation/core-api/cpu_hotplug.rst for the possible_cpus=NUM
kernel start parameter as well as more information on the various cpumasks.
linux-7.2.3/Documentation/admin-guide/dell_rbu.rst
+128
View File
@@ -0,0 +1,128 @@
=========================================
Dell Remote BIOS Update driver (dell_rbu)
=========================================
Purpose
=======
Document demonstrating the use of the Dell Remote BIOS Update driver
for updating BIOS images on Dell servers and desktops.
Scope
=====
This document discusses the functionality of the rbu driver only.
It does not cover the support needed from applications to enable the BIOS to
update itself with the image downloaded in to the memory.
Overview
========
This driver works with Dell OpenManage or Dell Update Packages for updating
the BIOS on Dell servers (starting from servers sold since 1999), desktops
and notebooks (starting from those sold in 2005).
Please go to http://support.dell.com register and you can find info on
OpenManage and Dell Update packages (DUP).
Libsmbios can also be used to update BIOS on Dell systems go to
https://linux.dell.com/libsmbios/ for details.
Dell_RBU driver supports BIOS update using the monolithic image and packetized
image methods. In case of monolithic the driver allocates a contiguous chunk
of physical pages having the BIOS image. In case of packetized the app
using the driver breaks the image in to packets of fixed sizes and the driver
would place each packet in contiguous physical memory. The driver also
maintains a link list of packets for reading them back.
If the dell_rbu driver is unloaded all the allocated memory is freed.
The rbu driver needs to have an application (as mentioned above) which will
inform the BIOS to enable the update in the next system reboot.
The user should not unload the rbu driver after downloading the BIOS image
or updating.
The driver load creates the following directories under the /sys file system::
/sys/class/firmware/dell_rbu/loading
/sys/class/firmware/dell_rbu/data
/sys/devices/platform/dell_rbu/image_type
/sys/devices/platform/dell_rbu/data
/sys/devices/platform/dell_rbu/packet_size
The driver supports two types of update mechanism; monolithic and packetized.
These update mechanism depends upon the BIOS currently running on the system.
Most of the Dell systems support a monolithic update where the BIOS image is
copied to a single contiguous block of physical memory.
In case of packet mechanism the single memory can be broken in smaller chunks
of contiguous memory and the BIOS image is scattered in these packets.
By default the driver uses monolithic memory for the update type. This can be
changed to packets during the driver load time by specifying the load
parameter image_type=packet. This can also be changed later as below::
echo packet > /sys/devices/platform/dell_rbu/image_type
In packet update mode the packet size has to be given before any packets can
be downloaded. It is done as below::
echo XXXX > /sys/devices/platform/dell_rbu/packet_size
In the packet update mechanism, the user needs to create a new file having
packets of data arranged back to back. It can be done as follows:
The user creates packets header, gets the chunk of the BIOS image and
places it next to the packetheader; now, the packetheader + BIOS image chunk
added together should match the specified packet_size. This makes one
packet, the user needs to create more such packets out of the entire BIOS
image file and then arrange all these packets back to back in to one single
file.
This file is then copied to /sys/class/firmware/dell_rbu/data.
Once this file gets to the driver, the driver extracts packet_size data from
the file and spreads it across the physical memory in contiguous packet_sized
space.
This method makes sure that all the packets get to the driver in a single operation.
In monolithic update the user simply get the BIOS image (.hdr file) and copies
to the data file as is without any change to the BIOS image itself.
Do the steps below to download the BIOS image.
1) echo 1 > /sys/class/firmware/dell_rbu/loading
2) cp bios_image.hdr /sys/class/firmware/dell_rbu/data
3) echo 0 > /sys/class/firmware/dell_rbu/loading
The /sys/class/firmware/dell_rbu/ entries will remain till the following is
done.
::
echo -1 > /sys/class/firmware/dell_rbu/loading
Until this step is completed the driver cannot be unloaded.
Also echoing either mono, packet or init in to image_type will free up the
memory allocated by the driver.
If a user by accident executes steps 1 and 3 above without executing step 2;
it will make the /sys/class/firmware/dell_rbu/ entries disappear.
The entries can be recreated by doing the following::
echo init > /sys/devices/platform/dell_rbu/image_type
.. note:: echoing init in image_type does not change its original value.
Also the driver provides /sys/devices/platform/dell_rbu/data readonly file to
read back the image downloaded.
.. note::
After updating the BIOS image a user mode application needs to execute
code which sends the BIOS update request to the BIOS. So on the next reboot
the BIOS knows about the new image downloaded and it updates itself.
Also don't unload the rbu driver if the image has to be updated.
linux-7.2.3/Documentation/admin-guide/device-mapper/cache-policies.rst
+131
View File
@@ -0,0 +1,131 @@
=============================
Guidance for writing policies
=============================
Try to keep transactionality out of it. The core is careful to
avoid asking about anything that is migrating. This is a pain, but
makes it easier to write the policies.
Mappings are loaded into the policy at construction time.
Every bio that is mapped by the target is referred to the policy.
The policy can return a simple HIT or MISS or issue a migration.
Currently there's no way for the policy to issue background work,
e.g. to start writing back dirty blocks that are going to be evicted
soon.
Because we map bios, rather than requests it's easy for the policy
to get fooled by many small bios. For this reason the core target
issues periodic ticks to the policy. It's suggested that the policy
doesn't update states (eg, hit counts) for a block more than once
for each tick. The core ticks by watching bios complete, and so
trying to see when the io scheduler has let the ios run.
Overview of supplied cache replacement policies
===============================================
multiqueue (mq)
---------------
This policy is now an alias for smq (see below).
The following tunables are accepted, but have no effect::
'sequential_threshold <#nr_sequential_ios>'
'random_threshold <#nr_random_ios>'
'read_promote_adjustment <value>'
'write_promote_adjustment <value>'
'discard_promote_adjustment <value>'
Stochastic multiqueue (smq)
---------------------------
This policy is the default.
The stochastic multi-queue (smq) policy addresses some of the problems
with the multiqueue (mq) policy.
The smq policy (vs mq) offers the promise of less memory utilization,
improved performance and increased adaptability in the face of changing
workloads. smq also does not have any cumbersome tuning knobs.
Users may switch from "mq" to "smq" simply by appropriately reloading a
DM table that is using the cache target. Doing so will cause all of the
mq policy's hints to be dropped. Also, performance of the cache may
degrade slightly until smq recalculates the origin device's hotspots
that should be cached.
Memory usage
^^^^^^^^^^^^
The mq policy used a lot of memory; 88 bytes per cache block on a 64
bit machine.
smq uses 28bit indexes to implement its data structures rather than
pointers. It avoids storing an explicit hit count for each block. It
has a 'hotspot' queue, rather than a pre-cache, which uses a quarter of
the entries (each hotspot block covers a larger area than a single
cache block).
All this means smq uses ~25bytes per cache block. Still a lot of
memory, but a substantial improvement nonetheless.
Level balancing
^^^^^^^^^^^^^^^
mq placed entries in different levels of the multiqueue structures
based on their hit count (~ln(hit count)). This meant the bottom
levels generally had the most entries, and the top ones had very
few. Having unbalanced levels like this reduced the efficacy of the
multiqueue.
smq does not maintain a hit count, instead it swaps hit entries with
the least recently used entry from the level above. The overall
ordering being a side effect of this stochastic process. With this
scheme we can decide how many entries occupy each multiqueue level,
resulting in better promotion/demotion decisions.
Adaptability:
The mq policy maintained a hit count for each cache block. For a
different block to get promoted to the cache its hit count has to
exceed the lowest currently in the cache. This meant it could take a
long time for the cache to adapt between varying IO patterns.
smq doesn't maintain hit counts, so a lot of this problem just goes
away. In addition it tracks performance of the hotspot queue, which
is used to decide which blocks to promote. If the hotspot queue is
performing badly then it starts moving entries more quickly between
levels. This lets it adapt to new IO patterns very quickly.
Performance
^^^^^^^^^^^
Testing smq shows substantially better performance than mq.
cleaner
-------
The cleaner writes back all dirty blocks in a cache to decommission it.
Examples
========
The syntax for a table is::
cache <metadata dev> <cache dev> <origin dev> <block size>
<#feature_args> [<feature arg>]*
<policy> <#policy_args> [<policy arg>]*
The syntax to send a message using the dmsetup command is::
dmsetup message <mapped device> 0 sequential_threshold 1024
dmsetup message <mapped device> 0 random_threshold 8
Using dmsetup::
dmsetup create blah --table "0 268435456 cache /dev/sdb /dev/sdc \
/dev/sdd 512 0 mq 4 sequential_threshold 1024 random_threshold 8"
creates a 128GB large mapped device named 'blah' with the
sequential threshold set to 1024 and the random_threshold set to 8.
linux-7.2.3/Documentation/admin-guide/device-mapper/cache.rst
+337
View File
@@ -0,0 +1,337 @@
=====
Cache
=====
Introduction
============
dm-cache is a device mapper target written by Joe Thornber, Heinz
Mauelshagen, and Mike Snitzer.
It aims to improve performance of a block device (eg, a spindle) by
dynamically migrating some of its data to a faster, smaller device
(eg, an SSD).
This device-mapper solution allows us to insert this caching at
different levels of the dm stack, for instance above the data device for
a thin-provisioning pool. Caching solutions that are integrated more
closely with the virtual memory system should give better performance.
The target reuses the metadata library used in the thin-provisioning
library.
The decision as to what data to migrate and when is left to a plug-in
policy module. Several of these have been written as we experiment,
and we hope other people will contribute others for specific io
scenarios (eg. a vm image server).
Glossary
========
Migration
Movement of the primary copy of a logical block from one
device to the other.
Promotion
Migration from slow device to fast device.
Demotion
Migration from fast device to slow device.
The origin device always contains a copy of the logical block, which
may be out of date or kept in sync with the copy on the cache device
(depending on policy).
Design
======
Sub-devices
-----------
The target is constructed by passing three devices to it (along with
other parameters detailed later):
1. An origin device - the big, slow one.
2. A cache device - the small, fast one.
3. A small metadata device - records which blocks are in the cache,
which are dirty, and extra hints for use by the policy object.
This information could be put on the cache device, but having it
separate allows the volume manager to configure it differently,
e.g. as a mirror for extra robustness. This metadata device may only
be used by a single cache device.
Fixed block size
----------------
The origin is divided up into blocks of a fixed size. This block size
is configurable when you first create the cache. Typically we've been
using block sizes of 256KB - 1024KB. The block size must be between 64
sectors (32KB) and 2097152 sectors (1GB) and a multiple of 64 sectors (32KB).
Having a fixed block size simplifies the target a lot. But it is
something of a compromise. For instance, a small part of a block may be
getting hit a lot, yet the whole block will be promoted to the cache.
So large block sizes are bad because they waste cache space. And small
block sizes are bad because they increase the amount of metadata (both
in core and on disk).
Cache operating modes
---------------------
The cache has three operating modes: writeback, writethrough and
passthrough.
If writeback, the default, is selected then a write to a block that is
cached will go only to the cache and the block will be marked dirty in
the metadata.
If writethrough is selected then a write to a cached block will not
complete until it has hit both the origin and cache devices. Clean
blocks should remain clean.
If passthrough is selected, useful when the cache contents are not known
to be coherent with the origin device, then all reads are served from
the origin device (all reads miss the cache) and all writes are
forwarded to the origin device; additionally, write hits cause cache
block invalidates. To enable passthrough mode the cache must be clean.
Passthrough mode allows a cache device to be activated without having to
worry about coherency. Coherency that exists is maintained, although
the cache will gradually cool as writes take place. If the coherency of
the cache can later be verified, or established through use of the
"invalidate_cblocks" message, the cache device can be transitioned to
writethrough or writeback mode while still warm. Otherwise, the cache
contents can be discarded prior to transitioning to the desired
operating mode.
A simple cleaner policy is provided, which will clean (write back) all
dirty blocks in a cache. Useful for decommissioning a cache or when
shrinking a cache. Shrinking the cache's fast device requires all cache
blocks, in the area of the cache being removed, to be clean. If the
area being removed from the cache still contains dirty blocks the resize
will fail. Care must be taken to never reduce the volume used for the
cache's fast device until the cache is clean. This is of particular
importance if writeback mode is used. Writethrough and passthrough
modes already maintain a clean cache. Future support to partially clean
the cache, above a specified threshold, will allow for keeping the cache
warm and in writeback mode during resize.
Migration throttling
--------------------
Migrating data between the origin and cache device uses bandwidth.
The user can set a throttle to prevent more than a certain amount of
migration occurring at any one time. Currently we're not taking any
account of normal io traffic going to the devices. More work needs
doing here to avoid migrating during those peak io moments.
For the time being, a message "migration_threshold <#sectors>"
can be used to set the maximum number of sectors being migrated,
the default being 2048 sectors (1MB).
Updating on-disk metadata
-------------------------
On-disk metadata is committed every time a FLUSH or FUA bio is written.
If no such requests are made then commits will occur every second. This
means the cache behaves like a physical disk that has a volatile write
cache. If power is lost you may lose some recent writes. The metadata
should always be consistent in spite of any crash.
The 'dirty' state for a cache block changes far too frequently for us
to keep updating it on the fly. So we treat it as a hint. In normal
operation it will be written when the dm device is suspended. If the
system crashes all cache blocks will be assumed dirty when restarted.
Per-block policy hints
----------------------
Policy plug-ins can store a chunk of data per cache block. It's up to
the policy how big this chunk is, but it should be kept small. Like the
dirty flags this data is lost if there's a crash so a safe fallback
value should always be possible.
Policy hints affect performance, not correctness.
Policy messaging
----------------
Policies will have different tunables, specific to each one, so we
need a generic way of getting and setting these. Device-mapper
messages are used. Refer to cache-policies.txt.
Discard bitset resolution
-------------------------
We can avoid copying data during migration if we know the block has
been discarded. A prime example of this is when mkfs discards the
whole block device. We store a bitset tracking the discard state of
blocks. However, we allow this bitset to have a different block size
from the cache blocks. This is because we need to track the discard
state for all of the origin device (compare with the dirty bitset
which is just for the smaller cache device).
Target interface
================
Constructor
-----------
::
cache <metadata dev> <cache dev> <origin dev> <block size>
<#feature args> [<feature arg>]*
<policy> <#policy args> [policy args]*
================ =======================================================
metadata dev fast device holding the persistent metadata
cache dev fast device holding cached data blocks
origin dev slow device holding original data blocks
block size cache unit size in sectors
#feature args number of feature arguments passed
feature args writethrough or passthrough (The default is writeback.)
policy the replacement policy to use
#policy args an even number of arguments corresponding to
key/value pairs passed to the policy
policy args key/value pairs passed to the policy
E.g. 'sequential_threshold 1024'
See cache-policies.txt for details.
================ =======================================================
Optional feature arguments are:
==================== ========================================================
writethrough write through caching that prohibits cache block
content from being different from origin block content.
Without this argument, the default behaviour is to write
back cache block contents later for performance reasons,
so they may differ from the corresponding origin blocks.
passthrough a degraded mode useful for various cache coherency
situations (e.g., rolling back snapshots of
underlying storage). Reads and writes always go to
the origin. If a write goes to a cached origin
block, then the cache block is invalidated.
To enable passthrough mode the cache must be clean.
metadata2 use version 2 of the metadata. This stores the dirty
bits in a separate btree, which improves speed of
shutting down the cache.
no_discard_passdown disable passing down discards from the cache
to the origin's data device.
==================== ========================================================
A policy called 'default' is always registered. This is an alias for
the policy we currently think is giving best all round performance.
As the default policy could vary between kernels, if you are relying on
the characteristics of a specific policy, always request it by name.
Status
------
::
<metadata block size> <#used metadata blocks>/<#total metadata blocks>
<cache block size> <#used cache blocks>/<#total cache blocks>
<#read hits> <#read misses> <#write hits> <#write misses>
<#demotions> <#promotions> <#dirty> <#features> <features>*
<#core args> <core args>* <policy name> <#policy args> <policy args>*
<cache metadata mode>
========================= =====================================================
metadata block size Fixed block size for each metadata block in
sectors
#used metadata blocks Number of metadata blocks used
#total metadata blocks Total number of metadata blocks
cache block size Configurable block size for the cache device
in sectors
#used cache blocks Number of blocks resident in the cache
#total cache blocks Total number of cache blocks
#read hits Number of times a READ bio has been mapped
to the cache
#read misses Number of times a READ bio has been mapped
to the origin
#write hits Number of times a WRITE bio has been mapped
to the cache
#write misses Number of times a WRITE bio has been
mapped to the origin
#demotions Number of times a block has been removed
from the cache
#promotions Number of times a block has been moved to
the cache
#dirty Number of blocks in the cache that differ
from the origin
#feature args Number of feature args to follow
feature args 'writethrough' (optional)
#core args Number of core arguments (must be even)
core args Key/value pairs for tuning the core
e.g. migration_threshold
policy name Name of the policy
#policy args Number of policy arguments to follow (must be even)
policy args Key/value pairs e.g. sequential_threshold
cache metadata mode ro if read-only, rw if read-write
In serious cases where even a read-only mode is
deemed unsafe no further I/O will be permitted and
the status will just contain the string 'Fail'.
The userspace recovery tools should then be used.
needs_check 'needs_check' if set, '-' if not set
A metadata operation has failed, resulting in the
needs_check flag being set in the metadata's
superblock. The metadata device must be
deactivated and checked/repaired before the
cache can be made fully operational again.
'-' indicates needs_check is not set.
========================= =====================================================
Messages
--------
Policies will have different tunables, specific to each one, so we
need a generic way of getting and setting these. Device-mapper
messages are used. (A sysfs interface would also be possible.)
The message format is::
<key> <value>
E.g.::
dmsetup message my_cache 0 sequential_threshold 1024
Invalidation is removing an entry from the cache without writing it
back. Cache blocks can be invalidated via the invalidate_cblocks
message, which takes an arbitrary number of cblock ranges. Each cblock
range's end value is "one past the end", meaning 5-10 expresses a range
of values from 5 to 9. Each cblock must be expressed as a decimal
value, in the future a variant message that takes cblock ranges
expressed in hexadecimal may be needed to better support efficient
invalidation of larger caches. The cache must be in passthrough mode
when invalidate_cblocks is used::
invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]*
E.g.::
dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789
Examples
========
The test suite can be found here:
https://github.com/jthornber/device-mapper-test-suite
::
dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
/dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0'
dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \
/dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \
mq 4 sequential_threshold 1024 random_threshold 8'
linux-7.2.3/Documentation/admin-guide/device-mapper/delay.rst
+54
View File
@@ -0,0 +1,54 @@
========
dm-delay
========
Device-Mapper's "delay" target delays reads and/or writes
and/or flushes and optionally maps them to different devices.
Arguments::
<device> <offset> <delay> [<write_device> <write_offset> <write_delay>
[<flush_device> <flush_offset> <flush_delay>]]
Table line has to either have 3, 6 or 9 arguments:
3: apply offset and delay to read, write and flush operations on device
6: apply offset and delay to device, also apply write_offset and write_delay
to write and flush operations on optionally different write_device with
optionally different sector offset
9: same as 6 arguments plus define flush_offset and flush_delay explicitly
on/with optionally different flush_device/flush_offset.
Offsets are specified in sectors.
Delays are specified in milliseconds.
Example scripts
===============
::
#!/bin/sh
#
# Create mapped device named "delayed" delaying read, write and flush operations for 500ms.
#
dmsetup create delayed --table "0 `blockdev --getsz $1` delay $1 0 500"
::
#!/bin/sh
#
# Create mapped device delaying write and flush operations for 400ms and
# splitting reads to device $1 but writes and flushes to different device $2
# to different offsets of 2048 and 4096 sectors respectively.
#
dmsetup create delayed --table "0 `blockdev --getsz $1` delay $1 2048 0 $2 4096 400"
::
#!/bin/sh
#
# Create mapped device delaying reads for 50ms, writes for 100ms and flushes for 333ms
# onto the same backing device at offset 0 sectors.
#
dmsetup create delayed --table "0 `blockdev --getsz $1` delay $1 0 50 $2 0 100 $1 0 333"
linux-7.2.3/Documentation/admin-guide/device-mapper/dm-clone.rst
+333
View File
@@ -0,0 +1,333 @@
.. SPDX-License-Identifier: GPL-2.0-only
========
dm-clone
========
Introduction
============
dm-clone is a device mapper target which produces a one-to-one copy of an
existing, read-only source device into a writable destination device: It
presents a virtual block device which makes all data appear immediately, and
redirects reads and writes accordingly.
The main use case of dm-clone is to clone a potentially remote, high-latency,
read-only, archival-type block device into a writable, fast, primary-type device
for fast, low-latency I/O. The cloned device is visible/mountable immediately
and the copy of the source device to the destination device happens in the
background, in parallel with user I/O.
For example, one could restore an application backup from a read-only copy,
accessible through a network storage protocol (NBD, Fibre Channel, iSCSI, AoE,
etc.), into a local SSD or NVMe device, and start using the device immediately,
without waiting for the restore to complete.
When the cloning completes, the dm-clone table can be removed altogether and be
replaced, e.g., by a linear table, mapping directly to the destination device.
The dm-clone target reuses the metadata library used by the thin-provisioning
target.
Glossary
========
Hydration
The process of filling a region of the destination device with data from
the same region of the source device, i.e., copying the region from the
source to the destination device.
Once a region gets hydrated we redirect all I/O regarding it to the destination
device.
Design
======
Sub-devices
-----------
The target is constructed by passing three devices to it (along with other
parameters detailed later):
1. A source device - the read-only device that gets cloned and source of the
hydration.
2. A destination device - the destination of the hydration, which will become a
clone of the source device.
3. A small metadata device - it records which regions are already valid in the
destination device, i.e., which regions have already been hydrated, or have
been written to directly, via user I/O.
The size of the destination device must be at least equal to the size of the
source device.
Regions
-------
dm-clone divides the source and destination devices in fixed sized regions.
Regions are the unit of hydration, i.e., the minimum amount of data copied from
the source to the destination device.
The region size is configurable when you first create the dm-clone device. The
recommended region size is the same as the file system block size, which usually
is 4KB. The region size must be between 8 sectors (4KB) and 2097152 sectors
(1GB) and a power of two.
Reads and writes from/to hydrated regions are serviced from the destination
device.
A read to a not yet hydrated region is serviced directly from the source device.
A write to a not yet hydrated region will be delayed until the corresponding
region has been hydrated and the hydration of the region starts immediately.
Note that a write request with size equal to region size will skip copying of
the corresponding region from the source device and overwrite the region of the
destination device directly.
Discards
--------
dm-clone interprets a discard request to a range that hasn't been hydrated yet
as a hint to skip hydration of the regions covered by the request, i.e., it
skips copying the region's data from the source to the destination device, and
only updates its metadata.
If the destination device supports discards, then by default dm-clone will pass
down discard requests to it.
Background Hydration
--------------------
dm-clone copies continuously from the source to the destination device, until
all of the device has been copied.
Copying data from the source to the destination device uses bandwidth. The user
can set a throttle to prevent more than a certain amount of copying occurring at
any one time. Moreover, dm-clone takes into account user I/O traffic going to
the devices and pauses the background hydration when there is I/O in-flight.
A message `hydration_threshold <#regions>` can be used to set the maximum number
of regions being copied, the default being 1 region.
dm-clone employs dm-kcopyd for copying portions of the source device to the
destination device. By default, we issue copy requests of size equal to the
region size. A message `hydration_batch_size <#regions>` can be used to tune the
size of these copy requests. Increasing the hydration batch size results in
dm-clone trying to batch together contiguous regions, so we copy the data in
batches of this many regions.
When the hydration of the destination device finishes, a dm event will be sent
to user space.
Updating on-disk metadata
-------------------------
On-disk metadata is committed every time a FLUSH or FUA bio is written. If no
such requests are made then commits will occur every second. This means the
dm-clone device behaves like a physical disk that has a volatile write cache. If
power is lost you may lose some recent writes. The metadata should always be
consistent in spite of any crash.
Target Interface
================
Constructor
-----------
::
clone <metadata dev> <destination dev> <source dev> <region size>
[<#feature args> [<feature arg>]* [<#core args> [<core arg>]*]]
================ ==============================================================
metadata dev Fast device holding the persistent metadata
destination dev The destination device, where the source will be cloned
source dev Read only device containing the data that gets cloned
region size The size of a region in sectors
#feature args Number of feature arguments passed
feature args no_hydration or no_discard_passdown
#core args An even number of arguments corresponding to key/value pairs
passed to dm-clone
core args Key/value pairs passed to dm-clone, e.g. `hydration_threshold
256`
================ ==============================================================
Optional feature arguments are:
==================== =========================================================
no_hydration Create a dm-clone instance with background hydration
disabled
no_discard_passdown Disable passing down discards to the destination device
==================== =========================================================
Optional core arguments are:
================================ ==============================================
hydration_threshold <#regions> Maximum number of regions being copied from
the source to the destination device at any
one time, during background hydration.
hydration_batch_size <#regions> During background hydration, try to batch
together contiguous regions, so we copy data
from the source to the destination device in
batches of this many regions.
================================ ==============================================
Status
------
::
<metadata block size> <#used metadata blocks>/<#total metadata blocks>
<region size> <#hydrated regions>/<#total regions> <#hydrating regions>
<#feature args> <feature args>* <#core args> <core args>*
<clone metadata mode>
======================= =======================================================
metadata block size Fixed block size for each metadata block in sectors
#used metadata blocks Number of metadata blocks used
#total metadata blocks Total number of metadata blocks
region size Configurable region size for the device in sectors
#hydrated regions Number of regions that have finished hydrating
#total regions Total number of regions to hydrate
#hydrating regions Number of regions currently hydrating
#feature args Number of feature arguments to follow
feature args Feature arguments, e.g. `no_hydration`
#core args Even number of core arguments to follow
core args Key/value pairs for tuning the core, e.g.
`hydration_threshold 256`
clone metadata mode ro if read-only, rw if read-write
In serious cases where even a read-only mode is deemed
unsafe no further I/O will be permitted and the status
will just contain the string 'Fail'. If the metadata
mode changes, a dm event will be sent to user space.
======================= =======================================================
Messages
--------
`disable_hydration`
Disable the background hydration of the destination device.
`enable_hydration`
Enable the background hydration of the destination device.
`hydration_threshold <#regions>`
Set background hydration threshold.
`hydration_batch_size <#regions>`
Set background hydration batch size.
Examples
========
Clone a device containing a file system
---------------------------------------
1. Create the dm-clone device.
::
dmsetup create clone --table "0 1048576000 clone $metadata_dev $dest_dev \
$source_dev 8 1 no_hydration"
2. Mount the device and trim the file system. dm-clone interprets the discards
sent by the file system and it will not hydrate the unused space.
::
mount /dev/mapper/clone /mnt/cloned-fs
fstrim /mnt/cloned-fs
3. Enable background hydration of the destination device.
::
dmsetup message clone 0 enable_hydration
4. When the hydration finishes, we can replace the dm-clone table with a linear
table.
::
dmsetup suspend clone
dmsetup load clone --table "0 1048576000 linear $dest_dev 0"
dmsetup resume clone
The metadata device is no longer needed and can be safely discarded or reused
for other purposes.
Known issues
============
1. We redirect reads, to not-yet-hydrated regions, to the source device. If
reading the source device has high latency and the user repeatedly reads from
the same regions, this behaviour could degrade performance. We should use
these reads as hints to hydrate the relevant regions sooner. Currently, we
rely on the page cache to cache these regions, so we hopefully don't end up
reading them multiple times from the source device.
2. Release in-core resources, i.e., the bitmaps tracking which regions are
hydrated, after the hydration has finished.
3. During background hydration, if we fail to read the source or write to the
destination device, we print an error message, but the hydration process
continues indefinitely, until it succeeds. We should stop the background
hydration after a number of failures and emit a dm event for user space to
notice.
Why not...?
===========
We explored the following alternatives before implementing dm-clone:
1. Use dm-cache with cache size equal to the source device and implement a new
cloning policy:
* The resulting cache device is not a one-to-one mirror of the source device
and thus we cannot remove the cache device once cloning completes.
* dm-cache writes to the source device, which violates our requirement that
the source device must be treated as read-only.
* Caching is semantically different from cloning.
2. Use dm-snapshot with a COW device equal to the source device:
* dm-snapshot stores its metadata in the COW device, so the resulting device
is not a one-to-one mirror of the source device.
* No background copying mechanism.
* dm-snapshot needs to commit its metadata whenever a pending exception
completes, to ensure snapshot consistency. In the case of cloning, we don't
need to be so strict and can rely on committing metadata every time a FLUSH
or FUA bio is written, or periodically, like dm-thin and dm-cache do. This
improves the performance significantly.
3. Use dm-mirror: The mirror target has a background copying/mirroring
mechanism, but it writes to all mirrors, thus violating our requirement that
the source device must be treated as read-only.
4. Use dm-thin's external snapshot functionality. This approach is the most
promising among all alternatives, as the thinly-provisioned volume is a
one-to-one mirror of the source device and handles reads and writes to
un-provisioned/not-yet-cloned areas the same way as dm-clone does.
Still:
* There is no background copying mechanism, though one could be implemented.
* Most importantly, we want to support arbitrary block devices as the
destination of the cloning process and not restrict ourselves to
thinly-provisioned volumes. Thin-provisioning has an inherent metadata
overhead, for maintaining the thin volume mappings, which significantly
degrades performance.
Moreover, cloning a device shouldn't force the use of thin-provisioning. On
the other hand, if we wish to use thin provisioning, we can just use a thin
LV as dm-clone's destination device.
linux-7.2.3/Documentation/admin-guide/device-mapper/dm-crypt.rst
+212
View File
@@ -0,0 +1,212 @@
========
dm-crypt
========
Device-Mapper's "crypt" target provides transparent encryption of block devices
using the kernel crypto API.
For a more detailed description of supported parameters see:
https://gitlab.com/cryptsetup/cryptsetup/wikis/DMCrypt
Parameters::
<cipher> <key> <iv_offset> <device path> \
<offset> [<#opt_params> <opt_params>]
<cipher>
Encryption cipher, encryption mode and Initial Vector (IV) generator.
The cipher specifications format is::
cipher[:keycount]-chainmode-ivmode[:ivopts]
Examples::
aes-cbc-essiv:sha256
aes-xts-plain64
serpent-xts-plain64
Cipher format also supports direct specification with kernel crypt API
format (selected by capi: prefix). The IV specification is the same
as for the first format type.
This format is mainly used for specification of authenticated modes.
The crypto API cipher specifications format is::
capi:cipher_api_spec-ivmode[:ivopts]
Examples::
capi:cbc(aes)-essiv:sha256
capi:xts(aes)-plain64
Examples of authenticated modes::
capi:gcm(aes)-random
capi:authenc(hmac(sha256),xts(aes))-random
capi:rfc7539(chacha20,poly1305)-random
The /proc/crypto contains a list of currently loaded crypto modes.
<key>
Key used for encryption. It is encoded either as a hexadecimal number
or it can be passed as <key_string> prefixed with single colon
character (':') for keys residing in kernel keyring service.
You can only use key sizes that are valid for the selected cipher
in combination with the selected iv mode.
Note that for some iv modes the key string can contain additional
keys (for example IV seed) so the key contains more parts concatenated
into a single string.
<key_string>
The kernel keyring key is identified by string in following format:
<key_size>:<key_type>:<key_description>.
<key_size>
The encryption key size in bytes. The kernel key payload size must match
the value passed in <key_size>.
<key_type>
Either 'logon', 'user', 'encrypted' or 'trusted' kernel key type.
<key_description>
The kernel keyring key description crypt target should look for
when loading key of <key_type>.
<keycount>
Multi-key compatibility mode. You can define <keycount> keys and
then sectors are encrypted according to their offsets (sector 0 uses key0;
sector 1 uses key1 etc.). <keycount> must be a power of two.
<iv_offset>
The IV offset is a sector count that is added to the sector number
before creating the IV.
<device path>
This is the device that is going to be used as backend and contains the
encrypted data. You can specify it as a path like /dev/xxx or a device
number <major>:<minor>.
<offset>
Starting sector within the device where the encrypted data begins.
<#opt_params>
Number of optional parameters. If there are no optional parameters,
the optional parameters section can be skipped or #opt_params can be zero.
Otherwise #opt_params is the number of following arguments.
Example of optional parameters section:
3 allow_discards same_cpu_crypt submit_from_crypt_cpus
allow_discards
Block discard requests (a.k.a. TRIM) are passed through the crypt device.
The default is to ignore discard requests.
WARNING: Assess the specific security risks carefully before enabling this
option. For example, allowing discards on encrypted devices may lead to
the leak of information about the ciphertext device (filesystem type,
used space etc.) if the discarded blocks can be located easily on the
device later.
same_cpu_crypt
Perform encryption using the same cpu that IO was submitted on.
The default is to use an unbound workqueue so that encryption work
is automatically balanced between available CPUs.
high_priority
Set dm-crypt workqueues and the writer thread to high priority. This
improves throughput and latency of dm-crypt while degrading general
responsiveness of the system.
submit_from_crypt_cpus
Disable offloading writes to a separate thread after encryption.
There are some situations where offloading write bios from the
encryption threads to a single thread degrades performance
significantly. The default is to offload write bios to the same
thread because it benefits CFQ to have writes submitted using the
same context.
no_read_workqueue
Bypass dm-crypt internal workqueue and process read requests synchronously.
no_write_workqueue
Bypass dm-crypt internal workqueue and process write requests synchronously.
This option is automatically enabled for host-managed zoned block devices
(e.g. host-managed SMR hard-disks).
integrity:<bytes>:<type>
The device requires additional <bytes> metadata per-sector stored
in per-bio integrity structure. This metadata must by provided
by underlying dm-integrity target.
The <type> can be "none" if metadata is used only for persistent IV.
For Authenticated Encryption with Additional Data (AEAD)
the <type> is "aead". An AEAD mode additionally calculates and verifies
integrity for the encrypted device. The additional space is then
used for storing authentication tag (and persistent IV if needed).
integrity_key_size:<bytes>
Optionally set the integrity key size if it differs from the digest size.
It allows the use of wrapped key algorithms where the key size is
independent of the cryptographic key size.
sector_size:<bytes>
Use <bytes> as the encryption unit instead of 512 bytes sectors.
This option can be in range 512 - 4096 bytes and must be power of two.
Virtual device will announce this size as a minimal IO and logical sector.
iv_large_sectors
IV generators will use sector number counted in <sector_size> units
instead of default 512 bytes sectors.
For example, if <sector_size> is 4096 bytes, plain64 IV for the second
sector will be 8 (without flag) and 1 if iv_large_sectors is present.
The <iv_offset> must be multiple of <sector_size> (in 512 bytes units)
if this flag is specified.
integrity_key_size:<bytes>
Use an integrity key of <bytes> size instead of using an integrity key size
of the digest size of the used HMAC algorithm.
Module parameters::
max_read_size
Maximum size of read requests. When a request larger than this size
is received, dm-crypt will split the request. The splitting improves
concurrency (the split requests could be encrypted in parallel by multiple
cores), but it also causes overhead. The user should tune this parameters to
fit the actual workload.
max_write_size
Maximum size of write requests. When a request larger than this size
is received, dm-crypt will split the request. The splitting improves
concurrency (the split requests could be encrypted in parallel by multiple
cores), but it also causes overhead. The user should tune this parameters to
fit the actual workload.
Example scripts
===============
LUKS (Linux Unified Key Setup) is now the preferred way to set up disk
encryption with dm-crypt using the 'cryptsetup' utility, see
https://gitlab.com/cryptsetup/cryptsetup
::
#!/bin/sh
# Create a crypt device using dmsetup
dmsetup create crypt1 --table "0 `blockdev --getsz $1` crypt aes-cbc-essiv:sha256 babebabebabebabebabebabebabebabe 0 $1 0"
::
#!/bin/sh
# Create a crypt device using dmsetup when encryption key is stored in keyring service
dmsetup create crypt2 --table "0 `blockdev --getsize $1` crypt aes-cbc-essiv:sha256 :32:logon:my_prefix:my_key 0 $1 0"
::
#!/bin/sh
# Create a crypt device using cryptsetup and LUKS header with default cipher
cryptsetup luksFormat $1
cryptsetup luksOpen $1 crypt1
linux-7.2.3/Documentation/admin-guide/device-mapper/dm-dust.rst
+305
View File
@@ -0,0 +1,305 @@
dm-dust
=======
This target emulates the behavior of bad sectors at arbitrary
locations, and the ability to enable the emulation of the failures
at an arbitrary time.
This target behaves similarly to a linear target. At a given time,
the user can send a message to the target to start failing read
requests on specific blocks (to emulate the behavior of a hard disk
drive with bad sectors).
When the failure behavior is enabled (i.e.: when the output of
"dmsetup status" displays "fail_read_on_bad_block"), reads of blocks
in the "bad block list" will fail with EIO ("Input/output error").
Writes of blocks in the "bad block list will result in the following:
1. Remove the block from the "bad block list".
2. Successfully complete the write.
This emulates the "remapped sector" behavior of a drive with bad
sectors.
Normally, a drive that is encountering bad sectors will most likely
encounter more bad sectors, at an unknown time or location.
With dm-dust, the user can use the "addbadblock" and "removebadblock"
messages to add arbitrary bad blocks at new locations, and the
"enable" and "disable" messages to modulate the state of whether the
configured "bad blocks" will be treated as bad, or bypassed.
This allows the pre-writing of test data and metadata prior to
simulating a "failure" event where bad sectors start to appear.
Table parameters
----------------
<device_path> <offset> <blksz>
Mandatory parameters:
<device_path>:
Path to the block device.
<offset>:
Offset to data area from start of device_path
<blksz>:
Block size in bytes
(minimum 512, maximum 1073741824, must be a power of 2)
Usage instructions
------------------
First, find the size (in 512-byte sectors) of the device to be used::
$ sudo blockdev --getsz /dev/vdb1
33552384
Create the dm-dust device:
(For a device with a block size of 512 bytes)
::
$ sudo dmsetup create dust1 --table '0 33552384 dust /dev/vdb1 0 512'
(For a device with a block size of 4096 bytes)
::
$ sudo dmsetup create dust1 --table '0 33552384 dust /dev/vdb1 0 4096'
Check the status of the read behavior ("bypass" indicates that all I/O
will be passed through to the underlying device; "verbose" indicates that
bad block additions, removals, and remaps will be verbosely logged)::
$ sudo dmsetup status dust1
0 33552384 dust 252:17 bypass verbose
$ sudo dd if=/dev/mapper/dust1 of=/dev/null bs=512 count=128 iflag=direct
128+0 records in
128+0 records out
$ sudo dd if=/dev/zero of=/dev/mapper/dust1 bs=512 count=128 oflag=direct
128+0 records in
128+0 records out
Adding and removing bad blocks
------------------------------
At any time (i.e.: whether the device has the "bad block" emulation
enabled or disabled), bad blocks may be added or removed from the
device via the "addbadblock" and "removebadblock" messages::
$ sudo dmsetup message dust1 0 addbadblock 60
kernel: device-mapper: dust: badblock added at block 60
$ sudo dmsetup message dust1 0 addbadblock 67
kernel: device-mapper: dust: badblock added at block 67
$ sudo dmsetup message dust1 0 addbadblock 72
kernel: device-mapper: dust: badblock added at block 72
These bad blocks will be stored in the "bad block list".
While the device is in "bypass" mode, reads and writes will succeed::
$ sudo dmsetup status dust1
0 33552384 dust 252:17 bypass
Enabling block read failures
----------------------------
To enable the "fail read on bad block" behavior, send the "enable" message::
$ sudo dmsetup message dust1 0 enable
kernel: device-mapper: dust: enabling read failures on bad sectors
$ sudo dmsetup status dust1
0 33552384 dust 252:17 fail_read_on_bad_block
With the device in "fail read on bad block" mode, attempting to read a
block will encounter an "Input/output error"::
$ sudo dd if=/dev/mapper/dust1 of=/dev/null bs=512 count=1 skip=67 iflag=direct
dd: error reading '/dev/mapper/dust1': Input/output error
0+0 records in
0+0 records out
0 bytes copied, 0.00040651 s, 0.0 kB/s
...and writing to the bad blocks will remove the blocks from the list,
therefore emulating the "remap" behavior of hard disk drives::
$ sudo dd if=/dev/zero of=/dev/mapper/dust1 bs=512 count=128 oflag=direct
128+0 records in
128+0 records out
kernel: device-mapper: dust: block 60 removed from badblocklist by write
kernel: device-mapper: dust: block 67 removed from badblocklist by write
kernel: device-mapper: dust: block 72 removed from badblocklist by write
kernel: device-mapper: dust: block 87 removed from badblocklist by write
Bad block add/remove error handling
-----------------------------------
Attempting to add a bad block that already exists in the list will
result in an "Invalid argument" error, as well as a helpful message::
$ sudo dmsetup message dust1 0 addbadblock 88
device-mapper: message ioctl on dust1 failed: Invalid argument
kernel: device-mapper: dust: block 88 already in badblocklist
Attempting to remove a bad block that doesn't exist in the list will
result in an "Invalid argument" error, as well as a helpful message::
$ sudo dmsetup message dust1 0 removebadblock 87
device-mapper: message ioctl on dust1 failed: Invalid argument
kernel: device-mapper: dust: block 87 not found in badblocklist
Counting the number of bad blocks in the bad block list
-------------------------------------------------------
To count the number of bad blocks configured in the device, run the
following message command::
$ sudo dmsetup message dust1 0 countbadblocks
A message will print with the number of bad blocks currently
configured on the device::
countbadblocks: 895 badblock(s) found
Querying for specific bad blocks
--------------------------------
To find out if a specific block is in the bad block list, run the
following message command::
$ sudo dmsetup message dust1 0 queryblock 72
The following message will print if the block is in the list::
dust_query_block: block 72 found in badblocklist
The following message will print if the block is not in the list::
dust_query_block: block 72 not found in badblocklist
The "queryblock" message command will work in both the "enabled"
and "disabled" modes, allowing the verification of whether a block
will be treated as "bad" without having to issue I/O to the device,
or having to "enable" the bad block emulation.
Clearing the bad block list
---------------------------
To clear the bad block list (without needing to individually run
a "removebadblock" message command for every block), run the
following message command::
$ sudo dmsetup message dust1 0 clearbadblocks
After clearing the bad block list, the following message will appear::
dust_clear_badblocks: badblocks cleared
If there were no bad blocks to clear, the following message will
appear::
dust_clear_badblocks: no badblocks found
Listing the bad block list
--------------------------
To list all bad blocks in the bad block list (using an example device
with blocks 1 and 2 in the bad block list), run the following message
command::
$ sudo dmsetup message dust1 0 listbadblocks
1
2
If there are no bad blocks in the bad block list, the command will
execute with no output::
$ sudo dmsetup message dust1 0 listbadblocks
Message commands list
---------------------
Below is a list of the messages that can be sent to a dust device:
Operations on blocks (requires a <blknum> argument)::
addbadblock <blknum>
queryblock <blknum>
removebadblock <blknum>
...where <blknum> is a block number within range of the device
(corresponding to the block size of the device.)
Single argument message commands::
countbadblocks
clearbadblocks
listbadblocks
disable
enable
quiet
Device removal
--------------
When finished, remove the device via the "dmsetup remove" command::
$ sudo dmsetup remove dust1
Quiet mode
----------
On test runs with many bad blocks, it may be desirable to avoid
excessive logging (from bad blocks added, removed, or "remapped").
This can be done by enabling "quiet mode" via the following message::
$ sudo dmsetup message dust1 0 quiet
This will suppress log messages from add / remove / removed by write
operations. Log messages from "countbadblocks" or "queryblock"
message commands will still print in quiet mode.
The status of quiet mode can be seen by running "dmsetup status"::
$ sudo dmsetup status dust1
0 33552384 dust 252:17 fail_read_on_bad_block quiet
To disable quiet mode, send the "quiet" message again::
$ sudo dmsetup message dust1 0 quiet
$ sudo dmsetup status dust1
0 33552384 dust 252:17 fail_read_on_bad_block verbose
(The presence of "verbose" indicates normal logging.)
"Why not...?"
-------------
scsi_debug has a "medium error" mode that can fail reads on one
specified sector (sector 0x1234, hardcoded in the source code), but
it uses RAM for the persistent storage, which drastically decreases
the potential device size.
dm-flakey fails all I/O from all block locations at a specified time
frequency, and not a given point in time.
When a bad sector occurs on a hard disk drive, reads to that sector
are failed by the device, usually resulting in an error code of EIO
("I/O error") or ENODATA ("No data available"). However, a write to
the sector may succeed, and result in the sector becoming readable
after the device controller no longer experiences errors reading the
sector (or after a reallocation of the sector). However, there may
be bad sectors that occur on the device in the future, in a different,
unpredictable location.
This target seeks to provide a device that can exhibit the behavior
of a bad sector at a known sector location, at a known time, based
on a large storage device (at least tens of gigabytes, not occupying
system memory).
linux-7.2.3/Documentation/admin-guide/device-mapper/dm-ebs.rst
+51
View File
@@ -0,0 +1,51 @@
======
dm-ebs
======
This target is similar to the linear target except that it emulates
a smaller logical block size on a device with a larger logical block
size. Its main purpose is to provide emulation of 512 byte sectors on
devices that do not provide this emulation (i.e. 4K native disks).
Supported emulated logical block sizes 512, 1024, 2048 and 4096.
Underlying block size can be set to > 4K to test buffering larger units.
Table parameters
----------------
<dev path> <offset> <emulated sectors> [<underlying sectors>]
Mandatory parameters:
<dev path>:
Full pathname to the underlying block-device,
or a "major:minor" device-number.
<offset>:
Starting sector within the device;
has to be a multiple of <emulated sectors>.
<emulated sectors>:
Number of sectors defining the logical block size to be emulated;
1, 2, 4, 8 sectors of 512 bytes supported.
Optional parameter:
<underlying sectors>:
Number of sectors defining the logical block size of <dev path>.
2^N supported, e.g. 8 = emulate 8 sectors of 512 bytes = 4KiB.
If not provided, the logical block size of <dev path> will be used.
Examples:
Emulate 1 sector = 512 bytes logical block size on /dev/sda starting at
offset 1024 sectors with underlying devices block size automatically set:
ebs /dev/sda 1024 1
Emulate 2 sector = 1KiB logical block size on /dev/sda starting at
offset 128 sectors, enforce 2KiB underlying device block size.
This presumes 2KiB logical blocksize on /dev/sda or less to work:
ebs /dev/sda 128 2 4
linux-7.2.3/Documentation/admin-guide/device-mapper/dm-flakey.rst
+88
View File
@@ -0,0 +1,88 @@
=========
dm-flakey
=========
This target is the same as the linear target except that it exhibits
unreliable behaviour periodically. It's been found useful in simulating
failing devices for testing purposes.
Starting from the time the table is loaded, the device is available for
<up interval> seconds, then exhibits unreliable behaviour for <down
interval> seconds, and then this cycle repeats.
Also, consider using this in combination with the dm-delay target too,
which can delay reads and writes and/or send them to different
underlying devices.
Table parameters
----------------
::
<dev path> <offset> <up interval> <down interval> \
[<num_features> [<feature arguments>]]
Mandatory parameters:
<dev path>:
Full pathname to the underlying block-device, or a
"major:minor" device-number.
<offset>:
Starting sector within the device.
<up interval>:
Number of seconds device is available.
<down interval>:
Number of seconds device returns errors.
Optional feature parameters:
If no feature parameters are present, during the periods of
unreliability, all I/O returns errors.
error_reads:
All read I/O is failed with an error signalled.
Write I/O is handled correctly.
drop_writes:
All write I/O is silently ignored.
Read I/O is handled correctly.
error_writes:
All write I/O is failed with an error signalled.
Read I/O is handled correctly.
corrupt_bio_byte <Nth_byte> <direction> <value> <flags>:
During <down interval>, replace <Nth_byte> of the data of
each matching bio with <value>.
<Nth_byte>:
The offset of the byte to replace.
Counting starts at 1, to replace the first byte.
<direction>:
Either 'r' to corrupt reads or 'w' to corrupt writes.
'w' is incompatible with drop_writes.
<value>:
The value (from 0-255) to write.
<flags>:
Perform the replacement only if bio->bi_opf has all the
selected flags set.
random_read_corrupt <probability>
During <down interval>, replace random byte in a read bio
with a random value. probability is an integer between
0 and 1000000000 meaning 0% to 100% probability of corruption.
random_write_corrupt <probability>
During <down interval>, replace random byte in a write bio
with a random value. probability is an integer between
0 and 1000000000 meaning 0% to 100% probability of corruption.
Examples:
Replaces the 32nd byte of READ bios with the value 1::
corrupt_bio_byte 32 r 1 0
Replaces the 224th byte of REQ_META (=32) bios with the value 0::
corrupt_bio_byte 224 w 0 32

Some files were not shown because too many files have changed in this diff Show More