Notes2024-01-06T16:29:00.00-05:00Yuan Fuhttps://archive.casouri.cc/favicon.pngurn:uuid:53fd03d4-ec1b-11eb-8cca-e7401fdbc2e2Classic Systems Papers: Notes for CSE 221urn:uuid:43cbedf8-ac05-11ed-b63d-1f5ec28d824a2024-01-05T19:45:00.00-05:00<p>During my time at UCSD, I enjoyed their
systems courses greatly. It’d be a shame to let those wonderful things I
learnt fade away from my memory. So I compiled my notes into this
article. I hope this can be helpful to future me and entertaining for
others.</p><p><span class="oldstyle-num">CSE
221</span> is the entry course, introducing students to reading
papers and the essential systems papers. The cast of papers is pretty
stable over the years. Here is a syllabus of a <span
class="oldstyle-num">CSE 221</span> similar to the one I took,
you can find links to the papers and extension reading there: <a
href="https://archive.casouri.cc/note/2024/cse221/https:/cseweb.ucsd.edu/classes/wi21/cse221-a/readings.html"><em>CSE
221: Reading List and Schedule, Winter
2021</em></a>.</p><h2 id="THE%20System"
class="section">THE System</h2><p><em>The Structure
of the “THE”-Multiprogramming System</em>, <span
class="oldstyle-num">1968</span>, by none other than Edsger W.
Dijkstra.</p><p>The main take away is “central abstraction in
a hierarchy”. The central abstraction is sequential process, and
hierarchy is basically “layers”. The benefit of layers is that it’s easy
to verify soundness and prove correctness for each individual layer,
which is essential to handle complexity.</p><p>To Dijkstra,
if a designer structures their system well, the possible test cases for
the system at each level would be so few such that it’s easy to cover
every possible case.</p><p>He then mentioned that “industrial
software makers” has mixed feelings of this methodology: they agree that
it’s the right thing to do, but doubt whether it’s applicable in the real
world, away from the shelter of academia. Dijkstra’s stance is that the
larger the project, the more essential the structuring. This stance is
apparent in <a id="footref:ewd1041" class="footref-anchor
obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Aewd1041">his other writings<sup
class="inline-footref">1</sup></a>.</p><div
id="footdef:ewd1041" class="footdef"><div class="def-footref
obviously-a-link"><a aria-label="Jump back to main text"
href="#footref%3Aewd1041">1</a></div><div
class="def-footdef"><a
href="https://archive.casouri.cc/note/2024/cse221/https:/www.cs.utexas.edu/users/EWD/transcriptions/EWD10xx/EWD1041.html"><em>EWD
1041</em></a>. Now, I don’t think it is realistic to write
proofs for every system you design (and how do you ensure the proof is
correct?), but good structuring, and designing with testing in mind are
certainly essential.</div></div><p>The downside of
layering is, of course, the potential loss of efficiently, due to either
the overhead added by layering, or the lack of details hidden by lower
layers. For example, the graphic subsystem in win<span
class="oldstyle-num">32</span> was moved into the kernel in
<span class="oldstyle-num">NT4</span>, because there were too
many boundary crossing.</p><p>And sometimes it’s hard to
separate the system into layers at all, eg, due to circular dependency,
etc. For example, in Linux, memory used by the scheduler is pinned and
never page.</p><p>We also learned some interesting
terminology used at the time; “harmonious cooperation” means no deadlock,
and “deadly embrace” means deadlock.</p><h2 id="Nucleus"
class="section">Nucleus</h2><p><em>The Nucleus of a
Multiprogramming System</em>, <span
class="oldstyle-num">1970</span>.</p><p>Basically
they want a “nucleus” (small kernal) that supports multiple simultaneous
operating system implementations. So the user can have their OS however
they want. (Another example of “mechanism instead of policy”, sort of.)
This school of thought would later reappear on exokernel and micro
kernel.</p><p>The nucleus provides a scheduler (for process
and I/O), communication (messages passing), and primitive for controlling
processes (create, start, stop, remove).</p><p>In their
design, the parent process is basically the OS of their child processes,
controlling allocation of resources for them: starting/stoping them,
allocating memory and storage to them, etc.</p><p>However,
the parent process doesn’t have full control over their children: it
doesn’t control scheduling for it’s children. Nucleus handles scheduling;
it divides computing time by round-robin scheduling among all active
processes.</p><p>A more “complete” abstraction would be
having nucleus schedule the top-level processes and let those processes
schedule their children themselves. Perhaps it would be too inconvenient
if you need to implement scheduler for every “OS” you want to
run.</p><h2 id="HYDRA"
class="section">HYDRA</h2><p><em>HYDRA: The Kernel
of a Multiprocessor Operating System</em>, <span
class="oldstyle-num">1974</span>.</p><p>The authros
have several design goals for HYDRA: a) separation of mechanism and
policy; b) reject strict hierarchy layering for access control, because
they consider access control more of a mesh than layers; c) an emphasize
on protection—not only comprehensive protection, but also flexible
protection. They provide protection mechanism that can be used for not
only for regular things like I/O, etc, but also arbitrary things that a
higher-level program want to protect/control. It surely would be nice if
UNIX has something similar to offer.</p><p>HYDRA structures
protection around <em>capabilities</em>. Capability is
basically the right to use some resource—a key to a door. Possessing the
capability means you have the right of using whatever resource it
references. For example, file descriptors in UNIX are capabilities: when
you open a file, the kernel checks if you are allowed to read/write that
file, and if the check passes, you get a file descriptor. Then you are
free to read/write to that file as long as you hold the file descriptor;
no need to go through access checks every time.</p><p>In
genreal, in an access-controlled OS, there are resources, like data or a
device; execution domains, like “execute as this user” or “execute as
this group”; and access control, controlling which domain can access
which resource.</p><p>In HYDRA, there is
<em>procedure</em>, LNS, and <em>process</em>.
Procedure is a executable program or a subroutine. LNS (local name space)
is the execution domain. Conceptually it is a collection of capabilities,
it determines what you can and cannot do. Each invocation of a procedure
has a LNS attached to it. To explain it in UNIX terms, when a user Alice
runs a program <code>ls</code>, the capabilities Alice has is
the LNS, and <code>ls</code> is the procedure. Finally, a
process is conceptually a (call) stack of procedures with their
accompanying LNS.</p><p>Since each invocation of procedures
have an accompanying LNS, the callee’s LNS could have more or different
capabilities from its caller, so HYDRA can support <em>right
amplification</em>.</p><p>Right amplification is when
caller has more privilege/capabilities than the caller. For example, in
UNIX, when a program uses a syscall, that syscall executed by the kernel
has far more privilege than the caller. For another example, when Alice
runs <code>passwd</code> to change her password, that program
can modify the password file which Alice has no access to, because
<code>passwd</code> has a euid (effective user id) with
higher privilege.</p><p>Another concept often used in
security is ACL (access control list). It’s basically a table recording
who has access to what. ACL and capabilities each have their pros and
cons. To use an ACL, you need to know the user; with capabilities, anyone
with the capability can have access, you don’t need to know the
particular user. Capabilities is easier to check, and useful for
distributed systems or very large systems, where storing information of
all users/entities is not possible.</p><p>However,
capabilities are unforgettable, ie, you can’t take it back. Maybe you can
make them expire, but that’s more complexity. Capabilities can also be
duplicated and given away, which has it’s own virtues and
vices.</p><p>Since ACL is easy to store and manage, and
capability is easy to check, they are often used together. In UNIX,
opening a file warrens a check in the ACL, and the file descriptor
returned to you is a capability.</p><p>It’s interesting to
think of the access control systems used around us. Windows certainly has
a more sophisticated ACL than UNIX. What about Google Docs, eh? On top of
the regular features, they also support “accessible through links”, “can
comment but not edit”, etc.</p><h2 id="TENEX"
class="section">TENEX</h2><p><em>TENEX, a Paged Time
Sharing System for the PDP-10</em>, <span
class="oldstyle-num">1972</span>.</p><p>TENEX is the
predecessor of MULTICS, which in turn is the predecessor of UNIX. It runs
on <span class="oldstyle-num">PDP-10</span>, a machine very
popular at the time: used by Harvard, MIT, CMU, to name a few. <span
class="oldstyle-num">PDP-10</span> was manufactured by BBN, a
military contractor at the time. It’s micro-coded, meaning its
instructions are programmable.</p><p>In BBN’s pager, each
page is <span class="oldstyle-num">512</span> words, the TLB
is called “associative register”. Their virtual memory supports <span
class="oldstyle-num">256</span>K words and copy-on-write. A
process in TENEX always have exactly one superior (parent) process and
any number of inferior (child) processes. Processes communicate through
a) sharing memory, b) direct control (parent to child only), and c)
pseudo (software simulated) interrupts. Theses are also the only ways of
IPC we have today in UNIX. Would be nice if we had message-passing
built-in to the OS. But maybe D-Bus is even better, since it can be
portable.</p><p>TENEX can run binary programs compiled for
<span class="oldstyle-num">DEC 10/50</span>, the vendor OS
for the <span class="oldstyle-num">PDP-10</span>. All the
TENEX syscalls “were implemented with the JSYS instruction, reserving all
old monitor [OS/kernel] calls for their previous use”. They also
implemented all of the <span class="oldstyle-num">DEC
10/50</span> syscalls as a compatibility package. The first time a
program calls a <span class="oldstyle-num">DEC 10/50</span>
syscall, that package is mapped “to a remote portion of the process
address space, an area not usually available on a <span
class="oldstyle-num">10/50</span>
system”.</p><p>TENEX uses balanced set scheduling to reduce
pagefaults. A balanced set is a set of highest priority processes whose
total working set fits in memory. And the working set of a process is the
set of pages this process reference.<br/>Guess what is an
“executive command language interpreter”? They descried it as “...which
provides direct access to a large variety of small, commonly used system
functions, and access to and control over all other subsystems and user
programs”. It’s a shell!</p><p>Some other interesting facts:
TENEX supports at most 5 levels in file paths; the paper mentions file
extensions; files in TENEX are versioned, a new version is created every
time you write to a file, old versions are automatically garbage
collected by the system over time; TENEX has five access rights:
directory listing, read, write, execute, and append; TENEX also has a
debugger residing in the core memory alongside the
kernel.</p><p>The file operations is the same as in UNIX,
opening a file gives you a file descriptor, called JFN (job file number),
and you can read or write the file. The effect of the write is seen
immediately by readers (so I guess no caching or buffering). They even
have “unthawed access”, meaning only one writer is allowed while multiple
reader can read from the file at the same time. UNIX really cut a lot of
corners, didn’t
it?</p><p><details><p><summary>Their
conclusion section is also
interesting…</summary></p><blockquote><p>One of
the most valuable results of our work was the knowledge we gained of how
to organize a hardware/software project of this size. Virtually all of
the work on TENEX from initial inception to a usable system was done over
a two year period. There were a total of six people principally involved
in the design and implementation. An <span
class="oldstyle-num">18</span> month part-time study, hardware
design and implementation culminated in a series of documents which
describe in considerable detail each of the important modules of the
system. These documents were carefully and closely followed during the
actual coding of the system. The first state of coding was completed in
<span class="oldstyle-num">6</span> months; at this point the
system was operating and capable of sustaining use by nonsystem users for
work on their individual projects. The key design document, the JSYS
Manual (extended machine code), was kept updated by a person who devoted
full time to insuring its consistency and coherence; and in retrospect,
it is out judgment that this contributed significantly to the overall
integrity of the system.</p><p>We felt it was extremely
important to optimize the size of the tasks and the number of people
working on the project. We felt that too many people working on a
particular task or too great an overlap of people on separate tasks would
result in serious inefficiency. Therefore, tasks given to each person
were as large as could reasonably be handled by that person, and insofar
as possible, tasks were independent or related in ways that were well
defined and documented. We believe that this procedure was a major factor
in the demonstrated integrity of the system as well as in the speed with
which it was
implemented.</p></blockquote></details></p><h2
id="MULTICS"
class="section">MULTICS</h2><p><em>Protection and
the Control of Information Sharing in Multics</em>, <span
class="oldstyle-num">1974</span>.</p><p>The almighty
MULTICS, running on the equally powerful Honeywell <span
class="oldstyle-num">6180</span>. There are multiple papers on
MULTIC, and this one is about its protection
system.</p><p>Their design principles
are</p><ol><li>Permission rather than exclusion (ie,
default is no permission)</li><li><a id="footref:selinux"
class="footref-anchor obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Aselinux">Check every access to every object<sup
class="inline-footref">2</sup></a></li><li>The
design is not secret (ie, security not by
obscurity)</li><li>Principle of least
privilege</li><li>Easy to use and understand (human
interface) is important to reduce human
mistakes</li></ol><div id="footdef:selinux"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Aselinux">2</a></div><div
class="def-footdef">Early-day Linux doesn’t do this, which led to
SELinux. It has merged into main Linux long
ago.</div></div><p>MULTICS has a concept of
<em>descriptor segments</em>. The virtual memory is made of
segments, and each segment has a descriptor, which contains
access-control information: access right, protection domain, etc. This
way, MULTICS can access-control memory. The access check are done by
hardware for performance. (Which means MULTICS depends on the hardware
and isn’t portable like UNIX).</p><p>MULTIC uses an regular
ACL for file-access-control. When opening a file, the kernel checks for
access rights, creates a segment descriptor, and maps the whole file into
virtual memory as a segment. In the paper, the ACL is described as the
first level access-control, and the hardware-based access-control the
second. Note that in MULTICS, you can’t read a file as a stream: the
whole file is mmaped into memory, essentially.</p><p>MULTICS
also has <em>protected subsystems</em>. It’s a collection of
procedure and data that can only be used through designated entry points
called “gates” (think of an API). To me, it’s like modules
(public/private functions and variables) in programming languages, but in
an OS. All subsystems are put in a hierarchy: Every subsystem within a
process gets a number, lower-numbered subsystems can use descriptors
containing higher-numbered subsystems. And the protection is guaranteed
by the hardware. They call it “rings of
protection”.</p><p>Speaking of rings, <span
class="oldstyle-num">x86</span> supports four ring levels, this
is how kernel protects itself from userspace programs. Traditionally
userspace is on ring <span class="oldstyle-num">3</span> and
kernel is on ring <span class="oldstyle-num">0</span>.
Nowadays with virtual machines, the guest OS is put on ring
1.</p><h2 id="Protection"
class="section">Protection</h2><p><em>Protection</em>,
<span
class="oldstyle-num">1974</span>.</p><p>This paper
by Butler Lampson gave an overview of protection in systems, and
introduces a couple useful concepts.</p><p>A
<em>protection domain</em> is anything that has certain
rights to do something and has some protection from other things, eg,
kernel or userspace, a process, a user. A lot of words are used to
describe it: protection context, environment, state, sphere, capability
list, ring, domain. Then there are <em>objects</em>, things
needs to be protected. Domains themselves can be
objects.</p><p>The relationship between domains and objects
form a matrix, the <em>access matrix</em>. Each relationship
between a domain and an object can be a list of <em>access
attributes</em>, like owner, control, call, read, write,
etc.</p><p>When implementing the access matrix, the system
might want to attach the list of accessible object of a domain to that
domain. Each element of this list is essentially a
capability.</p><p>Alternatively, the system can attach a list
of domains that can access an object to that object. An object would have
a procedure that takes a domain’s name as input and returns its access
rights to this object. The domain’s name shouldn’t be forge-able. One
idea is to use capability as the domain identifier: a domain would ask
the supervisor (kernel) for an identifier (so it can’t be forged), and
pass it to objects’ access-control procedure. An arbitrary procedure is
often an overkill, and an <em>access lock list</em> is used
instead.</p><p>Many system use a hybrid implementation in
which a domain first access an object by access-key to obtain a
capability, which is used for subsequent access. (Eg, opening a file and
geting a file descriptor.)</p><h2 id="UNIX"
class="section">UNIX</h2><p><em>The UNIX
Time-Sharing System</em>, <span
class="oldstyle-num">1974</span>.</p><p>The good ol’
UNIX! This paper describe the “modern” UNIX written in C, running on
<span
class="oldstyle-num">PDP-11</span>.</p><p>Comparing
to systems like TENEX and MULTICS, UNIX has a simpler design and does not
require special hardware supports, since it has always been designed for
rather limited machines, and for its creators’ own use
only.</p><p>The paper spends major portions describing the
file system, something we tend to take for granted from an operating
system and <a id="footref:filesystem" class="footref-anchor
obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Afilesystem">view as swappable nowadays<sup
class="inline-footref">3</sup></a>. We are all too
familiar with “everything as a file”. UNIX treats files as a linear
sequence of bytes, but that’s not the only possible way. IBM filesystems
has the notion of “records” <a id="footref:fs-database"
class="footref-anchor obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Afs-database">like in a database<sup
class="inline-footref">4</sup></a>. And on MULTICS, as
we’ve seen, the whole file is <a id="footref:fs-mmap"
class="footref-anchor obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Afs-mmap">mmaped to the memory<sup
class="inline-footref">5</sup></a>.</p><div
id="footdef:filesystem" class="footdef"><div class="def-footref
obviously-a-link"><a aria-label="Jump back to main text"
href="#footref%3Afilesystem">3</a></div><div
class="def-footdef">Because most filesystems we use expose the same
interface, namely the POSIX standard. They all have read, write, open,
close, seek, makedir, etc. I wish in the future we can plug in custom
filesystems to the OS and expose new interfaces for programs to use. For
example, a network filesystem that can tell the program “I’m downloading
this file from remote, the progress is xx%”. Right now network
filesystems can only choose between blocking and immediately error
out.</div></div><div id="footdef:fs-database"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Afs-database">4</a></div><div
class="def-footdef">As every idea in CS, this might be coming back in
another form. For example, Android uses (modified) SQLite for its
filesystem.</div></div><div id="footdef:fs-mmap"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Afs-mmap">5</a></div><div
class="def-footdef">Again, this might be coming back, in the form of
persistent memory.</div></div><p>UNIX uses mounting to
integrate multiple devices into a single namespace. On the other hand, MS
DOS uses filenames to represent devices.</p><p>This version
of UNIX only has seven protections bits, one of which switches
set-user-id, so there is no permission for “group”. set-user-id is
essentially effective user id (euid).</p><p>The paper talked
about the shell in detail, for example the <code>| <
> ; &</code> operators. Judging from the example,
the <code><</code> and
<code>></code> are clearly intended to be prefixes
rather than operators (that was one of the mysteries for me before
reading this paper):</p><pre class="code-block">ls
>temp1 pr -2 <temp1 >temp2 opr
<temp2</pre><h2 id="Plan%209" class="section">Plan
9</h2><p><em>Plan 9 From Bell Labs</em>, <span
class="oldstyle-num">1995</span>.</p><p>According to
the paper, by the mid <span
class="oldstyle-num">1980</span>’s, people have moved away from
centralized, powerful timesharing systems (on mainframes and
mini-computers) to small personal micro-computers. But a network of
machines have difficulty serving as seamlessly as the old timesharing
system. They want to build a system that feels like the old timesharing
system, but is made of a bunch of micro-computers. Instead of having a
single powerful computer that does everything, they will have individual
micro-computers for each task: a computing (CPU) server, a file server,
routers, terminals, etc.</p><p>The central idea is to expose
every service as files. Each user can compose their own private
namespace, mapping files, devices, and services (as files) into a single
hierarchy. Finally, all communication are made through a single protocol,
<span class="oldstyle-num">9P</span>. Compare that to what we
have now, where the interface is essentially C ABI plus web API, it
certainly sounds nice. But on the other hand, using text stream as the
sole interface for everything feels a bit shaky.</p><p>Their
file server has an interesting storage called WORM (write-once, read
many), it’s basically a time machine. Everyday at <span
class="oldstyle-num">5</span> AM, a snapshot of all the disks is
taken and put into the WORM storage. People can get back old versions of
their files by simply reading the WORM storage. Nowadays WORM snapshot is
often used to defend against ransom attacks. </p><h2 id="Medusa"
class="section">Medusa</h2><p><em>Medusa: An
Experiment in Distributed Operating Systems Structure</em>,
<span class="oldstyle-num">1980</span>.</p><p>A
distributed system made at CMU, to closely match and maximally exploit
its hardware: the distributed-processor Cm* system (Computer
Modules).</p><p>On a distributed processor hardware, they can
place the kernel code in memory in three
ways:</p><ol><li>Replicate the kernel on every
node</li><li>Kernel code on one node, other nodes’ processors
execute code remotely</li><li>Split the kernel onto multiple
nodes</li></ol><p>They chose the third approach: divide
the kernel into <em>utilities</em> (kernel module) and
distribute them among all the processors. When a running program needs to
invoke a certain utility (basically some syscall provided by some kernel
module), it migrates to the processor that has that utility. Different
processors can have the same utility, so programs don’t have to fight for
a single popular utility.</p><p>The design is primarily
influenced by efficiency given their particular hardware, not structural
purity, but some nice structure properties nonetheless arised. Boundaries
between utilities are rigidly enforced, since each utility can only send
messages to each other and can’t modify other’s memory. This improves
security and robustness. For example, error in one utility won’t affect
other utilities.</p><p>One problem that might occur when you
split the kernel into modules is circular dependency and deadlocks. If
the filesystem utility calls into the memory manager utility (eg, get a
buffer), and the memory manager utility calls into the filesystem utility
(eg, swap pages), you have a circular dependency. Mix in locks and you
might get a deadlock.</p><p>To be deadlock-free, Medusa
further divides each utility into <em>service classes</em>
such that service classes don’t have circular dependencies between each
other. It also makes sure each utility use separate and statistically
allocated resources.</p><p>Programs written to run on Medusa
are mostly concurrent in nature. Instead of conventional processes,
program execution are carried out by <em>task forces</em>,
which is a collection of <em>activities</em>. Each activity
is like a thread but runs on different
processors.</p><p>Activities access kernel objects (resources
like memory page, pipe, file, etc) through descriptors. Each activity has
a <em>private descriptor list</em> (PDL), and all activities
in a task force share a <em>shared descriptor list</em>
(SDL). There are also <em>utility descriptor list</em> (UDL)
for utility entry points (syscalls), and <em>external descriptor
list</em> (XDL) referencing remote UDL and PDL. Both UDL and XDL
are processor-specific.</p><p>The task force notion is useful
for scheduling: Medusa schedules activities that are in the same task
force to run in the same time. It’s often referred to as <em>gang
scheduling</em> or <em>coscheduling</em>, where you
schedule inter-communicating processes to run together, just like working
sets in paging. In addition, Medusa does not schedule out an activity
immediately when it starts waiting, and instead spin-waits for a short
while (<em>pause time</em>), in the hope that the wait is
short (shorter than context switch).</p><p>Utilities store
information for an activity alongside the activity, instead of storing it
on the utility’s processor. This way if an utilities fails, another
utility can come in, read the information, and carry on the work. The
utility <em>seals</em> the information stored with the
activity, so user programs can’t muddle with it. Only other utilities can
unseal and use that information. Implementation wise, unsealing means
mapping the kernel object into the XDL of the processor running the
utility; sealing it means removing it from the
XDL.</p><p>Medusa’s kernel also provide some handy utilities
like the exception reporter and a debugger/tracer. When an exception
occurs, the kernel on the processor sends exception data to the reporter,
which sends that information to other activities (<em>buddy
activity</em>) to handle. And you can use the debugger/tracer to
online-debug programs. <a id="footref:common-lisp"
class="footref-anchor obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Acommon-lisp">Must be nice if the kernel drops you
into a debugger when your program segfaults, no?<sup
class="inline-footref">6</sup></a> I feel that Ken
Thompson being too good a programmer negatively impacted the capability
of computing devices we have today. If he wasn’t that good, perhaps they
would add a kernel debugger in UNIX ;-)</p><div
id="footdef:common-lisp" class="footdef"><div class="def-footref
obviously-a-link"><a aria-label="Jump back to main text"
href="#footref%3Acommon-lisp">6</a></div><div
class="def-footdef">Common Lisp can do that, just
sayin.</div></div><h2 id="Pilot"
class="section">Pilot</h2><p><em>Pilot: An Operating
System for a Personal Computer</em>, <span
class="oldstyle-num">1980</span>.</p><p>A system
developed by Xerox PARC on their personal work stations. Since it is
intended for personal computing, they made some interesting design
choices. The kernel doesn’t worry about fairness in allocating resources,
and can take advices from userspace. For example, userspace programs can
mark some process as high priority for <a
id="footref:pilot-scheduling" class="footref-anchor obviously-a-link"
aria-label="Jump to footnote"
href="#footdef%3Apilot-scheduling">scheduling<sup
class="inline-footref">7</sup></a>, or pin some pages in
the memory so it’s never swapped out. (These are just examples, I don’t
know for sure if you can do these things in Pilot.)</p><div
id="footdef:pilot-scheduling" class="footdef"><div
class="def-footref obviously-a-link"><a aria-label="Jump back to
main text"
href="#footref%3Apilot-scheduling">7</a></div><div
class="def-footdef">Recently we start to see big/small cores in Apple
M1 and Intel 12th gen, and “quality of service” in
macOS.</div></div><p>Pilot uses the same language,
Mesa, for operating system and user programs. In result, the OS and user
programs are tightly coupled.</p><p>Pilot provides defense
(against errors) but not <a id="footref:absolute-protection"
class="footref-anchor obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Aabsolute-protection">absolute protection<sup
class="inline-footref">8</sup></a>. And protection is
language-based, provided by (and only by) type-checking in
Mesa.</p><div id="footdef:absolute-protection"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Aabsolute-protection">8</a></div><div
class="def-footdef">This is before Internet, and malicious program
isn’t a thing yet, I think?</div></div><p>Lastly, Pilot
has integrated support for networks. It is designed to be used in a
network (of Pilots). In fact, the first distributed email system is
created on Pilot.</p><p>The device on which Pilot runs is
also worth noting. ’Twas a powerful machine, with high-resolution bitmap
display, keyboard, and a “pointing device”. Xerox PARC basically invented
personal computer, plus GUI and mouse.</p><p>The filesystem
is flat (no directory hierarchy), though higher level software are free
to implement additional structure. Files are accessed through mapping its
pages (blocks) into virtual memory. Files and volumes (devices) are named
by a 64-bit unique id (uid), which means files created anywhere anytime
can be uniquely identified across different machines (and thus across the
network). They used a classic trick, unique serial number plus real-time
clock, to guarantee uniqueness.</p><p>A file can be marked
immutable. An immutable file can’t be modified ever again, and can be
shared across machines without changing its uid. This is useful for, eg,
sharing programs.</p><h2 id="Monitor"
class="section">Monitor</h2><p><em>Monitors: An
Operating System Structuring Concept</em>, <span
class="oldstyle-num">1974</span>, by C. A. R.
Hoare.</p><p><em>Experience with Processes and Monitors
in Mesa</em>, <span
class="oldstyle-num">1980</span>.</p><p><em>Monitor</em>
is a synchronization concept. Think of it as a class that manages some
resource and synchronizes automatically. In C, you would manually create
a mutex and lock/unlock it; in Java, you just add some keyword in front
of a variable and the runtime creates and manages the lock for you—that’s
a monitor.</p><p>The Hoare paper introduced the concept and
gave a bunch of examples. The Mesa paper describes how did they implement
and use monitors in Mesa. If you recall, Mesa is the system and
application language for Pilot.</p><p>Pilot uses monitors
provided by Mesa to implement synchronization in the kernel, another
example of the tight coupling of Pilot and Mesa.</p><p>I have
some notes on the differences between Mesa’s monitors and Hoare’s
monitors, but they aren’t very interesting. Basically Mesa folks needed
to figure out a lot of details for using monitors for Pilot, like nested
wait, creating monitor, handling exceptions in monitor, scheduling, class
level vs instance level, etc.</p><p>Pilot didn’t use mutual
monitors between devices. If two devices with orders of magnitude
difference in processing speed shares a monitor, the fast device could be
slowed down by waiting for the slower device to finish its critical
section.</p><h2 id="V%20Kernel" class="section">V
Kernel</h2><p><em>The Distributed V Kernel and its
Performance for Diskless Workstations</em>, <span
class="oldstyle-num">1983</span>.</p><p>Back in the
day, professors and their grad students work together to build an awesome
and cutting-edge system, and journals invite them to write down their
thoughts and experiences. Papers we’ve read up to this point are mostly
describing the system the authors built, and sharing their experiences
and lessons learned.</p><p>This paper is a bit different—it
presents performance measurements and use it to argue a claim. You see,
the conventional approach to build a distributed workstation is to use a
small local disk for caching, and these systems usually use specialized
protocols. This papar tries to build a distributed workstation without
local disks (diskless) and only use generic message-based IPC. The
authors argue that the overhead added by this two decisions are
ok.</p><p>The paper introduced V message. It’s synchronous
(request and response), has a small message size (<span
class="oldstyle-num">32</span> bytes), and has separate control
data messages. Though they also have a “control+data message”
(<code>ReplyWithSegment</code>), presumably to squeeze out
some performance.</p><p>They used various measures to reduce
the overhead. They put everything into the kernel, including the file
server. They didn’t use TCP but used Ethernet frames directly. There is
no separate ACK message, instead ACK is implied by a
response.</p><p>The paper analyzed what network penalty
consists of. When you send a message from one host to another, it goes
from RAM to the network interface, then it’s transferred on wire to the
destination interface, then it’s copied into RAM. Their argument is that
message layer doesn’t add much overhead comparing to the base network
penalty—copying between RAM and network interface, and waiting in the
interface before going onto the wire. They also argued that remote file
access adds little overhead comparing to already-slow disk
access.</p><p>Overall, their argument do have some cracks.
For example, they argue that there is no need for specialized message
protocol, but their protocol ends up specializing. They also argued that
no streaming is needed, but large data packet are effectively
streaming.</p><h2 id="Sprite"
class="section">Sprite</h2><p><em>The Sprite Network
Operating System</em>, <span
class="oldstyle-num">1988</span>.</p><p>Sprite is
another distributed system. It tries to use large memory cache to improve
file access; and do it transparently, giving the user the illusion of a
local system. It also has a very cool process migration feature. Sadly,
process migration never caught up in the
industry.</p><p>Several trends at the time influenced
Sprite’s design. Distributed system was popular (at least in academia);
memories are getting larger and larger; and more and more systems are
featuring multiple processors.</p><p>To present the illusion
of a local file system, Sprite uses <em>prefix tables</em>.
Here, prefix means path prefix. When the userspace access a file, the
kernel looks for a prefix of the path that’s in the prefix table. In the
prefix table, the prefix can either point to the local filesystem or a
remote filesystem. If it points to a remote filesystem, the kernel makes
RPC calls to the remote host, which then access the local filesystem of
that remote host.</p><p>Prefix table isn’t only useful for
distributed system. In general, OS that uses file paths usually cache the
file paths it reads in a prefix table, because resolving a file path is
very slow. When the OS resolves a file path, it needs to read each
directory in the path to find the next directory.</p><p>With
cache, the biggest issue is consistency: if two clients get a file and
stored it in their cache, and both write to their cache, you have a
problem. Sprite’s solution is to allow only one writer at a time and
track the current writer of every file. When a client needs to read a
file, it finds the current writer and requests the file from it. This is
<em>sequential write-sharing</em>.</p><p>If
multiple clients needs to write the same file (<em>concurrent
write-sharing</em>), Sprite just turns off caching. This is rare
enough to not worth complicating the system. (And you probably need a
substantially more complicated system to handle this.)</p><h2
id="Grapevine"
class="section">Grapevine</h2><p><em>Experience with
Grapevine: The Growth of a Distributed System</em>, <span
class="oldstyle-num">1984</span>.</p><p>A classic
paper in distributed systems, even considered the MULTICS of distributed
systems by some. Grapevine is a distributed email delivery and management
system; it provides message delivery, naming, authentication, resource
location, access control—you name it.</p><p>The main takeaway
is the experience they got from running Grapevine. To support scaling,
the cost of any computation/operation should not grow as the size of the
system grows. But on the other hand, sometimes you can afford to have
complete information—maybe that information can never get too large,
regardless of how large the system grows.</p><p>Grapevine
generally tries to hide the distributed nature of the system, but that
caused some problem. First of all, they can’t really hide everything:
update in the sytem takes time to propagate, and sometimes users get
duplicated messages, all of which are confusing for someone accustomed to
the mail service on time-sharing systems.</p><p>More
importantly, user sometimes needs to know more information of the
underlying system to understand what’s going on: when stuff doesn’t work,
people want to know why. For example, removing an inbox is an expensive
operation and removing a lot of them in the same time could overload the
system. System administrators needs to understand this, and to understand
this they need to understand roughly how the system works under the
hood.</p><p>The lesson is, complete transparency is usually
not possible, and often not what you want anyway. When you design a
system, it is important to decide what to make transparent and what not
to.</p><p>Finally, the paper mentioned some considerations
about managing the system. Maintaining a geographically dispersed system
involves on-site operators and system experts. On-site operators carry
out operations on-site, but has little to no understanding of the
underlying system. System experts has deep understanding of the system,
but are in short supply and are almost always remote from the servers
they need to work on. Grapevine has remote monitoring and debugging
features to help an expert to diagnose and repair a server
remotely.</p><figure><img alt="The system structure of
Grapevine."
src="https://archive.casouri.cc/note/2024/cse221/grapevine.jpg"/>
<figcaption>The system structure of
Grapevine.</figcaption></figure><h2 id="Global%20memory"
class="section">Global
memory</h2><p><em>Implementing Global Memory Management
in a Workstation Cluster</em>, <span
class="oldstyle-num">1995</span>.</p><p>This paper
is purely academic, but pretty cool nonetheless. They built a cluster
that shares physical memory at a very low level, below VM, paging,
file-mapping, etc. This allows the system to utilize the physical memory
much better and allows more file-caching. More file caches is nice
because CPU was becoming much faster than the
disk.</p><p>Each node in the cluster divides their memory
into local memory and global memory. Local memory stores pages requested
by local processes; global memory stores pages in behave of other nodes
in the cluster.</p><p>When a fault occurs on a node P, one of
four things could happen.</p><ol><li>If the requested
page is in the global memory of another node Q, P uses a random page in
its global memory to trade the desired page with Q. (See illustration 1.)
</li><li>If the requested page is in the global memory of
another node Q, but P doesn’t have any page in its global memory, P use
the least-recently used (LRU) local page to trade with
Q.</li><li>If the requested page is on local disk, P reads it
into its local memory, and evict the oldest page in the <em>entire
cluster</em> to make room for the new page. If the oldest page is
on P, evict that; if the oldest page is on a node Q, evict the page on Q,
and send a page of P to Q. This page is either a random global page on P,
or the LRU local page of P if P has no global pages. (See illustration
2.)</li><li>If the requested page is a local page of another
node Q, duplicate that page into the local memory of P, and evict the
oldest page in the entire cluster. Again, if the oldest page is on
another node R, send one of P’s global pages or P’s LRU page to trade
with R.</li></ol><figure><img alt="Illustration of
page exchange in case 1."
src="https://archive.casouri.cc/note/2024/cse221/global-memory-1.jpg"/>
<figcaption>Illustration 1: Page exchange in case
1.</figcaption></figure><figure><img
alt="Illustration of page exchange in case 3."
src="https://archive.casouri.cc/note/2024/cse221/global-memory-2.jpg"/>
<figcaption>Illustartion 2: Page exchange in case
3.</figcaption></figure><p>This whole dance can improve
performance of memory-intensive tasks because fetching a page from remote
memory is about two to ten times faster than disk access. However, local
hit is over three magnitudes faster than fetching remote memory, so the
algorithm has to be very careful not to evict the wrong
page.</p><p>The description above omits a crucial problem:
how does memory management code running on each node know which page is
the oldest page in the entire cluster?</p><p>Consider the
naive solution, where the system is managed by a single entity, a central
controller. The controller keeps track of every single page’s age and
tells each node which node to evict. Of course, this is impossible
because that’s way too slow, the controller has to be running at a much
faster speed than the other nodes and the communication speed between
nodes must be very fast.</p><p>Instead, each node must make
local independent decisions that combines to achieve a global goal (evict
the oldest page). The difficulty is that local nodes usually don’t have
complete, up-to-date information.</p><p>A beautiful approach
to this kind of problem is probability-based algorithm. We don’t aim to
make the optimal decision for every single case, but use probability to
approximate the optimal outcome.</p><p>We divide time into
epochs, in each epoch, the cluster expects to replace
<em>m</em> oldest pages. (<em>m</em> is predicted
from date from previous epochs.) At the beginning of each epoch, every
node sends a summary of its pages and their age to an <em>initiator
node</em> (central controller). The initiator node sorts all the
pages by their age, and finds the set of <em>m</em> oldest
pages in the cluster (call it <em>W</em>). Then, it assigns
each node <em>i</em> a weight
<em>w<sub>i</sub></em>, where
<em>w<sub>i</sub></em>
is</p><p><img alt="A math expression: the number of old
pages in W that are in node i, divided by W."
src="https://archive.casouri.cc/note/2024/cse221/global-memory-frac.png"/></p><p>Basically,
<em>w<sub>i</sub></em> means “among the
<em>M</em> oldest pages in the cluster, how many of them are
in node <em>i</em>”.</p><p>The initiator node
tells each node of every node’s weight, and when a node P encounters case
3 or 4 and wants to evict “the oldest page in the cluster”, it randomly
picks a node by each node’s weight, and tells that node to evict its
oldest page.</p><p>That takes care of finding which node to
evict pages from, but tracking page age isn’t easy either. For one, in a
mmaped file, memory access bypasses pagefault handler and goes straight
to the TLB. More importantly, the OS uses FIFO second-chance page caching
and hides many page request/eviction from their memory manager, because
the memory manager runs at a lower level (presumably in pagefault
handlers).</p><p>The authors resorted to hacking the TLB
handler of the machine with PALcode (microcode). This would’ve been
impossible on x86—it’s TLB is handled purely in
hardware.</p><p>Probability-based algorithms sometimes feels
outright magical—they seem to just bypass trade-offs. In reality, they
usually just add a new dimension to the trade-off. We’ll see this again
later in lottery scheduling.</p><h2 id="%CE%BC-kernel"
class="section">μ-kernel</h2><p><em>The Performance
of μ-Kernel-Based Systems</em>, <span
class="oldstyle-num">1997</span>.</p><p>This paper
is another measurement paper. It uses benchmarks to argue that a) micro
kernel can deliver comparable performance, and b) the performance doesn’t
depend on a particular hardware architecture.</p><p>The
authors built a micro kernel L4, and ported Linux to run on it (called
L⁴Linux). Then they ported L4 itself from Pentium to both Alpha and MIPS
architecture—to show that L4 is architecture-independent. They also
conducted some experiment to show L4’s extensibility and
performance.</p><p>The paper considers micro kernels like
Mach and Chrous to be first-generation, evolved out of earlier monolithic
kernels. It considers later kernels like L4 and QNX to be
second-generation, for that they are designed more rigorously from
scratch, ie, more “pure”.</p><p>L4 allows user programs to
control memory allocation like nucleus did: kernel manages top-level
tasks’ memory, top-level tasks manages their children’s memory. And
scheduling? Hard priorities with round-robin scheduling per priority, not
unlike nucleus.</p><p>L⁴Linux only modifies the
architecture-dependent part of Linux, meaning they didn’t have to modify
Linux. The authors also restricted themselves to not make any
Linux-specific change to L4, as a test for the design of L4. The result
is not bad: in micro benchmarks, L⁴Linux is ×<span
class="oldstyle-num">2.4</span> times slower than native Linux;
in macro benchmarks, L⁴Linux is about <span
class="oldstyle-num">5–10%</span> slower than native Linux. More
over, L⁴Linux is much faster than running Linux on top of other micro
kernels, like MkLinux (Linux + Mach 3.0).</p><p>The paper
also mentions supporting tagged TLBs. Normal TLB needs to be flashed on
context switch, which is a big reason why context switch is expensive.
But if you tag each entry in the TLB with a tag to associate that entry
with a specific process, you wouldn’t need to flush TLB anymore. The
downside is that, tagged TLB needs some form of software-managed TLB, so
not all architecture can support it. For example, x86 doesn’t support
software-managed TLB.</p><p>The benefit of micro kernels is
of course the extensibility. For example, when a page is swapped out,
instead of writing to disk, we can swap to a remote machine, or encrypt
the page and write to disk, or compress the page and write to page. A
database program could bypass the filesystem and file cache, and control
the layout of data on physical disk for optimization; it can control
caching and keep pages in memory and not swapped
out.</p><p>All of these are very nice perks, and the
performance doesn’t seem too bad, then why micro kernels never caught on?
Here’s our professor’s take: big companies can just hire kernel
developers to <a id="footref:mu-kernel-linux" class="footref-anchor
obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Amu-kernel-linux">modify Linux to their need<sup
class="inline-footref">9</sup></a>; smaller companies
don’t have special requirements and can just use Linux. That leaves only
the companies in the middle: have special requirements, but don’t want to
modify Linux. (Professor’s take ends here.) However, extending micro
kernel is still work, it might be easier than modifying Linux, but how
much easier? Plus, if there are a lot of Linux kernel developers, perhaps
modifying Linux is more easier afterall.</p><div
id="footdef:mu-kernel-linux" class="footdef"><div
class="def-footref obviously-a-link"><a aria-label="Jump back to
main text"
href="#footref%3Amu-kernel-linux">9</a></div><div
class="def-footdef">And they did. Since the paper has been written,
Linux has gained many features of L4 described in the
paper.</div></div><p>If we look at “we need a custom
OS” scenario today, Nintendo Switch and Playstation use modified BSD,
Steam Deck is built on top of Linux. And I’m sure most data centers run
some form of Linux.</p><p>Beyond monolithic and microkernel,
there are many other kernel designs: hybrid, exokernel, even virtual
machines. Hybrid kernels include Windows NT, NetWave, BeOS, etc. Hybrid
kernel leaves some modules in the kernel, like IPC, driver, VM,
scheduling, and put others in the userspace, like
filesystem.</p><h2 id="Exokernel"
class="section">Exokernel</h2><p><em>Exokernel: An
Operating System Architecture for Application-Level Resource
Management</em>, <span
class="oldstyle-num">1997</span>.</p><p>The idea is
to go one step further than microkernels and turn the kernel into a
library. Kernel exposes hardware resources, provide multiplexing and
protection, but leaves management to the application. The motivation is
that traditional kernel abstraction hides key information and obstructs
application-specific optimizations.</p><p>This idea can be
nicely applied to single-purpose applicants, when the whole purpose of a
machine is to run a single application, eg, a database, a web server, or
an embedded program. In this case, things that a traditional kernel
provides like users, permissions, fairness, are all unnecessary overhead.
(<a
href="https://archive.casouri.cc/note/2024/cse221/https:/dl.acm.org/doi/10.1145/2490301.2451167">Unikernel</a>
explored exactly this use-case.)</p><p>Exokernel exports
hardware resources and protection, and leaves management to the
(untrusted) application. Applications can request for resources and
handle events. Each application cooperatively share the limited resources
by participating in a <em>resource revocation</em> protocol.
Eg, the exokernel might tell an application to release some resources for
others to use. Finally, the exokernel can forcibly retract resources held
by uncooperative applications by the <em>abort
protocol</em>.</p><p>Exokenel doesn’t provide many of
the traditional abstractions, like VM or IPC, those are left for the
application to implement.</p><p>The protection provided by an
exokernel is inevitably weaker: an application error could corrupt
on-disk data; and because the kernel and application runs in the same VM,
application error could corrupt kernel memory!</p><p>The
existence of abort protocol kind of breaks the “no management”
principle—retracting resources from an application
<em>is</em> management.</p><p>Finally, their
benchmark isn’t very convincing: there are only micro benchmarks and no
macro benchmark; they only benchmarked mechanism (context switch,
exception handler, etc) and has no benchmark for
application.</p><h2 id="Xen"
class="section">Xen</h2><p><em>Xen and the Art of
Virtualization</em>, <span
class="oldstyle-num">2003</span>.</p><p>Xen is a
virtual machine monitor (VMM), also called hypervisor—the thing that sits
between an OS and the hardware. The goal of Xen is to be able to run
hundreds of guest OS’s in the same time.</p><p>Xen provides a
virtual machine abstraction (<em>paravirtualization</em>)
rather than a full virtual hardware (<em>full
virtualization</em>). Paravirtualization has better performance and
gives the VMM more control, but requires modification to the guest OS. On
the other hand, full virtualization VMM, for example VMWare, can work
with unmodified guest OS.</p><p>Nowadays there are a plethora
of virtual machine solutions, like VMWare, Hyper-V, VirtualBox, KVM, Xen.
On top of that, there are containers like LXC, docker, etc. The whole
stack contains OS, VMM/container engine, guest OS, and guest app. These
solutions all have different configurations: The VMM can sit on the host
OS or directly on the hardware; you can run one guest OS per app, or run
a single guest OS for multiple apps; on the old IBM and VMS systems, the
VMM supports both a batch processing OS and an interactive
OS.</p><p>Let’s look at how does Xen virtualize and how does
it compare to VMWare.</p><p>Scheduling virtualization: Xen
uses the Borrowed Virtual Time (BVT) algorithm. This algorithm allows a
guest OS to borrow future execution time to respond to latency-critical
tasks.</p><p>Instructions virtualization: Boring instructions
like <code>add</code> can just pass-through to the hardware,
but privileged instructions (like memory access) needs intervention from
the monitor.</p><p>In Xen, the guest OS is modified so that
it is aware of the VMM, and instead of doing privileged task by itself,
the guest OS delegates the work to the VMM by
<em>hypercalls</em>. In VMWare, since they can’t modify the
guest OS, privileged instructions simply trap into VMM. If you remember,
we talked about rings in the MULTICS section. On <span
class="oldstyle-num">x86</span>, The CPU will trap if it’s asked
to execute a privileged instruction when in a low ring
level.</p><p>Memory virtualization: The guest OS isn’t
managing physical memory anymore, though we still call it physical
memory. VMM has real access to the phyiscal memory, often called machine
memory.</p><p>Then, how is the virtual memory address in the
guest OS translated into machine memory address?</p><p>In
Xen, the guest OS is aware of the virtualization. It’s page table can map
directly from virtual address to machine address, and MMU can just read
off of guest OS’s page table. The VMM just need to verify writes to the
page table to enforce protection.</p><p>In VMWare, however,
the guest OS is unaware of the VMM, and its page table maps from virtual
address to physical address. Also, the guest OS writes to its page table
without bothering to notify anyone. VMM maintains a shadow page table
that maps virtual address to actual machine address. It also uses dirty
bits to make sure whenever the guest OS writs to the page table, it is
notified and can update its shadow page table accordingly. (I forgot
exactly how.) And MMU reads off the shadow page table. (Presumably by
trapping to VMM when the guest OS tries to modify the <span
class="oldstyle-num">CR3</span> register, and let VMM override
<span class="oldstyle-num">CR3</span> to its shadow page
table?)</p><figure><img alt="Diagram illustrating Xen and
VMWare’s memory remapping approach."
src="https://archive.casouri.cc/note/2024/cse221/xen.jpg"/>
<figcaption>Illustration of Xen and VMWare’s memory
virtualization.</figcaption></figure><p>Note that
VMWare needs all these complication only because <span
class="oldstyle-num">x86</span>’s memory management is
completely hardware-based—the kernel can only point the MMU to the page
table and has no other control over the MMU. Other “higher-end”
architectures usually support software-managed and tagged
TLB.</p><p>A clever trick that Xen uses is <em>balloon
driver</em>. It’s a drive whose whole purpose is to take up memory.
When the VMM wants to retract memory from the guest OS, it enlarges the
“balloon”, so the guest OS relinquishes memory to the
host.</p><h2 id="VMS"
class="section">VMS</h2><p><em>Virtual Memory
Management in VAX/VMS</em>, <span
class="oldstyle-num">1982</span>.</p><p>This paper
mainly concerns of the implementation of the virtual memory for VMS. VMS
has to run on a variety of low-end hardware with small memory and slow
CPU; it also needs to support drastically different use-cases: real time,
timeshared, and batch. These requirements all affected the design of
VMS.</p><p>VMS’s virtual memory has three regions: program
region, control region and system region. The highest two bits of an
address indicates the region, after that are the regular stuff: <span
class="oldstyle-num">20</span> bits of virtual page number and
<span class="oldstyle-num">8</span> bits of byte offset. The
system region (think of it as kernel stack) is shared by all processes;
program and control region are process-specific.</p><p>The
paper mentions a trick they used: they mark the first page in the VM as
no access, so that an uninitialized pointer (pointing to
<code>0x0</code>) causes an exception. I think Linux does the
same.</p><p>VMS uses a process-local page replacement policy.
When a process requests for memory that needs to be paged in, kernel
swaps out a page from this process’s resident set—the set of pages
currently used by that process. This way a heavily paging process can
only slow down itself.</p><p>When a page is removed from the
resident set, it doesn’t go out of the memory immediately; instead, it’s
appended to one of two lists. It goes to the free page list if it hasn’t
been modified; otherwise it goes to the modified page list. When kernel
needs a fresh page to swap data in, it takes a page from the head of the
free list. When kernel decides to write pages back to paging file (swap
file), it takes the page from the head of the modified
list.</p><p>So a page is appended to the end of the list, and
gradually moves to the head, until it’s consumed. But if the page is
requested again by the process while still in the list, it is pulled out
and put back into the the process’s resident set. This is basically
second chance caching: we keep the page in the memory for a while before
really discarding it, in case it is used again
soon.</p><p>Because VMS uses a relatively small <a
id="footref:vms-page" class="footref-anchor obviously-a-link"
aria-label="Jump to footnote" href="#footdef%3Avms-page"><span
class="oldstyle-num">512</span> byte page size<sup
class="inline-footref">10</sup></a>, pages causes a lot of
I/O, which is obviously not good. To reduce the number of disk
operations, they try to read and write several pages at once (they call
this clustering).</p><div id="footdef:vms-page"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Avms-page">10</a></div><div
class="def-footdef">To be compatible with <span
class="oldstyle-num">PDP-11</span> and because of the promise of
low-latency semiconductor disk technologies (which obviously didn’t
materialize on time).</div></div><p>The paper also
mentions some other nice features, like on-demand zeroed page, and
copy-on-reference page. On-demand zeroed page are only allocated and
zeroed when it’s actually referenced. Similarly, copy-on-reference pages
are only copied when it’s actually referenced. I wonder why didn’t they
make it copy-on-write though, they say it’s used for sharing executable
files.</p><p>Quiz time: does kernel know about every memory
access?</p><p>…The answer is no. Kernel only get to know
about memory use when there’s a pagefault, which runs the pagefault
handler provided by the kernel. If there’s no pagefault, memory access is
handled silently by the MMU.</p><h2 id="Mach"
class="section">Mach</h2><p><em>Machine-Independent
Virtual Memory Management for Paged Uniprocessor and Multiprocessor
Architectures</em>, <span
class="oldstyle-num">1987</span>.</p><p>Mach was a
popular research OS. In fact, our professor, Dr. Zhou, did her PhD on
Mach’s virtual memory. Mach actually influenced both Windows and Mac: one
of the prominent Mach researcher went to Microsoft and worked on Windows
NT, and Mac OSX was Mach plus BSD plus NextStep.</p><p>The
main topic of this paper is machine-independent VM. The idea is to treat
hardware information (machine-dependent, like TLB) as a cache of
machine-independent information.</p><p>Mach’s page table is a
sorted doubly linked list of <em>virtual regions</em>. Each
virtual region stores some machine-independent info like address range,
inheritance, protection, and some cache for the machine-dependent info.
The machine-dependent part is a cache because it can be re-constructed
from the machine-independent info. Also, since Mach uses doubly linked
list, it can support sparse addresses (VMS can’t).</p><p>Each
virtual region maps a virtual address range to a range in a
<em>memory object</em>. A memory object is an abstraction
over some data; it can be a piece of memory, secondary storage, and even
remote data, I think?</p><p>A memory object is associated
with a pager, which handles pagefault and page-out requests. This pager
is outside of the kernel and is customizable. And we can make it do
interesting things like encrypting memory, remote memory,
etc.</p><p>When performing a copy-on-write, Mac creates a
shadow memory object which only contains pages that have been modified.
Access to the unmodified page will be redirected to the original memory
object. Since shadow memory objects themselves can be shadowed,
sometimes, large chains of shadow objects will manifest. Mach has to
garbage collect intermediate shadow objects when the chain gets long.
Reading the paper, this seems to be tricky to implement and was quite an
annoyance to the designers.</p><p>When a task inherits memory
from its parent task, the parent can set the inheritance flag of any page
to either <em>shared</em> (read-write),
<em>copy</em> (copy-on-write), or <em>none</em>
(no access). To me, this would be very helpful for
sandboxing.</p><h2 id="FFS"
class="section">FFS</h2><p><em>A Fast File System
for UNIX</em>, <span
class="oldstyle-num">1984</span>.</p><p>This paper
literally describes a faster file system they implemented for UNIX. It
was widely adopted.</p><p>The author identifies a series of
shortcomings of the default file system of UNIX:</p><p>The
free list (a linked list of all free blocks) starts out ordered, but over
time becomes random, so when the file system allocates blocks for files,
those block are not physically continuous but rather scatter
around.</p><p>The inodes are stored in one place, and the
data (blocks) another. File operations (list directory, open, read,
write) involve editing meta information interleaved with writing data,
causing long seeks between the inodes and the
blocks.</p><p>The default block size of 512 bytes is too
small and creates indirection and fragmentation. Smaller block size also
means it takes more disk transactions to transfer the same amount of
data.</p><p>With all these combined, the default file system
can only produce <span class="oldstyle-num">2%</span> of the
full bandwidth.</p><p>FFS improves performance by creating
locality as much as possible. It divides a disk partition into
<em>cylinder groups</em>. Each cylinder group has its own
copy of the superblock, its own inodes, and a free list implemented with
a bitmap. This way inodes and data blocks are reasonably close to each
other. Each cylinder has a fixed number of inodes.</p><p>FFS
uses a smart allocation policy for allocating blocks for files and
directories. It tries to place inodes of files in the same directory in
the same cylinder group; it places new directories in a cylinder group
that has more free inocdes and less existing directories; it tries to
place all the data blocks of a file in the same cylinder group.
Basically, anthing that improves locality.</p><p>FFS uses a
larger block size since 512 bytes is too small. But larger block size
wastes space—most UNIX systems are composed of many small files that
would be smaller than a larger block size. FFS allows a block to be
splitted into <em>fragments</em>. A block can be broken into
2, 4, or 8 fragments. At the end, the author claims that FFS with
4096-byte blocks and 512-byte fragments has about the same disk
utilization as the old 512-byte block file system.</p><p>FFS
requires some percent of free space to maintain it’s performance. When
the disk is too full, it’s hard for FFS to keep the blocks of a file
localized. FFS performs best when there are around <span
class="oldstyle-num">10%</span> of free space. This applies to
most modern filesystems too.</p><p>To maximally optimize the
file system, FFS is parameterized so it can be tuned according to the
physical property of the disk (number of blocks on a track, spin speed),
processor speed (speed of interrupt and disk transfer),
etc.</p><p>Here’s one example of how these information could
improve performance. Two physically consecutive blocks on the disk can’t
be read consecutively, because it takes some time for the processor to
process the data after reading a block. FFS can calculate the number of
blocks to skip according to the processor speed and spin speed, such that
when the OS finished reading one block, the next block of the file comes
into position right under the disk head.</p><h2 id="LFS"
class="section">LFS</h2><p><em>The Design and
Implementation of a Log-Structured File System</em>, <span
class="oldstyle-num">1991</span>.</p><p>When this
paper came out, it stirred quote some controversy on LFS vs extent-based
FFS. Comparing to FFS, LFS has much faster writes, but it has slower read
and needs garbage collection.</p><p>The main idea is this:
since now machines have large RAMs, file cache should ensure read is
fast; so the filesystem should optimize for write speed. To optimize
write speed, we can buffer writes in the file cache and write them all at
once sequentially.</p><p>This approach solves several
shortcoming of FFS. In FFS, even though inodes are close to the data,
they are still separate and requires seeking when writing. And the same
goes for directories and files. The typical work load of the filesystem
alternates between writing metadata and data, producing a lot of separate
small writes. Further, most of the files are small, so most writes are
really writing metadata. Writing metadata is much slower than writing
files, because the filesystem has to do synchronous write for metadata,
to ensure consistency in case of unexpected failure (power outage,
etc).</p><p>On the other hand, LFS treats the whole disk as
an append-only log. When writing a file, the filssytem just appends what
it wants to write to the end of the log, followed by the new inodes
pointing to the newly written blocks, followed by the new inode map
pointing to the newly written inodes. The inode map is additionally
copied in the memory for fast access.</p><p>To read, LFS
looks into the inode map (always at the end of the log), finds the
inodes, reads the inode to find the blocks, and pieces together the parts
it wants to read.</p><p>When LFS has used the entire disk up,
how does it keep appending new blocks? LFS divides the disk into
<em>segments</em>, each consisting of a number of blocks.
Some of the blocks are still being referenced (live blocks), some are
free to be reused (free blocks). LFS will regularly perform garbage
collection and create segments that only contains free blocks—during
garbage collection, LFS copies all the live blocks in a segment to the
end of the log, then this segment becomes a free segment. Finally, when
LFS needs to write new logs, it writes them in free
segments.</p><p>The challenge of garbage collection is to
choose the best segment to clean. The authors first tried to clean least
utilized segment first, ie, clean the segment with the least amount of
live data. This didn’t go well, because segments don’t get cleaned until
they cross the threshold, and a lot of segments lingers around the
threshold, don’t get cleaned, and hold up a lot of
space.</p><p>The authors found that it’s best to categorize
segments into hot and cold segments. Hot segments are the ones that are
actively updated, where blocks are actively marked free. Cleaning hot
segments isn’t very valuable, because even if we don’t clean it, more and
more of its blocks will become free by themselves. On the other hand,
cold segments are valuable to clean, since it’s unlikely/slow to free up
blocks by itself.</p><p>The authors also mentioned some crash
recovery and checkpoint mechanism in the paper.</p><h2
id="Soft%20update" class="section">Soft
update</h2><p><em>Soft Updates: A Solution to the
Metadata Update Problem in File Systems</em>, <span
class="oldstyle-num">2000</span>.</p><p>In LFS we
mentioned that metadata edit requires synchronize writes. That’s because
you want to ensure the data on disk (or any persistent storage) is always
consistent. If the system writes only a partial of the data it wishes to
write, then crashed, the disk should be in a consistent or at least
recoverable state. For example, when adding a file to a directory, adding
the new inode must happen before adding the file entry to the
directory.</p><p>Folks has long sought to improve the
performance of updating metadata, this paper lists several existing
solutions.</p><dl><dt>Nonvolatile RAM
(NVRAM)</dt><dd>Use NVRAM to store metadata. Updating
metadata is as fast as accessing RAM, and it
persists.</dd><dt>Write-ahead logging</dt><dd>Ie,
journaling. The filesystem first log the operation it’s about to perform,
and performs it. If a crash happens, the filesystem can recover using the
log.</dd><dt>Scheduler-enforced
ordering</dt><dd>Modify disk request scheduler to enforce
synchronous edit of metadata. Meanwhile, the filesystem is free to edit
metadata asynchronously (since the disk request scheduler will take care
of it)</dd><dt>Interbuffer
dependencies</dt><dd>Use write cache, and let the cache
write-back code enforce metadata
ordering.</dd></dl><p>Soft update is similar to
“interbuffer dependencies”. It maintains a log of metadata updates, and
tracks dependencies at a fine granularity (per field or pointer), and can
move the order of operations around to avoid circular dependencies. Then
it can group some updates together and make less writes.</p><h2
id="Rio" class="section">Rio</h2><p><em>The Rio File
Cache: Surviving Operating System Crashes</em>, <span
class="oldstyle-num">1996</span>.</p><p>The main
point of Rio (RAM/IO) is to make memory survive crashes; then the OS
doesn’t have to consistently write the cache to persistent
storage.</p><p>Power outages can be solved by power supply
with battery and dumping memory to persistent storage when power outage
occurs. Alternatively, we can just use persistent memory. Then, during
reboot, the OS goes through the dumped memory file to recover data (file
cache). The authors call this “warm reboot”.</p><p>System
crash is the main challenge, because kernel crash can corrupt the memory.
The authors argue that the reason why people consider persistent storage
to be reliable and memory to be unreliable is because of their interface:
writing to disk needs drivers and explicit procedures, etc, while writing
to memory only takes a <code>mov</code>
instruction.</p><p>Then, protecting the file cache is just a
matter of write-protecting the memory. And there are a myriad of
techniques for that already. For example, you can use the protection that
virtual memory already provides. Just turn off the write-permission bits
in the page table for file cache pages. However, some systems allow
kernel to bypass virtual memory protection. The authors resorted to
disabling processor’s ability to bypass TLB. This is of course
architecture-dependent.</p><p>Another way is to install
checks for every kernel memory access, but that’s a heavy penalty on the
performance.</p><p>What’s more interesting is perhaps the
effect of having a reliable memory on the filesystem. First, you can turn
off reliable sync writes (this is the motivation for this paper in the
first place). But also, since memory is now permanent, metadata updates
must be ordered, so that a crash in the middle of an operation doesn’t
create an inconsistent state.</p><p>Nowadays, persistent
memory is getting larger and cheaper to the point that it seems possible
to use it to improve IO performance in datacenters. Problem is, every
update has to be ordered, and you can’t control L1/2/3 cache. They can
decide to write to memory at different orders than you
intended.</p><p>Currently there are two approaches: treat the
persistent memory as a super fast SSD, and slap a filesystem on it, the
filesystem will take care of the dirty work. Others don’t want to pay for
the overhead of a filesystem, and want to use it as a memory. To go this
route, the programmer have to deal with the complications of
consistency/ordering.</p><h2 id="Scheduler%20activation"
class="section">Scheduler
activation</h2><p><em>Scheduler Activations: Effective
Kernel Support for the User-level Management of Parallelism</em>,
<span
class="oldstyle-num">1991</span>.</p><p>Threading
can be implemented in either kernel or userspace. However, both have
their problems. If implemented in userspace, it has bad integration with
kernel schedular—userspace thread scheduler has no way to know when a
thread is going to run, and for how long. If implemented in kernel,
thread management now requires a context-switch into kernel, which is
very slow. Plus, like anthing else that goes into kernel, there won’t be
much customizability.</p><p>The authors present a new
abstraction as the solution—scheduler activation. The idea is to allow
more cooperation between kernel and userspace. Kernel allocates
processors, and notifies the userspace when it gives processors or takes
processors away. The userspace decides what to run on the provided
processors. Finally, the userspace can request or relinquish
processors.</p><p>This way we get the best of both worlds:
userspace thread scheduler has more information to make decisions,
meanwhile userspace can do their own scheduling, requiring less
context-switches.</p><p>When kernel notifies userspace of a
change, it “activates” the userspace thread scheduler (that’s where the
name “scheduler activation” comes from). A scheduler activation is like
an empty kernel thread. When kernel wants to notify userspace of
something, it creates a “scheduler activation”, assigns it a processor,
and runs userspace scheduler in this “scheduler activation”. The
userspace scheduler makes decisions by the information given in the
scheduler activation by the kernel, then proceeds to run some thread on
this scheduler activation.</p><p>The difference between a
scheduler activation and normal kernel thread is that, when the kernel
stops a scheduler activation, (maybe due to I/O), the kernel will create
another scheduler activation to notify the userspace that the other
scheduler activation has stopped; then the userspace scheduler can decide
which thread to run on this scheduler activation. When the original
scheduler activation is to be resumed (I/O completes), kernel blocks a
running scheduler activation and creates a new scheduler activation, and
let userspace scheduler decide which to run on this new scheduler
activation.</p><p>For normal kernel threads, the kernel stops
and resumes the thread without noticing userspace, and the kernel selects
what to run.</p><p>Critical sections (where the executing
program holds some locks) is a bit tricky in scheduler activation. When
the thread is in critical section when it is blocked or preempted,
performance might take a hit (no one else can run), or a deadlock might
even appear. The solution is to let the thread run a little bit until it
exits the critical section.</p><p>Scheduler activation is
basically the N:M thread we’re taught in undergrad OS classes.
Evidentally it isn’t very widely used, maybe because the performance
improvement isn’t worth the complexity.</p><h2
id="Lottery%20scheduling" class="section">Lottery
scheduling</h2><p><em>Lottery Scheduling: Flexible
Proportional-Share Resource Management</em>, <span
class="oldstyle-num">1994</span>.</p><p>Lottery
scheduling is another probability-based algorithm, it uses a simple
algorithm to solve a otherwise difficult problem. <a
id="footref:google-random" class="footref-anchor obviously-a-link"
aria-label="Jump to footnote" href="#footdef%3Agoogle-random">I really
like probability-based algorithms in general.<sup
class="inline-footref">11</sup></a></p><div
id="footdef:google-random" class="footdef"><div class="def-footref
obviously-a-link"><a aria-label="Jump back to main text"
href="#footref%3Agoogle-random">11</a></div><div
class="def-footdef">Another great example is Google’s HyperLogLog:
<a
href="https://archive.casouri.cc/note/2024/cse221/https:/www.youtube.com/watch?v=lJYufx0bfpw"><em>A
problem so hard even Google relies on Random
Chance</em></a></div></div><p>Scheduling is
hard. There are so many requirements to consider: fairness, overhead,
starvation, priority, <a id="footref:priority-inversion"
class="footref-anchor obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Apriority-inversion">priority inversion<sup
class="inline-footref">12</sup></a>. However, lottery
scheduling seemingly can have its cake and eat it too, solving all of the
above simultaneously (with a catch, of course). Even better, lottery
scheduling allows flexible distribution of resources, while normal
priority-based scheduler only has corase control over processes: <a
id="footref:fair-share" class="footref-anchor obviously-a-link"
aria-label="Jump to footnote" href="#footdef%3Afair-share">higher
priority always wins<sup
class="inline-footref">13</sup></a>. It’s also general
enough to apply to sharing other resources, like network bandwith or
memory.</p><div id="footdef:priority-inversion"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Apriority-inversion">12</a></div><div
class="def-footdef">Priority inversion is when a preempted
lower-priority process/thread holds a lock which the higher priority
process/thread needs to acquire in order to progress. In this case, the
lower-priority process is blocking the higher priority process,
effectively inverting the priority.</div></div><div
id="footdef:fair-share" class="footdef"><div class="def-footref
obviously-a-link"><a aria-label="Jump back to main text"
href="#footref%3Afair-share">13</a></div><div
class="def-footdef">Some schduler has static priorities, some
scheduler allows dynamically adjusting priorities. And fair-share
schedulers need to monitor CPU usage over time and adjust priorities
accordingly.</div></div><p>Here’s how it works. Suppose
we have some processes and want to allocate some proportion of execution
time to each. We create <span class="oldstyle-num">100</span>
tickets, and assign each process tickets based on their allocated
proportion. Eg, if we want alloacte <span
class="oldstyle-num">30%</span> of the execution time to process
A, we assign it <span class="oldstyle-num">30</span>
tickets.</p><p>Then, we divide time into epochs. At the start
of each epoch, we randomly draw a ticket out of the <span
class="oldstyle-num">100</span>, and run the process that owns
this ticket. Over a period of time, the total execution time of each
process should match the assigned proportion.</p><p>Lottery
scheduling is probabilistically fair. The shorter the epoch, and the
longer the measured duration, the more accurate and fair is the
scheduling. To ensure fairness, when a process wins lottery and executes
in an epoch, only to be blocked by I/O midway, the scheduler would give
it more ticket in the next epoch to compensate.</p><p>Lottery
scheduling doesn’t have starvation. As long as a process has some ticket,
the probability of it getting executed is not
zero.</p><p>Lottery scheduling is very responsive to changes
in configuration, because any change in the allocation proportion is
immediately reflected in the next epoch. Some scheduler, like the
fair-use scheduler mentioned earlier, might take longer to adjust
priorities.</p><p>Lottery scheduling has very low overhead.
It just need to generate a random number and find the process that owns
it. It takes <span class="oldstyle-num">~1000</span>
instructions to run scheduling; it takes <span
class="oldstyle-num">~10</span> for generating a random number,
and the rest for finding the process. The processes are stored in a
linked list, ordered by the number of tickets
held.</p><p>Lottery scheduling handles priority inversion by
allowing processes to transfer tickets to other process. Traditional
schedulers would use priority inheritance: the higher priority process
elevates the lower priority process temporarily to execute and release
the lock that the higher priority process needs. It’s the same principle,
but instead of elevating priority, a process lends its
tickets.</p><p>Of course, there’s always a catch. Lottery
scheduling isn’t very good at immediate, strict control over resources.
Eg, in a real-time system, a very high priority task has to be executed
immediately when it comes up. Lottery scheduling can’t run it immediately
(epoch), and it can’t guarantee to run it
(randomized).</p><p>Also, the simple lottery scheduling can’t
express response time (maybe something needs to run immediately but won’t
take a lot of CPU time). We can add another parameter to represent
response time, in addition to CPU time allocation. Not exactly sure how
that works though.</p><p>Nowadays, lottery scheduling isn’t
used so much for CPU scheduling, but widely used in
networking.</p><h2 id="Epilogue"
class="section">Epilogue</h2><p>That was the last classic
paper. For the rest of the course, we went through some more recent
literature like Android, GFS, MapReduce, Haystack. Those are no less
filled with interesting ideas, but this article is already so long and I
want to stop here.</p><p>Incidentally, as I’m writing this,
there’s only two days left in <span
class="oldstyle-num">2023</span>. Judging from the tags, I
started this article in February <span
class="oldstyle-num">15</span> this year. Back then I didn’t
know it’ll take a whole year to finish; during half way I thought I’ll
never finish this. But look where we are now! Persistence really do get
things done eventually.</p><p>I also started a programming
project at around the same time, and that project (after much
head-scratches and typing late at night) is also coming to fruition
around this time. Looking back, I can’t believe that I actually pulled
both of these off in <span class="oldstyle-num">2023</span>,
oh my!</p>Remap modifiers in Linux Desktop and Alacrittyurn:uuid:19af9006-b4d9-11ed-aec6-9fe7ebf475382023-02-24T22:53:00.00-05:00<p>I’m used to macOS’s key binding, that means
for a desktop environment, I want three
things:</p><ol><li>Caps lock act as
Control</li><li>System bindings are on the Command key (ie,
the Windows key), specifically, Command+C/V for
copy/paste</li><li>In the terminal emulator, Command+C/V
works as usual, and Ctrl+C/V sends respective control codes, as
usual</li></ol><p>I’m a simple man, and this is all I
want, but Thy Voice From Above hath spoken: <em>“thy shall not have
comfort!!”</em></p><h2
id="Command+C/V%20for%20copy%20and%20paste"
class="section">Command+C/V for copy and
paste</h2><p>Remapping Caps lock to Control is easy and there
are plenty tutorials online for it. However, there is
<em>absolutely no way</em> to change the default bindings of
copy/paste on a Linux desktop reliably. Because there is simply no
unified configuration for the keybinding of copy & paste. Qt
supports rebinding copy & paste and Gtk straight up <a
id="footref:gtk" class="footref-anchor obviously-a-link" aria-label="Jump
to footnote" href="#footdef%3Agtk">doesn’t support it<sup
class="inline-footref">1</sup></a>. On top of that,
applications bind their own keys and completely disregard the toolkit’s
setting, except in some toolkit widgets they use, then you have different
bindings within the same application.</p><div id="footdef:gtk"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Agtk">1</a></div><div
class="def-footdef">Gtk 3 seems to support it through CSS themes,
which is removed in Gtk 4. Anyway, I never got it to
work.</div></div><p>The whole situation is pretty
laughable, but live must go on. There are things like <a
id="footref:xkeysnail" class="footref-anchor obviously-a-link"
aria-label="Jump to footnote"
href="#footdef%3Axkeysnail">xkeysnail<sup
class="inline-footref">2</sup></a> that literally
intercepts every keystroke you type and translate them into other keys
depending on the application currently in focus. It requires some
nontrivial configuration and may or may not work reliably on X11, <a
id="footref:wayland" class="footref-anchor obviously-a-link"
aria-label="Jump to footnote" href="#footdef%3Awayland">definitely
doesn’t work on Wayland<sup
class="inline-footref">3</sup></a>, and I don’t know how
do I feel about a Python program running as root, intercepting and
translating every key I type. There are Rust alternatives, but I didn’t
have much luck with those either.</p><div id="footdef:xkeysnail"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Axkeysnail">2</a></div><div
class="def-footdef"><a
href="https://archive.casouri.cc/note/2023/alacritty-modifier/https:/github.com/mooz/xkeysnail">xkeysnail</a>.
There are also projects like <a
href="https://archive.casouri.cc/note/2023/alacritty-modifier/https:/github.com/rbreaves/kinto">kinto.sh</a>
that pre-configures it for you on both Linux and Windows. (On Windows it
uses AutoHotkey.)</div></div><div id="footdef:wayland"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Awayland">3</a></div><div
class="def-footdef">These type of program use X11 protocol, and
Wayland just doesn’t support program intercepting and translating other
program’s input.</div></div><p>The real way, the only
good way, to do it is to just swap Control with Super (ie, Command) at
X11 level. (Wayland picks it up so it works on Wayland too, or so I’m
told). Since we also want to swap Caps lock and Control, we actually do a
three-way swap:</p><ul><li>Super →
Control</li><li>Control → Caps lock</li><li>Caps
lock → Super</li></ul><p>So now when you press
Command+C, the application gets Control+C.</p><p>To actually
swap the modifiers, we edit</p><p><span
class="mono">/usr/share/X11/xkb/keycodes/evdev</span></p><p>and
reboot—no adding command to X init or some config file or some other
crap. You edit the file, reboot, and it works, and keeps working. I
learned this from a <a
href="https://archive.casouri.cc/note/2023/alacritty-modifier/https:/askubuntu.com/questions/929744/how-to-remap-key-in-ubuntu-17-10-wayland-up-key-to-shift">StackExchange
question</a>.</p><p>Below are the exact edit you need
to make in that file, and their effect:</p><p>To map Left
Control (keycode 37) to Caps lock:<br/>Change
<code><CAPS> = 66</code> to
<code><CAPS> = 37</code></p><p>To
map Left Super (keycode 133) to Control:<br/>Change
<code><LCTL> = 37</code> to
<code><LCTL> 133</code></p><p>To
map Caps lock (keycode 66) to Left Super:<br/>Change
<code><LWIN> = 133</code> to
<code><LWIN> = 66</code></p><p>If
you use Emacs, you need to swap Super and Control back. Add this to your
<span class="mono">init.el</span>:</p><pre
class="code-block">(setq x-super-keysym 'ctrl) (setq x-ctrl-keysym
'super)</pre><h2 id="Command+C/V%20in%20terminal"
class="section">Command+C/V in terminal</h2><p>Now
Command+C/V works in normal applications, but in terminal, Caps lock+C/V
(appears as Super+C/V) will not send control keys and Command+C/V
(appears as Control+C/V) will not do what you want—again, you need to
swap Super and Control back, as we did for Emacs.</p><p>I
looked at every terminal emulator on Linux, and <a
href="https://archive.casouri.cc/note/2023/alacritty-modifier/https:/github.com/alacritty/alacritty">Alacritty</a>
is the only one that allows remapping modifier keys, has sane
configuration so that I can actually configure the remap, and has sane
dependencies.</p><p>You want to remap all
Control+<em>x</em> keys to simply <em>x</em>,
except for Control+C/V/F, etc, which are bind to actions like Copy,
Paste, SearchForward. And you want to remap all
Super+<em>x</em> keys to Control+<em>x</em>. In
effect, you have:</p><ul><li>Command+C/V → Control+C/V
→ Copy/Paste</li><li>Caps lock+C/V → Super+C/V →
Control+C/V</li></ul><p>To do that, add this to the
beginning of <span
class="mono">~/.config/alacritty/alacritty.yml</span>:</p><pre
class="code-block">key_bindings: - { key: At, mods: Control, chars:
"@" } - { key: A, mods: Control, chars: "a" } - { key: B, mods: Control,
chars: "b" } - { key: C, mods: Control, action: Copy } - { key: D, mods:
Control, chars: "d" } - { key: E, mods: Control, chars: "e" } - { key: F,
mods: Control, action: SearchForward } - { key: F, mods: Control, mode:
~Search, action: SearchForward } - { key: F, mods: Control|Shift, action:
SearchBackward } - { key: F, mods: Control|Shift, mode: ~Search, action:
SearchBackward } - { key: G, mods: Control, chars: "g" } - { key: H,
mods: Control, chars: "h" } - { key: I, mods: Control, chars: "i" } - {
key: J, mods: Control, chars: "j" } - { key: K, mods: Control, chars: "k"
} - { key: L, mods: Control, chars: "l" } - { key: M, mods: Control,
chars: "m" } - { key: N, mods: Control, action: CreateNewWindow } - {
key: O, mods: Control, chars: "o" } - { key: P, mods: Control, chars: "p"
} - { key: Q, mods: Control, action: Quit } - { key: R, mods: Control,
chars: "r" } - { key: S, mods: Control, chars: "s" } - { key: T, mods:
Control, chars: "t" } - { key: U, mods: Control, chars: "u" } - { key: V,
mods: Control, action: Paste } - { key: W, mods: Control, action: Quit }
- { key: X, mods: Control, chars: Cut } - { key: Y, mods: Control, chars:
"y" } - { key: Z, mods: Control, chars: "z" } - { key: LBracket, mods:
Control, chars: "[" } - { key: Backslash, mods: Control, chars: "\\" } -
{ key: RBracket, mods: Control, chars: "]" } - { key: Grave, mods:
Control, chars: "^" } - { key: Underline, mods: Control, chars: "_" } - {
key: At, mods: Super, chars: "\x00" } - { key: A, mods: Super, chars:
"\x01" } - { key: B, mods: Super, chars: "\x02" } - { key: C, mods:
Super, chars: "\x03" } - { key: D, mods: Super, chars: "\x04" } - { key:
E, mods: Super, chars: "\x05" } - { key: F, mods: Super, chars: "\x06" }
- { key: G, mods: Super, chars: "\x07" } - { key: H, mods: Super, chars:
"\x08" } - { key: I, mods: Super, chars: "\x09" } - { key: J, mods:
Super, chars: "\x0a" } - { key: K, mods: Super, chars: "\x0b" } - { key:
L, mods: Super, chars: "\x0c" } - { key: M, mods: Super, chars: "\x0d" }
- { key: N, mods: Super, chars: "\x0e" } - { key: O, mods: Super, chars:
"\x0f" } - { key: P, mods: Super, chars: "\x10" } - { key: Q, mods:
Super, chars: "\x11" } - { key: R, mods: Super, chars: "\x12" } - { key:
S, mods: Super, chars: "\x13" } - { key: T, mods: Super, chars: "\x14" }
- { key: U, mods: Super, chars: "\x15" } - { key: V, mods: Super, chars:
"\x16" } - { key: W, mods: Super, chars: "\x17" } - { key: X, mods:
Super, chars: "\x18" } - { key: Y, mods: Super, chars: "\x19" } - { key:
Z, mods: Super, chars: "\x1a" } - { key: LBracket, mods: Super, chars:
"\x1b" } - { key: Backslash, mods: Super, chars: "\x1c" } - { key:
RBracket, mods: Super, chars: "\x1d" } - { key: Grave, mods: Super,
chars: "\x1e" } - { key: Underline, mods: Super, chars: "\x1f"
}</pre><p>This configuration remaps <a id="footref:ascii"
class="footref-anchor obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Aascii">all possible modifier keybindings available in
a terminal environment<sup
class="inline-footref">4</sup></a>.</p><div
id="footdef:ascii" class="footdef"><div class="def-footref
obviously-a-link"><a aria-label="Jump back to main text"
href="#footref%3Aascii">4</a></div><div
class="def-footdef">See this <a
href="https://archive.casouri.cc/note/2023/alacritty-modifier/https:/www.physics.udel.edu/~watson/scen103/ascii.html">ASCII
table</a>.</div></div><h2 id="Conclusion"
class="section">Conclusion</h2><p>At this point you should
be able to copy & paste with Command+C/V in every application and
terminal, and use Caps lock as Control in Emacs and terminal,
<em>as it should be</em>.</p>Bonjour Crash Courseurn:uuid:4a6f4716-b355-11ed-8033-5f84d04364942023-02-23T00:37:00.00-05:00<p>Bonjour is Apple’s implementation of
zeroconf networking. With Bonjour, you can plug in a printer into the
local network and expect it to show up on computers in the network,
without manually configuring anything. Linux’s implementation is
Avahi.</p><p>I recently needed to use Bonjour for some
project and read some documentation. This is an article summarizing some
concepts one needs to know in order to use a Bonjour library. This
article assumes some basic network knowledge (TPC/IP, DHCP, DNS,
multicast, unicast, network layers, etc).</p><p>Everything in
this article is based on Apple’s documentation at <a
href="https://archive.casouri.cc/note/2023/bonjour/https:/developer.apple.com/library/archive/documentation/Cocoa/Conceptual/NetServices/Introduction.html#%2F%2Fapple_ref%2Fdoc%2Fuid%2FTP40002445-SW1"><em>Bonjour
Overview</em></a>. (If you want to read it, I recommend
starting with the “Bonjour Operations”
section.)</p><p>Bonjour operates in the link-local network
and provides three operations:</p><ol><li>Registering
services</li><li>Discovering available
services</li><li>Resolving a service instance name to an
address and port</li></ol><h2
id="Registering%20a%20service" class="section">Registering a
service</h2><p>When registering/publishing a service, you (or
rather the library) creates a mDNS (multicast DNS) responder with three
records: a service (SRV) record, a pointer (PTR) record, and a text (TXT)
record. The text record is for providing additional information and is
usually empty.</p><h3 id="Service%20records"
class="subsection">Service records</h3><p>A service record
maps the service name to the host name and the port of the service. It
uses host name rather than an IP address so that the service can be on
multiple IP addresses at the same time (eg, on both IPv4 and
IPv6).</p><p>The full name of a service is made of three
parts, the <em>service instance name</em>, the
<em>service type</em>, and the <em>domain</em>,
in the form of:</p><pre class="code-block"><service
instance name>.<service
type>.<domain></pre><p>The
<em>service instance name</em> is a human-readable string
showed to end-users, encoded in <span
class="oldstyle-num">utf-8</span>, and can be up to <span
class="oldstyle-num">63</span> bytes long.</p><p>The
<em>service type</em> is made of the <em>service
type</em> and the <em>transport protocol</em>, in the
form of <code>_type._protocol.</code>. Eg,
<code>_ftp._tcp.</code>. The underscore prefix is to
distinguish from domain names. Bonjour basically uses the format
described in <a
href="https://archive.casouri.cc/note/2023/bonjour/https:/www.ietf.org/rfc/rfc2782.txt"><span
class="oldstyle-num">RFC
2782</span></a>.</p><p>Technically, both the type
and the protocol are standardized. If you want to add a service type, you
need to register it with <a
href="https://archive.casouri.cc/note/2023/bonjour/https:/www.iana.org/form/ports-services">IANA</a>.</p><p>The
<em>domain name</em> is just like an Internet domain name,
eg, <code>www.apple.com.</code>. In addition, there is a
pseudo domain, <code>local.</code>, which refers to the
link-local network. (So you have Bonjour to thank when you ssh to LAN
hosts with
<code><host>.local</code>.)</p><p>Service
instance name, service type, and domain name together make up the full
name of a service instance. For example,</p><pre
class="code-block">Alice’s Music
library._music._tcp.local.</pre><h3 id="Pointer%20records"
class="subsection">Pointer records</h3><p>A pointer record
basically maps service types to full service names. Ie, it
maps</p><pre class="code-block"><service
type>.<domain></pre><p>to</p><pre
class="code-block"><service instance
name>.<service
type>.<domain></pre><p>This way you can
search for a type of service and get a list of available service
instances.</p><h3 id="Publishing%20(advertising)"
class="subsection">Publishing (advertising)</h3><p>When
publishing a service, a host will first make sure the intended service
instance name is not taken by someone else, by broadcasting request to
that service instance name: if there is a response, the name is taken. If
someone else has taken it, the host will append a number to the service
instance name and increment the number until it gets a name that no one
is using.</p><p>If you use a library, this part is taken care
for you. But it’s good to know how does Bonjour avoid name
conflicts.</p><h2 id="Discovering%20services"
class="section">Discovering services</h2><p>To discover
service instances, you first request PTR records by mDNS, and get back a
list of service instance names. …And that’s it. The host will save those
names, and resolve a service name into actual address and port every time
it needs to use the service.</p><h2
id="Resolving%20service%20names" class="section">Resolving service
names</h2><p>By the discovery step, we collected some service
instance names that are available for us in the local network. The next
step is to pick one, resolve it into an actual address and connect to
it.</p><p>The host will send out a mDNS request for the
service instance name, and get back a host name and a port. It then sends
out a mDNS request for the host name and get an IP address. Now it can
connect to the address on the port and start using the
service.</p>Tree-sitter Starter Guideurn:uuid:afb76ba2-8a39-11ed-998c-8f06c8e638dc2023-01-15T00:00:00.00-05:00<p>This guide gives you a starting point on
writing a tree-sitter major mode. Remember, don’t panic and check your
manuals!</p><h2 id="Build%20Emacs%20with%20tree-sitter"
class="section">Build Emacs with tree-sitter</h2><p>You
can either install tree-sitter by your package manager, or
from<br/>source:</p><pre class="code-block">git clone
https://github.com/tree-sitter/tree-sitter.git cd tree-sitter make make
install</pre><p>To build and run Emacs 29:</p><pre
class="code-block">git clone
https://git.savannah.gnu.org/git/emacs.git -b emacs-29 cd emacs
./autogen.sh ./configure make src/emacs</pre><p>Require the
tree-sitter package with <code>(require 'treesit)</code>.
Note that tree-sitter always appear as <code>treesit</code>
in symbols. Now check if Emacs is successfully built with tree-sitter
library by evaluating
<code>(treesit-available-p)</code>.</p><p>Tree-sitter
stuff in Emacs can be categorized into two parts: the tree-sitter API
itself, and integration with fontification, indentation, Imenu, etc. You
can use shortdoc to glance over all the tree-sitter API functions by
typing <code>M-x shortdoc RET treesit RET</code>. The
integration are described in the rest of the post.</p><h2
id="Install%20language%20definitions" class="section">Install language
definitions</h2><p>Tree-sitter by itself doesn’t know how to
parse any particular language. It needs the language grammar (a dynamic
library) for a language to be able to parse it.</p><p>First,
find the repository for the language grammar, eg, <a
href="https://archive.casouri.cc/note/2023/tree-sitter-starter-guide/https:/github.com/tree-sitter/tree-sitter-python">tree-sitter-python</a>.
Take note of the Git clone URL of it, eg,
<code>https://github.com/tree-sitter/tree-sitter-python.git</code>.
Now check where is the parser.c file in that repository, usually it’s in
<code>src</code>.</p><p>Make sure you have Git, C
and C++ compiler, and run the
<code>treesit-install-grammar</code> command, it will prompt
for the URL and the directory of parser.c, leave other prompts at default
unless you know what you are doing.</p><p>You can also
manually clone the repository and compile it, and put the dynamic library
at a standard library location. Emacs will be able to find it. If you
wish to put it somewhere else, set
<code>treesit-extra-load-path</code> so Emacs can find
it.</p><h2 id="Tree-sitter%20major%20modes"
class="section">Tree-sitter major modes</h2><p>Tree-sitter
modes should be separate major modes, usually named
<code>xxx-ts-mode</code>. I know I said tree-sitter always
appear as <code>treesit</code> in symbols, this is the only
exception.</p><p>If the tree-sitter mode and the “native”
mode could share some setup code, you can create a “base mode”, which
only contains the common setup. For example, there is python-base-mode
(shared), and both python-mode (native), and python-ts-mode (tree-sitter)
derives from it.</p><p>In the tree-sitter mode, check if we
can use tree-sitter with <code>treesit-ready-p</code>, it
will emit a warning if tree-sitter is not ready (tree-sitter not built
with Emacs, can’t find the language grammar, buffer too large,
etc).</p><h2 id="Fontification"
class="section">Fontification</h2><p>Tree-sitter works
like this: It parses the buffer and produces a <a
href="https://archive.casouri.cc/note/2023/tree-sitter-starter-guide/https:/en.wikipedia.org/wiki/Parse_tree"><em>parse
tree</em></a>. You provide a query made of patterns and
capture names, tree-sitter finds the nodes that match these patterns, tag
the corresponding capture names onto the nodes and return them to you.
The query function returns a list of <code>(capture-name .
node)</code>.</p><p>For fontification, we simply use
face names as capture names. And the captured node will be fontified in
their capture name (the face).</p><p>The capture name could
also be a function, in which case <code>(NODE OVERRIDE START
END)</code> is passed to the function for fontification.
<code>START</code> and <code>END</code> are the
start and end of the region to be fontified. The function should only
fontify within that region. The function should also allow more optional
arguments with <code>&rest _</code>, for future
extensibility. For <code>OVERRIDE</code> check out the
docstring of
<code>treesit-font-lock-rules</code>.</p><h3
id="Query%20syntax" class="subsection">Query
syntax</h3><p>There are two types of nodes: “named nodes”,
like <code>(identifier)</code>,
<code>(function_definition)</code>, and “anonymous nodes”,
like <code>"return"</code>, <code>"def"</code>,
<code>"("</code>, <code>";"</code>. Parent-child
relationship is expressed as</p><pre
class="code-block">(parent (child) (child) (child
(grand_child)))</pre><p>Eg, an argument list <code>(1,
"3", 1)</code> would be:</p><pre
class="code-block">(argument_list "(" (number) (string) (number)
")")</pre><p>Children could have field
names:</p><pre class="code-block">(function_definition name:
(identifier) type: (identifier))</pre><p>To match any one in
the list:</p><pre class="code-block">["true" "false"
"none"]</pre><p>Capture names can come after any node in the
pattern:</p><pre class="code-block">(parent (child) @child)
@parent</pre><p>The query above captures both the parent and
the child.</p><p>The query below captures all the keywords
with capture
name<br/><code>"keyword"</code>:</p><pre
class="code-block">["return" "continue" "break"]
@keyword</pre><p>These are the common syntax, check out the
full syntax in the manual: <a
href="https://archive.casouri.cc/note/2023/tree-sitter-starter-guide/html-manual/Pattern-Matching.html">Pattern
Matching</a>.</p><h3 id="Query%20references"
class="subsection">Query references</h3><p>But how do one
come up with the queries? Take python for an example, open any python
source file, type <code>M-x treesit-explore-mode RET</code>.
You should see the parse tree in a separate window, automatically updated
as you select text or edit the buffer. Besides this, you can consult the
grammar of the language definition. For example, Python’s grammar file is
at</p><p><a
href="https://archive.casouri.cc/note/2023/tree-sitter-starter-guide/https:/github.com/tree-sitter/tree-sitter-python/blob/master/grammar.js">https://github.com/tree-sitter/tree-sitter-python/blob/master/grammar.js</a></p><p>Neovim
also has a bunch of <a
href="https://archive.casouri.cc/note/2023/tree-sitter-starter-guide/https:/github.com/nvim-treesitter/nvim-treesitter/tree/master/queries">queries
to reference from</a>.</p><p>The manual explains how to
read grammar files in the bottom of <a
href="https://archive.casouri.cc/note/2023/tree-sitter-starter-guide/html-manual/Language-Grammar.html">Language
Grammar</a>.</p><h3 id="Debugging%20queries"
class="subsection">Debugging queries</h3><p>If your query
has problems, use <code>treesit-query-validate</code> to
debug the query. It will pop a buffer containing the query (in text
format) and mark the offending part in red. Set
<code>treesit--font-lock-verbose</code> to
<code>t</code> if you want the font-lock function to report
what it’s doing.</p><h3 id="Set%20up%20font-lock"
class="subsection">Set up font-lock</h3><p>To enable
tree-sitter font-lock, set
<code>treesit-font-lock-settings</code> and
<code>treesit-font-lock-feature-list</code> buffer-locally
and call <code>treesit-major-mode-setup</code>. For example,
see <code>python--treesit-settings</code> in python.el. Below
is a snippet of it.</p><p>Note that like the current
font-lock system, if the to-be-fontified region already has a face (ie,
an earlier match fontified part/all of the region), the new face is
discarded rather than applied. If you want later matches always override
earlier matches, use the <code>:override</code>
keyword.</p><p>Each rule should have a
<code>:feature</code>, like
<code>function-name</code>,
<code>string-interpolation</code>,
<code>builtin</code>, etc. This way users can enable/disable
each feature individually.</p><p>Read the manual section
<a
href="https://archive.casouri.cc/note/2023/tree-sitter-starter-guide/html-manual/Parser_002dbased-Font-Lock.html">Parser-based
Font-Lock</a> for more detail.</p><p>Example from
python.el:</p><pre class="code-block">(defvar
python--treesit-settings (treesit-font-lock-rules :feature 'comment
:language 'python '((comment) @font-lock-comment-face) :feature 'string
:language 'python '((string) @python--treesit-fontify-string) :feature
'string-interpolation :language 'python :override t '((interpolation
(identifier) @font-lock-variable-name-face)) ...))</pre><p>In
<code>python-ts-mode</code>:</p><pre
class="code-block">(treesit-parser-create 'python) (setq-local
treesit-font-lock-settings python--treesit-settings) (setq-local
treesit-font-lock-feature-list '(( comment definition) ( keyword string
type) ( assignment builtin constant decorator escape-sequence number
property string-interpolation ) ( bracket delimiter function operator
variable))) ...
(treesit-major-mode-setup)</pre><p>Concretely, something like
this:</p><pre class="code-block">(define-derived-mode
python-ts-mode python-base-mode "Python" "Major mode for editing Python
files, using tree-sitter library. \\{python-ts-mode-map}" :syntax-table
python-mode-syntax-table (when (treesit-ready-p 'python)
(treesit-parser-create 'python) (setq-local
treesit-font-lock-feature-list '(( comment definition) ( keyword string
type) ( assignment builtin constant decorator escape-sequence number
property string-interpolation ) ( bracket delimiter function operator
variable))) (setq-local treesit-font-lock-settings
python--treesit-settings) (setq-local imenu-create-index-function
#'python-imenu-treesit-create-index) (setq-local
treesit-defun-type-regexp (rx (or "function" "class") "_definition"))
(setq-local treesit-defun-name-function #'python--treesit-defun-name)
(treesit-major-mode-setup) (when python-indent-guess-indent-offset
(python-indent-guess-indent-offset))))</pre><h2 id="Indentation"
class="section">Indentation</h2><p>Indentation works like
this: We have a bunch of rules that look like</p><pre
class="code-block">(MATCHER ANCHOR OFFSET)</pre><p>When
the indenting a line, let <code>NODE</code> be the node at
the beginning of the current line, we pass this node to the
<code>MATCHER</code> of each rule, one of them will match the
node (eg, “this node is a closing bracket!”). Then we pass the node to
the <code>ANCHOR</code>, which returns a point (eg, the
beginning of <code>NODE</code>’s parent). We find the column
number of that point (eg, 4), add <code>OFFSET</code> to it
(eg, 0), and that is the column we want to indent the current line to (4
+ 0 = 4).</p><p>Matchers and anchors are functions that takes
<code>(NODE PARENT BOL &rest _)</code>. Matches
return nil/non-nil for no match/match, and anchors return the anchor
point. An Offset is usually a number or a variable, but it can also be a
function. Below are some convenient builtin matchers and
anchors.</p><p>For <code>MATHCER</code> we
have</p><pre class="code-block">(parent-is TYPE) =>
matches if PARENT’s type matches TYPE as regexp (node-is TYPE) =>
matches NODE’s type (query QUERY) => matches if querying PARENT
with QUERY captures NODE. (match NODE-TYPE PARENT-TYPE NODE-FIELD
NODE-INDEX-MIN NODE-INDEX-MAX) => checks everything. If an
argument is nil, don’t match that. Eg, (match nil TYPE) is the same as
(parent-is TYPE)</pre><p>For <code>ANCHOR</code>
we have</p><pre class="code-block">first-sibling =>
start of the first sibling parent => start of parent parent-bol
=> BOL of the line parent is on. prev-sibling => start of
previous sibling no-indent => current position (don’t indent)
prev-line => start of previous line</pre><p>There is
also a manual section for indent: <a
href="https://archive.casouri.cc/note/2023/tree-sitter-starter-guide/html-manual/Parser_002dbased-Indentation.html">Parser-based
Indentation</a>.</p><p>When writing indent rules, you
can use <code>treesit-check-indent</code> to<br/>check
if your indentation is correct. To debug what went wrong,
set<br/><code>treesit--indent-verbose</code> to
<code>t</code>. Then when you indent, Emacs<br/>tells
you which rule is applied in the echo area.</p><p>Here is an
example:</p><pre class="code-block">(defvar
typescript-mode-indent-rules (let ((offset 'typescript-indent-offset))
`((typescript ;; This rule matches if node at point is ")", ANCHOR is the
;; parent node’s BOL, and offset is 0. ((node-is ")") parent-bol 0)
((node-is "]") parent-bol 0) ((node-is ">") parent-bol 0)
((node-is "\\.") parent-bol ,offset) ((parent-is "ternary_expression")
parent-bol ,offset) ((parent-is "named_imports") parent-bol ,offset)
((parent-is "statement_block") parent-bol ,offset) ((parent-is
"type_arguments") parent-bol ,offset) ((parent-is "variable_declarator")
parent-bol ,offset) ((parent-is "arguments") parent-bol ,offset)
((parent-is "array") parent-bol ,offset) ((parent-is "formal_parameters")
parent-bol ,offset) ((parent-is "template_substitution") parent-bol
,offset) ((parent-is "object_pattern") parent-bol ,offset) ((parent-is
"object") parent-bol ,offset) ((parent-is "object_type") parent-bol
,offset) ((parent-is "enum_body") parent-bol ,offset) ((parent-is
"arrow_function") parent-bol ,offset) ((parent-is
"parenthesized_expression") parent-bol ,offset)
...))))</pre><p>Then you set
<code>treesit-simple-indent-rules</code> to your rules, and
call <code>treesit-major-mode-setup</code>.</p><h2
id="Imenu" class="section">Imenu</h2><p>Set
<code>treesit-simple-imenu-settings</code> and call
<code>treesit-major-mode-setup</code>.</p><h2
id="Navigation" class="section">Navigation</h2><p>Set
<code>treesit-defun-type-regexp</code>,
<code>treesit-defun-name-function</code>, and call
<code>treesit-major-mode-setup</code>.</p><h2
id="C-like%20languages" class="section">C-like
languages</h2><p>[Update: Common functions described in this
section have been moved from c-ts-mode.el to c-ts-common.el. I also made
some changes to the functions and variables
themselves.]</p><p>c-ts-common.el has some goodies for
handling indenting and filling block comments.</p><p>These
two rules should take care of indenting block comments.</p><pre
class="code-block">((and (parent-is "comment")
c-ts-common-looking-at-star) c-ts-common-comment-start-after-first-star
-1) ((parent-is "comment") prev-adaptive-prefix
0)</pre><p><code>standalone-parent</code> should
be enough for most of the cases where you want to "indent one level
further", for example, a statement inside a block. Normally
<code>standalone-parent</code> returns the parent’s start
position as the anchor, but if the parent doesn’t start on its own line,
it returns the parent’s parent instead, and so on and so forth. This
works pretty well in practice. For example, indentation rules for
statements and brackets would look like:</p><pre
class="code-block">;; Statements in {} block. ((parent-is
"compound_statement") standalone-parent x-mode-indent-offset) ;; Closing
bracket. ((node-is "}") standalone-parent x-mode-indent-offset) ;;
Opening bracket. ((node-is "compound_statement") standalone-parent
x-mode-indent-offset)</pre><p>You’ll need additional rules
for “brackless” if/for/while statements, eg</p><pre
class="code-block">if (true) return 0; else return
1;</pre><p>You need rules like these:</p><pre
class="code-block">((parent-is "if_statement") standalone-parent
x-mode-indent-offset)</pre><p>Finally,
<code>c-ts-common-comment-setup</code> will set up comment
and filling for you.</p><h2 id="Multi-language%20modes"
class="section">Multi-language modes</h2><p>Refer to the
manual: <a
href="https://archive.casouri.cc/note/2023/tree-sitter-starter-guide/html-manual/Multiple-Languages.html">Multiple
Languages</a>.</p><h2 id="Common%20Tasks"
class="section">Common Tasks</h2><p><code>M-x
shortdoc RET treesit RET</code> will give you a complete
list.</p><p>How to...</p><p><b>Get the
buffer text corresponding to a node?</b></p><pre
class="code-block">(treesit-node-text node)</pre><p>Don’t
confuse this with
<code>treesit-node-string</code>.</p><p><b>Scan
the whole tree for stuff?</b></p><pre
class="code-block">(treesit-search-subtree) (treesit-search-forward)
(treesit-induce-sparse-tree)</pre><p><b>Find/move to to
next node that...?</b></p><pre
class="code-block">(treesit-search-forward node ...)
(treesit-search-forward-goto node ...)</pre><p><b>Get
the root node?</b></p><pre
class="code-block">(treesit-buffer-root-node)</pre><p><b>Get
the node at point?</b></p><pre
class="code-block">(treesit-node-at (point))</pre>Tree-sitter in Emacs 29 and Beyondurn:uuid:fac62c4a-8599-11ed-a0db-5f97535421d32023-01-15T00:00:00.00-05:00<p>Emacs’ release branch is now on complete
feature freeze, meaning absolutely only bug fixes can happen on it. Now
is a good time to talk about the state of <a
href="https://archive.casouri.cc/note/2023/tree-sitter-in-emacs-29/https:/tree-sitter.github.io/tree-sitter">tree-sitter</a>
in Emacs: what do you get in Emacs 29, what you don’t, and what would
happen going forward.</p><h2
id="What%E2%80%99s%20in%20Emacs%2029" class="section">What’s in Emacs
29</h2><p>From a pure user’s perspective, Emacs 29 just adds
some new built-in major modes which look more-or-less identical to the
old ones. There aren’t any flashy cool features either. That sounds
disappointing, but there are a lot of new stuff under the hood, a solid
base upon which exciting things can emerge.</p><p>If Emacs 29
is built with the tree-sitter library, you have access to most of the
functions in its C API, including creating parsers, parsing text,
retrieving nodes from the parse tree, finding the parent/child/sibling
node, pattern matching nodes with a DSL, etc. You also get a bunch of
convenient functions built upon the primitive functions, like searching
for a particular node in the parse tree, cherry picking nodes and
building a sparse tree out of the parse tree, getting the node at point,
etc. You can type <code>M-x shortdoc RET treesit RET</code>
to view a list of tree-sitter functions. And because it’s Emacs, there is
comprehensive manual coverage for everything you need to know. It’s in
“Section 37, Parsing Program Source” of Emacs Lisp Reference
Manual.</p><p>Emacs 29 has built-in tree-sitter major modes
for C, C++, C#, Java, Rust, Go, Python, Javascript, Typescript, JSON,
YAML, TOML, CSS, Bash, Dockerfile, CMake file. We tried to extend
existing modes with tree-sitter at first but it didn’t work out too well,
so now tree-sitter lives in separate major modes. The tree-sitter modes
are usually called <code>xxx-ts-mode</code>, like
<code>c-ts-mode</code> and
<code>python-ts-mode</code>. The simplest way to enable them
is to use <code>major-mode-remap-alist</code>. For
example,</p><pre class="code-block">(add-to-list
'major-mode-remap-alist '(c-mode . c-ts-mode))</pre><p>The
built-in tree-sitter major modes have support for font-lock (syntax
highlight), indentation, Imenu, which-func, and defun
navigation.</p><p>For major mode developers, Emacs 29
includes integration for these features for tree-sitter, so major modes
only need to supply language-specific information, and Emacs takes care
of plugging tree-sitter into font-lock, indent, Imenu,
etc.</p><h3 id="Fontification"
class="subsection">Fontification</h3><p>In tree-sitter
major modes, fontification is categorized into “features”, like
“builtin”, “function”, “variable”, “operator”, etc. You can choose what
“features” to enable for a mode. If you are feeling adventurous, it is
also possible to add your own fontification rules.</p><p>To
add/remove features for a major mode, use
<code>treesit-font-lock-recompute-features</code> in its mode
hook. For example,</p><pre class="code-block">(defun
c-ts-mode-setup () (treesit-font-lock-recompute-features '(function
variable) '(definition))) (add-hook 'c-ts-mode-hook
#'c-ts-mode-setup)</pre><p>Features are grouped into
decoration levels, right now there are 4 levels and the default level is
3. If you want to program in skittles, set
<code>treesit-font-lock-level</code> to 4 ;-)</p><h3
id="Language%20grammars" class="subsection">Language
grammars</h3><p>Tree-sitter major modes need corresponding
langauge grammar to work. These grammars come in the form of dynamic
libraries. Ideally the package manager will build them when building
Emacs, like with any other dynamic libraries. But they can’t cover every
language grammar out there, so you probably need to build them yourself
from time to time. Emacs has a command for it:
<code>treesit-install-language-grammar</code>. It asks you
for the Git repository and other stuff and builds the dynamic library.
Third-party major modes can instruct their users to add the recipe for
building a language grammar like this:</p><pre
class="code-block">(add-to-list 'treesit-language-source-alist
'(python
"https://github.com/tree-sitter/tree-sitter-python.git"))</pre><p>Then
typing <code>M-x treesit-install-language-grammar RET
python</code> builds the language grammar without
user-input.</p><h3 id="Other%20features"
class="subsection">Other features</h3><p>Things like
indentation, Imenu, navigation, etc, should just
work.</p><p>There is no code-folding, selection expansion,
and structural navigation (except for defun) in Emacs 29. Folding and
expansion should be trivial to implement in existing third-party
packages. Structural navigation needs careful design and nontrivial
changes to existing commands (ie, more work). So not in 29,
unfortunately.</p><h2 id="Future%20plans"
class="section">Future plans</h2><h3 id="Navigation"
class="subsection">Navigation</h3><p>The tree-sitter
integration is far from complete. As mentioned earlier, structural
navigation is still in the works. Right now Emacs allows you to define a
“thing” by a regexp that matches node types, plus optionally a filter
function that filters out nodes that matches the regexp but isn’t really
the “thing”. Given the definition of a “thing”, Emacs has functions for
finding the “things” around point
(<code>treesit--things-around</code>), finding the “thing” at
point (<code>treesit--thing-at-point</code>), and navigating
around “things” (<code>treesit--navigate-thing</code>).
Besides moving around, these functions should be also useful for other
things like folding blocks. Beware that, as the double dash suggests,
these functions are experimental and could change.</p><p>I
also have an idea for “abstract list elements”. Basically an abstract
list element is anything repeatable in a grammar: defun, statement,
arguments in argument list, etc. These things appear at every level of
the grammar and seems like a very good unit for
navigation.</p><h3 id="Context%20extraction"
class="subsection">Context extraction</h3><p>There is also
potential for language-agnostic “context extraction” (for the lack of a
better term) with tree-sitter. Right now we can get the name and span of
the defun at point, but it doesn’t have to stop there, we can also get
the parameter list, the type of the return value, the class/trait of the
function, etc. Because it’s language agnostic, any tool using this
feature will work on many languages all at once.</p><p>In
fact, you can already extract useful things, to some degree, with the
fontification queries written by major modes: using the query intended
for the <code>variable</code> query, I can get all the
variable nodes in a given range.</p><p>There are some
unanswered questions though: (1) What would be the best function
interface and data structure for such a feature? Should it use a plist
like <code>(:name ... :params ...)</code>, or a cl-struct?
(2) If a language is different enough from the “common pattern”, how
useful does this feature remains? For example, there isn’t a clear
parameter list in Haskell, and there could be several defun bodies that
defines the same function. (3) Is this feature genuinely useful, or is it
just something that looks cool? Only time and experiments can tell, I’m
looking forward to see what people will do with tree-sitter in the wild
:-)</p><h3 id="Major%20mode%20fallback"
class="subsection">Major mode fallback</h3><p>Right now
there is no automatic falling back from tree-sitter major modes to
“native” major modes when the tree-sitter library or language grammar is
missing. Doing it right requires some change to the auto-mode facility.
Hopefully we’ll see a good solution for it in Emacs 30. Right now, if you
need automatic fallback, try something like this:</p><pre
class="code-block">(define-derived-mode python-auto-mode prog-mode
"Python Auto" "Automatically decide which Python mode to use." (if
(treesit-ready-p 'python t) (python-ts-mode)
(python-mode)))</pre><h3 id="Other%20plans"
class="subsection">Other plans</h3><p>Existing tree-sitter
major modes are pretty basic and doesn’t have many bells and whistles,
and I’m sure there are rough corners here and there. Of course, these
things will improve over time.</p><p>Tree-sitter is very
different and very new, and touches many parts of Emacs, so no one has
experience with it and no one knows exactly how should it look like.
Emacs 29 will give us valuable experience and feedback, and we can make
it better and better in the future.</p><p>If you are
interested, get involved! Read <a
href="https://archive.casouri.cc/note/2023/tree-sitter-in-emacs-29/note/2020/contributing-to-emacs/index.html">Contributing
to Emacs</a> for some tips in getting involved with the Emacs
development. Read <a
href="https://archive.casouri.cc/note/2023/tree-sitter-in-emacs-29/note/2023/tree-sitter-starter-guide/index.html">Tree-sitter
Starter Guide</a> if you want to write a major mode using
tree-sitter. And of course, docstrings and the manual is always your
friend. If you have questions, you can ask on Reddit, or comment in this
post’s public inbox (see the footer).</p>This Site is Changing its Domainurn:uuid:ff14d65c-6726-11ed-b927-c76ea0aa44992022-11-18T00:00:00.00-05:00<p>Right now this site resides on <a
href="https://archive.casouri.cc/note/2022/domain-change/https:/archive.casouri.cat">archive.casouri.cat</a>,
I really love the .cat TLD. Alas, .cat was never meant for generic use
and my site doesn't comply to its requirements, which is to use and
promote Catalan language and culture. I don’t want to wake up one day
recieving a take down notice, however slim the possibility is. Plus, the
longer this site uses this domain, the more backlinks to it, the harder
to move on.</p><p>Moving forward, this site will be on <a
href="https://archive.casouri.cc/note/2022/domain-change/https:/archive.casouri.cc">archive.casouri.cc</a>.
I'll keep the .cat domain around for a few years. In the meantime the
.cat domain will redirect to the .cc domain by 301 redirect. The whole
site is archived on the Wayback Machine. Hopefully someone in the future
clicking on my .cat link knows about Wayback Machine and can view the
page.</p><p>If you have a link to the .cat domain, you might
want to edit the link to point to the new domain. Sorry for the
incovenience! Fortunately there are so few links to my site
:-)</p>NAT traversal: STUN, TURN, ICE, what do they actually do?urn:uuid:84e3d816-a8c6-11ec-bcbd-af8069bc83612022-03-20T20:26:00.00-05:00<p>When searching for NAT traversal I found
all these protocols but no one can tell me what do they essentially do to
traverse NAT, surely not by magic? Turns out it’s conceptually very
simple.</p><p>What NAT traversal does is not really “punching
holes” on the NAT, or delivering message through some tunnel, or some
demotic portals, but to simply find the public address:port that can can
reach <em>me</em>.</p><p>If I’m behind a NAT or
even multiple NAT’s, my packets are relayed by these NAT’s and they
appear on the public Internet at the out-most NAT’s address and port. And
reply packets going to that address:port are relayed back to me. So, in
some sense, I still got a public address:port that can reach me on the
public Internet. The purpose of NAT traversal is to find that public
address:port.</p><p>That’s basically what the initial/classic
STUN (<a
href="https://archive.casouri.cc/note/2022/nat-what-do-they-do/https:/datatracker.ietf.org/doc/html/rfc3489"><span
class="oldstyle-num">RFC 3489</span></a>) does. You send a
STUN server a message, the STUN server looks at the source IP address and
port of the IP packet, and reply that back to you. Voilà, you know you
public <code>address:port</code>!</p><p>Sometimes
having that address:port isn’t enough, because many NAT poses <a
id="footref:extra" class="footref-anchor obviously-a-link"
aria-label="Jump to footnote" href="#footdef%3Aextra">extra
restrictions<sup class="inline-footref">1</sup></a>.
Then we have to resort to having a public-visible relay server in the
middle, which is what TURN (<a
href="https://archive.casouri.cc/note/2022/nat-what-do-they-do/https:/datatracker.ietf.org/doc/html/rfc5766"><span
class="oldstyle-num">RFC 5766</span></a>)
does.</p><div id="footdef:extra" class="footdef"><div
class="def-footref obviously-a-link"><a aria-label="Jump back to
main text" href="#footref%3Aextra">1</a></div><div
class="def-footdef">Some NAT wouldn’t let a packet from an external
host through if the host inside never sent a packet to that external host
before. There are many ways a NAT could make your life difficult, check
out “full cone”, “restricted cone”, “symmetric NAT”,
etc.</div></div><p>As time goes by, STUN and TURN turns
out to be still not enough. For one, one can usually find multiple
address that could possibly work, but then which one to use? Eg, maybe a
host has an IP assigned by a VPN, if the other host is also in the VPN,
we should use this IP over the others; similarly, if the other host is in
the same LAN, we should use the local IP; even over NAT, there could be
multiple IP’s that can reach us.</p><p>ICE fills that gap. It
gathers a bunch of <code>address:port</code>’s that possibly
works (through STUN messages with STUN servers), sorts them by
preference, and <a id="footref:trickle" class="footref-anchor
obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Atrickle">tries them one-by-one according to some
algorithm<sup class="inline-footref">2</sup></a>, and
reports to you the best one. If none works, it tries to establish a relay
through TURN.</p><div id="footdef:trickle"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Atrickle">2</a></div><div
class="def-footdef">Or even gather candidates and try them out in the
same time, instead of waiting for full candidates list before trying each
out. This speeds up establishing connections and is called trickle
ICE.</div></div><p>And here is where the new STUN comes
in. People threw away the algorithm for finding
<code>address:port</code> in classic STUN, and kept and
extended the STUN message format. Now ICE runs a more thorough algorithm
that uses STUN messages to communicate with STUN servers. And the new
STUN (<a
href="https://archive.casouri.cc/note/2022/nat-what-do-they-do/https:/datatracker.ietf.org/doc/html/rfc5389"><span
class="oldstyle-num">RFC 5389</span></a>) just defines the
STUN message format. There is a even newer version (<a
href="https://archive.casouri.cc/note/2022/nat-what-do-they-do/https:/datatracker.ietf.org/doc/html/rfc8489"><span
class="oldstyle-num">RFC 8489</span></a>) that updated
<span class="oldstyle-num">RFC 5389</span> slightly, but with
no fundamental changes.</p><p>Similarly, TURN is updated in
<a
href="https://archive.casouri.cc/note/2022/nat-what-do-they-do/https:/datatracker.ietf.org/doc/html/rfc8656"><span
class="oldstyle-num">RFC 8656</span></a> and now is a
message protocol used by ICE rather than a standalone
solution.</p>Using Fontsets in Emacsurn:uuid:336b8c2c-4d8b-11ec-967b-17e83717b0eb2021-11-24T17:01:00.00-05:00<h2 id="Fontset"
class="section">Fontset?</h2><p>Fontset is a feature of
Emacs that allows you to bundle together multiple fonts and use them as a
single font, such that it covers more characters than a single font could
have. For example, you can combine a Latin font, a Greek font and a
Chinese font together.</p><p>With fontsets, we can use
different Unicode fonts for different faces. For example, serif Latin and
Chinese font for a “serif” face, and sans serif Latin and Chinese font
for a “sans” face. Without fontsets, we can only set different Latin
fonts to faces and use a single fall-back Chinese
font.</p><p><img class="half" alt="A graph showing
different fonts with different faces"
src="https://archive.casouri.cc/note/2021/fontset/fonts%20&%20faces.svg"/></p><h2
id="Create%20a%20fontset" class="section">Create a
fontset</h2><p>A fontset is recognized by its name. Each
fontset has two names, one short and one long. The short name looks like
<code>fontset-xxx</code>. The long name is a <a
href="https://archive.casouri.cc/note/2021/fontset/https:/wiki.archlinux.org/title/X_Logical_Font_Description">X
Logical Font Description</a> with last two fields being
<code>fontset</code> and <code>xxx</code>. For
example,</p><pre class="code-block">-*-ibm plex
mono-medium-*-*-*-13-*-*-*-*-*-fontset-my
fontset</pre><p>Emacs come with three fontsets by default:
<code>fontset-startup</code>,
<code>fontset-standard</code> and
<code>fontset-default</code>. We only care about
<code>fontset-default</code>; it is the ultimate fall-back
when Emacs cannot find a font to display a character. But more on that
later.</p><p>To create a fontset, you can use
<code>create-fontset-from-fontset-spec</code> and pass it a
bunch of X Logical Font Descriptions, each for a font you want to
include. I find that tedious. Instead, I like to create a fontset with a
single ASCII font and use <code>set-fontset-font</code> to
add other fonts later, like this:</p><pre
class="code-block">(create-fontset-from-fontset-spec (font-xlfd-name
(font-spec :family "IBM Plex Mono" :size 13 :registry "fontset-my
fontset")))</pre><p>Make sure you put the short fontset name
under the <code>:registry</code> spec. The code above creates
the fontset, and returns its long name,</p><pre
class="code-block">-*-ibm plex mono-*-*-*-*-13-*-*-*-*-*-fontset-my
fontset</pre><p>Now we can add a Chinese font and a Greek
font:</p><pre class="code-block">(set-fontset-font
"fontset-my fontset" 'han (font-spec :family "Source Han Serif" :size
12)) (set-fontset-font "fontset-my fontset" 'greek (font-spec :family
"Academica"))</pre><p>If you are not familiar with
<code>set-fontset-font</code>, <a
href="https://archive.casouri.cc/note/2021/fontset/http:/idiocy.org/emacs-fonts-and-fontsets.html"><em>Emacs,
fonts and fontsets</em></a> is a good read.</p><h2
id="Apply%20a%20fonset" class="section">Apply a
fonset</h2><p>Although the manual says we can use a fontset
wherever a font is appropriate, it is not entirely true. If you pass your
fontset through the <code>:font</code> attribute in
<code>set-face-attribute</code>, <a
id="footref:takes-ascii" class="footref-anchor obviously-a-link"
aria-label="Jump to footnote" href="#footdef%3Atakes-ascii">Emacs
takes the ASCII font from the fontset and only uses the ASCII font for
the face<sup class="inline-footref">1</sup></a>. The
real way to do it is to use the undocumented
<code>:fontset</code> attribute:</p><pre
class="code-block">(set-face-attribute 'some-face nil :fontset
"fontset-my fontset")</pre><p>That’s not all. While the above
code works for most faces, setting <code>:fontset</code> for
<code>default</code> will not work as you expected, because
Emacs again <a id="footref:default" class="footref-anchor
obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Adefault">only takes the ASCII font, even if you use
the <code>fontset</code> attribute<sup
class="inline-footref">2</sup></a>. So don’t set the
fontset for the <code>default</code> face; instead, just
modify <code>fontset-default</code> (it’s the ultimate
fall-back fontset we mentioned earlier) for Unicode fonts, and use
whatever method you like for ASCII font. If you read <a
href="https://archive.casouri.cc/note/2021/fontset/http:/idiocy.org/emacs-fonts-and-fontsets.html"><em>Emacs,
fonts and fontsets</em></a>, you’ll know we can modify
<code>fontset-default</code> by either</p><pre
class="code-block">(set-fontset-font "fontset-default"
...)</pre><p>or</p><pre
class="code-block">(set-fontset-font t
...)</pre><p>Technically you could set the
<code>font</code> attribute of a frame to a fontset by
<code>set-frame-font</code> and it works fine. But as soon as
you change any font-related attributes in
<code>default</code> face, like font size, your fontset in
the frame attribute will be overwritten by the font derived from
<code>default</code> face. So the best way is still to just
modify <code>fontset-default</code>.</p><div
id="footdef:takes-ascii" class="footdef"><div class="def-footref
obviously-a-link"><a aria-label="Jump back to main text"
href="#footref%3Atakes-ascii">1</a></div><div
class="def-footdef">According to <a
href="https://archive.casouri.cc/note/2021/fontset/https:/github.com/emacs-mirror/emacs/blob/11e5c7d8ca58cc946930048b5c88c8f582d4d5d8/src/xfaces.c#L3391">the
source</a>.</div></div><div id="footdef:default"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Adefault">2</a></div><div
class="def-footdef">Basically, if the face is
<code>default</code>,
<code>set-face-attribute</code> calls
<code>set_font_frame_param</code> (<a
href="https://archive.casouri.cc/note/2021/fontset/https:/github.com/emacs-mirror/emacs/blob/11e5c7d8ca58cc946930048b5c88c8f582d4d5d8/src/xfaces.c#L3514">source</a>),
which only looks at the <code>:font</code> attribute (<a
href="https://archive.casouri.cc/note/2021/fontset/https:/github.com/emacs-mirror/emacs/blob/11e5c7d8ca58cc946930048b5c88c8f582d4d5d8/src/xfaces.c#L3685">source</a>).</div></div><h2
id="Further%20reading" class="section">Further
reading</h2><ul><li>Command
<code>list-fontsets</code> lists all the defined
fontsets.</li><li>Command
<code>describe-fontset</code> shows which font is each
character assigned to in a fontset.</li><li>Manual page:
<a
href="https://archive.casouri.cc/note/2021/fontset/https:/www.gnu.org/software/emacs/manual/html_node/emacs/Fontsets.html"><em>Fontsets,
Emacs User Manual</em></a></li><li>Another manual
page: <a
href="https://archive.casouri.cc/note/2021/fontset/https:/www.gnu.org/software/emacs/manual/html_node/elisp/Fontsets.html"><em>Fontsets,
Emacs Lisp Manual</em></a></li></ul>Code Page 437urn:uuid:97279de0-28c5-11ec-b9e1-e3d1f57bb2952021-10-23T00:11:00.00-05:00<p>So I was installing a new OS on my desktop
machine, and for some technical reasons I need to install the OS
manually. That means typing in a console. I couldn’t help but wonder:
what the font is it showing?</p><figure><img
alt="Screenshot of the console"
src="https://archive.casouri.cc/note/2021/code-page-437/console.jpeg"/>
<figcaption>I was typing in
this</figcaption></figure><p>Turns out the typeface
isn’t even a typeface. It is a encoding that extends ASCII. It maps 8-bit
patterns to characters. For example, <code>10000110</code>
corresponds to “å”. According to Wikipedia, It is the“standard character
set of the original IBM PC”, and it “remains the primary set in the core
of any EGA and VGA-compatible graphic cards”. Basically this is the most
basic font on a personal computer, stored directly in
hardware.</p><p>This character set is supposed to contain
many characters including fancy ones like “⌠”, “☺”, “§”, etc. But my
graphic card is missing most of the non-basic characters. (How
disappointing!)</p><figure><img alt="A screenshot
specimen"
src="https://archive.casouri.cc/note/2021/code-page-437/specimen.jpeg"/>
<figcaption>Many characters are
missing</figcaption></figure><p>I don’t think this font
is pretty or anything. What makes it so interesting to me is that it is
such ubiquitous yet most people never notice it. Next time <a
id="footref:PC" class="footref-anchor obviously-a-link" aria-label="Jump
to footnote" href="#footdef%3APC">when your PC starts up or
crashes<sup class="inline-footref">1</sup></a>, see if
you can spot any message printed in this font.</p><p>You can
even download the font file for this font: <a
href="https://archive.casouri.cc/note/2021/code-page-437/https:/cp437.github.io"><em>Code
Page 437</em></a>.</p><p>PS. this makes me wonder
if Mac has something similar, and sure enough, there is. <a
href="https://archive.casouri.cc/note/2021/code-page-437/https:/apple.stackexchange.com/questions/157038/what-font-is-used-during-verbose-boot-mode">Someone
asked about it on StackExchange</a>. I bet even less people know
about this one. I for one have never seen it despite using a MacBook for
years (That’s probably a good thing, as one only see it when something
goes hopelessly wrong.)</p><p>PPS. On Linux, you can drop
yourself into a console by typing Ctrl+Alt+F1/F2/etc. Usually that screen
is printed in <a
href="https://archive.casouri.cc/note/2021/code-page-437/http:/terminus-font.sourceforge.net">Terminus</a>.</p><div
id="footdef:PC" class="footdef"><div class="def-footref
obviously-a-link"><a aria-label="Jump back to main text"
href="#footref%3APC">1</a></div><div
class="def-footdef">You still use a PC, do
you?</div></div>Dutch 801 Headlineurn:uuid:78a32c96-227f-11ec-9fc2-63ecfe8296922021-09-30T23:18:00.00-05:00<p>Today’s typeface isn’t really interesting
in itself, but in the way I came across it. It’s a long story, are you
ready? Ok, so I was reading assigned papers for my OS class, and I
started on this one:</p><p><img alt="A clip of the paper"
src="https://archive.casouri.cc/note/2021/dutch-801/hydra.jpeg"/></p><p>The
title immediately caught my attention: it’s a elegant, graceful font. So
I clipped an image and searched on myfont.com, and it turns out to be ...
Dutch 801 Headline.</p><p>The end.</p><p>I don’t
know about you, but isn’t it a rather strange name for a typeface? Why
Dutch? Why 801? I still don’t know the answer. Anyway, I think its a cool
name. Maybe the one who named it thought the
same.</p><p>Despite its interesting name, information about
this typeface is quite scarce. I only know it is Bitstream’s version of
Times New Roman (ie, clone). I was kind of surprised when I found out,
because I never associated Times New Roman with elegance. Maybe enlarging
a font naturally releases it from its humble form, and brings out its
gracefulness.</p><figure><img alt="A larger clip of the
title"
src="https://archive.casouri.cc/note/2021/dutch-801/hydra-large.jpeg"/>
<figcaption>The title in its full
glory</figcaption></figure><figure><img alt="Clip
for another title"
src="https://archive.casouri.cc/note/2021/dutch-801/tenex.jpeg"/>
<figcaption>Another article
title</figcaption></figure><p>As for the body text, I
can only assume it to be Dutch 801 Text. I didn’t bother to check
though.</p><p><img alt="A clip of the body text"
src="https://archive.casouri.cc/note/2021/dutch-801/body.jpeg"/></p>Academicaurn:uuid:8a5678de-2181-11ec-9b77-8b26fea6a6932021-09-29T17:01:00.00-05:00<p><a
href="https://archive.casouri.cc/note/2021/academica/https:/www.stormtype.com/families/academica">Academica</a>
is a typeface I found out when reading <a
href="https://archive.casouri.cc/note/2021/academica/https:/aeon.co"><em>aeon</em></a>
(a digital magazine in Science and Humanities). Academica is designed by
Josef Týfa for scientific texts. The original design was cut and cast in
metal in 1968, and in 2003, Týfa and František Štorm worked together to
rework it for digital printing.</p><p>Academica shares some
similarities with Charter in tall x‑height and emphasize on legibility,
but the similarity pretty much ends there. Comparing to Charter,
Academica is considerably blacker. And comparing to Charter’s <a
id="footref:stoic" class="footref-anchor obviously-a-link"
aria-label="Jump to footnote" href="#footdef%3Astoic">stoic stint on
curves<sup class="inline-footref">1</sup></a>,
Academica is lavishly rounded, tapered, bent, squished and stretched. In
fact, I don’t even know why am I comparing it to Charter, Academica
reminds me more of another typeface (that I dig), <a
href="https://archive.casouri.cc/note/2021/academica/https:/en.wikipedia.org/wiki/Cooper_Black">Cooper
Black</a>.</p><p><img alt="A specimen for some
lower-case Latin letters"
src="https://archive.casouri.cc/note/2021/academica/specimen1.png"/></p><p>The
alien-looking 0 is perhaps the most salient character (pun intended) in
Academica. Instead of simply narrowing 0 to distinguish it from capital
O, Academica “flipped” it such that the horizontal stroke is thicker than
the vertical. The 0 is really the culmination of the overall vibe of
Academica—little roundish goofiness here and there, slightly throwing the
reader off; but when you zoom away, you see a legible, realistic academic
typeface.</p><p><img alt="A specimen for text “2001”"
src="https://archive.casouri.cc/note/2021/academica/specimen2.png"/></p><p>I
love the color of Academica, it’s thiccc ;-) Use it for body text, and
the dense, full color is beautiful. Looking at a block of Academica, you
can almost feel the energy of live imbued in every corner. Also, the tall
x-height means you can pack more lines into a page, increasing the
information density.</p><p>Overall, Academica feels humane to
me. It is a practical typeface for serious scientific publications, but
in the same time has its very own quirky character. I’m very fond of it.
It isn’t that expensive either. If you buy it on <a
href="https://archive.casouri.cc/note/2021/academica/https:/www.myfonts.com">myfonts.com</a>,
each font costs $44 (at the time of writing). <a id="footref:need"
class="footref-anchor obviously-a-link" aria-label="Jump to footnote"
href="#footdef%3Aneed">So regular, italic and bold<sup
class="inline-footref">2</sup></a> combined costs $132.
That’s more than a cup of coffee, but still less than 20 cups (I
think?)</p><p>Some more
specimen:</p><figure><img alt="A specimen for body text"
src="https://archive.casouri.cc/note/2021/academica/specimen3.png"/>
<figcaption>Academica Text (Regular) in body
text</figcaption></figure><figure><img alt="A
specimen for Light/Book weight"
src="https://archive.casouri.cc/note/2021/academica/specimen4.png"/>
<figcaption>Academica Book (Light) in slightly larger
size</figcaption></figure><p>Further
reading:</p><ul><li><a
href="https://archive.casouri.cc/note/2021/academica/https:/fontsinuse.com/typefaces/13032/academica"><em>Fonts
in Use: Academica</em></a></li><li>The <a
href="https://archive.casouri.cc/note/2021/academica/StormType-AcademicaSpecimenA4.pdf">Official
specimen</a></li></ul><div id="footdef:stoic"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Astoic">1</a></div><div
class="def-footdef">Of course, there is nothing bad about stoic
outlines. Matthew Carter designed Charter for low-resolution laser
printers (which will muddle any delicate detail on the character), and
aimed for economic use of curves to accommodate low-memory computers and
printers. More over, the crisp, direct, clean outline is actually one of
Charter’s virtues. (Edit: Actually, economic use of curves turns out to
be a premature optimization, but Carter liked the style anyway and kept
the design.)</div></div><div id="footdef:need"
class="footdef"><div class="def-footref obviously-a-link"><a
aria-label="Jump back to main text"
href="#footref%3Aneed">2</a></div><div
class="def-footdef">Regular, italic and bold are really all you need,
sometimes you don’t even need bold. If you want to use Academica in a
larger size (larger than 13pt), then Light, Light italic and Regular is
also a good combination.</div></div>RFC: Emacs tree-sitter integrationurn:uuid:484e573e-207f-11ec-bd91-975a51a5f3f12021-09-28T10:12:00.00-05:00<p><a
href="https://archive.casouri.cc/note/2021/emacs-tree-sitter/https:/tree-sitter.github.io/tree-sitter">Tree-sitter</a>
is a incremental parser that can provide a concrete syntax tree for the
source code and is fast enough to parse on each key press. It has
supported a wide range of languages, and support for more languages is on
the way.</p><p>I’ve been working on a integration of
tree-sitter library into Emacs’ core. The integration consists of two
parts, first the direct translate of tree-sitter’s API, second the
integration with Emacs’ font-lock and indent system. The first part is
completed and is rather uncontentious. I’d appreciate comments on the
second: Is the interface easy to understand? Is it easy to use? Is it
flexible enough for every language?</p><p>Whether you are a
major mode author or just a interested Emacs user, I invite you to try
hacking with this tree-sitter integration—recreate existing major mode
features (font-lock, indent), create new features (structured editing,
etc)—and tell me how well it works. Better yet, provide some suggestions
on improving the interface.</p><h2
id="Building%20Emacs%20with%20tree-sitter%20support"
class="section">Building Emacs with tree-sitter
support</h2><h3 id="Install%20tree-sittter"
class="subsection">Install tree-sittter</h3><p>First,
install libtree-sitter, either by a package manager, or from
source:</p><pre class="code-block">git clone
https://github.com/tree-sitter/tree-sitter.git cd tree-sitter make make
install</pre><p>This should install libtree-sitter in
standard location.</p><h3 id="Build%20Emacs"
class="subsection">Build Emacs</h3><p>Then, build Emacs
from my GitHub repository. Make sure you clone the
<code>ts</code> branch.</p><pre
class="code-block">git clone https://github.com/casouri/emacs.git
--branch ts ./autogen.sh ./configure make</pre><p>No need for
special configure flags, tree-sitter is enabled automatically if
libtree-sitter is present on the system. Now Emacs can be started
by</p><pre class="code-block">src/emacs</pre><h3
id="Get%20language%20definitions" class="subsection">Get language
definitions</h3><p>To use tree-sitter features in any
meaningful way, we also need the language definition, eg,
libtree-sitter-c for C. I wrote a script for automatically retrieving and
compiling some of the libraries. The following commands</p><pre
class="code-block">git clone
https://github.com/casouri/tree-sitter-module.git cd tree-sitter-module
./batch-new.sh</pre><p>should produce libraries for C, JSON,
Go, HTML, JavaScript, CSS and Python and store them in
<code>dist</code> directory. From there you can copy these
libraries to a standard path, or add that directory to
<code>LD_LIBRARY_PATH</code>.</p><p>You can also
find pre-built libraries in the release page: <a
href="https://archive.casouri.cc/note/2021/emacs-tree-sitter/https:/github.com/casouri/tree-sitter-module/releases/tag/v2,0"><em>tree-sitter-module
release v2.0</em></a>.</p><h2
id="Basic%20tree-sitter%20features" class="section">Basic tree-sitter
features</h2><p>I suggest reading the tree-sitter node in the
manual first, it covers how to create a parser, how to retrieve a node,
how to pattern match nodes, and more. You can access the manual by
typing</p><pre class="code-block">C-h i m elisp RET g Parsing
Program Source RET</pre><p>The command(s) above opens the
Info reader, goes to <em>Elisp Reference Manual</em>, and
opens the “Parsing Program Source” node, which contains manual for
tree-sitter. Alternatively, you can read <a
href="https://archive.casouri.cc/note/2021/emacs-tree-sitter/Parsing-Program-Source.html">the
tree-sitter node</a> that I clipped from the HTML
manuel.</p><p>Once you’ve read the manual, you can
<code>(require 'tree-sitter)</code> and hack
away!</p><p>The manual only documents basic features of
tree-sitter, leaving out font-lock and indent integration, because I
expect the latter to change. They are instead documented
below.</p><h2 id="Font-lock%20interface"
class="section">Font-lock interface</h2><p>(From now on, I
assume you have read the manual and I will use concepts introduced in the
manual without explanation.)</p><p>If you are familiar with
font-lock in Emacs, you know it is primarily configured by
<code>font-lock-defaults</code>: major mode sets this
variable with language-specific configuration, font-lock takes that
variable and populate <code>font-lock-keywords</code>, which
directly defines the pattern to fontify.</p><h3
id="tree-sitter-font-lock-settings"
class="subsection"><code>tree-sitter-font-lock-settings</code></h3><p><a
id="footref:ts-name" class="footref-anchor obviously-a-link"
aria-label="Jump to footnote"
href="#footdef%3Ats-name">Tree-sitter<sup
class="inline-footref">1</sup></a> provides two analogues
variables, <code>tree-sitter-font-lock-defaults</code> and
<code>tree-sitter-font-lock-settings</code>.
<code>tree-sitter-font-lock-settings</code> is a list of
<code>SETTING</code>s where each
<code>SETTING</code> looks like</p><pre
class="code-block">(LANGUAGE
QUERY)</pre><p><code>LANGUAGE</code> is the
language this setting should use, and <code>QUERY</code> is
either a string or a sexp query. Each capture name in
<code>QUERY</code> is either a face name, in which case the
captured node is fontified in that face, or a function name, in which
case the captured node is passed to the function for fontification.
Specifically, the function is passed three arguments <code>(BEG END
NODE)</code>, where <code>BEG</code> and
<code>END</code> is the beginning and end position of the
node in the buffer, for convenience.</p><p>An example
<code>SETTING</code> for C is</p><pre
class="code-block">(tree-sitter-c ; LANGUAGE ((null)
@font-lock-constant-face (true) @font-lock-constant-face (false)
@font-lock-constant-face)) ; QUERY</pre><div
id="footdef:ts-name" class="footdef"><div class="def-footref
obviously-a-link"><a aria-label="Jump back to main text"
href="#footref%3Ats-name">1</a></div><div
class="def-footdef">From now on, “tree-sitter” refers to the Emacs
integration of tree-sitter.</div></div><h3
id="tree-sitter-font-lock-defaults"
class="subsection"><code>tree-sitter-font-lock-defaults</code></h3><p>Tree-sitter
font-lock, like font-lock, support fontification at different levels of
decoration (controlled by
<code>font-lock-maximum-decoration</code>). And this is the
primary purpose of
<code>tree-sitter-font-lock-defaults</code>. Its value is a
list of</p><pre class="code-block">(DEFAULT :KEYWORD
VALUE...)</pre><p>Where each <code>DEFAULT</code>
may be a symbol or a list of symbols. The symbol should be either a
variable containing <code>(LANGUAGE QUERY)</code>, or a
function that returns that. If <code>DEFAULT</code> is a
list, each symbol corresponds to a decoration level. For example, if I
want to implement three levels of decoration for C, I would populate
<code>tree-sitter-font-lock-defaults</code>
with</p><pre class="code-block">(((c-font-lock-settings-1
c-font-lock-settings-2 c-font-lock-settings-3) :KEYWORD
VALUE...))</pre><p>where
<code>c-font-lock-settings-1</code> would contain,
say,</p><pre class="code-block">(tree-sitter-c ((null)
@font-lock-constant-face (true) @font-lock-constant-face (false)
@font-lock-constant-face))</pre><p>for those who need no
more. And the other two levels could be for the rest mortals. As for
<code>:KEYWORD</code> and <code>VALUE</code>,
they are analogues to that in
<code>font-lock-defaults</code>, used for specifying other
configurations. Currently they are not used for tree-sitter
font-lock.</p><p>To enable tree-sitter font-lock, a major
mode should first assign
<code>tree-sitter-font-lock-defaults</code>, then call
<code>tree-sitter-font-lock-enable</code>. For
example,</p><pre class="code-block">(define-derived-mode
ts-c-mode prog-mode "tree-sitter C" (setq-local
tree-sitter-font-lock-defaults '((ts-c-tree-sitter-settings-1)))
(tree-sitter-enable-font-lock))</pre><h2 id="Indentation"
class="section">Indentation</h2><p>In Emacs, indentation
is provided by <code>indent-line-function</code>. Tree-sitter
provides a convenient system,
<em>tree-sitter-simple-indent</em>, to simplify the
implementation of a indenting function. To use it, bind
<code>indent-line-function</code> to
<code>tree-sitter-indent</code>, and fill in indentation
configurations in
<code>tree-sitter-simple-indent-rules</code>.</p><p><code>tree-sitter-simple-indent-rules</code>
is a list of rules, and each rule looks like</p><pre
class="code-block">(MATCHER ANCHOR OFFSET)</pre><p>When
indenting, <em>tree-sitter-simple-indent</em> finds the
largest node that starts at the beginning of the current line, and
matches it against each <code>MATCHER</code> in
<code>tree-sitter-simple-indent-rules</code>. If
<code>MATCHER</code> matches that node,
<code>ANCHOR</code> and <code>OFFSET</code>
determines how to indent—find the column of
<code>ANCHOR</code> (which represents a point), and add
<code>OFFSET</code> to it.</p><p>By now you must
be wondering what the heck is <code>MATCHER</code>. It is a
function that takes <code>(NODE PARENT BOL &rest
_)</code> as arguments, if the rule should apply to
<code>NODE</code>, it returns non-nil.
<code>PARENT</code> and <code>BOL</code>
(position of beginning of line) are provided just for convenience. The
“<code>&rest _</code>” part is required to allow the
possibility to extend the interface in the future.</p><p>This
function can do anything: check the type of that node, check the type of
its parent, check whether this node is the first child node of its
parent, etc. <code>ANCHOR</code> is also a function that
takes theses arguments, but it returns a point, the “anchor”. If the rule
determines that the node should be indented two columns inward comparing
to its parent, <code>ANCHOR</code> should return the start of
the parent node, and <code>OFFSET</code> should be
2.</p><p>For example, the following rule matches any line
that starts with the <code>null</code> keyword, and indents
the line inwards by two columns against the
<code>null</code>’s parent node.</p><pre
class="code-block">((lambda (n p bol &rest _) (equal
(tree-sitter-node-type n) "null")) ; MATCHER (lambda (n p bol
&rest _) (tree-sitter-node-start (tree-sitter-node-parent n))) ;
ANCHOR 2) ; OFFSET</pre><p>Of course, it is terribly tedious
to write out every <code>MATCHER</code> and
<code>ANCHOR</code> explicitly.
<em>tree-sitter-simple-indent</em> provides some predefined
<code>MATCHER</code> and <code>ANCHOR</code>
functions. Most of them are higher-order functions: they takes an
argument and returns a
function.</p><p><code>MATCHER</code>
presets:</p><dl><dt><code>(parent-is
TYPE)</code></dt><dd>Check that the parent has type
<code>TYPE</code>.</dd><dt><code>(node-is
TYPE)</code></dt><dd>Check that node has type
<code>TYPE</code>.</dd><dt><code>(match
NODE-TYPE PARENT-TYPE NODE-FIELD NODE-INDEX-MIN
NODE-INDEX-MAX)</code></dt><dd><code>NODE-TYPE</code>
checks for node’s type, <code>PARENT-TYPE</code> checks for
parent’s type, <code>NODE-FIELD</code> checks for the field
name for node int the parent, <code>NODE-INDEX-MIN</code> and
<code>NODE-INDEX-MAX</code> limits the node’s index in the
parent. Any argument left as nil are not checked. For example, to match
the node that is the first child and has a parent of type
<code>argument_list</code>, use<br/><code>(match
nil "argument_list" nil nil 0
0)</code></dd><dt><code>(query
QUERY)</code></dt><dd>Queries the parent with
<code>QUERY</code>. Matches if the node is captured by any
capture
name.</dd><dt><code>no-node</code></dt><dd>Matches
null node. When the current line is empty, there is no node at the
beginning, so the node is
nil.</dd></dl><p><code>ANCHOR</code>
presets:</p><dl><dt><code>first-child</code></dt><dd>Finds
the first sibling of node, ie, the first child of the
parent.</dd><dt><code>parent</code></dt><dd>Finds
the parent
node.</dd><dt><code>prev-sibling</code></dt><dd>Finds
node’s first
sibling.</dd><dt><code>no-indent</code></dt><dd>Do
nothing, don’t indent. This is useful for a indenting a line inside a
multiline string, where masterful inactivity is most
preferred.</dd><dt><code>prev-line</code></dt><dd>Find
the named node on the previous line. This can be used when indenting an
empty line: just indent like the previous
node.</dd></dl><h2 id="Some%20handy%20tools"
class="section">Some handy tools</h2><p>I have two handy
tools for you to work with tree-sitter more easily: first,
<code>tree-sitter-inspect-mode</code> will show the relevant
information of the node at point in the mode-line; second,
<code>tree-sitter-check-indent</code> can check the indent
result against a stock major mode. Check out their docstring for more
detail.</p><h2 id="Feedback"
class="section">Feedback</h2><p>You can send a message to
<a
href="https://archive.casouri.cc/note/2021/emacs-tree-sitter/https:/lists.gnu.org/mailman/listinfo/emacs-devel"><em>emacs-devel</em></a>,
or open an issue on the <a
href="https://archive.casouri.cc/note/2021/emacs-tree-sitter/https:/github.com/casouri/emacs">GitHub
repository</a>.</p><h2 id="An%20example"
class="section">An example</h2><p>All these must be pretty
confusing without seeing a concrete example, so here it is. This example
code is for a demo C major mode, <code>ts-c-mode</code>,
defined in the “<code>;;; Lab</code>” section in
<code>tree-sitter.el</code>. (Here is a <a
href="https://archive.casouri.cc/note/2021/emacs-tree-sitter/https:/github.com/casouri/emacs/blob/350ae9cc19e478f08468443843f63bdf005d9d92/lisp/tree-sitter.el#L640">link
to the file on
GitHub</a>.)</p><p>Indent:</p><pre
class="code-block">(defvar ts-c-tree-sitter-indent-rules
`((tree-sitter-c ;; Empty line. (no-node prev-line 0) ;; Function/struct
definition body {}. ((match nil "function_definition" "body") parent 0)
((node-is "field_declaration_list") parent 0) ;; Call expression.
((parent-is "call_expression") parent 2) ;; If-else. ((match nil
"if_statement" "condition") parent 2) ((match nil "if_statement"
"consequence") parent 2) ((match nil "if_statement" "alternative") parent
2) ((match nil "switch_statement" "condition") parent 2) ((node-is
"else") parent 0) ;; Switch case. ((parent-is "case_statement") parent 2)
((node-is "case_statement") parent 0) ;; { and }. ((node-is
"compound_statement") parent 2) ((node-is "}") parent 0) ;; Multi-line
string. ((parent-is "string_literal") no-indent 0) ;; List. ,@(cl-loop
for type in '("compound_statement" "initializer_list" "argument_list"
"parameter_list" "field_declaration_list") collect `((match nil ,type nil
0 0) parent 2) collect `((match nil ,type nil 1) first-sibling
0)))))</pre><p>Font-lock:</p><pre
class="code-block">(defvar ts-c-tree-sitter-settings-1 '(tree-sitter-c
((null) @font-lock-constant-face (true) @font-lock-constant-face (false)
@font-lock-constant-face (comment) @font-lock-comment-face
(system_lib_string) @ts-c-fontify-system-lib (unary_expression operator:
_ @font-lock-negation-char-face) (string_literal) @font-lock-string-face
(char_literal) @font-lock-string-face (function_definition declarator:
(identifier) @font-lock-function-name-face) (declaration declarator:
(identifier) @font-lock-function-name-face) (function_declarator
declarator: (identifier) @font-lock-function-name-face) (init_declarator
declarator: (identifier) @font-lock-variable-name-face)
(parameter_declaration declarator: (identifier)
@font-lock-variable-name-face) (preproc_def name: (identifier)
@font-lock-variable-name-face) (enumerator name: (identifier)
@font-lock-variable-name-face) (field_identifier)
@font-lock-variable-name-face (parameter_list (parameter_declaration
(identifier) @font-lock-variable-name-face)) (pointer_declarator
declarator: (identifier) @font-lock-variable-name-face) (array_declarator
declarator: (identifier) @font-lock-variable-name-face)
(preproc_function_def name: (identifier) @font-lock-variable-name-face
parameters: (preproc_params (identifier) @font-lock-variable-name-face))
(type_identifier) @font-lock-type-face (primitive_type)
@font-lock-type-face "auto" @font-lock-keyword-face "break"
@font-lock-keyword-face "case" @font-lock-keyword-face "const"
@font-lock-keyword-face "continue" @font-lock-keyword-face "default"
@font-lock-keyword-face "do" @font-lock-keyword-face "else"
@font-lock-keyword-face "enum" @font-lock-keyword-face "extern"
@font-lock-keyword-face "for" @font-lock-keyword-face "goto"
@font-lock-keyword-face "if" @font-lock-keyword-face "register"
@font-lock-keyword-face "return" @font-lock-keyword-face "sizeof"
@font-lock-keyword-face "static" @font-lock-keyword-face "struct"
@font-lock-keyword-face "switch" @font-lock-keyword-face "typedef"
@font-lock-keyword-face "union" @font-lock-keyword-face "volatile"
@font-lock-keyword-face "while" @font-lock-keyword-face "long"
@font-lock-type-face "short" @font-lock-type-face "signed"
@font-lock-type-face "unsigned" @font-lock-type-face "#include"
@font-lock-preprocessor-face "#define" @font-lock-preprocessor-face
"#ifdef" @font-lock-preprocessor-face "#ifndef"
@font-lock-preprocessor-face "#endif" @font-lock-preprocessor-face
"#else" @font-lock-preprocessor-face "#elif" @font-lock-preprocessor-face
)))</pre>Don’t Use Rubber Pin Backings on Backpacksurn:uuid:093e8b2e-1515-11ec-b47c-a70e3cf20eb92021-09-13T22:01:00.00-05:00<p>If you like enamel pins, you know there are
three types of backings: rubber, butterfly, and “secure/locking backing”.
I grew up with butterfly backings, but nowadays, when you buy a enamel
pin, more often than not, it comes with rubber
backings.</p><p>Rubber backings—they feel insecure at first
sight, but then the difficulty to remove one from the packaging might
give you a false sense of security. Let me tell you: don’t trust them. My
mistrust lost me a pin on my backpack. Thankfully I only lost one—the
others are at most loose or missing one of two backings. Still, that’s
enough proof that rubber backings are not suitable for surfaces that see
a lot of movement, for example, a backpack.</p><p>Butterfly
backings have their own problems: under stress or repeated use, they
might loose the little metal butterfly wings. On top of that, they aren’t
that much more secure than rubber backings. They won’t gradually loosen
by time like the rubber backings do, but they can come loose or break
under force.</p><p>That left the “secure/locking” backings.
They can be a bit of pain when putting on and taking off—you need to get
a feel for them. But they are secure. Buy a box of them from Amazon for a
couple bucks, and your precious pins won’t gone missing from your
backpack again.</p>自动处理网页里的全角引号和标点挤压urn:uuid:05c234e6-d6b3-11eb-b625-f744d82912722021-09-03T13:15:00.00-05:00<h2 id="%E5%85%A8%E8%A7%92%E5%BC%95%E5%8F%B7"
class="section">全角引号</h2><p>在 Unicode 里<span
class="full-width-mark">,</span>问号<span
class="full-width-mark">、</span>叹号<span
class="full-width-mark">、</span>各种括号都有全角半角两种版本<span
class="full-width-mark">,</span>各自有独立的编码<span
class="full-width-mark">;</span>但因为莫名的原因<span
class="full-width-mark">,</span>最常用的引号却不在此列<span
class="full-width-mark">。</span>中英混排的时候想要正确显示直角和半角的引号就很头疼<span
class="full-width-mark">;</span>搞不好的话<span
class="full-width-mark">,</span>中文里显示半角引号还不算太违和<span
class="full-width-mark">,</span>英文里蹦出来一个全角引号就太丑了<span
class="full-width-mark">。</span></p><p>CSS
没法自动区别什么时候用全角引号<span
class="full-width-mark">、</span>什么时候用半角<span
class="full-width-mark">,</span>只能靠标记<span
class="full-width-mark">。</span>好在还没复杂到需要手工标记的地步<span
class="full-width-mark">,</span>只要用程序检查引号前后的字是中文还是英文<span
class="full-width-mark">,</span>以此标记全角还是半角<span
class="full-width-mark">,</span>就基本不会出错<span
class="full-width-mark">。</span>我现在的办法是这样<span
class="full-width-mark">,</span>默认字体还是英文先中文后<span
class="full-width-mark">:</span></p><pre
class="code-block">body { font-family: Charter, Source Han Serif CN,
serif; }</pre><p>需要全角的引号用 <code>span</code>
标签包起来<span class="full-width-mark">:</span></p><pre
class="code-block"><span
class="full-width-quote">“</span></pre><p>然后用
CSS 指定中文字体<span
class="full-width-mark">:</span></p><pre
class="code-block">span.full-width-quote { font-family: Srouce Han
Serif CN, serif; }</pre><p>怎么区别一个引号应该全角还是半角呢<span
class="full-width-mark">?</span>我用了一个简单的判断方法<span
class="full-width-mark">:</span>如果前或后紧挨着中文字符<span
class="full-width-mark">,</span>就全角<span
class="full-width-mark">;</span>如果前后都不是中文字符<span
class="full-width-mark">,</span>就半角<span
class="full-width-mark">。</span>我目前还没发现这个简单判断不够用的情况<span
class="full-width-mark">。</span>这样一来还需要判断一个字符是不是中文<span
class="full-width-mark">,</span>最简单的办法是检查字符的 Unicode codepoint
在不在中文区间内<span class="full-width-mark">。</span>常用汉字和标点符号在
<code>0x4E00</code>–<code>0x9FFF</code> 和
<code>0x3000</code>–<code>0x303F</code>
两个区间里<span class="full-width-mark">,</span>检查这两个就够了<span
class="full-width-mark">,</span>其他的区间里都是生僻字<span
class="full-width-mark">。</span></p><h2
id="%E6%A0%87%E7%82%B9%E6%8C%A4%E5%8E%8B"
class="section">标点挤压</h2><p>全角引号搞好了<span
class="full-width-mark">,</span>又会贪心标点挤压<span
class="full-width-mark">。</span>没有标点挤压的时候<span
class="full-width-mark">,</span>几个标点排在一起确实不大好看<span
class="full-width-mark">:</span></p><figure><img
class="half300" alt="没有标点挤压的样子"
src="https://archive.casouri.cc/note/2021/full-width-quote/%E4%BE%8B%E5%AD%901.png"/>
<figcaption><a
href="https://archive.casouri.cc/note/2021/full-width-quote/https:/archive.casouri.cat/rock/day/day-48/index.html">余日摇滚第48期</a></figcaption></figure><p>挤压以后就不那么空了<span
class="full-width-mark">:</span></p><figure><img
class="half300" alt="有标点挤压的样子"
src="https://archive.casouri.cc/note/2021/full-width-quote/%E4%BE%8B%E5%AD%902.png"/>
<figcaption><a
href="https://archive.casouri.cc/note/2021/full-width-quote/https:/archive.casouri.cat/rock/day/day-48/index.html">余日摇滚第48期</a></figcaption></figure><p>原理是设置
CSS 属性 <code>font-feature-settings: "halt"</code><span
class="full-width-mark">,</span>启用 OpenType 的
<code>halt</code> 特性<span
class="full-width-mark">。</span>和全角引号一样<span
class="full-width-mark">,</span>用程序自动识别需要挤压的标点<span
class="full-width-mark">,</span>包在 <code>span</code>
标签里<span class="full-width-mark">。</span>要注意的是<span
class="full-width-mark">,</span>你用的字体要有
<code>halt</code> 这个特性才行<span
class="full-width-mark">,</span>我用的思源宋体是有的<span
class="full-width-mark">。</span></p><p>具体怎么挤压标点符号<span
class="full-width-mark">,</span>我没找到现成的标准或者算法<span
class="full-width-mark">,</span>下面是我的方法<span
class="full-width-mark">。</span>这个方法并不完整<span
class="full-width-mark">,</span>只处理比较常见的情况<span
class="full-width-mark">,</span>但对我来说够用了<span
class="full-width-mark">。</span>如果读者知道更好的算法<span
class="full-width-mark">,</span>请一定告诉我<span
class="full-width-mark">。</span></p><p>首先<span
class="full-width-mark">,</span>能挤压的标点符号可以分为三类<span
class="full-width-mark">:</span>靠左<span
class="full-width-mark">,</span>靠右<span
class="full-width-mark">,</span>居中<span
class="full-width-mark">:</span></p><figure><img
alt="各种类型的标点符号"
src="https://archive.casouri.cc/note/2021/full-width-quote/%E5%90%84%E7%B1%BB%E7%AC%A6%E5%8F%B7.png"/>
<figcaption><a
href="https://archive.casouri.cc/note/2021/full-width-quote/https:/www.w3.org/TR/2020/WD-clreq-20201101"><span
class="full-width-mark">《</span>中文排版需求<span class="squeeze
full-width-mark">》</span><span
class="full-width-mark">,</span>W3C Working Draft 01 November
2020<span class="full-width-mark">,</span>3.1.6
标点符号的宽度调整<span
class="full-width-mark">,</span>有修改</a></figcaption></figure><p>我们不考虑居中的符号<span
class="full-width-mark">,</span>因为简体中文普遍不用<span
class="full-width-mark">,</span>而我以简体中文写作<span
class="full-width-mark">。</span>程序从头到尾遍历每个字符<span
class="full-width-mark">,</span>决定每个字符要不要挤压<span
class="full-width-mark">。</span>挤不挤压取决于这个字符和其前后的字符<span
class="full-width-mark">,</span>以伪码表达为<span
class="full-width-mark">:</span></p><pre
class="code-block">遍历 字符: 如果 此字符为靠左标点 且 后一字符为标点: 挤压此字符 如果 此字符为靠右标点 且
前一字符为靠右标点: 挤压此字符</pre><p>这个算法运行的结果是这样<span class="squeeze
full-width-mark">:</span><span
class="full-width-mark">(</span><span class="squeeze
full-width-mark">(</span>文字<span class="squeeze
full-width-mark">)</span><span class="squeeze
full-width-mark">)</span><span class="squeeze
full-width-mark">,</span><span
class="full-width-mark">(</span>文<span class="squeeze
full-width-mark">)</span><span
class="full-width-mark">「</span>字<span class="squeeze
full-width-mark">」</span><span
class="full-width-mark">。</span></p><p><a
id="footref:subset" class="footref-anchor obviously-a-link"
aria-label="Jump to footnote" href="#footdef%3Asubset">如果你用
<code>pyftsubset</code> 压缩过字体文件<sup
class="inline-footref">1</sup></a><span
class="full-width-mark">,</span>注意它默认会把
<code>halt</code> 这样的 OTF 特性扔掉<span
class="full-width-mark">,</span>这样一来即使加上挤压标签也没有效果<span
class="full-width-mark">。</span>压缩的时候加上
<code>--layout-features='*'</code> 这个选项就可以保留所有 OTF
特性了<span class="full-width-mark">。</span>也可以用
<code>--layout-features='halt'</code> 只保留
<code>halt</code> 特性<span
class="full-width-mark">。</span></p><div
id="footdef:subset" class="footdef"><div class="def-footref
obviously-a-link"><a aria-label="Jump back to main text"
href="#footref%3Asubset">1</a></div><div
class="def-footdef">参见 <a
href="https://archive.casouri.cc/note/2019/reduce-font-loading-time-in-my-blog/index.html"><em>Reduce
Font Loading Time in My Blog</em></a><span
class="full-width-mark">。</span></div></div><h2
id="%E7%A0%B4%E6%8A%98%E5%8F%B7"
class="section">破折号</h2><p>我还发现破折号有时会显示成 em dash<span
class="full-width-mark">(</span>因为破折号在 Unicode 里其实就是 em
dash<span class="squeeze full-width-mark">)</span><span
class="full-width-mark">。</span>解决方法和全角引号一样<span
class="full-width-mark">,</span>包上全角的
<code>span</code> 标签就可以了<span
class="full-width-mark">——</span>这样就能正确显示破折号<span
class="full-width-mark">。</span></p>