scratch

Image sealing with composefs

Fri, 05 Jun 2026 08:00:00 +0000

Composefs achieves whole-filesystem integrity verification through image sealing: a single cryptographic digest authenticates an entire filesystem, covering both file contents and metadata (directory structure, permissions, ownership, symlinks, and xattrs). This goes further than fs-verity alone, which can only verify individual file contents, and avoids the fixed-partition requirement of dm-verity. The mechanism combines an EROFS image for metadata, a content-addressed object store for file data, and overlayfs with verity=require to enforce integrity checks on every file access at the kernel level.

GitChronicler: Write commit messages with AI

Wed, 30 Apr 2025 16:12:00 +0200

I started working on GitChronicler mostly to learn how I could integrate AI into my workflow in a way that would actually spare me doing boring stuff, like writing the git commit message. The tool feeds a patch to a language model via the OpenRouter API and gets back a commit message that reflects what the code actually does — saving the mechanical step of describing changes that are already fully visible in the diff, while still leaving the developer in control of what gets committed.

Why do I have two /sys/fs/cgroup in my container

Fri, 26 Jan 2024 14:12:00 +0200

It happened a few times in the past that users wonder why they see two /sys/fs/cgroup mounts in their unprivileged container. When working with unprivileged containers in Podman, users often notice two /sys/fs/cgroup mounts if the container is not using a new network namespace. The duplication is not a bug but an intentional consequence of how the kernel handles bind mounts that cross user namespace boundaries, combined with the need to provide the container with a writable cgroup view that is scoped to its own slice.

Hide the current process executable file

Wed, 21 Dec 2022 22:15:00 +0200

I have been working on a new functionality for the prctl syscall that addresses a common security concern with container runtimes. The /proc/self/exe symlink, which points to the executable of the running process, was the key ingredient in CVE-2019-5736, a vulnerability that allowed a malicious container to overwrite the container runtime binary on the host. The workaround deployed at the time — re-execing from a copy or using a read-only bind mount — treats the symptom rather than the cause.

The journey to speed up running OCI containers

Wed, 21 Sep 2022 16:30:00 +0200

When I started working on crun, I was looking at a faster way to start up and stop containers by improving the OCI runtime, the component in the OCI stack that is responsible for talking to the kernel and setting up the environment where the container runs. Over roughly five years, a combination of kernel patches and userspace fixes reduced the time to start and stop a container from around 160 ms to just over 5 ms — nearly a 30x improvement — through targeted work on network namespace teardown, mqueue mount overhead, IPC namespace cleanup, and seccomp profile compilation.

An interesting issue handling the seccomp listener

Mon, 05 Sep 2022 21:59:12 +0200

A bug report filed against crun a few days ago exposed a deadlock: under certain seccomp profiles, the runtime would hang indefinitely before the container process ever started. The root cause is a subtle sequencing problem between installing a seccomp filter that intercepts a syscall and then immediately using that same syscall to hand off the resulting listener file descriptor to the userspace handler — the very handler that has not yet received the descriptor it needs to process the interception.

Composefs - a file system for container images

Tue, 26 Oct 2021 16:02:56 +0200

For the last couple of weeks, I’ve been playing on a PoC implementation of a file system for the Linux kernel. The goal is to address a fundamental limitation in how container images are stored: the existing overlay model deduplicates at the layer level, but once you want per-file deduplication — so that identical files across different images share a single copy on disk and in the page cache — the current architecture gets in the way and requires awkward workarounds involving hard links or filesystem-specific reflinks.

Seccomp made easy

Sat, 30 Jan 2021 21:10:14 +0200

Seccomp is a kernel feature that restricts what syscalls can be used by a process. The allowed syscalls are described as a BPF program that the kernel evaluates on every syscall entry. While effective, writing and maintaining seccomp profiles in the JSON format expected by OCI runtimes is tedious, and the underlying libseccomp API has surprising constraints — particularly around combining per-argument rules for the same syscall — that make complex policies difficult to express correctly.

Almost every container runs with seccomp enabled to restrict its access to syscalls.

Cgroup v2 OOM group

Fri, 14 Aug 2020 19:49:32 +0200

One annoying issue with setting a memory limit for a container is that the OOM killer can leave the container in an inconsistent state with only some of its processes terminated. When a cgroup hits its memory limit, the kernel selects a single process to kill based on a badness score, not all the processes in the cgroup. This means that a multi-process container — for example, one running a web server and several worker processes — may continue running in a broken state after the OOM event rather than being cleanly torn down.

Playing with seccomp notifications in the OCI runtime

Mon, 10 Aug 2020 10:40:19 +0200

A couple weekends ago I’ve played with seccomp user notifications and how they can be used in the OCI containers stack. Seccomp user notifications are a Linux kernel feature that lets a privileged monitor process intercept specific syscalls made by a less-privileged container, inspect the arguments, and either emulate the syscall or return an error. This opens up possibilities for safely expanding what unprivileged containers can do — for example, emulating mknod — without granting broad kernel capabilities to the container itself.

Seccomp user notifications are a powerful Linux kernel feature, that delegates syscalls handling to a userland program.

Avoid a memory page allocation on mount(2)

Fri, 27 Dec 2019 16:16:33 +0000

While working on crun, I got surprised by how much time the kernel spent in the copy_mount_options function. A container runtime issues a large number of mount(2) syscalls during startup — bind mounts, proc, sysfs, devtmpfs, and more — many of them with no extra options to pass. It turned out that passing an empty string instead of NULL for the data argument caused the kernel to allocate a full memory page and attempt a copy from user space on every one of those calls, adding measurable overhead.

Run containers without pulling images

Thu, 24 Oct 2019 18:37:23 +0000

CRFS is a Google project that aims at running a container without pre-pulling the image first. The key insight is that in practice a container process only accesses a small fraction of the files in its image, so fetching the entire image before startup wastes both time and disk space. CRFS achieves this through the stargz (Seekable tar.gz) format, which restructures each compressed layer so that individual files can be fetched on demand rather than requiring the entire tarball to be downloaded and extracted upfront.

Crun moved to github.com/containers

Mon, 12 Aug 2019 09:54:25 +0000

The giuseppe/crun github project was moved under https://github.com/containers/crun. Moving to the containers organization means the project is no longer a personal experiment but a community-maintained component of the container stack, alongside tools like Podman, Buildah, and fuse-overlayfs. This makes it easier to coordinate changes across the ecosystem and signals that crun is a supported alternative OCI runtime for production use.

Similarly libocispec, used internally by crun for parsing the OCI configuration file was moved to https://github.com/containers/libocispec

Rootless resources management with Podman on Fedora 30

Sun, 12 May 2019 20:36:59 +0000

I have finally opened some PRs for conmon and libpod that enable resources management for Podman rootless containers on Fedora 30 when using crun. This builds on the cgroups v2 delegation support added to crun earlier: Fedora 30 ships a kernel and systemd new enough to support the unified cgroup hierarchy, so with a single kernel command-line option and a small systemd drop-in, unprivileged users can now set memory and CPU limits on their containers without root access.

Resources management with rootless containers and cgroups v2

Tue, 26 Feb 2019 21:22:10 +0000

cgroups v2 will finally allow unprivileged users to manage a cgroup hierarchy in a safe manner without requiring any additional permission. In the cgroups v1 model, writing to cgroup control files requires root, which means rootless containers cannot enforce memory limits or CPU quotas. The unified cgroups v2 hierarchy introduces a delegation mechanism where systemd can hand ownership of a subtree to a user process, enabling the OCI runtime to configure resource limits directly without any privileged helper.

Rootless containers @ devconf.cz

Sun, 24 Feb 2019 22:26:15 +0000

The video of the rootless containers talk from Devconf.cz 2019 is finally available on YouTube. The talk covers how user namespaces, fuse-overlayfs, and slirp4netns come together to allow running containers entirely as an unprivileged user, without any setuid helpers beyond newuidmap and newgidmap, and discusses the remaining challenges around cgroup resource management and overlay storage performance that still need to be addressed for rootless containers to reach full feature parity.

SUID binaries from a user namespace

Thu, 10 Jan 2019 21:49:30 +0000

Additional IDs that are allocated to a user through /etc/subuid and /etc/subgid must be considered as permanently allocated and never reused for any other user. The reason is that a setuid binary created inside a user namespace can retain access to any UID that was mapped in that namespace, even after the namespace is destroyed. If the same UID range is later assigned to a different user, that new user would inherit access to files owned by the old user’s containers.

Disposable rootless sessions

Wed, 09 Jan 2019 22:01:08 +0000

Would be nice to have a way to “fork” the current session and be able to revert all the changes done, without any leftover on the file system. With fuse-overlayfs, a user-space overlay filesystem that unprivileged users can mount, this turns out to be surprisingly straightforward: mount the entire root filesystem as the lower layer of an overlay, point the upper layer at a temporary directory, and every write is captured there and can be discarded at the end of the session, leaving the underlying system untouched.

An Emacs mode for Rust

Tue, 18 Dec 2018 20:57:11 +0000

I was looking for an Emacs mode that could help me to hack on Rust. The built-in rust-mode provides syntax highlighting and basic indentation, but for a language with a complex type system and borrow checker it is useful to have editor integration that can navigate to definitions, show type information, and offer completions. This post covers setting up racer-mode, which drives the racer code-completion engine to provide those features inside Emacs.

Rust-mode itself has not enough features to help me with a language I am not really proficient with yet.

Rootless Podman from upstream on CentOS 7

Fri, 12 Oct 2018 09:14:21 +0000

This is the recipe I use to build podman from upstream on Centos 7 and use rootless containers. We need an updated version of the shadow utils as newuidmap and newgidmap are not present on Centos 7. The shadow utils are installed using “make install” which is not the clean way to install packages and it also overwrites the existing binaries, but it is fine on a development system. Podman is already present on Centos 7 and in facts we install it so we don’t have to worry about conmon and other dependencies.

Network namespaces for unprivileged users

Sun, 05 Aug 2018 13:54:44 +0000

A couple of weekends ago I’ve played with libslirp and put together slirp-forwarder. The challenge with network namespaces for unprivileged users is that creating TAP or TUN devices requires privileges in the host network namespace. SliRP sidesteps this by emulating a full TCP/IP stack entirely in user space, so the helper process can forward traffic to the outside world using only normal socket operations, without needing any elevated capability.

SliRP emulates in userspace a TCP/IP stack. It can be used to circumvent the limitation of creating TAP/TUN devices in the host namespace for an unprivileged user. The program could run in the host namespace, receive messages from the network namespace where a TAP device is configured, and forward them to the outside world using unprivileged operations such as opening another connection to the destination host. Privileged operations are still not possible outside of the emulated network, as the helper program doesn’t gain any additional privilege that running as an unprivileged user.

Become-root in a user namespace

Thu, 19 Jul 2018 08:28:06 +0000

I’ve cleaned up some C files I was using locally for hacking with user namespaces and uploaded them to a new repository on github: https://github.com/giuseppe/become-root. The tool creates a new user namespace and maps the caller to UID 0 inside it, while also mapping additional UIDs and GIDs from the ranges allocated in /etc/subuid and /etc/subgid. This is the foundation needed for rootless containers, which require a full UID/GID mapping — not just the single-UID mapping that unshare -r provides — to correctly represent file ownership inside container images.

Fuse-overlayfs moved to github.com/containers

Fri, 13 Jul 2018 22:00:59 +0000

The fuse-overlayfs project I was working on in the last weeks was moved under the github.com/containers umbrella. fuse-overlayfs is a user-space implementation of the overlay filesystem that can be mounted without root privileges, which is essential for rootless containers. With Linux 4.18 introducing the ability to mount FUSE filesystems inside user namespaces, this makes overlay-based storage finally usable by unprivileged container runtimes such as Podman.

With Linux 4.18 it will be possible to mount a FUSE file system in an user namespace. fuse-overlayfs is an implementation in user space of the overlay file system already present in the Linux kernel, but that can be mounted only by the root user. Union file systems were around for a long time, allowing multiple layers to be stacked on top of each other where usually the last one is the only writeable.
Overlay is an union file system widely used for mounting OCI image. Each OCI image is made up of different layers, each layer can be used by different images. A list of layers, stacked on each other gives the final image that is used by a container. The last level, that is writeable, is specific for the container. This model enables different containers to use the same image that is accessible as read-only from the lower layers of the overlay file system.

Current status (and problems) of running Buildah as non root

Sun, 25 Feb 2018 13:59:14 +0000

Having Buildah running in a user namespace opens the possibility of building container images as a non-root user. I’ve done some work to get Buildah running inside a user container, where it can still create and modify container images without any elevated privileges on the host. This is useful for CI environments and shared systems where granting root or setuid access is not acceptable.

There are still some open issues to get it fully working. The biggest open one is that overlayfs cannot be currently used as non root user. There is some work going on, but this will require changes in the kernel and the way extended attributes work for overlay. The alternative is far from ideal and it is to use the vfs storage driver, but it is a good starting point to get things moving and see how far we get. (Another possibility that doesn’t require changes in the kernel would be an OSTree storage for Buildah, but that is a different story).

New COPR repository for crun

Wed, 15 Nov 2017 19:25:46 +0000

I made a new COPR repository for crun so that it can be easily tested on Fedora without having to build from source. crun is a lightweight OCI container runtime written in C, intended as a faster and lower-overhead alternative to runC. The COPR repository tracks the upstream development branch, making it straightforward to try out new features and report issues before they land in a distribution package.

https://copr.fedorainfracloud.org/coprs/gscrivano/crun/

To install crun on Fedora, it is enough to:

C is a better fit for tools like an OCI runtime

Mon, 23 Oct 2017 21:21:19 +0000

I’ve spent some of the last weeks working on a replacement for runC, the most used/known OCI runtime for running containers. It might not be very well known, but it is a key component for running containers. Every Docker container ultimately runs through runC. The OCI runtime is the thin layer between the container engine and the kernel: it reads a JSON configuration file, creates the necessary namespaces and cgroups, sets up mounts and capabilities, and finally execs the container process. Because it runs for such a short time and its workload is almost entirely syscalls, the implementation language matters for startup latency.

OpenShift on system containers

Thu, 23 Feb 2017 18:26:43 +0000

It is still an ongoing work not ready for production, but the upstream version of OpenShift origin has already an experimental support for running OpenShift Origin using system containers. The “latest” Docker image for origin, node and openvswitch, the 3 components we need, are automatically pushed to docker.io, so we can use these for our test. The rhel7/etcd system container image instead is pulled from the Red Hat registry.

This demo is based on these blog posts www.projectatomic.io/blog/2016/12/part1-install-origin-on-f25-atomic-host/ and www.projectatomic.io/blog/2016/12/part2-install-origin-on-f25-atomic-host/ with some differences for the provision of the VMs and obviously running system containers instead of Docker containers.

System containers presentation

Mon, 30 Jan 2017 18:41:37 +0000

Here are the slides for the Atomic System Containers talk I gave at Devconf.cz 2017. System containers are a way to run infrastructure services — such as etcd and Flannel — outside of Docker, managed directly by runc and systemd, which removes the circular dependency that arises when a container runtime depends on components that must themselves be running inside containers.

http://scrivano.org/static/system-containers-demo/

If you are interested in the video, it is on YouTube:

Facebook detox?

Tue, 27 Dec 2016 20:52:47 +0000

I have been using Facebook for the last years to fill every dead time:waiting for the bus, ads on TV, compiling, etc. The quality of the information coming from Facebook is inferior to any other social network, at least to my experience (it can be I follow/know the wrong people), though the part of the brain that controls procrastination seems addicted to this lower quality information and the chattering there. Also, I don’t want to simply delete my Facebook account and move on, most of the people I know are present only there, neither I want to be more “asocial”.

Use bubblewrap as an unprivileged user to run systemd images

Sat, 22 Oct 2016 13:21:25 +0000

bubblewrap is a sandboxing tool that allows unprivileged users to run containers. I was recently working on a way to allow unprivileged users to take advantage of bubblewrap to run regular system images that use systemd. To do so, it was necessary to modify bubblewrap to retain a controlled set of Linux capabilities inside the sandbox. Without those capabilities, systemd cannot perform the privilege-separation steps it needs at startup, even when running as UID 0 inside a user namespace.

Brainfucd brainfk

Wed, 11 May 2016 21:15:43 +0000

Every programmer at some point gets in touch with the Brainfuck programming language and how surprising is that very few instructions are needed to have a Turing complete language, 6 is the case of Brainfuck (plus other 2 for I/O operations). The language operates on a tape of memory cells using only increment, decrement, pointer movement, and conditional loop instructions. Writing a GCC frontend for it turns out to be a manageable project, and the result is a good template for understanding how language frontends hook into the GCC middle-end and optimization passes.

Refactoring a function name across several patches with git rebase

Fri, 22 Apr 2016 17:32:23 +0000

git rebase is one of my favorite git commands. It allows to update a set of local patches against another git branch and also to rework, through the -i flag, some previous patches. A lesser-known capability is the –exec flag, which runs an arbitrary shell command after each patch is applied during the rebase. Combined with the -X theirs merge strategy to silently resolve conflicts, this makes it straightforward to apply mechanical transformations — such as a symbol rename — across an entire patch series without manual intervention.

System containers for Atomic

Thu, 24 Mar 2016 15:41:50 +0000

The main reason behind system containers was the inability to run Flannel in a Docker container as Flannel is required by Docker itself. CoreOS solved this chicken and egg problem by using another instance of Docker (called early-docker) that is used to setup only Etcd and Flannel. Atomic system containers take a different approach: instead of a second Docker daemon, they are managed directly by runc and systemd, so the dependency on Docker is removed entirely and the chicken-and-egg problem simply does not arise.

Ostree-docker-builder

Wed, 30 Sep 2015 13:25:56 +0000

rpm-ostree, used together with OStree, is a powerful tool to generate immutable images for .rpm based systems, so why not use it for generating Docker images as well? rpm-ostree already has support for composing a container tree, but the steps to go from an OStree commit to a ready-to-run Docker image involve several manual operations. ostree-docker-builder is a small tool that automates this workflow, taking a JSON package description and producing a tagged Docker image with a minimal Dockerfile.

Summer of Code 2015 for wget

Thu, 30 Apr 2015 21:40:25 +0000

Coming as a surprise, this year we have got 4 students to work full-time during the summer on wget. More than all the students who have ever worked for wget before during a Summer of Code! The projects span a broad range of improvements: HSTS and FTPS security hardening, TCP Fast Open and conditional GET for performance, HTTP/2 support on top of nghttp2, and an expanded FTP test suite. Each student is expected to keep their work rebased on the current development branch to ease eventual integration.

Create a QCOW2 image for Fedora 22 Atomic

Mon, 20 Apr 2015 15:50:30 +0000

This tutorial shows how to create a QCOW2 image that can be directly imported via virt-install to test out Fedora 22 Atomic starting from a custom OStree repo. The process involves composing an OStree repository from a Fedora Atomic tree definition, serving it over HTTP so the installer can reach it, then driving an unattended installation through a kickstart file. This gives you full control over the package set and tree composition rather than relying on a pre-built official image.

How to deploy a WordPress Docker container using docker-compose

Sun, 19 Apr 2015 23:25:49 +0000

These are the steps to setup the current website in a Docker container. The setup uses docker-compose to declaratively describe a two-container application: a MySQL 5.5 database and a WordPress frontend. Docker links tie them together so that the web container can reach the database by hostname, without needing to hard-code any IP addresses or manage networking manually. A single docker-compose up command downloads the required images and starts both containers, with port 80 on the host forwarded into the WordPress container.

About

Mon, 01 Jan 0001 00:00:00 +0000

Software engineer at Red Hat, based in Rome, Italy. Most of my work in the last years was on container runtimes, especially Podman and the crun OCI runtime.

A more detailed list of my contributions is on GitHub.

To contact me, you can write an email at giuseppe@ followed by the same domain here.

My PGP key fingerprint is AC40 4C1C 0BF7 35C6 3FF4 D562 263D 6DF2 E163 E1EA (keys.openpgp.org).

github.com/giuseppe · linkedin.com/in/gscrivano · fosstodon.org/@giuseppe

*scratch*