Skip to content

Frequently asked questions (FAQ)

Errors

How do I read the clearly run error messages?

clearly run error messages look like this:

$ clearly run foo -- echo hello
clearly run[25750]: can’t find image: foo: No such file or directory (clearly run.c:107 2)

There is a lot of information here, and it comes in this order:

1. Name of the executable; always clearly run.

2. Process ID in square brackets; here 25750. This is useful when debugging parallel clearly run invocations.

3. Colon.

4. Main error message; here can’t find image: foo. This should be informative as to what went wrong, and if it’s not, please file an issue, because you may have found a usability bug. Note that in some cases you may encounter the default message error; if this happens and you’re not doing something very strange, that’s also a usability bug.

5. Colon (but note that the main error itself can contain colons too), if and only if the next item is present.

6. Operating system’s description of the the value of errno; here No such file or directory. Omitted if not applicable.

7. Open parenthesis.

8. Name of the source file where the error occurred; here clearly run.c. This and the following item tell developers exactly where clearly run became confused, which greatly improves our ability to provide help and/or debug.

9. Source line where the error occurred.

10. Value of errno (see C error codes in Linux for the full list of possibilities).

11. Close parenthesis.

Note: Despite the structured format, the error messages are not guaranteed to be machine-readable.

clearly run fails with “can’t re-mount image read-only”

Normally, clearly run re-mounts the image directory read-only within the container. This fails if the image resides on certain filesystems, such as NFS There are two solutions:

1. Unpack the image into a different filesystem, such as tmpfs or local disk. Consult your local admins for a recommendation. Note that Lustre is probably not a good idea because it can give poor performance for you and also everyone else on the system.

2. Use the -w switch to leave the image mounted read-write. This may have an impact on reproducibility (because the application can change the image between runs) and/or stability (if there are multiple application processes and one writes a file in the image that another is reading or writing).

clearly image fails with "certificate verify failed"

When clearly image interacts with a remote registry (e.g., via push or pull subcommands), it will verify the registry’s HTTPS certificate. If this fails, clearly image will exit with the error "certificate verify failed".

This situation tends to arise with self-signed or institutionally-signed certificates, even if the OS is configured to trust them. We use the Python HTTP library Requests, which on many platforms includes its own CA certificates bundle, ignoring the bundle installed by the OS.

Requests can be directed to use an alternate bundle of trusted CAs by setting environment variable REQUESTS_CA_BUNDLE to the bundle path. (See the Requests documentation for details.) For example:

$ export REQUESTS_CA_BUNDLE=/usr/local/share/ca-certificates/registry.crt
$ clearly image pull registry.example.com/image:tag

Alternatively, certificate verification can be disabled entirely with the --tls-no-verify flag. However, users should enable this option only if they have other means to be confident in the registry’s identity.

"storage directory seems invalid"

Clearly uses its storage directory (/var/lib/clearly by default) for various internal uses. As such, Clearly needs complete control over this directory’s contents. This error happens when the storage directory exists but its contents do not match what’s expected, including if it’s an empty directory, which is to protect against using common temporary directories like /tmp or /var/tmp as the storage directory.

Let Clearly create the storage directory. For example, if you want to use /big/containers/$USER/clearly for the storage directory (e.g., by setting CLEARLY_IMAGE_STORAGE), ensure /big/containers/$USER exists but do not create the final directory clearly.

"Transport endpoint is not connected"

This error likely means that the SquashFS mount process has exited or been killed and you’re attempting to access the mount location. This is most often seen when a parallel launcher like srun is used to run the mount command. srun will see that the mount command has exited successfully and clean up all child processes, including that of the active mount. A workaround is to use a tool like pdsh.

"fatal: $HOME not set" from Git, or similar

For example:

$ cat Dockerfile
FROM alpine:3.17
RUN apk add git
RUN git config --global http.sslVerify false
$ clearly image build -t foo -f Dockerfile .
  1 FROM alpine:3.17
  2 RUN ['/bin/sh', '-c', 'apk add git']
[...]
  3 RUN ['/bin/sh', '-c', 'git config --global http.sslVerify false']
fatal: $HOME not set
error: build failed: RUN command exited with 128

The reason this happens is that clearly image build executes RUN instructions with clearly run options including the absence of --home, under which the environment variable $HOME is unset. Thus, tools like Git that try to use it will fail.

The reasoning for leaving the variable unset is that because Clearly runs unprivileged, it isn’t really meaningful for a container to have multiple users, and thus building images with things in the home directory is an antipattern. In fact, with --home specified, clearly run sets $HOME to /home/$USER and bind-mounts the user’s host home directory at that path.

The concern with setting $HOME to some default value during build is that it could simply hide the problem until runtime later, where it would be even more confusing. (That said, if this pattern is desired, it can be implemented with an ARG or ENV instruction.)

The recommended workaround and best practice is to put configuration at the system level, not the user level. In the example above, this means changing git config --global to git config --system.

See the man page for clearly run for more on environment variable handling.

clearly run fails with “can’t execve(2): permission denied”

For example:

$ clearly run /var/tmp/hello -- /bin/echo foo
clearly run[154334]: error: can’t execve(2): /bin/echo: Permission denied (core.c:387 13)

But /bin/echo does have execute permission:

$ ls -lh /var/tmp/hello/bin/echo
-rwxr-xr-x 1 clearly clearly 51 Oct  8  2021 /var/tmp/hello/bin/echo

In this case, the error indicates the container image is on a filesystem mounted with noexec. To verify this, you can use e.g. findmnt(8):

$ findmnt
TARGET      SOURCE  FSTYPE  OPTIONS
[...]
└─/var/tmp  tmpfs   tmpfs   rw,noexec,relatime,size=8675309k

Note noexec under OPTIONS.

To fix this, you can:

1. Use a different filesystem mounted exec (i.e., the opposite of noexec and typically the default).

2. Change the mount options for the filesystem (e.g., update /etc/fstab or remount with exec).

3. Use SquashFS format images (only for images exported from Clearly’s storage directory).

Unexpected behavior

What do the version numbers mean?

Released versions of Clearly have a pretty standard version number, e.g. 0.9.7.

Work leading up to a released version also has version numbers, to satisfy tools that require them and to give the executables something useful to report on --version, but these can be quite messy. We refer to such versions informally as pre-releases, but Clearly does not have formal pre-releases such as alpha, beta, or release candidate.

Pre-release version numbers are not in order, because this work is in a DAG rather than linear, except they precede the version we are working towards. If you’re dealing with these versions, use Git.

Pre-release version numbers are the version we are working towards, followed by: ~pre, the branch name if not master with non-alphanumerics removed, the commit hash, and finally dirty if the working directory had uncommitted changes.

Examples:

• 0.2.0 : Version 0.2.0. Released versions don’t include Git information, even if built in a Git working directory.

• 0.2.1~pre : Some snapshot of work leading up to 0.2.1, built from source code where the Git information has been lost, e.g. the tarballs Github provides. This should make you wary because you don’t have any provenance. It might even be uncommitted work or an abandoned branch.

• 0.2.1~pre+1a99f42 : Master branch commit 1a99f42, built from a clean working directory (i.e., no changes since that commit).

• 0.2.1~pre+foo1.0729a78 : Commit 0729a78 on branch foo-1, foo_1, etc. built from clean working directory.

• 0.2.1~pre+foo1.0729a78.dirty : Commit 0729a78 on one of those branches, plus un-committed changes.

--uid 0 lets me read files I can’t otherwise!

Some permission bits can give a surprising result with a container UID of 0. For example:

$ whoami
reidpr
$ echo surprise > ~/cantreadme
$ chmod 000 ~/cantreadme
$ ls -l ~/cantreadme
---------- 1 reidpr reidpr 9 Oct  3 15:03 /home/reidpr/cantreadme
$ cat ~/cantreadme
cat: /home/reidpr/cantreadme: Permission denied
$ clearly run /var/tmp/hello cat ~/cantreadme
cat: /home/reidpr/cantreadme: Permission denied
$ clearly run --uid 0 /var/tmp/hello cat ~/cantreadme
surprise

At first glance, it seems that we’ve found an escalation -- we were able to read a file inside a container that we could not read on the host! That seems bad.

However, what is really going on here is more prosaic but complicated:

1. After unshare(CLONE_NEWUSER), clearly run gains all capabilities inside the namespace. (Outside, capabilities are unchanged.)

2. This include CAP_DAC_OVERRIDE, which enables a process to read/write/execute a file or directory mostly regardless of its permission bits. (This is why root isn’t limited by permissions.)

3. Within the container, exec(2) capability rules are followed. Normally, this basically means that all capabilities are dropped when clearly run replaces itself with the user command. However, if EUID is 0, which it is inside the namespace given --uid 0, then the subprocess keeps all its capabilities. (This makes sense: if root creates a new process, it stays root.)

4. CAP_DAC_OVERRIDE within a user namespace is honored for a file or directory only if its UID and GID are both mapped. In this case, clearly run maps reidpr to container root and group reidpr to itself.

5. Thus, files and directories owned by the host EUID and EGID (here reidpr:reidpr) are available for all access with clearly run --uid 0.

This is not an escalation. The quirk applies only to files owned by the invoking user, because clearly run is unprivileged outside the namespace, and thus he or she could simply chmod the file to read it. Access inside and outside the container remains equivalent.

References:

• http://man7.org/linux/man-pages/man7/capabilities.7.html

• http://lxr.free-electrons.com/source/kernel/capability.c?v=4.2#L442

• http://lxr.free-electrons.com/source/fs/namei.c?v=4.2#L328

--bind creates mount points within un-writeable directories!

Consider this image:

$ ls /var/tmp/image
bin  dev  home  media  opt   root  sbin  sys  usr
clearly   etc  lib   mnt    proc  run   srv   tmp  var
$ ls -ld /var/tmp/image/mnt
drwxr-xr-x 4 root root 80 Jan  5 09:52 /var/tmp/image/mnt
$ ls /var/tmp/image/mnt
bar  foo

That is, /mnt is owned by root, un-writeable by us even considering the prior question, and contains two subdirectories. Indeed, we cannot create a new directory there:

$ mkdir /var/tmp/image/mnt/baz
mkdir: cannot create directory ‘/var/tmp/image/mnt/baz’: Permission denied

Recall that bind-mounting to a path that does not exist in a read-only image fails:

$ clearly run -b /tmp/baz:/mnt/baz /var/tmp/image -- ls /mnt
clearly run[40498]: error: can't mkdir: /var/tmp/image/mnt/baz: Read-only file system (misc.c:582 30)

That’s fine; we’ll just use --write-fake to create a writeable overlay on the container. Then we can make any mount points we need. Right?

$ clearly run -W /var/tmp/image -- mkdir /qux      # succeeds
$ clearly run -W /var/tmp/image -- mkdir /mnt/baz  # fails
mkdir: can't create directory '/mnt/baz': Permission denied

Wait — why could we create a subdirectory of (container path) / but not a subdirectory of /mnt? This is because the latter, which is at host path /var/tmp/image/mnt, is not writeable by us: the overlayfs propagates the directory’s no-write permissions. Despite this, we can in fact use paths that do not yet exist for bind-mount destinations:

$ clearly run -W -b /tmp/baz:/mnt/baz /var/tmp/image -- ls /mnt
bar  baz  foo

What’s happening is bind-mount trickery and a symlink ranch. clearly run creates a new directory on the overlaid tmpfs, bind-mounts the old (host path) /var/tmp/images/mnt to a subdirectory of it, symlinks the old contents, and finally overmounts the old, un-writeable directory with the new one:

$ clearly run -W -b /tmp/baz:/mnt/baz /var/tmp/image -- ls -la /mnt
drwxr-x---    4 reidpr   reidpr         120 Jan  5 17:11 .
drwx------    1 reidpr   reidpr          40 Jan  5 17:11 ..
drwxr-xr-x    4 nobody   nogroup         80 Jan  5 16:52 .orig
lrwxrwxrwx    1 reidpr   reidpr           9 Jan  5 17:11 bar -> .orig/bar
drwxr-x---    2 reidpr   reidpr          40 Jan  3 23:49 baz
lrwxrwxrwx    1 reidpr   reidpr           9 Jan  5 17:11 foo -> .orig/foo
$ clearly run -W -b /tmp/baz:/mnt/baz /var/tmp/image -- cat /proc/mounts | fgrep ' /mnt'
none /mnt tmpfs rw,relatime,size=3943804k,uid=1000,gid=1000,inode64 0 0
none /mnt/.orig overlay rw,relatime,lowerdir=/var/tmp/image,upperdir=/mnt/upper,workdir=/mnt/work,volatile,userxattr 0 0
tmpfs /mnt/baz tmpfs rw,relatime,size=8388608k,inode64 0 0

This new directory is writeable, and mkdir(2) succeeds. (The overlaid tmpfs is mounted on host /mnt during container assembly, which is why it appears in mount options.)

There are differences from the original directory, of course. Most notably:

• The ranched symlinks can be deleted by the user within the container, contrary to the old directory’s read-only permissions.

• The contents of the “ranched” directory become symlinks rather than their original file type.

Software that cares about these things may break.

Why does ping not work?

ping fails with “permission denied” or similar under Clearly, even if you’re UID 0 inside the container:

$ clearly run $IMG -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: permission denied (are you root?)
$ clearly run --uid=0 $IMG -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: permission denied (are you root?)

This is because ping needs a raw socket to construct the needed ICMP ECHO packets, which requires capability CAP_NET_RAW or root. Unprivileged users can normally use ping because it’s a setuid or setcap binary: it raises privilege using the filesystem bits on the executable to obtain a raw socket.

Under Clearly, there are multiple reasons ping can’t get a raw socket. First, images are unpacked without privilege, meaning that setuid and setcap bits are lost. But even if you do get privilege in the container (e.g., with --uid=0), this only applies in the container. Clearly uses the host’s network namespace, where your unprivileged host identity applies and ping still can’t get a raw socket.

The recommended alternative is to simply try the thing you want to do, without testing connectivity using ping first.

Why is MATLAB trying and failing to change the group of /dev/pts/0?

MATLAB and some other programs want pseudo-TTY (PTY) files to be group-owned by tty. If it’s not, Matlab will attempt to chown(2) the file, which fails inside a container.

The scenario in more detail is this. Assume you’re user clearly (UID=1000), your primary group is nerds (GID=1001), /dev/pts/0 is the PTY file in question, and its ownership is clearly:tty (1000:5), as it should be. What happens in the container by default is:

1. MATLAB stat(2)s /dev/pts/0 and checks the GID.

2. This GID is nogroup (65534) because tty (5) is not mapped on the host side (and cannot be, because only one’s EGID can be mapped in an unprivileged user namespace).

3. MATLAB concludes this is bad.

4. MATLAB executes chown("/dev/pts/0", 1000, 5).

5. This fails because GID 5 is not mapped on the guest side.

6. MATLAB pukes.

The workaround is to map your EGID of 1001 to 5 inside the container (instead of the default 1001:1001), i.e. --gid=5. Then, step 4 succeeds because the call is mapped to chown("/dev/pts/0", 1000, 1001) and MATLAB is happy.

clearly convert from Docker incorrect image sizes

When converting from Docker, clearly convert often finishes before the progress bar is complete. For example:

$ clearly convert -i docker foo /var/tmp/foo.tar.gz
input:   docker    foo
output:  tar       /var/tmp/foo.tar.gz
exporting ...
 373MiB 0:00:21 [============================>                 ] 65%
[...]

In this case, the .tar.gz contains 392 MB uncompressed:

$ zcat /var/tmp/foo.tar.gz | wc
2740966 14631550 392145408

But Docker thinks the image is 597 MB:

$ sudo docker image inspect foo | fgrep -i size
        "Size": 596952928,
        "VirtualSize": 596952928,

We’ve also seen cases where the Docker-reported size is an underestimate:

$ clearly convert -i docker bar /var/tmp/bar.tar.gz
input:   docker    bar
output:  tar       /var/tmp/bar.tar.gz
exporting ...
 423MiB 0:00:22 [============================================>] 102%
[...]
$ zcat /var/tmp/bar.tar.gz | wc
4181186 20317858 444212736
$ sudo docker image inspect bar | fgrep -i size
        "Size": 433812403,
        "VirtualSize": 433812403,

We think that this is because Docker is computing size based on the size of the layers rather than the unpacked image. We do not currently have a fix;

My password that contains digits doesn’t work in VirtualBox console

VirtualBox has confusing Num Lock behavior. Thus, you may be typing arrows, page up/down, etc. instead of digits, without noticing because console password fields give no feedback, not even whether a character has been typed.

Try using the number row instead, toggling Num Lock key, or SSHing into the virtual machine.

Mode bits (permission bits) are lost

Clearly preserves only some mode bits, specifically user, group, and world permissions, and the restricted deletion flag on directories; i.e. 777 on files and 1777 on directories.

The setuid (4000) and setgid (2000) bits are not preserved because ownership of files within Clearly images is that of the user who unpacks the image. Leaving these bits set could therefore surprise that user by unexpectedly creating files and directories setuid/gid to them.

The sticky bit (1000) is not preserved for files because unsquashfs(1) unsets it even with umask 000. However, this is bit is largely obsolete for files.

Note the non-preserved bits may sometimes be retained, but this is undefined behavior. The specified behavior is that they may be zeroed at any time.

Why is my wildcard in clearly run not working?

Be aware that wildcards in the clearly run command are interpreted by the host, not the container, unless protected. One workaround is to use a sub-shell. For example:

$ ls /usr/bin/oldfind
ls: cannot access '/usr/bin/oldfind': No such file or directory
$ clearly run /var/tmp/hello.sqfs -- ls /usr/bin/oldfind
/usr/bin/oldfind
$ ls /usr/bin/oldf*
ls: cannot access '/usr/bin/oldf*': No such file or directory
$ clearly run /var/tmp/hello.sqfs -- ls /usr/bin/oldf*
ls: cannot access /usr/bin/oldf*: No such file or directory
$ clearly run /var/tmp/hello.sqfs -- sh -c 'ls /usr/bin/oldf*'
/usr/bin/oldfind

How do I ...

My app needs to write to /var/log, /run, etc.

Because the image is mounted read-only by default, log files, caches, and other stuff cannot be written anywhere in the image. You have three options:

1. Configure the application to use a different directory. /tmp is often a good choice, because it’s shared with the host and fast.

2. Use RUN commands in your Dockerfile to create symlinks that point somewhere writeable, e.g. /tmp, or /mnt/0 with clearly run --bind.

3. Run the image read-write with clearly run -w. Be careful that multiple containers do not try to write to the same files.

OpenMPI Clearly jobs don’t work

MPI can be finicky. This section documents some of the problems we’ve seen.

mpirun can’t launch jobs

For example, you might see:

$ mpirun -np 1 clearly run /var/tmp/mpihello-openmpi -- /hello/hello
App launch reported: 2 (out of 2) daemons - 0 (out of 1) procs
[cn001:27101] PMIX ERROR: BAD-PARAM in file src/dstore/pmix_esh.c at line 996

We’re not yet sure why this happens — it may be a mismatch between the OpenMPI builds inside and outside the container — but in our experience launching with srun often works when mpirun doesn’t, so try that.

Communication between ranks on the same node fails

OpenMPI has many ways to transfer messages between ranks. If the ranks are on the same node, it is faster to do these transfers using shared memory rather than involving the network stack. There are two ways to use shared memory.

The first and older method is to use POSIX or SysV shared memory segments. This approach uses two copies: one from Rank A to shared memory, and a second from shared memory to Rank B. For example, the sm byte transport layer (BTL) does this.

The second and newer method is to use the process_vm_readv(2) and/or process_vm_writev(2)) system calls to transfer messages directly from Rank A’s virtual memory to Rank B’s. This approach is known as cross-memory attach (CMA). It gives significant performance improvements in benchmarks, though of course the real-world impact depends on the application. For example, the vader BTL (enabled by default in OpenMPI 2.0) and psm2 matching transport layer (MTL) do this.

The problem in Clearly is that the second approach does not work by default.

We can demonstrate the problem with LAMMPS molecular dynamics application:

$ srun --cpus-per-task 1 clearly run /var/tmp/lammps_mpi -- \
  lmp_mpi -log none -in /lammps/examples/melt/in.melt
[cn002:21512] Read -1, expected 6144, errno = 1
[cn001:23947] Read -1, expected 6144, errno = 1
[cn002:21517] Read -1, expected 9792, errno = 1
[... repeat thousands of times ...]

With strace(1), one can isolate the problem to the system call noted above:

process_vm_readv(...) = -1 EPERM (Operation not permitted)
write(33, "[cn001:27673] Read -1, expected 6"..., 48) = 48

The man page reveals that these system calls require that the process have permission to ptrace(2) one another, but sibling user namespaces do not. (You can ptrace(2) into a child namespace, which is why gdb doesn’t require anything special in Clearly.)

This problem is not specific to containers; for example, many settings of kernels with YAMA enabled will similarly disallow this access.

So what can you do? There are a few options:

• We recommend simply using the --join family of arguments to clearly run. This puts a group of clearly run peers in the same namespaces; then, the system calls work. See the clearly run man page for details.

• You can also sometimes turn off single-copy. For example, for vader, set the MCA variable btl_vader_single_copy_mechanism to none, e.g. with an environment variable:

  $ export OMPI_MCA_btl_vader_single_copy_mechanism=none
  

  psm2 does not let you turn off CMA, but it does fall back to two-copy if CMA doesn’t work. However, this fallback crashed when we tried it.

• The kernel module XPMEM enables a different single-copy approach. We have not yet tried this, and the module needs to be evaluated for user namespace safety, but it’s quite a bit faster than CMA on benchmarks.

I get a bunch of independent rank-0 processes when launching with srun

For example, you might be seeing this:

$ srun clearly run /var/tmp/mpihello-openmpi -- /hello/hello
0: init ok cn036.localdomain, 1 ranks, userns 4026554634
0: send/receive ok
0: finalize ok
0: init ok cn035.localdomain, 1 ranks, userns 4026554634
0: send/receive ok
0: finalize ok

We were expecting a two-rank MPI job, but instead we got two independent one-rank jobs that did not coordinate.

MPI ranks start as normal, independent processes that must find one another somehow in order to sync up and begin the coupled parallel program; this happens in MPI_Init().

There are lots of ways to do this coordination. Because we are launching with the host’s Slurm, we need it to provide something for the containerized processes for such coordination. OpenMPI must be compiled to use what that Slurm has to offer, and Slurm must be told to offer it. What works for us is a something called "PMIx". You can see if your Slurm supports it with:

$ srun --mpi=list
  cray_shasta
  none
  pmi2
  pmix

If pmix is not in the list, you must either (a) ask your admins to enable Slurm’s PMIx support, or (b) rebuild your container MPI against an PMI in the list. If it is in the list, but you’re seeing this problem, that means it is not the default, and you need to tell Slurm you want it. Try:

$ srun --mpi=pmix clearly run /var/tmp/mpihello-openmpi -- /hello/hello
0: init ok wc035.localdomain, 2 ranks, userns 4026554634
1: init ok wc036.localdomain, 2 ranks, userns 4026554634
0: send/receive ok
0: finalize ok

How do I run X11 apps?

X11 applications should “just work”. For example, try this Dockerfile:

FROM debian:stretch
RUN    apt-get update \
    && apt-get install -y xterm

Build it and unpack it to /var/tmp. Then:

$ clearly run /scratch/clearly/xterm -- xterm

should pop an xterm.

If your X11 application doesn’t work, please file an issue so we can figure out why.

How do I specify an image reference?

You must specify an image for many use cases, including FROM instructions, the source of an image pull (e.g. clearly image pull or docker pull), the destination of an image push, and adding image tags. Clearly calls this an image reference, but there appears to be no established name for this concept.

The syntax of an image reference is not well documented. This FAQ represents our understanding, which is cobbled together from the Dockerfile reference, the docker tag documentation, and various forum posts. It is not a precise match for how Docker implements it, but it should be close enough.

We’ll start with two complete examples with all the bells and whistles:

1. example.com:8080/foo/bar/hello-world:version1.0

2. example.com:8080/foo/bar/hello-world@sha256:f6c68e2ad82a

These references parse into the following components, in this order:

1. A valid hostname; we assume this matches the regular expression [A-Za-z0-9.-]+, which is very approximate. Optional; here example.com.

2. A colon followed by a decimal port number. If hostname is given, optional; otherwise disallowed; here 8080.

3. If hostname given, a slash.

4. A path, with one or more components separated by slash. Components match the regex [a-z0-9_.-]+. Optional; here foo/bar. Pedantic details:

   • Under the hood, the default path is library, but this is generally not exposed to users.

   • Three or more underscores in a row is disallowed by Docker, but we don’t check this.

5. If path given, a slash.

6. The image name (tag), which matches [a-z0-9_.-]+. Required; here hello-world.

7. Zero or one of:

   • A tag matching the regular expression [A-Za-z0-9_.-]+ and preceded by a colon. Here version1.0 (example 1).

   • A hexadecimal hash preceded by the string @sha256:. Here f6c68e2ad82a (example 2).

     • Note: Digest algorithms other than SHA-256 are in principle allowed, but we have not yet seen any.

Detail-oriented readers may have noticed the following gotchas:

• A hostname without port number is ambiguous with the leading component of a path. For example, in the reference foo/bar/baz, it is ambiguous whether foo is a hostname or the first (and only) component of the path foo/bar. The resolution rule is: if the ambiguous substring contains a dot, assume it’s a hostname; otherwise, assume it’s a path component.

• The only character that cannot go in a POSIX filename is slash. Thus, Clearly uses image references in filenames, replacing slash with percent (%). Because this character cannot appear in image references, the transformation is reversible.

  Git branch names do not allow a colon. Thus, to maintain the image reference as both the image filename and git branch in storage, we replace the colon with plus (+).

  An alternate approach would be to replicate the reference path in the filesystem, i.e., path components in the reference would correspond directly to a filesystem path. This would yield a clearer filesystem structure. However, we elected not to do it because it complicates the code to save and clean up image reference-related data, and it does not address a few related questions, e.g. should the host and port also be a directory level.

Usually, most of the components are omitted. For example, you’ll more commonly see image references like:

• debian, which refers to the tag latest of image debian from Docker Hub.

• debian:stretch, which is the same except for tag stretch.

• fedora/httpd, which is tag latest of fedora/httpd from Docker Hub.

See clearly.py for a specific grammar that implements this.

Can I build or pull images using a tool Clearly doesn’t know about?

Yes. Clearly deals in well-known UNIX formats like directories, tarballs, and SquashFS images. So, once you get your image into some format Clearly likes, you can enter the workflow.

For example, skopeo is a tool to pull images to OCI format, and umoci can flatten an OCI image to a directory. Thus, you can use the following commands to run an Alpine 3.9 image pulled from Docker hub:

$ skopeo copy docker://alpine:3.17 oci:/tmp/oci:img
[...]
$ ls /tmp/oci
blobs  index.json  oci-layout
$ umoci unpack --rootless --image /tmp/oci:img /tmp/alpine:3.17
[...]
$ ls /tmp/alpine:3.17
config.json
rootfs
sha256_2ca27acab3a0f4057852d9a8b775791ad8ff62fbedfc99529754549d33965941.mtree
umoci.json
$ ls /tmp/alpine:3.17/rootfs
bin  etc   lib    mnt  proc  run   srv  tmp  var
dev  home  media  opt  root  sbin  sys  usr
$ clearly run /tmp/alpine:3.17/rootfs -- cat /etc/alpine-release
3.9.5

How do I authenticate with SSH during clearly image build?

The simplest approach is to run the SSH agent on the host. clearly image then leverages this with two steps:

1. pass environment variable SSH_AUTH_SOCK into the build, with no need to put ARG in the Dockerfile or specify --build-arg on the command line; and

2. bind-mount host /tmp to guest /tmp, which is where the SSH agent’s listening socket usually resides.

Thus, SSH within the container will use this existing SSH agent on the host to authenticate without further intervention.

For example, after making ssh-agent available on the host, which is OS and site-specific:

$ echo $SSH_AUTH_SOCK
/tmp/ssh-rHsFFqwwqh/agent.49041
$ ssh-add
Enter passphrase for /home/clearly/.ssh/id_rsa:
Identity added: /home/clearly/.ssh/id_rsa (/home/clearly/.ssh/id_rsa)
$ ssh-add -l
4096 SHA256:aN4n2JeMah2ekwhyHnb0Ug9bYMASmY+5uGg6MrieaQ /home/clearly/.ssh/id_rsa (RSA)
$ cat ./Dockerfile
FROM alpine:latest
RUN apk add openssh
RUN echo $SSH_AUTH_SOCK
RUN ssh git@github.com
$ clearly image build -t foo -f ./Dockerfile .
[...]
  3 RUN ['/bin/sh', '-c', 'echo $SSH_AUTH_SOCK']
  /tmp/ssh-rHsFFqwwqh/agent.49041
  4 RUN ['/bin/sh', '-c', 'ssh git@github.com']
[...]
Hi clearly! You’ve successfully authenticated, but GitHub does not provide shell access.

Note this example is rather contrived — bare SSH sessions in a Dockerfile rarely make sense. In practice, SSH is used as a transport to fetch something, e.g. with scp(1) or git(1). See the next entry for a more realistic example.

SSH stops clearly image build with interactive queries

This often occurs during an SSH-based Git clone. For example:

FROM alpine:latest
RUN apk add git openssh
RUN git clone git@github.com:example/example.git

$ clearly image build -t foo -f ./Dockerfile .
[...]
3 RUN ['/bin/sh', '-c', 'git clone git@github.com:example/example.git']
Cloning into 'example'...
The authenticity of host 'github.com (140.82.113.3)' can’t be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

At this point, the build stops while SSH waits for input.

This happens even if you have github.com in your ~/.ssh/known_hosts. This file is not available to the build because clearly image runs clearly run without --home, so RUN instructions can’t see anything in your home directory.

Solutions include:

1. Change to anonymous HTTPS clone, if available. Most public repositories will support this. For example:

   FROM alpine:latest
   RUN apk add git
   RUN git clone https://github.com/example/example.git
   

2. Approve the connection interactively by typing yes. Note this will record details of the connection within the image, including IP address and the fingerprint. The build also remains interactive.

3. Edit the image’s system SSH config to turn off host key checking. Note this can be rather hairy, because the SSH config language is quite flexible and the first instance of a directive is the one used. However, often the changes can be simply appended:

   FROM alpine:latest
   RUN apk add git openssh
   RUN printf 'StrictHostKeyChecking=no\nUserKnownHostsFile=/dev/null\n' \
       >> /etc/ssh/ssh_config
   RUN git clone git@github.com:example/example.git
   

   Check your institutional policy on whether this is permissible, though it’s worth noting that users almost never verify the host fingerprints anyway.

   This will not record details of the connection in the image.

4. Turn off host key checking on the SSH command line. (See caveats in the previous item.) The wrapping tool should provide a way to configure this command line. For example, for Git:

   FROM alpine:latest
   RUN apk add git openssh
   ARG GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
   RUN git clone git@github.com:example/example.git
   

5. Add the remote host to the system known hosts file, e.g.:

   FROM alpine:latest
   RUN apk add git openssh
   RUN echo 'github.com,140.82.112.4 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==' >> /etc/ssh/ssh_known_hosts
   RUN git clone git@github.com:example/example.git
   

   This records connection details in both the Dockerfile and the image.

Other approaches could be found with web searches such as "automate unattended SSH" or "SSH in cron jobs".

How do I use Docker to build Clearly images?

The short version is to run Docker commands like docker build and docker pull like usual, and then use clearly convert to copy the image from Docker storage to a SquashFS archive, tarball, or directory. If you are behind an HTTP proxy, that requires some extra setup for Docker; see below.

Security implications of Docker

Because Docker (a) makes installing random crap from the internet simple and (b) is easy to deploy insecurely, you should take care. Some of the implications are below. This list should not be considered comprehensive nor a substitute for appropriate expertise; adhere to your ethical and institutional responsibilities.

• Docker equals root. Anyone who can run the docker command or interact with the Docker daemon can trivially escalate to root. This is considered a feature.

  For this reason, don’t create the docker group, as this will allow passwordless, unlogged escalation for anyone in the group. Run it with sudo docker.

  Also, Docker runs container processes as root by default. In addition to being poor hygiene, this can be an escalation path, e.g. if you bind-mount host directories.

• Docker alters your network configuration. To see what it did:

  $ ifconfig    # note docker0 interface
  $ brctl show  # note docker0 bridge
  $ route -n
  

• Docker installs services. If you don’t want the Docker service starting automatically at boot, e.g.:

  $ systemctl is-enabled docker
  enabled
  $ systemctl disable docker
  $ systemctl is-enabled docker
  disabled
  

Configuring for a proxy

By default, Docker does not work if you are behind a proxy, and it fails in two different ways.

The first problem is that Docker itself must be told to use a proxy. This manifests as:

$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
Pulling repository hello-world
Get https://index.docker.io/v1/repositories/library/hello-world/images: dial tcp 54.152.161.54:443: connection refused

If you have a systemd system, the Docker documentation explains how to configure this. If you don’t have a systemd system, then /etc/default/docker might be the place to go?

The second problem is that programs executed during build (RUN instructions) need to know about the proxy as well. This manifests as images failing to build because they can’t download stuff from the internet.

One fix is to configure your .bashrc or equivalent to:

1. Set the proxy environment variables:

   export HTTP_PROXY=http://proxy.example.com:8088
   export http_proxy=$HTTP_PROXY
   export HTTPS_PROXY=$HTTP_PROXY
   export https_proxy=$HTTP_PROXY
   export NO_PROXY='localhost,127.0.0.1,.example.com'
   export no_proxy=$NO_PROXY
   

2. Configure a docker wrapper, e.g. this one for fish:

     # This file defines a docker(1) wrapper that (unconditionally) pre-pends
     # sudo(8) and also sets proxy environment variables if needed. Save it as
     # “~/.config/fish/functions.docker.fish”. I think it’s better than configuring
     # a proxy in config.json [1] because it’s easier to switch between the proxy
     # and no-proxy states.
   
     # List of environment variables we want to pass through to Docker.
     set -g _docker_vars HTTP_PROXY HTTPS_PROXY NO_PROXY
     set -a _docker_vars (string lower $_docker_vars)
   
     function docker \
              -d 'docker(1) wrapper with auto-sudo and proxy variables'
        set cmd $argv[1]
        set -e argv[1]
        # Build pass-thru arguments if needed.
        if set -q HTTP_PROXY
          switch $cmd
            case build
              set ea --build-arg
            case run
              set ea --env
          end
        if set -q ea
          set_color da1884
          echo "note: proxy variables found; passing through with $ea"
          set_color normal
          for v in $_docker_vars
            set -a evs "$ea=$v=$$v"
          end
        end
      end
      # Run Docker.
      sudo docker $cmd $evs $argv
   end
   

How can I build images for a foreign architecture?

QEMU

Suppose you want to build Clearly containers on a system which has a different architecture from the target system.

It’s straightforward as long as you can install suitable packages on the build system (your personal computer?). You just need the magic of QEMU via a distribution package with a name like Debian’s qemu-user-static. For use in an image root this needs to be the -static version, not plain qemu-user, and contain a qemu-*-static executable for your target architecture. In case it doesn’t install “binfmt” hooks (telling Linux how to run foreign binaries), you’ll need to make that work — perhaps it’s in another package.

That’s all you need to make building with clearly image work with a base foreign architecture image and the --arch option. It’s significantly slower than native, but quite usable — about half the speed of native for the ppc64le target with a build taking minutes on a laptop with a magnetic disc. There’s a catch that images in clearly image storage aren’t distinguished by architecture except by any name you give them, e.g., a base image like debian:11 pulled with --arch ppc64le will overwrite a native x86 one.

For example, to build a ppc64le image on a Debian Buster amd64 host:

$ uname -m
x86_64
$ sudo apt install qemu-user-static
$ clearly image pull --arch ppc64le alpine:3.17
$ printf 'FROM alpine:3.17\nRUN apk add coreutils\n' | clearly image build -t foo -
$ clearly convert alpine:3.17 /var/tmp/foo
$ clearly run /var/tmp/foo -- uname -m
ppc64le

PRoot

Another way to build a foreign image, which works even without sudo to install qemu-*-static, is to populate a chroot for it with the PRoot tool, whose -q option allows specifying a qemu-*-static binary (perhaps obtained by unpacking a distribution package).

How can I use tarball base images from e.g. linuxcontainers.org?

If you can’t find an image repository from which to pull for the distribution and architecture of interest, it is worth looking at the extensive collection of rootfs archives maintained by linuxcontainers.org. They are meant for LXC, but are fine as a basis for Clearly.

For example, this would leave a ppc64le/alpine:3.17 image du jour in the registry for use in a Dockerfile FROM line. Note that linuxcontainers.org uses the opposite order for “le” in the architecture name.

$ wget https://uk.lxd.images.canonical.com/images/alpine/3.15/ppc64el/default/20220304_13:00/rootfs.tar.xz
$ clearly image import rootfs.tar.xz ppc64le/alpine:3.17

How can I control Clearly’s quietness or verbosity?

Clearly logs various chatter about what is going on to standard error. This is distinct from output, e.g., clearly image list prints the list of images to standard output. We use reasonably standard log levels:

1. Error. Some error condition that makes it impossible to proceed. The program exits soon after printing the error. Examples: unknown image type, Dockerfile parse error. (There is an internal distinction between “fatal” and “error” levels, but this isn’t really meaningful to users.)

2. Warning. Unexpected condition the user needs to know about but that should not stop the program. Examples: clearly run --mount with a directory image (which does not use a mount point), unsupported Dockerfile instructions that are ignored.

3. Info. Chatter useful enough to be printed by default. Example: progress messages during image download and unpacking. (clearly run is silent during normal operations and does not have any “info” logging.)

4. Verbose. Diagnostic information useful for debugging user containers, the Clearly installation, and Clearly itself. Examples: clearly run --join coordination progress, clearly image internal paths, Dockerfile parse tree.

5. Debug. More detailed diagnostic information useful for debugging Clearly. Examples: data structures unserialized from image registry metadata JSON, image reference parse tree.

6. Trace; printed if -vvv. Grotesquely detailed diagnostic information for debugging Clearly, to the extent it interferes with normal use. A sensible person might use a debugger instead. Examples: component-by-component progress of bind-mount target directory analysis/creation, text of image registry JSON, every single file unpacked from image layers.

Clearly also runs sub-programs at various times, notably commands in RUN instructions and git(1) to manage the build cache. These programs have their own standard error and standard output streams, which Clearly either suppresses or passes through depending on verbosity level.

Most Clearly programs accept -v to increase logging verbosity and -q to decrease it. Generally:

• Each -v (up to three) makes Clearly noisier.

• -q suppresses normal logging.

• -qq also suppresses stdout for the program and its subprocesses, and warnings from the program.

• -qqq also suppresses subprocess stderr. (This means subprocesses are completely silenced no matter what goes wrong!)

This table list which logging is printed at which verbosity levels (✅ indicates printed, ❌ suppressed).

	-vvv	-vv	-v	def.	-q	-qq	-qqq
trace	✅	❌	❌	❌	❌	❌	❌
debug	✅	✅	❌	❌	❌	❌	❌
verbose	✅	✅	✅	❌	❌	❌	❌
info	✅	✅	✅	✅	❌	❌	❌
program stdout	✅	✅	[1]	[1]	[1]	❌	❌
subprocess stdout	✅	✅	[1]	[1]	[1] [2]	❌	❌
warning	✅	✅	✅	✅	✅	❌	❌
subprocess stderr	✅	✅	✅	✅	✅	✅	❌
error	✅	✅	✅	✅	✅	✅	✅

Notes:

1. Clearly handles subprocess stdout on case-by-case basis for these log levels. For example, sometimes it’s passed through by default (e.g., RUN) and sometimes it’s captured for internal use (e.g., many git(1) invocations).

2. In the case of clearly run, the user command is considered a subprocess, e.g. clearly run -q example -- echo foo will produce no output.

How do I handle extended attributes in Clearly?

As noted in section Build cache, Clearly doesn’t support extended attributes (xattrs) by default. Support for xattrs can be enabled for clearly image and clearly convert by specifying --xattrs or setting $CLEARLY_XATTRS. This will make clearly image save and restore xattrs via the build cache, and will make clearly convert preserve xattrs on conversion. Important caveats include:

1. clearly image and clearly convert cannot read xattrs in privileged namespaces (e.g. trusted and security). Extended attributes in these namespaces will never be saved or restored via the cache, and will never be preserved when converting between image formats.

2. clearly image import cannot handle xattrs. This is a limitation of the Python tarfile library, which as of version 3.12.1 doesn’t support xattrs (see CPython issue #113293).

3. clearly convert -o clearly image uses clearly image import under the hood. This in conjunction with (2) means that clearly convert cannot preserve xattrs when converting to the clearly image format.

4. clearly image pull uses the tarfile library, so xattrs will be lost when pulling from a registry.

5. Support for xattrs varies among filesystems, e.g. tmpfs didn’t support xattrs in the user namespace prior to Linux kernel upstream 6.6 (Oct 2023).

How do I source a file automatically when container starts?

There are ways to have Bash source an arbitrary script at start-up (docs).

For example, clearly run ... -- bash --rcfile /my/start.sh will give you an interactive shell with /my/start.sh sourced first.

Or: clearly run ... -- bash --login will give you an interactive login shell and so it will source /etc/profile.

Finally, you can have a file $HOME/.bashrc (where $HOME is set by default, as described here), which Bash will source automatically with clearly run ... -- bash.

Alternately, if you want the interactive Bash shell to source .bashrc of your host system, you can use clearly run --home, which binds your home directory into the container.