How to run cancer in using a different distro ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are different altneratives to experiment with. 1.a. almost-chroot pro: no root required at all; fs, GUI, hardware sharing is automatic con: cancer-system depends on host kernel version Install a Devuan or Debian in a chroot using debootstrap and install bloatware using chroot. Then have a wrapper script for the bloatware that runs it from within that deb installation without chrooting into it. The wrapper script should look like this: --- #!/bin/sh export R=/opt/cancer/web2.0 export LD_LIBRARY_PATH=$R/lib64:$R/usr/local/lib:$R/usr/lib:$R/usr/lib/x86_64-linux-gnu:$R/usr/lib/x86_64-linux-gnu/pulseaudio export PATH=$R/usr/local/sbin:$R/usr/local/bin:$R/usr/sbin:$R/usr/bin:$R/sbin:$R/bin exec $R/usr/bin/bloatware "$@" --- A major problem is that the deb installation will have a different glibc version with a different ld.so, while all the executables have the path to ld.so hardwired in the elf, running host system's ld.so. The simplest solution is having a static linked dispatcher ld.so installed from the conj package that normally installs ld.so. This dispatcher would look at the path of the first argument and decide to exec the original ld.so or one of the cancer systems'. 1.b. almost-chroot with qemu-user The -L option of qemu-user replaces ld.so in execve, bypassin the normal binfmt mechanism: --- export R=/opt/cancer/web2.0 /usr/bin/qemu-x86_64 -L /opt/cancer/web2.0/usr \ -E LD_LIBRARY_PATH=$R/lib64:$R/usr/local/lib:$R/usr/lib:$R/usr/lib/x86_64-linux-gnu:$R/usr/lib/x86_64-linux-gnu/pulseaudio/opt/cancer/web2.0 \ -E PATH=$R/usr/local/sbin:$R/usr/local/bin:$R/usr/sbin:$R/usr/bin:$R/sbin:$R/bin \ $R/bin/bash --- Unfortunately it doesn't seem to recurse properly so anything started from that bash will use the the host system's ld.so. To test this, from the above bash run: - /opt/cancer/web2.0/bin/ls ->fails with libc incompatibility error - /opt/cancer/web2.0/lib64/ld-linux-x86-64.so.2 /opt/cancer/web2.0/bin/ls -> OK Qemu could be patched to work around this problem; the problem is in qemu's execve() implementation. 1.c. almost-chroot with ld.so dispatcher TODO: Write a static linked small C program that replaces host's ld.so, looks at $0's path and depending on wheter it's in cancer/, chain-exec*() the original ld.so or the cancer system's ld.so 1.d. almost-chroot with LD_PRELOAD pro: less intrusive than ld.so dispatching con: won't work with static linked executables Write an LD_PRELOAD lib, perhaps injected using /etc/ld.so.preload, that hooks exec*() of libc, looks at $0 path and when an executable is in a cancer root, prefix the whole argv[] array with a new $0 for the given cancer ld.so. Cancer system's /etc/ld.so.preload needs editing too (after install). Note: ld.so removes itself from $0 so the prefixing is invisible to the child process; test with executing main in /work/cancer/argv0 prefixed with ld.so If child process is a static link executable, prefixing it with ld.so will fail with an error code. If static executables are ran directly, their exec() calls are not hooked so subsequent dynamic child processes will use the wrong ld.so 1.e. almost-chroot with clone and namespaces pro: implements partial chroot cleanly con: requires root (suid?) clone() can be used to create a new mount namespace in which each main directory in / is mounted from the host, except for /usr and /lib* which are mounted from the cancer "chroot". These mounts are not visible in the hsot system (?). Processes ran within sich a partial chroot would have access only to its own ld.so. 1.f. almost-chroot with ptrace pro: implements partial chroot without root con: hackish, ptrace syscall API is arch-dependent Parent process running as user forks a child and waits for sync; parent attaches ptrace to the child to hook fs related syscalls and syncs so the child can proceed; child calls exec() on the target process. The parent does mainly two things: - on an exec syscall prefixes argv[0] with the local ld.so - on any other syscall that has a path argument, it changes the path so that /lib, lib32, /lib64, /usr, /bin and /sbin are redirected silently 1.g. almost-chroot with ld.so audit pro: no root, no messing with host ld.so, can be activated selectively for cancer software invocation con: hackish; depends on env var (LD_AUDIT); fails with chromium ld.so offers an audit API which loads all .so files listed in env var LD_AUDIT as plugins (not linked to the target program). ld.so calls functions of those .so files upon linking events. It's possible to write an audit .so that doesn't use the ld.so API at all but looks at the executable path being linked and calls exec() using cancer ld.so when it detects the executable is under cancer root. It needs to avoid prefixing cancer ld.so to avoid infinite loop. The path to the executable and the original command line can be acquired from /proc. LD_LIBRARY_PATH can not be exported but the audit.so's exec() needs to pass the library path list to cancer ld.so using the --library-path argument. This is because the sequence of calls is: I. invoke /cancer/bin/sh II. triggers host ld.so III. audit figures and calls exec using /cancer/lib/ld.so IV. /cancer/bin/sh is linked and started using /cancer/lib/ld.so and /cancer/lib libs V. /cancer/bin/ls is executed from this cancer shell VI. first host ld.so is invoked again on /cancer/bin/ls because /lib/ld.so is hardwired in the elf VII. audit figures and calls exec using /cancer/lib/ld.so VIII. /cancer/bin/ls is linked and started using /cancer/lib/ld.so and /cancer/lib libs If there's LD_LIBRARY_PATH in effect at step VI, host ld.so will attempt to link audit.so against cancer libc which will fail. Test results: generally works, but chromium fails to start up. 2. uml pro: no root required, kernel version independence con: needs fs image --- #!/bin/sh export TMP=/tmp export PATH="/usr/lib64/uml:$PATH" linux mem=1000M "$@" --- TMP is required so that /dev/shm noexec permission not an error; the extra PATH is for the console xterm. The file system image, without partitions, shall be in ./root_fs. FAILS because chromium thinks there's no sse3. If that'd work, X would probably be hard to get through, the same way as with qemu-system, because of the extra kernel in between. 3. qemu-system modern firefox-esr and chroium are unable to use remote X sessions, thye drop to 0.001 fps. Tried with: - ssh -X - ssh -Y - direct TCP using DISPLAY=ip:8 (Xephyr)