The graphics-core20 Snap interface

alan_g · 2021-06-30T13:15:03.317Z

The problem with snapping userspace graphics drivers

Over the past few years developing graphical snaps to run on Ubuntu Core we’ve encountered various issues. One of these is the need to make userspace graphics drivers available inside snap confinement. There’s a partial solution in snapd that, on desktop systems can make these available from the host system but among the cases where this doesn’t work is on Ubuntu Core.

Until recently, we have been including the mesa drivers inside the snap as these have the least licensing issues, and cover a wide range of hardware. But it is not an ideal solution:

Both server (e.g. mir-kiosk) and client (e.g. mir-kiosk-kodi) snaps contain these drivers;
Security updates to mesa require all the server and client snaps to be rebuilt and distributed; and
There’s no way to introduce other drivers, even if Mir supports them.

This led to some frustration as there are a number of graphics stacks that Mir supports that were not supported by mir-kiosk. With Ubuntu Frame we have adopted a more flexible approach.

Introducing graphics-core20

Fundamentally, userspace graphics drivers are just files. And snapd has a way to share files between snaps: content interfaces. Care is needed to ensure that the binaries are ABI compatible, but as the binaries come from a stable archive (20.04LTS) that isn’t a problem here.

The Mir team are now maintaining a mesa-core20 content snap with the current mesa drivers from the archive:

lib contains the mesa shared libraries (add to LD_LIBRARY_PATH)
drv contains the mesa drivers (set LIBGL_DRIVERS_PATH/LIBVA_DRIVERS_PATH to this)
glvnd/egl_vendor.d contains the mesa ICD (set __EGL_VENDOR_LIBRARY_DIRS to this)
libdrm contains mesa configuration for driver support (layout to /usr/share/libdrm)
drirc.d contains mesa app-specific workarounds (layout to /usr/share/drirc.d)
etc/mir-quirks contains any Mir configuration for driver support (none for mesa)

We’ve also built and tested a number of snaps to prove this approach. As well as setting the above environment variables, they need the following plug declaration:

  graphics-core20:
    interface: content
    target: $SNAP/graphics
    default-provider: mesa-core20

There’s also two mesa specific directories that needs to be bind mounted:

layout:
  /usr/share/libdrm:  # Needed by mesa-core20 on AMD GPUs
    bind: $SNAP/graphics/libdrm
  /usr/share/drirc.d:  # Used by mesa-core20 for app specific workarounds
    bind: $SNAP/graphics/drirc.d

Finally, it is desirable to avoid shipping anything from mesa that gets pulled into the snap with a “cleanup” part similar to this:

  cleanup:
    after: [kodi, mir-kiosk-snap-launch]
    plugin: nil
    build-snaps: [ mesa-core20 ]
    override-prime: |
      set -eux
      cd /snap/mesa-core20/current/egl/lib
      find . -type f,l -exec rm -f $SNAPCRAFT_PRIME/usr/lib/${SNAPCRAFT_ARCH_TRIPLET}/{} \;
      rm -fr "$SNAPCRAFT_PRIME/usr/lib/${SNAPCRAFT_ARCH_TRIPLET}/dri"
      for CRUFT in bug drirc.d glvnd libdrm lintian man; do
        rm -rf "$SNAPCRAFT_PRIME/usr/share/$CRUFT"
      done
      # Workaround for https://bugs.launchpad.net/snapd/+bug/2055273
      mkdir -p "$SNAPCRAFT_PRIME/graphics"

(This example is taken from a mir-kiosk-kodi PR).

The `graphics-core20` libraries

As we’ve gained experience with this interface we’ve added components to the point we need to document the libraries that consumers can depend upon in .../lib and which providers must supply.
(As opposed to additional libraries that are here to support these).

Actual graphics interface

libEGL.so.1
libva.so.2
libvulkan.so.1
libGLESv2.so.2

Libraries we need for gbm/kms

Libdrm.so.2
libgbm.so.1

Wayland support

libwayland-client.so.0
libwayland-server.so.0

X support (optional)

libX11-xcb.so.1
libXau.so.6
libxcb-dri2.so.0
libxcb-dri3.so.0
libxcb-present.so.0
libxcb.so.1
libxcb-sync.so.1
libxcb-xfixes.so.0
libXdmcp.so.6
libxshmfence.so.1

The result

With this approach:

There’s only one copy of the drivers in one snap;
Mesa updates only require one snap to be rebuilt and distributed; and,
It is possible to replace mesa with other (compatible) drivers

Now and the future

We have been using this approach with Ubuntu Frame both with mesa-core20 and other drivers providing KMS, libgbm and an EGL supporting EGL_WL_bind_wayland_display.

Use of the graphics-core20 interface is fully integrated into both Ubuntu Frame and the examples we provide for packaging IoT GUIs.

Longer term, we’re hopeful that the experience gained will feed into improvements in graphics support in snapd that make this unnecessary when packaging snaps.

alan_g · 2021-08-03T09:04:16.736Z

libGLU: A note

Some graphics applications use libGLU.so.1 which is supplied by the libglu1-mesa apt package. We considered adding this to mesa-core20 and to the requirements for graphics-core20.

We decided not to add it as, although it is provided by mesa, it doesn’t appear to be driver specific and would impose additional requirements on implementing alternative implementations of the interface.

Application snaps that need libGLU should continuing staging libglu1-mesa themselves.

Motivating example: https://github.com/MirServer/mir-kiosk-scummvm/pull/9

alan_g · 2021-08-09T16:00:03.504Z

AMD GPUs: A note

/usr/share/libdrm/amdgpu.ids is a file referenced via a hard coded path in libdrm_amdgpu.so.1.0.0 which is included in the mesa-core20 snap providing this interface. Failing to find this file causes problems on AMD GPUs. (See, https://github.com/snapcrafters/scummvm/issues/31 for this problem in a different context.)

The file is accessed in Mesa’s libdrm (amdgpu/amdgpu_asic_id.c), where there’s no customization point (e.g. environment variable) enabling the path to be updated point a path provided by graphics-core20.

When /usr/share/libdrm/amdgpu.ids is included in a confined snap, it can be bind mounted using “layout” (see https://github.com/snapcrafters/scummvm/pull/32), but that gets messy with the graphics-core20 interface supplying either Mesa, or another provider that may not include this file.

After some experimentation this didn’t prove as messy as anticipated. Layouts don’t complain if the directory doesn’t exist. So, I’m proposing a solution: expose the directory via graphics-core20 and use layouts to put it where amdgpu_asic_id.c expects it:

https://github.com/MirServer/mesa-core20/pull/11

https://github.com/MirServer/mir-kiosk-kodi/pull/24

ogra · 2022-05-04T12:19:14.386Z

you might want to include vaapi and vdpau libs too since they are usually tied to the mesa/GL(ES) libs so that accelerated video playback is still functional when possible …

alan_g · 2022-05-04T12:58:31.027Z

Are you talking about the minimum that needs to be provided by content snaps that support this interface? Or what the mesa content snap provides:

    stage-packages:
      - libgl1-mesa-dri
      - va-driver-all
      - libegl1
      - libgles2

Note that va-driver-all is already included.

ogra · 2022-05-04T12:59:44.876Z

ah, sorry, i only looked at the list from the first post, not actually at the snapcraft.yaml …

there seem to be some more bits when i.e. looking at the steam snap:

https://github.com/canonical/steam-snap/blob/main/snap/snapcraft.yaml#L168

alan_g · 2022-05-24T11:37:44.183Z

After investigating some (not directly related) Nvidia suport issues we came across some more poorly documented environment variables. These may need some consideration too:

"…two environment variables that might be helpful here. __EGL_EXTERNAL_PLATFORM_CONFIG_FILENAMES specifies a list of platform config files to load and if that is not set, __EGL_EXTERNAL_PLATFORM_CONFIG_DIRS specifies a list of directories to search (defaulting to /etc/egl/egl_external_platform.d and /usr/share/egl/egl_external_platform.d). Both use colon as a separator.

jamesh · 2022-07-14T08:28:29.052Z

We recently tried integrating this snap with the Steam snap, but found it didn’t quite meet our requirements. It still seems like a great idea, and it’d be nice to be able to upgrade the graphics drivers independent of e.g. the GNOME platform snaps.

The main problems we encountered were:

libGLX_mesa.so.0 is not included in the snap. I know that you’re primarily focusing on the needs of Wayland clients and display servers, but it’d be nice to also support X11 clients (possibly going through Xwayland). The snap already bundles a bunch of X libraries, so the incremental cost of including the GLX driver is relatively minor.
Only one architecture. In the general case this is fine, but some snaps will need both the 32-bit and 64-bit drivers. For Steam this was necessary because it is launching a mix of 32-bit and 64-bit games. I imagine it could be useful for snaps that use Wine too. I’ll note that @alan_g’s kisak-core20 snap addresses this with additional lib and dri directories for i386 drivers.

I kind of wonder if it would have been better to keep the architecture based subdirectories (e.g. x86_64-linux-gnu) rather than flattening everything.

I also wonder if it’d make sense to have a “greedy plugs” style interface similar to what we use for GTK themes, to allow multiple drivers to be plugged at once. This’d be useful if we ever decide to distribute the Nvidia user space drivers as a snap, for instance.

And if we do want to support Nvidia drivers, we’ll probably also need to specify egl_external_platform.d / __EGL_EXTERNAL_PLATFORM_CONFIG_DIRS config files used by Nvidia’s egl-wayland library.

It’s probably not worth making big changes to the graphics-core20 interface contract you’ve created, but maybe we should review it for a core22 version?

fabioamd87 · 2022-09-28T08:12:31.626Z

Hello,
Today while trying to update snaps I got this error message:

fabio@abu:~$ sudo snap refresh
error: cannot perform the following tasks:
- Run configure hook of "snap-store" snap if present (run hook "configure": 
-----
cannot update snap namespace: cannot create symlink in "/usr/share/libdrm/amdgpu.ids": existing file in the way
snap-update-ns failed with code 1
-----)

Could it be related to this issue?
Thanks

alan_g · 2022-09-28T08:20:06.418Z

fabioamd87:

Could it be related to this issue?

The error you pasted looks like a problem with either the snap-store snap or snapd. It is not related to the graphics-core20 Snap interface.

fabioamd87 · 2022-09-28T11:12:12.707Z

reinstalling snap-store seems to fix the issue.