Resource Manager¶

Introduction¶

Industrial Edge's Resource Manager enables apps to access special kinds of resources, which can be hardware devices, external interfaces, software entities or the like. For each kind of resource, there can be multiple instances representing the actual devices etc., where the mapping to containers is done by the resource manager depending on an app's demands. In the following, we describe the necessary steps for device builders to enable resource management on Industrial Edge.

Prerequisites¶

Support for the Resource Manager requires a Linux kernel version of at least version 5.12 and IEDK of at least version 1.16.0.

Additional requirements are placed on the Docker version and installation approach. For the Resource Manager from IEDK versions less than or equal to version 1.23, the maximal supported Docker version is 20.10.24. Conversely, the Resource Manager from IEDK versions greater than or equal to 1.24 are tested with Docker versions up to 26.1.5. While higher Docker versions may also be compatible, they have not been explicitly tested with the Resource Manager.

The Resource Manager does not support installation of Docker via snap or in "Rootless mode". The recommended installation approach is to use the standard package manager for the target Linux distribution, utilizing the distribution's official Docker packages. To give an example for a Debian 12 machine: It is advised to install the official Debian Docker package rather than the Docker-provided Debian package.

For Debian-based systems, the major version of Docker remains stable throughout the release of the Debian distribution. This provides a consistent and stable platform for integration and deploying Docker-based applications using the Resource Manager. However, for the Docker packages provided directly by the Docker project (rather than through the Debian repositories), new Docker major versions are released independently of the IEDK and Debian distribution release. These newer major versions of Docker may not be compatible with the Resource Manager, thus installing or upgrading Docker in this case could lead to incompatibility issues.

Overview¶

The Resource Manager consists of multiple components which are shipped as part of the Industrial Edge Device Kit (IEDK):

File	Package(s)	Resource Class	Description
`resource-manager.tar.gz`	`ie-resource-manager.deb` `ie-resource-discovery.deb` `ie-runc-shim*.deb`	n/a	Resource Manager core components (resource discovery, abstraction, and claim management)
`resource-enablement-cpu.tar.gz`	`ie-cpu-resource-plugin*.deb`	`siemens.com/isolated_core`	Plugin for CPU isolation
`resource-enablement-network.tar.gz`	`ie-network-resource-plugin.deb` `ie-docker-network-plugin.deb`	`siemens.com/isolated_network` (via Docker network settings)	Plugins for network interface isolation
`resource-enablement-gpu.tar.gz`	`ie-nvidia-gpu-resource-plugin*.deb`	`nvidia.com/gpu`	Plugin for Nvidia GPUs
`resource-enablement-shared-memory.tar.gz`	`ie-shared-memory-resource-plugin*.deb`	`siemens.com/shared_memory`	Plugins for shared memory allocation
`resource-enablement-gpio.tar.gz`	`ie-gpio-resource-plugin*.deb`	`siemens.com/gpio`	Plugin for GPIO chip allocation

The packages contained in resource-manager.tar.gz are mandatory for managing any kind of resource. Hence, you need to install them on your system before proceeding with the steps below. The remaining packages are optional and responsible for managing certain resource classes via plugins.

NOTICE

Although the resource manager makes use of concepts developed for Kubernetes, Kubernetes need not and should not be installed on the edge device.

CPU Isolation¶

Using the CPU isolation plugin, selected CPUs can be allocated exclusively for an app to avoid interferences with other workloads, which is essential for real-time applications.

NOTICE

Installing the plugin contained in resource-enablement-cpu.tar.gz is not sufficient to guarantee real-time behavior. Make sure that your device is prepared to support real-time applications.

Configuration¶

As a device builder, you can control the allocation of CPUs. The device builder's default configuration is specified in the file /usr/lib/ie-cpu-resource-plugin/ie-cpu-resource-plugin.yaml. Any changes during operation are only applied to the user's configuration file /etc/ie-resource-plugins/ie-cpu-resource-plugin.yaml.

The CPU isolation plugin extracts the static parameters from the device builder's default configuration file and the non-static parameters from the user's configuration file (the user can only set non-static parameters). Both sets of parameters are then merged to obtain the actual configuration. In case of a firmware update, the user's configuration is kept and subsequently merged with a potentially updated device builder's configuration.

NOTICE

Make sure that /etc/ie-resource-plugins/ is empty before shipping the image.

The following settings are currently supported:

# CPUs that should be reserved for system tasks and are not isolatable
cpuset_system: "2"

# Statically configured system CPUs
cpuset_system_static: "0-1"

# Statically configured CPUs that should be ignored by Industrial Edge
cpuset_ignore_static: "0,4"

Each setting takes a CPU set defined as comma-separated list of ranges, where a range can also be a single CPU. Examples are: 0, 0,2,4, 0-1,3,8-9. Spaces are not allowed and ranges must not be decreasing, i.e., the CPU set 0, 4-3 is invalid in both respects.

NOTICE

cpuset_system can be changed dynamically by the administrator of an edge device, whereas both static configuration options, cpuset_system_static and cpuset_ignore_static, can only be changed by editing the configuration file. This allows device builders to taylor the basic configuration to their needs, while giving administrators sufficient freedom to adapt to the actual workloads.

In the following, the effective system CPU set (cpuset_system_effective) refers to the union of cpuset_system and cpuset_system_static.

cpuset_system_effective defines the set of CPUs that are reserved for housekeeping and are not given to Industrial Edge apps as isolated CPUs. They are, however, allowed to run Industrial Edge apps that aren't given dedicated isolated CPUs.
cpuset_ignore_static defines the set of CPUs that must not be touched by Industrial Edge altogether. If this option is not set (or set to the empty string), it defaults to the empty set. The set of CPUs usable by Industrial Edge is the difference between the set of online CPUs and the CPUs to ignore (online CPU set cpuset_ignore); it therefore includes the CPUs in cpuset_system_effective. All Industrial Edge CPUs, except for the CPUs in cpuset_system_effective, are isolatable.

NOTICE

cpuset_system_effective and cpuset_ignore_static may overlap. CPUs that are in both sets are used to run system processes, but they are not used to run Industrial Edge apps.

The following constraints exist on the aforementioned configuration options:

cpuset_system_static must contain CPU 0 and therefore is never empty (CPU 0 is often special as it may run important system processes and kernel threads, or handle interrupts or hardware-related tasks)
cpuset_system_effective must neither equal cpuset_ignore_static nor be a subset thereof (if all Industrial Edge CPUs were to be isolated, regular apps that do not require isolation would no longer be able to run)

For most use cases, the following values, which provide a high degree of flexibility for system administrators, are recommended as default for device builders (not taking into account hyperthreading as discussed below):

cpuset_system: "", cpuset_system_static: "0", cpuset_ignore_static: ""

If the target device is supposed to run special tasks on one or more dedicated CPUs, you may exclude them from being used by Industrial Edge using cpuset_ignore_static. As an example, if your device has many CPUs, where each may run a real-time application, it is recommended to reserve at least one CPU for system tasks such as interrupt handling:

cpuset_system: "", cpuset_system_static: "0,1", cpuset_ignore_static: "0"

In the above example, cpuset_system_static has been extended to also include CPU 1.

NOTICE

Changing the configuration requires a system restart, as the ie-cpu-partition service depends on these configuration values to set the allowed CPU sets of the default systemd cgroups on startup and to update the tuning profile.

Hyperthreaded Systems¶

The CPU sets specified under cpuset_system, cpuset_system_static, and cpuset_ignore_static map to logical CPUs, i.e., processing units that the operating system recognizes and can schedule tasks on. These logical CPUs don't necessarily coincide with the system's physical cores and can correspond to hyperthreads depending on the system's hardware and configuration.

The set of online logical CPUs is listed under /sys/devices/system/cpu/online. The values found under /sys/devices/system/cpu/cpu<number>/topology/thread_siblings_list, where cpu<number> is any of the online logical CPUs, provide information on the association of hyperthreads (also called thread siblings) with the system's physical cores.

For example, /sys/devices/system/cpu/cpu0/topology/thread_siblings_list shows the thread siblings that logical CPU 0 shares a physical core with (including itself). If only a single CPU is listed under a logical CPU's thread_siblings_list, then that logical CPU maps directly to a physical core. If multiple logical CPUs are listed, then that logical CPU shares a physical core with other logical CPUs. For example, if /sys/devices/system/cpu/cpu0/topology/thread_siblings_list contains CPU set 0-1, then logical CPU 0 shares a physical core with CPU 1 (/sys/devices/system/cpu/cpu1/topology/thread_siblings_list contains the same CPU set).

NOTICE

It need not be the case that consecutive logical CPUs are each other's thread siblings (e.g., logical CPU 1 and logical CPU 5 could be thread siblings).

The CPU resource plugin only allows for the isolation of physical cores. That is, in determining which logical CPUs are isolatable, it does not allow for the separation of thread siblings. Concretely, this means that if cpuset_online \ cpuset_ignore_static \ cpuset_system_effective = cpuset_isolatable contains a proper subset (i.e., not all) of a physical core's thread siblings, then none of the logical CPUs that make up that proper subset are considered isolatable.

For example, on a quad-core hyperthreaded system (with /sys/devices/system/cpu/online listing CPU set 0-7), if each two consecutive logical CPUs are thread siblings (i.e., logical CPU 0 and logical CPU 1 are thread siblings, logical CPU 2 and logical CPU 3 are thread siblings, etc.), and if the CPU resource plugin is configured with cpuset_system: "2,4", cpuset_system_static: "0", and cpuset_ignore_static: "6", then, in effect, no logical CPU is considered isolatable, because 0-7 \ 6 \ 0,2,4 = 1,3,5,7, and no combination of logical CPUs from 1,3,5,7 resolves to all of the thread siblings of a physical core.

When isolating a CPU on a hyperthreaded system, the CPU resource plugin intentionally only hands out a single thread sibling of a physical core to avoid that a real-time application interferes with another process running on the same core. The other thread siblings remain unused; they are not handed out to the application and are removed from the best-effort set.

Network Interface Isolation¶

The network interface isolation allows Industrial Edge applications to obtain direct access to a network interface inside the application container. When an application successfully claims an isolatable network interface, the resource management moves the corresponding network interface from the host system into the network namespace of the application container. No other application or process in the host system will be able to access or generate traffic on the network interface When listing the network interfaces in the host system, the network interface will not be visible until the application is stopped and the interface is moved back into the host system. Also, the network settings in the Industrial Edge user interface temporarily no longer lists the corresponding interface. Applications can use network interface isolation for example to implement real-time capable network stacks like Profinet, or for getting access to a PTP clock associated with a network interface. It is thus not only useful for real-time applications and consequently, it does not strictly require the underlying system to be tuned or CPU isolation to be present.

In order to support such scenarios, install the packages contained in resource-enablement-network.tar.gz. These packages are not needed if apps are supposed to access general-purpose I/Os only. The network interface isolation support consists of a network resource plugin (ie-network-resource-plugin*.deb) providing the integration into the Resource Manager including, for example, dynamic configuration by the operators, and a Docker network plugin (ie-docker-network-plugin*.deb) providing the integration into Docker such that network claims can be specified in the networks section of compose manifests.

Configuration¶

The device builder's default configuration is specified in the file /usr/lib/ie-network-resource-plugin/ie-network-resource-plugin.yaml. Any changes during operation are only applied to the user's configuration file /etc/ie-resource-plugins/ie-network-resource-plugin.yaml.

The network isolation plugin extracts the static parameters from the device builder's default configuration file and the non-static parameters from the user's configuration file (the user can only set non-static parameters). Both sets of parameters are then merged to obtain the actual configuration. In case of a firmware update, the user's configuration is kept and subsequently merged with a potentially updated device builder's configuration.

NOTICE

Make sure that /etc/ie-resource-plugins/ is empty before shipping the image.

The following example shows the structure of the configuration files:

networks_isolatable:
  - name: eno2
    annotations:
      - label: mylabel
  - name: eno3
    annotations:
      - label: foo
        vlan: 42
      - label: bar
        vlan: 100
networks_ignore_static:
  - eno1

The field networks_isolatable specifies an array of network interfaces that can be claimed exclusively by Industrial Edge apps. Each isolatable interface is identified by the network interface name assigned by the host operating system (e.g., eno2 or eno3). Only these interfaces can be claimed by Industrial Edge apps. Device builders can leave the array of isolatable networks empty unless they would like to provide defaults. Operators can add isolatable interfaces by changing the configuration in the Resource Manager user interface. More information including explanations for the annotations can be found in the Get Started & Operate section.

The only network configuration setting specific to device builders is networks_ignore_static. It can be used to disallow a list of network interfaces of being configured as isolatable by the operator. This setting cannot be overriden by any user/operator configuration changes. Any interface marked as isolatable through the networks_isolatable field but appearing in the ignore list is filtered out.

NOTICE

If networks_ignore_static is left empty, users may accidentally isolate all available network interfaces, thus making the device inaccessible. For this reason, it is strongly recommended to define one or more interfaces as non-isolatable, at least on physical (non-virtual) devices. For example, if eno1 is the main interface used for communicating with the Industrial Edge Management System (IEM), specify it in the plugin's configuration file as shown above. In case of virtual edge devices, naming the main interface upfront without doubt might not be possible, however. When leaving networks_ignore_static empty, there should be a low-effort solution to recover from accidental misconfigurations. For virtual edge devices, make sure that a new virtual network interface can be added temporarily allowing the device administrator to revert a misconfiguration.

PTP Device Support¶

The Precision Time Protocol (PTP) is a protocol to synchronize clocks between devices. Some Network Interface Cards have hardware support for this protocol to achieve accurate time synchronization over the network. The device on the Network Interface Card which enables this hardware support is referred to as "PTP Device". If the Network Interface Card of an isolated network has a PTP device, then the PTP device is also mounted to the requesting Docker container as a read-write device.

Docker Root Directory¶

By default, Docker stores most of its data under the /var/lib/docker directory. If you relocate Docker's root directory, you have to communicate the new path to both plugins via an environment variable. To this end, set DOCKER_ROOT_DIRECTORY in the following systemd service files:

/lib/systemd/system/ie-docker-network-plugin.service
/lib/systemd/system/ie-network-resource-plugin.service

Real-time Applications¶

Basic Prerequisites¶

Linux Kernel¶

To enable execution of apps with real-time (low latency) requirements on Industrial Edge, you need a real-time kernel. Most Linux distributions offer packages, e.g., on Debian you can simply apt-get install linux-image-rt-amd64. To verify, check that the output of uname -a contains the string PREEMPT_RT or PREEMPT RT, or that the file /sys/kernel/realtime exists and contains the value 1.

Cgroups v2¶

The CPU isolation plugin relies on kernel control groups (cgroups) to control the CPU sets available to apps. There currently exist two interfaces, v1 and v2, and in general, a system can be operated in one of three ways:

cgroup v1 only
cgroup v2 only
hybrid, with both v1 and v2 enabled

The cgroup v1 interface is deprecated, and major projects like systemd have already announced that they will drop support for it. This is why the CPU isolation feature only supports the cgroup v2 interface. To verify, run mount | grep cgroup. You should see only one entry, it should be of type cgroup2, and the mount location should be /sys/fs/cgroup, e.g.:

$ mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)

As mentioned above, legacy setups where cgroup v2 is mounted in a different location (/sys/fs/cgroup/unified) are not supported.

Power Management¶

In addition to cgroup v2 support, Power Management Quality of Service (PM QoS) should be available on the host, and the corresponding device /dev/cpu_dma_latency should exist so that it can be mounted into containers. This gives apps an interface to configure power management on their own. Otherwise, make sure that power management is configured system-wide in a way that minimizes timing variations, e.g., due to sleep states.

System Tuning¶

To ensure proper real-time behavior, additional measures are required that need to be addressed at system level. We refer to this as "system tuning". Defining optimal tuning measures depends on the platform and the applications, i.e., there is no "one-size-fits-all" solution. As Industrial Edge runs on different platforms and is meant to execute arbitrary apps, the recommendations given here are a compromise:

They aim to provide acceptable latencies for most apps and were developed using measurements for selected benchmark applications.
They focus on generic (non device-dependent) measures that are not overly intrusive, i.e., don't require global changes/settings that can impact other applications in the system. Hence, we change as few system defaults as possible.

As a consequence, there may be additional (more aggressive) tuning measures that improve application latencies for specific platforms.

TuneD¶

We recommend TuneD for system tuning. Using TuneD to perform system tuning has advantages, both technical and non-technical:

TuneD is an established Open Source project, and using it means you don't have to implement and rely on own custom-built tuning scripts.
Tuning profiles offer a flexible way of describing tuning measures.
TuneD monitors the system for thread-creation events. This way, it can adapt the affinity and/or priority of newly created kernel threads (cf. the last item on the list above).
TuneD can perform other tuning tasks beyond CPU isolation. If your platform and/or application requires additional measures, you can simply extend the tuning profile.
TuneD enables applications to dynamically tune the per-cpu kernel threads of isolated cores and the network I/O path of isolated networks.

The steps to install TuneD depend on the Linux distribution. The minimum version we support is v2.20.0. For Debian, distributions since Bookworm include a recent version of TuneD. You can simply install it with apt-get install tuned. For Debian Bullseye, the included TuneD version v2.15.0 is not supported, but v2.20.0 can be installed from the backport repositories (bullseye-backports).

To allow applications to dynamically tune per-CPU kernel threads and network I/O path, the CPU and network isolation plugins require a newer version of TuneD that includes the kthread plugin. At the time of writing, however, the kthread plugin is not yet included in any official TuneD release. We recommend building a custom TuneD Debian package based on v2.24.0 using the kthread plugin pull request.

Linux Perf¶

To monitor the system and get notified whenever a new process or thread is created, TuneD uses Linux perf. The functionality offered by perf is implemented in part in the kernel itself, and in part by supporting userland libraries and tools. It's those libraries and tools that we're talking about here.

When using Debian 12 (Bookworm), simply install the package linux-perf. On Debian 11 (Bullseye), use the package from the backport repositories (bullseye-backports) as the one provided by Debian 11 will not work. In both cases, you can skip the following subsections and proceed with enabling the tuning profile.

Python Bindings¶

As TuneD is implemented in Python, it also requires the Python bindings of the Linux perf userland libraries and tools. Whether the perf package includes the required Python bindings depends on your distribution. For example, the RedHat flavors of Linux (RHEL, CentOS, Fedora) include the bindings, whereas Ubuntu does not. The change in Debian that includes the bindings can be found here.

To test if you have the Python bindings (after installing perf in the way preferred by your distribution), run:

python3 -c "import perf"

If the response is for example

$ python3 -c "import perf"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'perf'

then TuneD will not work as expected.

If the bindings are not included in the packages supplied by your distribution, you may need to build your own perf package with that patch.

Version Compatibility and Versioned Packages¶

The source code of the userland perf libraries and tools is part of the Linux kernel source tree, and its build and packaging are closely linked to that of the kernel itself. Thus, the required steps depend on how you build and/or integrate the kernel for your platform.

For kernels up to versions of around 5.16 or 5.18, the version (major.minor) of the userland libraries and tools must match that of the kernel. For more recent kernel versions, that strict requirement has been removed, so a newer version of the userland libraries and tools (the package linux-perf) will also work on older kernels. In case of Debian that means that the situation has improved as Debian 11 (Bullseye, with bullseye-backported kernel 6.1) and Debian 12 (Bookworm, with kernel 6.1) use the aforementioned more recent kernel versions.

Since Debian allows the installation of multiple kernel packages at the same time, they deliver versioned packages of Linux perf, so the installed binaries don't conflict. So, you could find packages linux-perf_4.19 and linux-perf_6.1, which can be installed simultaneously, and the correct binaries are selected at runtime based on the currently running kernel. If you're using a kernel before 5.16/5.18, and you need to build your own linux-perf package for your Debian system, then you need to build such a versioned package. However, as mentioned in the "Prerequisites" section, the minimal supported kernel version for the Resource Manager is 5.12. And if you're not using the Debian kernel sources, you need to apply two patches from Debian:

Find the patches Debian applies for your kernel version, e.g., for kernel 6.1.
You need the patches tools-perf-install.patch and tools-perf-version.patch.

In addition, when using a versioned package of linux-perf, you need to install a wrapper that lets you import perf from Python and selects the correct version based on the currently running kernel. You can find the file perf.py in this merge request. Install that code in the Python path, e.g., in /usr/lib/python3/dist-packages/.

Tuning Profile¶

The ie-cpu-resource-plugin package delivers the iedge TuneD profile, located in /lib/tuned/iedge/tuned.conf. The profile is configured using the file /etc/tuned/iedge-variables.conf, where it is possible to define the relevant CPU sets. The ie-cpu-partition service ensures that this configuration file is up-to-date and in sync with the configuration used by the ie-cpu-resource-plugin.

After installation, the iedge tuning profile needs to be activated:

tuned-adm profile iedge

GPU Allocation¶

Accessing Graphics Processing Units (GPUs) from Industrial Edge apps requires the host system to be prepared accordingly. This includes, for example, the installation of vendor-specific device drivers (Linux kernel modules) and low-level libraries. In the following, we describe the necessary steps for Nvidia GPUs. Support for further GPUs may be added in the future.

Nvidia GPUs¶

As the first step, unpack resource-enablement-nvidia-gpu.tar.gz and install the contained Resource Manager plugin. Unlike for CPU and network interface isolation, the Nvidia GPU plugin does not provide any configuration options.

If the target device's operating system does not yet include device drivers for Nvidia GPUs, you have to install them manually. There are three different flavors:

Closed-source drivers distributed as binaries under a proprietary license by Nvidia. Most Linux distributions provide ready-to-use packages. On Debian, for example, the drivers are contained in the package nvidia-driver.
Open-source drivers published by Nvidia under MIT/GPLv2 license. These drivers are still relatively new and not yet part of all Linux distributions.
Open-source nouveau drivers developed by independent software engineers, partly by reverse-engineering the closed-source drivers. These drivers are mainly used for graphics and do not support Nvidia CUDA, which is used, for example, by AI applications to execute neural networks. For this reason, they are not supported by Industrial Edge's Resource Manager.

We recommend using Nvidia's open-source drivers, which support CUDA and are not subject to licensing restrictions. These restrictions arise from the fact that the Linux kernel is licensed under GPLv2. As a consequence, any derived work must be licensed under the GPLv2 as well, which conflicts with the proprietary license of the closed-source drivers.

If the open-source drivers are not shipped with your Linux distribution, download them from GitHub. To compile and install the drivers, follow the instructions in the README file.

NOTICE

Nvidia does not officially support Linux kernels with PREEMPT-RT patch. However, you can override related checks by setting the environment variable IGNORE_PREEMPT_RT_PRESENCE to 1 before compiling the drivers.

To avoid problems, make sure that the nouveau driver is disabled. On Debian, add the following lines to /etc/modprobe.d/blacklist-nouveau.conf:

blacklist nouveau
options nouveau modeset=0

After that, regenerate the kernel's initial RAM file system (sudo update-initramfs -u) and reboot the system.

Finally, you have to install CUDA. For a full installation, download a package matching your system from the Nvidia website and follow the given instructions. This will install a variety of tools and libraries including, among other things, support for graphics.

NOTICE

The CUDA package must match the driver version. For example, if you installed version 535.104.05 of the drivers (on a 64-bit x86 Debian 11 system), install cuda-repo-debian11-12-2-local_12.2.2-535.104.05-1_amd64.deb.

Alternatively, you may want to restrict yourself to components actually needed by the users of your device. The following table summarizes a minimal installation on Debian targeting compute applications:

Package Name	Description
`libcuda1`	Runtime support for CUDA applications
`libnvidia-ptxjitcompiler1`	JIT compiler generating GPU machine code
`libnvidia-nvvm4`	Library for JIT link-time optimization
`libnvidia-ml1`	Library providing a monitoring and management API
`libnvidia-allocator1`	Library used internally by other driver components
`nvidia-smi`	NVIDIA System Management Interface

These packages can also be obtained from Nvidia.

NOTICE

As for the closed-source drivers, redistributing CUDA or components thereof may be subject to licensing restrictions (see above).

Carefully check any legal implications and mitigations.

The plugin for Nvidia GPUs assumes that the above components are located in the following directories (as it is the case for Debian packages):

Components	Directory	Minimally required set of files
Kernel modules	`/lib/modules/<kernel-release>/kernel/drivers/video`	`nvidia.ko`, `nvidia-modeset.ko`, `nvidia-uvm.ko`, `nvidia-drm.ko`
Tools	`/usr/lib/nvidia/current`	`nvidia-smi`
Libraries	`/usr/lib/x86_64-linux-gnu/nvidia/current`	`libcuda.so`, `libnvidia-ptxjitcompiler.so`, `libnvidia-nvvm.so`, `libnvidia-ml.so`, `libnvidia-allocator.so*`

NOTICE

Nvidia provides an installer script which places some of the files in other directories. Therefore, it is recommended to install the components as packages or manually (device drivers).

Shared Memory Allocation¶

The Shared Memory Resource Plugin allows Industrial Edge applications to obtain direct access to a shared memory filesystem inside the application container. When an application successfully requests a shared memory filesystem, the resource management system provides the corresponding shared memory filesystem and makes it accessible to the application container. The memory is created by the resource management system, and the application can only access and mount this pre-created memory. Other applications on the host system can not also access or write to the shared memory filesystem without mounting. Shared memory resources can be used between applications; that is, multiple applications can use the same resource at the same time, provided they request the same resource. Applications can use shared memory, for example, to implement high-speed inter-process communication or for sharing large amounts of data between processes.

In order to support such scenarios, install the packages contained in resource-enablement-shared-memory.tar.gz. The shared memory support consists of a shared memory resource plugin (ie-shared-memory-resource-plugin*.deb) providing the integration into the Resource Manager including, for example, dynamic configuration by the operators.

Configuration¶

The Device Builder should not apply any specific configuration for the shared memory plugin. Only the default configuration should exist under the directory /usr/lib/ie-shared-memory-resource-plugin/. Any changes made during operation should only be applied to the user's configuration file /etc/ie-resource-plugins/ie-shared-memory-resource-plugin.yaml. The shared memory plugin extracts the static parameters from the device builder's default configuration file and the non-static parameters from the user's configuration file (the user can only set non-static parameters). Both sets of parameters are then merged to obtain the actual configuration. In case of a firmware update, the user's configuration is kept and subsequently merged with a potentially updated device builder's configuration.

NOTICE

Make sure that /etc/ie-resource-plugins/ is empty before shipping the image.

The default configuration is as follows:

shared_memories: []
shared_memory_limit_in_mb_static: 4096

GPIO Chip Allocation¶

General Purpose I/O (GPIO) chips available on an Industrial Edge device can be made available to applications using the GPIO resource plugin Access to GPIO chips is enabled via the character devices /dev/gpiochip* provided by the kernel. This means that the host system must ensure that the drivers (kernel modules) for the GPIO chips are installed before the GPIO chips can be allocated to apps.

To detect GPIO chips present in the system, use the tools gpiodetect and gpioinfo provided by libgpiod. On Debian, they are available in the gpiod package:

$ gpiodetect
gpiochip0 [pinctrl-bcm2711] (58 lines)
gpiochip1 [raspberrypi-exp-gpio] (8 lines)
$ gpioinfo raspberrypi-exp-gpio
gpiochip1 - 8 lines:
    line   0:      "BT_ON"   "shutdown"  output  active-high [used]
    line   1:      "WL_ON"       unused  output  active-high
    line   2: "PWR_LED_OFF" "PWR" output active-low [used]
    line   3: "GLOBAL_RESET" unused output active-high
    line   4: "VDD_SD_IO_SEL" "vdd-sd-io" output active-high [used]
    line   5:   "CAM_GPIO" "cam1_regulator" output active-high [used]
    line   6:  "SD_PWR_ON" "regulator-sd-vcc" output active-high [used]
    line   7:    "SD_OC_N"       unused   input  active-high

The GPIO resource plugin identifies GPIO chips by their labels (pinctrl-bcm2711 and raspberrypi-exp-gpio in the example above)

Configuration¶

As a device builder, you can control the allocation of GPIO chips. The device builder's default configuration is specified in the file /usr/lib/ie-gpio-resource-plugin/ie-gpio-resource-plugin.yaml. Any changes during operation are only applied to the user's configuration file /etc/ie-resource-plugins/ie-gpio-resource-plugin.yaml.

The GPIO plugin extracts the static parameters from the device builder's default configuration file and the non-static parameters from the user's configuration file (the user can only set non-static parameters). Both sets of parameters are then merged to obtain the actual configuration. In case of a firmware update, the user's configuration is kept and subsequently merged with a potentially updated device builder's configuration.

NOTICE

Make sure that /etc/ie-resource-plugins/ is empty before shipping the image.

The following settings are currently supported:

gpiochips:
  - name: pinctrl-bcm2711
    labels:
      - label1
gpiochips_ignore_static:
  - raspberrypi-exp-gpio

The field gpiochips is used to attach (optional) labels to GPIO chips. Operators can add labels by changing the configuration in the Resource Manager user interface. More information can be found in the Get Started & Operate section.

The only configuration setting specific to device builders is gpiochips_ignore_static. It can be used to exclude a list of GPIO chips from being allocated to Industrial Edge apps. This setting cannot be overridden by any user/operator configuration changes. The list may contain non-existent chips, in which case the corresponding entries are silently ignored.

Configuring Capabilities¶

Users can view the resources being managed and their status via the IE Runtime UI. As a device builder, you need to include the below JSON object in the capabilities.json file of the device. Setting the defaultStatus field to enabled configures the UI to display resource management details.

For example:

{
    "capabilityId": "host.resourcemanager",
    "name": "Resource Manager Plugin Support",
    "description": "Specifies support for resource manager plugins on the device and notifies Industrial Edge to use these.",
    "scope": "protected",
    "property": "optional",
    "details": {
        "defaultStatus": "disabled",
        "ie.device.hardware.cpu_allocation": "false",
        "ie.device.hardware.nic_allocation": "false",
        "ie.device.hardware.gpu_allocation": "false",
        "ie.device.hardware.memory_allocation":"false"
    }
}

The fields ie.device.hardware.cpu_allocation, ie.device.hardware.nic_allocation, ie.device.hardware.gpu_allocation, and ie.device.hardware.memory_allocation must be set to true when the respective plugins are installed on the device.

The following table associates the above capabilities with the corresponding resource classes:

Capability	Resource Class
`ie.device.hardware.cpu_allocation`	`siemens.com/isolated_core`
`ie.device.hardware.nic_allocation`	`siemens.com/isolated_network`
`ie.device.hardware.gpu_allocation`	`nvidia.com/gpu`
`ie.device.hardware.memory_allocation`	`siemens.com/shared_memory`

These capabilities and their configuration are subject to change.

NOTICE

The values specified in the host.runtime.nfr capability hold for configurations without isolated CPUs. If you are using the Resource Manager's CPU plugin, the cpu property, which imposes an upper limit on the apps' CPU usage, may no longer be valid. Similarly, apps isolating CPUs are not bound to the memory property. To ensure that the specified limits are obeyed system-wide, any apps that have claimed isolated CPUs must be stopped.

Configuring the Resource Manager¶

It is possible to enable or disable the Resource Manager as a whole or individual resource classes after installation. This is complementary to the settings in the file /etc/capabilities.json and needed for the actual enablement and disablement of Resource Manager components. Both the actual enable/disable and the entries in /etc/capabilities.json should be synchronized.

The script /usr/lib/ie-resource-manager/ie-resource-manager-control.sh can be used to

check the status of the Resource Manager,
completely enable or disable the Resource Manager, and to
enable or disable individual resource classes.

Usage¶

The script is used as follows:

/usr/lib/ie-resource-manager/ie-resource-manager-control.sh {status|enable|disable|enable-and-start|enable-swagger-ui|disable-swagger-ui} [resource-class-name]

The options are:

status: Displays the status of the Resource Manager and each resource class. When invoked with root privileges, it also shows the status of the Docker runtime.
enable: Enables the Resource Manager or one of the resource classes.

!!! info "NOTICE" This only enables the services, but does not actually start them. This option can be used to configure the system in the early boot phase (before running systemd) or just before a (scheduled) reboot. The setting is persistent across reboots and requires root privileges.

disable: Disables the Resource Manager or one of its resource classes. This will stop and disable all related services. The setting is persistent across reboots and requires root privileges.
enable: Enables the Resource Manager or one of the resource classes. This only enables the services, but does not actually start them. This option can be used to configure the system before a (scheduled) reboot. The setting is persistent across reboots and requires root privileges. This option is not supported during an active systemd boot process.
disable: Disables the Resource Manager or one of its resource classes. This will stop and disable all related services. It is possible to use this option during an active systemd boot process provided it is used before Resource Manager services and containerd are started. The setting is persistent across reboots and requires root privileges.
enable-and-start: Same as enable, but also starts the services. This can be used by developers in the terminal to avoid circular dependencies or deadlocks. Requires root privileges.
enable-swagger-ui and disable-swagger-ui: Enables or disables the Resource Manager debug interface. If enabled, the Resource Manager debug interface can be reached on port 8123. Normally disabled. Do not use in production systems for security reasons. Requires root privileges.

NOTICE

The script is supposed to be executed during startup. Calling it during operation to dynamically enable/disable the Resource Manager as a whole or individual resource classes is not supported. Calling the script during system operation to dynamically enable/disable the Resource Manager as a whole or individual resource classes is not supported. The script is supposed to be executed during system configuration. That includes usage during system set-up (e.g. image building), during first boot configuration (for disabling resource manager entirely or parts of it), before a reboot (e.g. part of a hard reset), or usage as a developer utility.

NOTICE

Contrary to disabling, enabling services during systemd boot is not supported. Systemd (which is used by the script under the hood) will not enable services during an active boot and enabling an already enabled service may disturb the service start ordering. As all Resource Manager services are enabled by default, selectively disabling them during first boot is sufficient to configure Resource Manager. This works provided the configuration is done very early in boot before Resource Manager services and containerd are started (ie-cpu-partition.service is the first Resource Manager service running in basic.target).

Examples¶

Status Check¶

$ sudo /usr/lib/ie-resource-manager/ie-resource-manager-control.sh status

Resource Manager Core:
   enabled and running
Docker Runtime 'iedge':
   enabled and running
Resource Class "siemens.com/isolated_core":
   enabled and running
Resource Class "siemens.com/isolated_network":
   enabled and running

NOTICE

The script needs not necessarily be executed with sudo, but in that case may not display all information about the Docker runtime status.

Disable Resource Manager¶

$ sudo /usr/lib/ie-resource-manager/ie-resource-manager-control.sh disable
Resource Class "siemens.com/isolated_core": disabling "ie-cpu-resource-plugin.service"
Resource Class "siemens.com/isolated_core": disabling "ie-cpu-partition.service"
Resource Class "siemens.com/isolated_network": disabling "ie-network-resource-plugin.service"
Resource Class "siemens.com/isolated_network": disabling "ie-docker-network-plugin.service"
Resource Manager Core: disabling "ie-resource-manager.service"
Resource Manager Core: disabling "ie-resource-discovery.service"

$ sudo /usr/lib/ie-resource-manager/ie-resource-manager-control.sh status

Resource Manager Core:
   disabled and stopped
Docker Runtime 'iedge':
   disabled, but running
Resource Class "siemens.com/isolated_core":
   disabled and stopped
Resource Class "siemens.com/isolated_network":
   disabled and stopped

Docker is not restarted for performance reasons. The docker runtime status can be updated by sudo systemctl restart docker.service or will be correct at next boot.

It is possible to run /usr/lib/ie-resource-manager/ie-resource-manager-control.sh disable from within a systemd service, provided it is done before all Resource Manager services and containerd.service are to be started.

Enable Resource Manager (again)¶

$ sudo /usr/lib/ie-resource-manager/ie-resource-manager-control.sh enable
Resource Manager Core: enabling "ie-resource-discovery.service"
Resource Manager Core: enabling "ie-resource-manager.service"
Resource Class "siemens.com/isolated_core": enabling "ie-cpu-resource-plugin.service"
Resource Class "siemens.com/isolated_core": enabling "ie-cpu-partition.service"
Resource Class "siemens.com/isolated_network": enabling "ie-network-resource-plugin.service"
Resource Class "siemens.com/isolated_network": enabling "ie-docker-network-plugin.service"

$ sudo /usr/lib/ie-resource-manager/ie-resource-manager-control.sh status

Resource Manager Core:
   enabled but stopped (start required)
Docker Runtime 'iedge':
   enabled and running
Resource Class "siemens.com/isolated_core":
   enabled but stopped (start required)
Resource Class "siemens.com/isolated_network":
   enabled but stopped (start required)

The services are enabled but not started.

After a reboot everything is correct. Alternatively, one can start the services by issuing the enable-and-start command without having to wait for a reboot:

$ sudo /usr/lib/ie-resource-manager/ie-resource-manager-control.sh enable-and-start
Resource Manager Core: enabling and starting "ie-resource-discovery.service"
Resource Manager Core: enabling and starting "ie-resource-manager.service"
Resource Class "siemens.com/isolated_core": enabling and starting "ie-cpu-resource-plugin.service"
Resource Class "siemens.com/isolated_core": enabling and starting "ie-cpu-partition.service"
Resource Class "siemens.com/isolated_network": enabling and starting "ie-network-resource-plugin.service"
Resource Class "siemens.com/isolated_network": enabling and starting "ie-docker-network-plugin.service"

$ sudo /usr/lib/ie-resource-manager/ie-resource-manager-control.sh status

Resource Manager Core:
   enabled and running
Docker Runtime 'iedge':
   enabled and running
Resource Class "siemens.com/isolated_core":
   enabled and running
Resource Class "siemens.com/isolated_network":
   enabled and running

This does permanently enable the relevant services and will immediately start them. In case the resource manager as a whole is enabled, docker (including all running containers) is restarted as well.

Disable a Single Resource Class¶

To disable a single resource class, run the following commands:

$ sudo /usr/lib/ie-resource-manager/ie-resource-manager-control.sh disable siemens.com/isolated_core
Resource Class "siemens.com/isolated_core": disabling "ie-cpu-resource-plugin.service"
Resource Class "siemens.com/isolated_core": disabling "ie-cpu-partition.service"

$ sudo /usr/lib/ie-resource-manager/ie-resource-manager-control.sh status

Resource Manager Core:
   enabled and running
Docker Runtime 'iedge':
   enabled and running
Resource Class "siemens.com/isolated_core":
   disabled and stopped
Resource Class "siemens.com/isolated_network":
   enabled and running

It is possible to run /usr/lib/ie-resource-manager/ie-resource-manager-control.sh disable <resource-name> from within a systemd service, provided it is done before all Resource Manager services and containerd.service are to be started. Enabling resource manager or parts of it is not supported from within a systemd service.

Resource Manager State Persistence¶

The ie-resource-manager.service maintains the resource allocation state (mapping of resources to containers) and the user data of all resource plug-ins. This information needs to be persisted to protect against either forced (by developer) or unwanted (service crash) resource manager service restarts.

Hence, all state information is stored in /var/lib/ie-resource-manager/Allocation_Persistence.json on disk. When an application starts or stops, the corresponding state in the json file is updated. The ie-resource-manager.service program reads the state from the json file and restores the state in memory whenever it starts. The device builder should never make any changes to this file.

The resource manager performs cleanup action after (re-)start of the docker daemon to prevent resources being incorrectly marked as occupied. This is important to properly handle unclean shutdown scenarios (e.g. system power outage) for all app restart policies.