Docker-Base installiation with Podman for GPU

Problems and bugs
1

I installed datalore with podman by using docker-compose.yaml file provided and did very few changes to make it work.

I’m also able to attach external agents since configuration of external agents accepts a parameter called: additionalOptions where you can pass devices and environment variables.

First of all your documentation for Set up external agents is wrong!

podman run -it --rm -e NVIDIA_VISIBLE_DEVICES=all jetbrains/datalore-agent:2023.3 nvidia-smi
Error: runc: runc create failed: unable to start container process: error during container init: exec: "nvidia-smi": executable file not found in $PATH: OCI runtime attempted to invoke a command that was not found

Without having --device nvidia.com/gpu=all in the podman command it will fail and it’s failing because devices are not getting attached to the container.

Second problem is that: It looks like docker-base agent, which is datalore server itself I believe, does not have additionalOptions fields where you can put --device nvidia.com/gpu=all and since I use podman for docker-base agent as well, it sends --gpus all which podman does not understand from the looks of it. We should have had something similar to external agents have, so that we would pass --device nvidia.com/gpu=all on docker-base agents as well, then GPUs would be usable on podman installations too.

2

Also please note that:

image
image735×142 13.1 KB

This section of the documentation is also wrong becase:

on the other hand for podman:

  • -e NVIDIA_VISIBLE_DEVICES=all is just an environment variable and without --devices nvidia.com/gpu=all it will have no effect.

I assume this is probably the reason why podman cannot start a GPU attached container on master agents but it can do on external agents where you can do this:

  • additionalOptions: "-e NVIDIA_VISIBLE_DEVICES=all --device nvidia.com/gpu=all"