Custom agent image not work

Problems and bugs
1

Hi,

I follow Configure agents | Datalore Documentation, and built my own image and tag it like

localhost:5000/my-agent:2023.1

and configurated to use it in agents-config.yaml, like

docker:
  network: datalore-agents-network
  dataloreHost: datalore
  instances:
    - my-agent
      image:  localhost:5000/my-agent:2023.1

and the machine type my-agent show in dropdown list,
however, when chosing to use this type,
the machine never get start successfully.
can anyone help:

  1. where can i find the log about the agent machine starting detail?
  2. what is the correct image url for local image?
2

The default given in the documentation works all right for me.

3

hi barcastaff,

could you give me some process detail on how you built the agent docker image, and an example of agents-config.yaml file?

many thanks.

4

Something like this:

docker:
  network: datalore-agents-network
  dataloreHost: datalore
  instances:
    - id: 1
      default: true
      label: "your-name"
      description: "your-description"
      image: docker.io/jetbrains/datalore-agent:2023.1
      deviceRequests:
        # I put in a GPU here. You don't have to.

Everything is basically the default recommendation from the documentation.

5

thanks!
finally i made it out, it is something conflict on my image definition(I guess).
for anyone come across this, following is my procedure:

  1. build image using “docker build -t my-agent:xxx .”
    1.1 becareful with the Dockerfile content(I am not quite sure and not know the cause, but it did help me)
    * not to change the owner of path /home/datalore
    * change back user to datalore (DO NOT leave root in last statement)
    1.2 make small steps every time in the Dockerfiler, and give enough test(build and start machine using the custom image)
    1.3 some experiences:
    * for programe need to create path in home dir, use root to mkdir then chown the path, and change back to datalore
    the Dockerfile example:
FROM jetbrains/datalore-agent:2023.1
LABEL maintainer="xxxx"


USER root
RUN mkdir -p /opt/software
RUN chown datalore:datalore /opt/software
# end of ====tag:mkdir======

USER datalore

# RUN curl -LO  https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3-scala2.13.tgz
COPY --chown=datalore:datalore ./spark-3.3.0-bin-hadoop3-scala2.13.tgz /opt/software
RUN tar -xvf /opt/software/spark-3.3.0-bin-hadoop3-scala2.13.tgz -C /opt/software
RUN ln -s /opt/software/spark-3.3.0-bin-hadoop3-scala2.13 /opt/software/spark
ENV SPARK_HOME=/opt/software/spark

USER datalore
# RUN /opt/python/envs/minimal/bin/pip install pyspark==3.3.0
COPY ./pyspark-3.3.0.tar.gz /opt/software
RUN /opt/python/envs/minimal/bin/pip install /opt/software/pyspark-3.3.0.tar.gz
# end of ====tag:spark======

USER datalore
# RUN /opt/python/envs/minimal/bin/pip install playwright==1.31.1
COPY ./playwright-1.30.0-py3-none-manylinux1_x86_64.whl /opt/software
RUN /opt/python/envs/minimal/bin/pip install /opt/software/playwright-1.30.0-py3-none-manylinux1_x86_64.whl
USER root
RUN mkdir -p /home/datalore/.cache
RUN chown datalore:datalore /home/datalore/.cache
USER datalore
RUN /opt/python/envs/minimal/bin/playwright install chromium
# end of ====tag:playwright======
  1. tag the image to another name and use it in the agents-config.yaml
    • using mounted volume in docker-compose.yaml(not agents-config, it is used by datalore server not agent) to do this , docker compose file like
version: "3.9"
services:
  datalore:
    image: jetbrains/datalore-server:2023.1
    ports:
      - "18082:8080"
    expose: [ "8081", "5050", "4060" ]
    networks:
      - datalore-agents-network
      - datalore-backend-network
    volumes:
      - ./dldata:/opt/data
      - /var/run/docker.sock:/var/run/docker.sock
      - ./datalore-agent/agents-config.yaml:/opt/datalore/configs/agents-config.yaml
    environment:
1 Like