Issues with running notebooks on Datalore Enterprise on-premise

patrick_h · April 11, 2023, 1:35pm

I have a new installation that I have setup using docker on our local infrastructure and I keep receiving the same error shown here:

Any help would be greatly appreciated.

Here is a snippet from the log dump showing the exceptions:


00:02:39.566 ERROR [Output OT] j.d.b.c.t.ThrowableHandler               Exception handled at ThrowableHandler: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@212ddeb2[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@57a89230[Wrapped task = jetbrains.datalore.base.jvm.edt.ExecutorEdtManager$ExecutorEdt$$Lambda$464/0x00000008011a24c8@7a4f754]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@512f3079[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
jetbrains.datalore.base.common.edt.EdtException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@212ddeb2[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@57a89230[Wrapped task = jetbrains.datalore.base.jvm.edt.ExecutorEdtManager$ExecutorEdt$$Lambda$464/0x00000008011a24c8@7a4f754]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@512f3079[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
	at jetbrains.datalore.base.jvm.edt.ExecutorEdtManager.schedule(ExecutorEdtManager.java:58)
	at jetbrains.datalore.base.jvm.edt.EdtManagerPool$EdtManagerAdapter.schedule(EdtManagerPool.java:17)
	at jetbrains.ocelot.ot.model.protocol.OtChannelExecutingDelegate.commandReceived(OtChannelExecutingDelegate.java:15)
	at jetbrains.ocelot.ot.model.protocol.OtChannelDelegate.commandReceived(OtChannelDelegate.java:21)
	at jetbrains.ocelot.ot.model.protocol.server.java.migration.MigrationOtPort$MigrationOutput.a(MigrationOtPort.java:89)
	at jetbrains.ocelot.ot.model.protocol.server.java.migration.MigrationOtPort$MigrationOutput.commandReceived(MigrationOtPort.java:69)
	at jetbrains.ocelot.ot.model.protocol.OtChannelDelegate.commandReceived(OtChannelDelegate.java:21)
	at jetbrains.ocelot.ot.model.protocol.server.BaseOtServerPort$1.commandReceived(BaseOtServerPort.java:2)
	at jetbrains.ocelot.ot.model.protocol.server.BaseOtServerPort.sendCommand(BaseOtServerPort.java:15)
	at jetbrains.ocelot.ot.model.protocol.server.BaseOtServer$BaseOtPortServer.fireChange(BaseOtServer.java:24)
	at jetbrains.ocelot.ot.model.protocol.server.DefaultOtServerPort.commandReceived(DefaultOtServerPort.java:77)
	at jetbrains.ocelot.ot.model.protocol.server.BaseOtServerPort$MyInput.commandReceived(BaseOtServerPort.java:62)
	at jetbrains.ocelot.ot.model.protocol.OtChannelDelegate.commandReceived(OtChannelDelegate.java:21)
	at jetbrains.ocelot.ot.model.protocol.server.java.migration.MigrationOtPort$1.commandReceived(MigrationOtPort.java:7)
	at jetbrains.ocelot.ot.model.protocol.OtChannelExecutingDelegate.lambda$commandReceived$0(OtChannelExecutingDelegate.java:54)
	at jetbrains.datalore.base.jvm.edt.ExecutorEdtManager$ExecutorEdt.lambda$wrap$1(ExecutorEdtManager.java:49)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@212ddeb2[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@57a89230[Wrapped task = jetbrains.datalore.base.jvm.edt.ExecutorEdtManager$ExecutorEdt$$Lambda$464/0x00000008011a24c8@7a4f754]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@512f3079[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:715)
	at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:748)
	at jetbrains.datalore.base.jvm.edt.ExecutorEdtManager$ExecutorEdt.schedule(ExecutorEdtManager.java:37)
	at jetbrains.datalore.base.jvm.edt.ExecutorEdtManager.schedule(ExecutorEdtManager.java:25)
	... 21 common frames omitted
00:02:39.625 ERROR [Output OT] j.d.b.c.t.ThrowableHandler               Exception handled at ThrowableHandler: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@607083b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@7df009d0[Wrapped task = jetbrains.datalore.base.jvm.edt.ExecutorEdtManager$ExecutorEdt$$Lambda$464/0x00000008011a24c8@1c5026a3]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@512f3079[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
jetbrains.datalore.base.common.edt.EdtException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@607083b[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@7df009d0[Wrapped task = jetbrains.datalore.base.jvm.edt.ExecutorEdtManager$ExecutorEdt$$Lambda$464/0x00000008011a24c8@1c5026a3]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@512f3079[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
	at jetbrains.datalore.base.jvm.edt.ExecutorEdtManager.schedule(ExecutorEdtManager.java:58)
	at jetbrains.datalore.base.jvm.edt.EdtManagerPool$EdtManagerAdapter.schedule(EdtManagerPool.java:17)
	at jetbrains.ocelot.ot.model.protocol.OtChannelExecutingDelegate.commandReceived(OtChannelExecutingDelegate.java:15)
	at jetbrains.ocelot.ot.model.protocol.OtChannelDelegate.commandReceived(OtChannelDelegate.java:21)
	at jetbrains.ocelot.ot.model.protocol.server.java.migration.MigrationOtPort$MigrationOutput.a(MigrationOtPort.java:89)
	at jetbrains.ocelot.ot.model.protocol.server.java.migration.MigrationOtPort$MigrationOutput.commandReceived(MigrationOtPort.java:69)
	at jetbrains.ocelot.ot.model.protocol.OtChannelDelegate.commandReceived(OtChannelDelegate.java:21)
	at jetbrains.ocelot.ot.model.protocol.server.BaseOtServerPort$1.commandReceived(BaseOtServerPort.java:2)
	at jetbrains.ocelot.ot.model.protocol.server.BaseOtServerPort.sendCommand(BaseOtServerPort.java:15)
	at jetbrains.ocelot.ot.model.protocol.server.BaseOtServer$BaseOtPortServer.fireChange(BaseOtServer.java:24)
	at jetbrains.ocelot.ot.model.protocol.server.DefaultOtServerPort.commandReceived(DefaultOtServerPort.java:77)
	at jetbrains.ocelot.ot.model.protocol.server.BaseOtServerPort$MyInput.commandReceived(BaseOtServerPort.java:62)
	at jetbrains.ocelot.ot.model.protocol.OtChannelDelegate.commandReceived(OtChannelDelegate.java:21)
	at jetbrains.ocelot.ot.model.protocol.server.java.migration.MigrationOtPort$1.commandReceived(MigrationOtPort.java:7)
	at jetbrains.ocelot.ot.model.protocol.OtChannelExecutingDelegate.lambda$commandReceived$0(OtChannelExecutingDelegate.java:54)
	at jetbrains.datalore.base.jvm.edt.ExecutorEdtManager$ExecutorEdt.lambda$wrap$1(ExecutorEdtManager.java:49)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

igro · April 13, 2023, 8:35am

Hello @patrick_h!

Are there any other error entries? Could you please send me the entire log via DM? Please also send me your config (without passwords).

Thank you!

Best regards,
Igor Medovolkin
QA Engineer in Datalore

Mikhail_Obidin · April 17, 2023, 10:57am

Hi,

As far as I can see in your logs you could have a problem with pulling the agent docker image (it hadn’t finished in 10 minutes timeout). Could you check the output of the command docker images jetbrains/datalore-agent:2023.1? It should output something like this:

docker images jetbrains/datalore-agent:2023.1
REPOSITORY                 TAG       IMAGE ID       CREATED        SIZE
jetbrains/datalore-agent   2023.1    58f1eaaa37a4   53 years ago   3.61GB