Either Gateway Timeout or Docker Agents Dose not Start after upgrade to 2023.1

My docker based datalore was previously working nice, and after seeing that there is a new version 2023.1 i decided to update datalore to the latest version, and after doing that, I am facing strange behaviours like (Those appears randomly) :

  • Agent machine dose not start
  • Visiting with domain ends up Gateway error
  • Stuck on initialization of docker machines (agents )

I tried restarting the datalore docker compose, but it still happens like above.

1 Like

here are some logs

Starting datalore 2023.1
Using default docker.agents-config.yaml. If you want to customize in please mount your's agents-config.yaml into /opt/datalore/configs/agents-config.yaml
WARN  : Forking second JVM due to forking module(s): [logging-jul-capture]. Use --dry-run to generate the command line to avoid forking.
08:24:34.688 INFO  [main] o.s.web.context.ContextLoader            Root WebApplicationContext: initialization started
08:24:38.943 INFO  [main] o.f.core.internal.command.DbValidate     Successfully validated 102 migrations (execution time 00:00.110s)
08:24:38.960 INFO  [main] o.f.core.internal.command.DbMigrate      Current version of schema "infra": 101
08:24:38.962 INFO  [main] o.f.core.internal.command.DbMigrate      Schema "infra" is up to date. No migration necessary.
08:24:44.037 WARN  [Datalore EDT Manager] j.d.n.s.c.r.ConnectionManagerBasedAgentChannelProvider Failed to recover agent for computation ITlB3aEM9oxh7WBX8B0tPX
08:24:44.059 INFO  [Datalore EDT Manager] j.d.n.s.c.c.ComputationIdProviderImpl    Start computation with id 2KxrMkWCVrMhkBesqD2mkw for session NOTEBOOK session Ke0vb1Mbl5XQjrpXh1evtL owned by 56NFJzAjHAMgIyWyQW7V7F [with source 56NFJzAjHAMgIyWyQW7V7F/iVY8lJDJZFEymlls6fWIRd] by user 56NFJzAjHAMgIyWyQW7V7F
08:24:44.124 INFO  [Datalore EDT Manager] j.d.n.s.c.c.ComputationIdProviderImpl    Started computation with id 2KxrMkWCVrMhkBesqD2mkw for session NOTEBOOK session Ke0vb1Mbl5XQjrpXh1evtL owned by 56NFJzAjHAMgIyWyQW7V7F [with source 56NFJzAjHAMgIyWyQW7V7F/iVY8lJDJZFEymlls6fWIRd] by user 56NFJzAjHAMgIyWyQW7V7F
08:24:45.694 INFO  [Datalore EDT Manager] j.d.n.s.c.a.i.i.core.BaseInstanceManager Agent 3f8d7ca4058a has been destroyed
08:24:48.077 INFO  [main] o.s.web.context.ContextLoader            Root WebApplicationContext initialized in 13385 ms
08:26:34.537 INFO  [Computation edt_0_0] j.d.n.s.c.p.c.CodeInsightVersionDependentMessageHelper$CodeInsightRequestsController First code_actions took 228 ms for source 56NFJzAjHAMgIyWyQW7V7F/iVY8lJDJZFEymlls6fWIRd
WARN: frameworks.jetbrains.com
WARN: frameworks.jetbrains.com
java.net.UnknownHostException: frameworks.jetbrains.com
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:567)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:633)
        at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
        at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636)
        at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
        at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
        at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308)
        at com.intellij.util.io.HttpRequests.openConnection(HttpRequests.java:621)
        at com.intellij.util.io.HttpRequests$RequestImpl.getConnection(HttpRequests.java:362)
        at com.intellij.util.io.HttpRequests$RequestImpl.doReadBytes(HttpRequests.java:438)
        at com.intellij.util.io.HttpRequests$RequestImpl.readBytes(HttpRequests.java:422)
        at com.intellij.util.io.RequestBuilder.lambda$readBytes$4(RequestBuilder.java:81)
        at com.intellij.util.io.HttpRequests.doProcess(HttpRequests.java:525)
        at com.intellij.util.io.HttpRequests.process(HttpRequests.java:507)
        at com.intellij.util.io.HttpRequests$RequestBuilderImpl.connect(HttpRequests.java:336)
        at com.intellij.util.io.RequestBuilder.readBytes(RequestBuilder.java:81)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager.a(DatabaseArtifactManager.java:7)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager.f(DatabaseArtifactManager.java:146)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager.a(DatabaseArtifactManager.java:22)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager.lambda$new$0(DatabaseArtifactManager.java:89)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:702)
        at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:699)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
        at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:699)
        at java.base/java.lang.Thread.run(Thread.java:833)
java.net.UnknownHostException: frameworks.jetbrains.com
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:567)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:633)
        at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
        at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636)
        at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
        at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
        at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308)
        at com.intellij.util.io.HttpRequests.openConnection(HttpRequests.java:621)
        at com.intellij.util.io.HttpRequests$RequestImpl.getConnection(HttpRequests.java:362)
        at com.intellij.util.io.HttpRequests$RequestImpl.doReadBytes(HttpRequests.java:438)
        at com.intellij.util.io.HttpRequests$RequestImpl.readBytes(HttpRequests.java:422)
        at com.intellij.util.io.RequestBuilder.lambda$readBytes$4(RequestBuilder.java:81)
        at com.intellij.util.io.HttpRequests.doProcess(HttpRequests.java:525)
        at com.intellij.util.io.HttpRequests.process(HttpRequests.java:507)
        at com.intellij.util.io.HttpRequests$RequestBuilderImpl.connect(HttpRequests.java:336)
        at com.intellij.util.io.RequestBuilder.readBytes(RequestBuilder.java:81)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager.a(DatabaseArtifactManager.java:7)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager.f(DatabaseArtifactManager.java:146)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager.d(DatabaseArtifactManager.java:24)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager$1.run(DatabaseArtifactManager.java:4)
        at com.intellij.openapi.progress.impl.CoreProgressManager.startTask(CoreProgressManager.java:429)
        at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcessWithProgressInCurrentThread$11(CoreProgressManager.java:557)
        at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:186)
        at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$13(CoreProgressManager.java:604)
        at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:679)
        at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:635)
        at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:603)
        at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:173)
        at com.intellij.openapi.progress.impl.CoreProgressManager.runProcessWithProgressInCurrentThread(CoreProgressManager.java:557)
        at com.intellij.openapi.progress.impl.CoreProgressManager.run(CoreProgressManager.java:363)
        at com.intellij.openapi.progress.Task.queue(Task.java:122)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager.a(DatabaseArtifactManager.java:61)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager.checkForUpdates(DatabaseArtifactManager.java:144)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager.getArtifactList(DatabaseArtifactManager.java:154)
        at com.intellij.database.dataSource.artifacts.DatabaseArtifactManager.getArtifacts(DatabaseArtifactManager.java:110)
        at com.intellij.database.dataSource.DatabaseDriverImpl$ArtifactRefImpl.getProperty(DatabaseDriverImpl.java:24)
        at com.intellij.database.dataSource.DatabaseDriverImpl.getArtifactsProperty(DatabaseDriverImpl.java:405)
        at com.intellij.database.dataSource.DatabaseDriverImpl.b(DatabaseDriverImpl.java:581)
        at com.intellij.database.dataSource.DatabaseDriverImpl.getDriverClass(DatabaseDriverImpl.java:458)
        at jetbrains.datalore.vfs.resources.databases.server.impl.DatabaseManager.a(DatabaseManager.kt:6)
        at jetbrains.datalore.vfs.resources.databases.server.impl.DatabaseManager.a(DatabaseManager.kt:52)
        at jetbrains.datalore.vfs.resources.databases.server.impl.DatabaseManager.createDbDriver$default(DatabaseManager.kt:29)
        at jetbrains.datalore.vfs.resources.databases.server.impl.DatabaseManager.getAllDrivers(DatabaseManager.kt:16)
        at jetbrains.datalore.vfs.resources.databases.server.impl.DatabaseController.getAllDrivers(DatabaseController.kt:35)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:523)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
        at org.eclipse.jetty.websocket.servlet.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:170)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at jetbrains.datalore.userManagement.server.api.audit.AuditHttpAccessFilter.doFilter(AuditHttpAccessFilter.kt:18)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at jetbrains.datalore.services.local.InternalApiFilter.doFilter(InternalApiFilter.kt:10)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141)
        at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1571)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1544)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
        at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:139)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1077)
        at java.base/java.lang.Thread.run(Thread.java:833)
08:27:13.796 INFO  [Datalore EDT Manager] j.d.n.s.c.a.i.i.core.BaseInstanceManager Agent ae8ab703f018 has been destroyed
08:28:00.819 INFO  [Computation edt_1_1] j.d.n.s.c.c.ComputationIdProviderImpl    Start computation with id NnoF2hwdQ0L9LbdCflg2p0 for session NOTEBOOK session c0NWsthN15N44EgwySIgG2 owned by 56NFJzAjHAMgIyWyQW7V7F [with source 56NFJzAjHAMgIyWyQW7V7F/r3YW26dU4Ps4XptmgPb9On] by user 56NFJzAjHAMgIyWyQW7V7F
08:28:00.830 INFO  [Computation edt_1_1] j.d.n.s.c.c.ComputationIdProviderImpl    Started computation with id NnoF2hwdQ0L9LbdCflg2p0 for session NOTEBOOK session c0NWsthN15N44EgwySIgG2 owned by 56NFJzAjHAMgIyWyQW7V7F [with source 56NFJzAjHAMgIyWyQW7V7F/r3YW26dU4Ps4XptmgPb9On] by user 56NFJzAjHAMgIyWyQW7V7F
08:28:39.746 INFO  [Computation edt_0_0] j.d.n.s.c.c.ComputationIdProviderImpl    Start computation with id zHkpQEi9JIBDJZWEyWe0ZA for session NOTEBOOK session Ke0vb1Mbl5XQjrpXh1evtL owned by 56NFJzAjHAMgIyWyQW7V7F [with source 56NFJzAjHAMgIyWyQW7V7F/iVY8lJDJZFEymlls6fWIRd] by user 56NFJzAjHAMgIyWyQW7V7F
08:28:39.756 INFO  [Computation edt_0_0] j.d.n.s.c.c.ComputationIdProviderImpl    Started computation with id zHkpQEi9JIBDJZWEyWe0ZA for session NOTEBOOK session Ke0vb1Mbl5XQjrpXh1evtL owned by 56NFJzAjHAMgIyWyQW7V7F [with source 56NFJzAjHAMgIyWyQW7V7F/iVY8lJDJZFEymlls6fWIRd] by user 56NFJzAjHAMgIyWyQW7V7F

I have the same issue(or very similar):
I have a basic installation of Datalore in docker using a provided docker-compose.
To reproduce:
I open a Python notebook. And nothing can be executed:

I’d like to reply with a solution to the problem.
The issue is that Datalore BY DEFAULT assumes the name of the docker container to be “datalore”. Thus, all swarm containers/agents are trying to connect to their main container( datalore server) using that name. IF you change the name in your docker-compose(or docker run) to something different than “datalore”, you must change to that name in several places:

  1. agent-config.yaml: “dataloreHost”
  2. Set below ENV vars in datalore-server docker-compose file(or docker run command)
  • AGENT_MANAGER_HOST
  • AGENT_DISK_HOST
  • COMPUTATION_HOST