Hi everyone,
I'm working on a project using TensorFlow for model training on a Datalore A10G machine and have hit some persistent GPU compatibility walls. I'm hoping someone in the community might have encountered similar issues or could offer some insights.
**My Datalore A10G Setup:**
* GPU: NVIDIA A10G
* NVIDIA Driver: `550.127.05` (from `nvidia-smi` )
* CUDA Version (reported by driver): `12.4`
* Notebook: `[Your Notebook Name/Link if shareable]`
I've tried a couple of TensorFlow versions with different results:
**Scenario 1: Datalore's Default Environment (TensorFlow 2.19.0)**
* When I use the default Datalore environment, which seems to provide TF 2.19.0, I immediately run into a `CUDA_ERROR_UNSUPPORTED_PTX_VERSION` when trying to build/run a Keras model.
* My understanding is TF 2.19.0 is built with CUDA 12.5. It seems the installed driver (compatible up to CUDA 12.4) can't handle the PTX from this newer CUDA toolkit.
* **Question for community/Datalore users:** Has anyone successfully used TF 2.19.0 on an A10G in Datalore, and if so, what driver/CUDA setup was present?
**Scenario 2: Downgrading to TensorFlow 2.17.0 (via `environment.yml` )**
* To align with the driver's CUDA 12.4 compatibility (TF 2.17.0 uses CUDA 12.3), I specified `tensorflow==2.17.0` in my `environment.yml` .
* While TF 2.17.0 installs, I then hit these issues during `model.fit()` :
* A persistent warning: `Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice.` (even when trying to set `XLA_FLAGS=--xla_gpu_cuda_data_dir=/usr/local/cuda-12.3` ).
* **The critical error:** `Loaded runtime CuDNN library: 8.5.0 but source was compiled with: 8.9.6.`
* This leads to: `FAILED_PRECONDITION: DNN library initialization failed.`
* It appears the runtime CuDNN 8.5.0 is too old for TF 2.17.0, which expects cuDNN 8.9.x (for CUDA 12.3).
* **Question for community/Datalore users:** If you've used TF 2.17.0 on a Datalore A10G, what cuDNN version was present in your environment? How was `libdevice` located?
**Current Situation:** I'm effectively blocked from GPU training. With TF 2.19.0, it's a PTX/driver issue. With TF 2.17.0, it's a critical cuDNN version mismatch and a `libdevice` path problem.
Has anyone faced similar TensorFlow/CUDA/CuDNN alignment challenges on Datalore A10G machines? Any suggestions for environment configurations, `environment.yml` tweaks, or ways to ensure all CUDA components are correctly versioned and accessible would be greatly appreciated!
Thanks in advance!Preformatted text