What is the custom option to turn off ssl certificate validation for s3 connection?

Tips & Tricks
1

I can turn off the ssl certification validation of self-signed certifcate in boto3 with verify=False

s3 = session.resource(
    's3', 
    endpoint_url = 'https://netappS3.local.domain.de', 
    verify=False
)

Is there a custom option which i can set for datalore UI Cloud storage to create s3 connection?

2

Hello @Yingding,

Please try adding --no-verify-ssl to the “Custom options” field.

3

Unfortunately, the --no-verify-ssl in the “Custom options” doesn’t work. As i tested connection from notebook, it is failed.

These was an issue with s3fs-fuse using --no-verify-ssl on some old distro (Not work --no-verify-ssl option for aws cli on some Runner · Issue #1812 · s3fs-fuse/s3fs-fuse · GitHub). What client does datalore use to connect S3 internally?

I have an enterpise datalore deployed with helm chart 0.2.6 on k8s:

NAME             	CHART VERSION	APP VERSION	DESCRIPTION
datalore/datalore	0.2.6        	2022.3.1
4

@igro Thanks again for your hint to connect a local s3 bucket with --no-verify-ssl in datalore. I updated today datalore enterprise on-prem to version 2023.1.

Now i got the error:

S3 options error: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@22fc31ea: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@1777bbb1: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@6e70c1c6: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@452d9df8: Unable to contact EC2 metadata service.]

Can you give me an advice, what I can do to make a local s3 like bucket mountable. I have left the region blank for previous 2022.3.1 version, but it doesn’t seem to work now for the 2023.1 datalore version.

1 Like
5

@Yingding,

Thank you for reporting this! We will add an option to leave the region empty.

This error should only occur during the connection check and not for mounting – as a temporary workaround please specify any valid AWS region (e.g. us-east-1).

Thank you!
Igor

6

@igro Thanks for your feedback.
After adding the region dummy input us-east-1 with the suggest custom option --no-verify-ssl, now i got the error

S3 options error: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Does it means the custom option --no-verify-ssl doesn’t work for an on-prem s3 bucket?

7

@Yingding,

Could you try to ignore this error in and mount the bucket to your notebook?
Most likely, this error is caused by AWS SDK* used for the connection check, but shouldn’t affect connection itself.

*we’re going to update AWS SDK in the upcoming builds, it should fix this and other related issues.

8

Unfortunately, I still get not mounted for this on-prem s3 bucket using attach data to notebook with dummy region and custom options --no-verify-ssl, the boto3 code works fine.

9

@Yingding, hmm, could you also try to mount without a placeholder in the region field?

UPD: my colleagues also corrected me, the required parameter is called ssl_verify_hostname=0 in s3fs, and --no-verify-ssl is only used in AWS CLI.

10

@igro Thanks so much for your help, I also found out that the current s3fs version in ubuntu 22.04 notebook terminal is 1.9.0
I am happy to share that i finally be able to mount my netapp on-prem s3 bucket in the options field with
use_path_request_style,no_check_certificate,ssl_verify_hostname=0
and a dummy region us-east-1

I do need all these three options for s3fs. I hope this can help others to mount on-prem s3 buckets at ease.

UPD: I can also mount without the dummy region field. Mount works, but the connection test shows error, with region, connection test shows also error with region and custom options with ssl certifcate validation turned off.

1 Like