Installation DVC on MinIO storage

Question

Does anybody install DVC on MinIO storage?

I have read docs but not all clear for me.

Which command should I use for setup MinIO storage with this entrance parameters:

storage url: https://minio.mysite.com/minio/bucket-name/ login: my_login password: my_password

Inviolable

Answer 1

Install

I usually use it as a Python package, in this case you need to install:

pip install "dvc[s3]"

Setup remote

By default DVC supports AWS S3 storage and it works fine.
It also supports "S3-compatible storage", and MinIO in particular. In this case you have a bucket - a directory on a MinIO server where actual data is stored (it is similar to an AWS bucket). DVC uses AWS CLI to authenticate with AWS and in case of MinIO you need to pass credentials to dvc (not to the minio package).

The commands to setup MinIO as your DVC remote:

# setup default remote (change "bucket-name" to your minio backet name)
dvc remote add -d minio s3://bucket-name -f

# add information about storage url (where "https://minio.mysite.com" is your MinIO url)
dvc remote modify minio endpointurl https://minio.mysite.com

#  add MinIO credentials (e.g. from env. variables)
dvc remote modify minio access_key_id my_login
dvc remote modify minio secret_access_key my_password

If you move from old remote, use the following commands to move your data:

Before setup (download old remote cache to the local machine - note it may take a long time):

dvc pull -r <old_remote_name> --all-commits --all-tags --all-branches

After setup (upload all local cache data to a new remote):

dvc push -r <new_remote_name> --all-commits --all-tags --all-branches

Astro

Doy you know how to issue commands like dvc list-/import- when using such s3 compatible backends? (Credentials wise I mean) – Trilby

Hello! Don't you know why I can get WARNING: failed to collect '<commit_hash>', skipping after doing dvc push (last command)? – Congregational

Answer 2

May be useful: I have self-hosted MinIO and when i'd tried to push I had an error ERROR: unexpected error - [Errno 22] Bad Request: An error occurred (400) when calling the HeadObject operation: Bad Request

The solution is to add use_ssl = False to my dvc config file, so my config file looks like

[core]
    remote = minio
    autostage = true
['remote "minio"']
    url = s3://tags-by-review/DVC
    endpointurl = http://...:9000/
    version_aware = true
    use_ssl = False

Bract