Cross Cloud Storage Adapter? [closed]

Question

Do any APIs/Libraries/tools exist that act as adapters/provider interfaces for accessing different cloud storage services through a common interface? Something similar to ODBC or OLE-DB, except for cloud storage instead of databases.

Such that, if I wrote a front end for taking notes, and I utilized such an API, and let the user provide configuration for which cloud storage provider they have an account with, the API library would handle translating my cloud.Save() call into the commands specific to whiever provider was being utilized. This would allow my front-end app to be cloud storage provider agnostic.

So maybe I wrote some chrome extension or portable thumb drive app for storing notes, or encrypting and storing passwords, or some such, and you tell it which cloud storage provider you have an account with, and it uses it for syncing. This way your use of that tool doesn't tie you to a specific cloud provider. As long as you backup your data, you could migrate to another provider and just reconfigure the app should you become unhappy with that provider or they go bankrupt.

WebDAV for example is one potential candidate since it seems some storage services offer it, but that is not quite what I have in mind, since it depends on the storage providers to offer that as an option. I also don't know enough about WebDAV to know if it really would serve in the capacity I'm imagining. But feel free to post that as an option with pros/cons for comment/discussion.

I more imagine something that is a middle layer external to each cloud provider. Of course since each provider offers a different web service for interacting with files, the middle layer would have adapter for each backend. But on the front-end, it would expose a common API that is provider agnostic.

Does anything of this type exist?

Even just an open source GUI that allows you to store files in any provider, which would imply that in its source code exists the beginnings of such a middle layer. I would think someone has already made a tool that helps you unify all the free GB that you can get from various services. Sort of a JBOD layer for the cloud(although that is not the goal of this post, the point being such a tool accessing many different services would imply it has the beginnings of a middle layer for standardizing access to them).

My main interest though is in abstractions for personal cloud storage services, that would be appropriate for applications used by individuals, to put the control of storage in the hands of the individual so that they can have the freedom to move between personal cloud storage services. It seems what I've found so far is more oriented for CDN, websites, or services.

Please make seperate posts per suggestion so that votes and comments/discussion can take place specific to that suggestion.

Regeneracy

It never fails that I spend 30 minutes looking for something, and then 30 seconds after I post I realize the combination of keywords that get me some results. Posting some potential answers now, but will be interested in comments from anyone who has used these tools. Literally it occurred to me to search for odbc for the cloud and found a blog on the exact same subject: janakiramm.net/blog/do-we-need-odbc-for-the-cloud – Regeneracy

check the open source solution for Java here – Frederickafredericks

Google Drive allows multiple files with the same name in the same folder. This is just an example of how storage providers can differ, and it would make such a unified interface challenging. However I'm sure it's possible, it's been done and it's a good idea. – Reynoso

@Reynoso Indeed. Two approaches are to either have a minimal common denominator of features, or allow applications to check for features. Kind of similar to how a graphics library allows applications to check for hardware support of features such as bump mapping or shaders, and then conditionally leverage those features.. – Regeneracy

There is no good solution to this problem yet. Maybe we can close this topic after the internets find a solution. – Deemphasize

@ErikAronesty There are actually several good solutions depending on your language and scenario, but the problem with my question was it is a type of "List all the possible solutions" which is generally considered a too broad question for stackoverflow. – Regeneracy

Answer 1

Kloudless provides a common API to several different cloud storage APIs (Dropbox, Box, GDrive, OneDrive, etc.). Kloudless also provides SDKs in popular languages and UI widgets to handle authentication and other user interactions.

You can find more information and sign up here: https://kloudless.com/

Disclosure: I work at Kloudless.

Footprint

Disclosure: This is a centralized server, not an API you can deploy on client machines. – Deemphasize

@ErikAronesty Yes, it's also available as a Docker container to self-host or run on on-premises client networks, but rarely on an end-user's laptop for example. – Footprint

Kloudless looks like exactly what I was looking for but they got sold (kloudless.com/blog/kloudless-is-now-part-of-netskope) to a company which took their products off the market a few years ago. – Isaisaac

Answer 2

Apache jclouds presents cloud-agnostic abstractions, with stable implementations of ComputeService and BlobStore.

visit https://jclouds.apache.org/

Apache jclouds® is an open source multi-cloud toolkit for the Java platform that gives you the freedom to create applications that are portable across clouds while giving you full control to use cloud-specific features.

Regeneracy

This is not cloud storage, it's cloud computing, doesn't answer the question. – Deemphasize

@ErikAronesty BlobStore is a form of cloud storage. – Regeneracy

Answer 3

Apache Libcloud: "a unified interface to the cloud"

http://libcloud.apache.org/

Regeneracy

This is not cloud storage, it's cloud computing, doesn't answer the question. – Deemphasize

@ErikAronesty It does include a storage API: libcloud.readthedocs.io/en/latest/storage/api.html – Regeneracy

which doesn't work with cloud storage providers: dropbox, drive, box, etc. It's not suitable for syncing - which was the original question. – Deemphasize

That's true, yes. Those personal services are not covered by this particular library, which indeed was the context of the original question. But that's different than you saying "this is not cloud storage". Honestly it was not a well defined question, hence why it is closed. At the time I asked it, I had a hard time finding provider agnostic libraries, and wanted to tap into other's knowledge. I could probably ask a more well defined question now that I've seen a good number of these types of libraries/products, but it'd still be out of scope of stackoverflow. – Regeneracy

Answer 4

A couple of months ago I did a survey of personal cloud storage aggregator services and applications. And one seems relevant to your question.

Oxtio is a service that connects multiple cloud storage services and includes a WebDAV service for accessing it's own service.

Tartan

Oxtio bascially have developed code to talk to virtually ALL the cloud providers (they support LOADS) and built a web app on top of it to let users administrate credentials. They briefly mentioned an API some years ago but doesn't look like it's been launched. blog.otixo.com/tag/api-2 – Toothpick

Unfortunately this is also a central server, not an api wrapper or something you can deploy to clients. – Deemphasize

Answer 5

Cloud storage providers each have different specifics which makes it hard to use exactly one interface for all (or even some) of them. CloudBlackbox package of our SecureBlackbox product offers a unified interface for major storage providers (S3, Azure, Google Drive, SkyDrive/OneDrive, Dropbox) with focus on security of the data, but due to mentioned specifics we have individual classes (descendants of one superclass) to serve each provider. SecureBlackbox is available for use from .NET, Java, C++ on Windows and Delphi.

Incorporated

This is the only response that actually answers the question as asked. Thanks! – Deemphasize

Answer 6

Check out Boto, a highly regarded Python library which provides an abstraction layer atop Amazon's S3 and Google Cloud Storage.

https://github.com/boto/boto

Olomouc

Answer 7

-StorageMadeEasy (SME)
-Otixo (But they do not offer FREE tier anymore since Feb 2013)
-Joukuu
-Gladinet
-Egistec CloudHub
...

All of above allows you to connect several cloud storages, but they do not actually combine it.

If you wan to combine several personal cloud storages, you need to make it yourself, which is what I am doing for the past few months.

So far I have combined several clouds (Dropbox, Box, Google Drive, Skydrive) using their Android API/SDK, then I process the data splitting/merging/compression/encryption inside my Android application (not a good choice, just for the sake of prototype)

In the future, maybe I will add more providers that has an API, such as Amazon S3, SugarSync, but right now there is lack of manpower.

If you just want to connect multiple clouds on Android (not combining), then you can try ES File Explorer or ASTRO File Manager, and several other applications

Dekeles

out of curiosity, did you write all the authorization and background syncing for every cloud service, or were you able to find a library to do it for you? – Gailey

Answer 8

I think webdav is the ultimate protocol:

webdav->dropdav->dropbox
webdav->box.net
webdav->DAV-pocket->google drive
webdav->Otixo(free for 14 days)->SugarSync

Glamour