BeyondStorage 2021Q3 Roadmap Draft

In this topic, we will discuss what to do in 2021Q3 (from Jul to Sept).


Background

Before starting my draft, let’s revisit our architecture.

In the past year:

  • go-storage API has been more and more stable, with no big break changes anymore.
  • Up to 21 services have been or are under development.
  • BeyondTP has released several beta versions.

So we have to think about where should BeyondStorage going on.

I have many ideas so far, let me outline all of them here.

Those ideas only include projects under BeyondStorage, other organizations or projects like ipfs repo upon go-storage will not be covered.

Ideas

go-storage

go-configure

Abstraction for configure.

With this support, all of our storage services can implement this API to support configure.

We can implement this via:

  • go-service-s3 provide schema in json or toml
  • go-configure will parse and generate related struct and API
  • go-service-s3 will export the API
  • Other tools can use this API.

Extension of Idea: Storage Configuration Support · Issue #634 · beyondstorage/go-storage · GitHub

rs-storage

Take go-storage to rust ecosystem.

With rs-storage support, we can:

  • Allow user use storage abstraction in more underlying services like filesystem or database
  • Allow user use storage abstraction in C/C++ or WASM

BeyondTP

BeyondTP’s progress is not as expected as I thought.

We need to releases versions that can be used by real users.

BeyondFS

Let’s build a POSIX alike filesystem upon go-storage.

Some works have been done in GitHub - beyondstorage/beyond-fs: A high-performance, POSIX-ish File System based on beyondstorage/go-storage

BeyondCTL

We can build command-line tools by go-storage, like s3cli, but for all storage services.

BeyondFileService

Build a file management service upon go-storage?

BeyondWebDAV

We can build a webdav service via go-storage, bring WebDAV support for all storage services.

BeyondBackup

We can build a backup system via go-storage, allow users to backup data from&to other storage services.

Roadmap

More Services Connected

For now, we have 21 supported services: 9 of them are stable, 1 of them is beta, 10 of them are under development.

In 2021Q3, we will expand them to 30 services:

  • Already implemented services will become stable.
  • And we will implement 9 more services.

After 2021Q3, BeyondStorage will connect almost all major storage services in the world.

More Releases for BeyondTP

BeyondTP is a neutral data migration service that is built upon go-storage. In the past several months, our development speed is too slow. In 2021Q3, we will accelerate iteration speed.

After 2021Q3, we will release a usable version of BeyondTP.

More Scene Expansion

In 2021Q3, we will expand more scenes of BeyondStorage.

First of all, we will try to build BeyondFS: a high-performance, POSIX-ish File System based on go-storage. In 2021Q3, we plan to build a first beta release version of BeyondFS.

Then, we will build an FTP server based on go-storage: BeyondFTP. This service could behave like an FTP gateway of all storage services.

Finally, we will start an incubator project: BeyondCTL. This service is a command-line tool for storage management. In this project, we will support multiple profiles that each profile can connect to different storage services.

2 Likes

I’m working on Fully support about connection string, maybe I can try to carry out Support init storager via config struct later.
I’m also interested in rs-storage and BeyondBackup, and I’m a little confused about the difference between BeyondBackup and BeyondTP.

2 Likes

BeyondTP is designed for data migration but BeyondBackup is used to backup users’ data.

The major different is backup need to take care about both backup and restore. And backup can use it’s own data format instead of the same with original.

2 Likes