Add new `Combine` Remote #5600

nkitsaini · 2021-09-14T11:34:34Z

What is your current rclone version (output from `rclone version`)?

rclone v1.53.3-DEV

os/arch: linux/amd64
go version: go1.15.4

What problem are you are trying to solve?

Managing multiple remotes.
I use a few google drive + backblaze remotes and it is not quite easy to work with all of them.
Most of the time I want to mount all of them in one particular directory, roughly as follows:

rclone mount backblaze:   mount/backblaze
rclone mount gdrive_home:   mount/gdrive/home
rclone mount gdrive_office:    mount/gdrive/office
etc..
# also if mount fails then `fusermount -uz` and stuff

I've a script to handle mounting but I think it's better if rclone handles this internally.

How do you think rclone should be changed to solve that?

Rclone can handle all of the mountings as a new remote maybe combine or join.
So config can be something like:

[all]
type=combine
remotes="backblaze: backblaze, gdrive_home: gdrive/home, gdrive_office: gdrive/office"

Now I just have to manage a single mount.
As a start I think we should not support nested remotes. And also no remote specific configurations like seperate --dir-cache-duration. So all remotes will be leaf in structure and whatever mount config is used for combine will apply to all of the remotes.

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

The text was updated successfully, but these errors were encountered:

ivandeex · 2021-09-14T14:24:15Z

In the past a user requested to add such a mode in the Union backend - #4730
Quite a time has passed and I'm feeling it's just not possible to implement within Union...
I'd be quite happy if someone makes it into new backend.

@ncw

By the way, does your fs.CommaSepList support optional whitespace after comma?

remotes="backblaze: backblaze, gdrive_home: gdrive/home, gdrive_office: gdrive/office"

ncw · 2021-09-14T19:17:34Z

In the past a user requested to add such a mode in the Union backend - #4730
Quite a time has passed and I'm feeling it's just not possible to implement within Union...
I'd be quite happy if someone makes it into new backend.

I've had multiple requests for a drive backend which combines lots of different drives, eg all the available team drives, so a project like this could work for that. Maybe if it had a script interface so it could generate the directory structure from a tree.

@ivandeex wrote

By the way, does your fs.CommaSepList support optional whitespace after comma?
remotes="backblaze: backblaze, gdrive_home: gdrive/home, gdrive_office: gdrive/office"

package main

import (
	"fmt"

	"github.com/rclone/rclone/fs"
)

func main() {
	var x fs.CommaSepList
	err := x.Set("backblaze: backblaze, gdrive_home: gdrive/home, gdrive_office: gdrive/office")
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Printf("x=%#v\n", x)
}

x=fs.CommaSepList{"backblaze: backblaze", " gdrive_home: gdrive/home", " gdrive_office: gdrive/office"}

It parsed it, but it didn't strip the space

ivandeex · 2021-09-15T06:00:45Z

Maybe if it had a script interface so it could generate the directory structure from a tree.

Could you elaborate?

By the way, does your fs.CommaSepList support optional whitespace after comma?

It parsed it, but it didn't strip the space

Would you mind if I add fs.CommaSpaceSepList that strips? - #5587 (comment)

ncw · 2021-09-15T11:16:19Z

Maybe if it had a script interface so it could generate the directory structure from a tree.

Could you elaborate?

For example you could have script which used rclone backend drives drive: to build the config for all the shared drives you have access to. If this could be wired in so that this script ran each time the remote was started then it could build a dynamic remote which had all your shared drives in.

By the way, does your fs.CommaSepList support optional whitespace after comma?

It parsed it, but it didn't strip the space

Would you mind if I add fs.CommaSpaceSepList that strips? - #5587 (comment)

Note that these use https://pkg.go.dev/encoding/csv to parse things so they have an escape mechanism for including commas and spaces. I guess an option to set TrimLeadingSpace would be OK!

ivandeex · 2021-10-04T20:57:22Z

For example you could have script which used rclone backend drives drive: to build the config for all the shared drives you have access to. If this could be wired in so that this script ran each time the remote was started then it could build a dynamic remote which had all your shared drives in.

https://forum.rclone.org/t/bulk-create-remotes-to-existing-google-shared-drives/26837/2

ncw · 2021-10-05T13:49:56Z

As far as team drives go I had this idea...

It occurred to me that I could make an option for rclone backend drives drive: which would output in config file format with the names of the drives...

I had a go at exactly this

v1.57.0-beta.5681.c135f93dd.fix-drive-drives on branch fix-drive-drives (uploaded in 15-30 mins)

Use it like

rclone backend -o config drives gdrive:

And it will output an rclone config file configuring aliases for all your team drives.

ivandeex · 2021-10-06T18:20:35Z

related to #4506

nkitsaini · 2021-11-04T05:29:47Z

If it's not yet started, should I try implementing it?
I don't have much idea about how remotes are implemented. I'll read the code and get back with questions if I have any. But let me know if there's anything I should keep in mind. Also I didn't get much of gdrive teams conversations, so do we need anything special other then the proposed Combine remote to support gdrive teams ?

dandar · 2022-04-10T21:40:27Z

From #4506 (comment)

As far as team drives go I had this idea...

It occurred to me that I could make an option for rclone backend drives drive: which would output in config file format with the names of the drives...

I had a go at exactly this

v1.57.0-beta.5681.c135f93dd.fix-drive-drives on branch fix-drive-drives (uploaded in 15-30 mins)

Use it like
rclone backend -o config drives gdrive:
And it will output an rclone config file configuring aliases for all your team drives.

For anyone wondering how to use
rclone backend -o config drives gdrive:
"gdrive" is the name of your google MyDrive in your config file
And it will output an rclone config file configuring aliases (I didn't know you can configure team shared drives as aliases, very helpful)

Thank you Nick. Can we somehow mount all remotes at once, mount every all remotes using one mount command?
If we can wrap several remotes using one name in the config file, then a single mount command to mount several specific remotes would be very helpful also. Mount and unmount (fusermount -u) should create and delete the remote root folder for each remote under one mount point.

… - FIXME WIP Needs - docs - integration tests Fixes #5600

ncw · 2022-04-21T09:06:02Z

I have made a first attempt at a combine backend here

v1.59.0-beta.6081.b0c54538b.fix-5600-bind on branch fix-5600-bind (uploaded in 15-30 mins)

You configure it like this

[TestCombine]
type = combine
upstreams = dir1=/tmp/dir1 dir2=/tmp/dir2 dir3=/tmp/dir3

I've also added a little to rclone backend drives -o config drive: so it makes config which combines all the shared drives into one

With the -o config parameter it will output the list in a format
suitable for adding to a config file to make aliases for all the
drives found and a combined drive.

[My Drive]
type = alias
remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:

[Test Drive]
type = alias
remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:

[AllDrives]
type = combine
remote = "My Drive=My Drive:" "Test Drive=Test Drive:"

Adding this to the rclone config file will cause those team drives to
be accessible with the aliases shown. Any illegal charactes will be
substituted with "_" and duplicate names will have numbers suffixed.
It will also add a remote called AllDrives which shows all the shared
drives combined into one directory tree.

showkubii · 2022-05-31T02:35:49Z

I have made a first attempt at a combine backend here

v1.59.0-beta.6081.b0c54538b.fix-5600-bind on branch fix-5600-bind (uploaded in 15-30 mins)

You configure it like this
[TestCombine]
type = combine
upstreams = dir1=/tmp/dir1 dir2=/tmp/dir2 dir3=/tmp/dir3
I've also added a little to rclone backend drives -o config drive: so it makes config which combines all the shared drives into one

With the -o config parameter it will output the list in a format suitable for adding to a config file to make aliases for all the drives found and a combined drive.
[My Drive]
type = alias
remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:

[Test Drive]
type = alias
remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:

[AllDrives]
type = combine
remote = "My Drive=My Drive:" "Test Drive=Test Drive:"
Adding this to the rclone config file will cause those team drives to be accessible with the aliases shown. Any illegal charactes will be substituted with "_" and duplicate names will have numbers suffixed. It will also add a remote called AllDrives which shows all the shared drives combined into one directory tree.

@ncw

rclone v1.59.0-beta.6167.5d6a6dd6c
- os/version: slackware 15.0 (64 bit)
- os/kernel: 5.15.43-Unraid (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.2
- go/linking: static
- go/tags: none
root@xxx:~# rclone lsf all: --config /mnt/cache/appdata/rclone/rclone.conf -vv
2022/05/31 09:34:00 DEBUG : rclone: Version "v1.59.0-beta.6167.5d6a6dd6c" starting with parameters ["rclone" "lsf" "all:" "--config" "/mnt/cache/appdata/rclone/rclone.conf" "-vv"]
2022/05/31 09:34:00 DEBUG : Creating backend with remote "all:"
2022/05/31 09:34:00 DEBUG : Using config file from "/mnt/cache/appdata/rclone/rclone.conf"
2022/05/31 09:34:00 Failed to create file system for "all:": didn't find backend called "combine"

[diynez-1]
type = alias
remote = gdrive,team_drive=xxx,root_folder_id=:

[diynez-2]
type = alias
remote = gdrive,team_drive=xxx,root_folder_id=:

[all]
type = combine
upstreams = diynez-1=diynez-1: diynez-2=diynez-2:



[v1.59.0-beta.6081.b0c54538b.fix-5600-bind](https://beta.rclone.org/branch/fix-5600-bind/v1.59.0-beta.6081.b0c54538b.fix-5600-bind/) 

It works properly.

… - FIXME WIP Needs - docs - integration tests Fixes #5600

ncw · 2022-05-31T10:36:40Z

Next test - with working changenotify hopefully.

v1.59.0-beta.6171.7dcf7cf55.fix-5600-combine on branch fix-5600-combine (uploaded in 15-30 mins)

Fixes #5600

showkubii · 2022-05-31T13:15:37Z

Next test - with working changenotify hopefully.

v1.59.0-beta.6171.7dcf7cf55.fix-5600-combine on branch fix-5600-combine (uploaded in 15-30 mins)

v1.59.0-beta.6171.7dcf7cf55.fix-5600-combine

Working, What commands can I use to get automatic updates?

rclone v1.59.0-beta.6171.7dcf7cf55.fix-5600-combine
- os/version: slackware 15.0 (64 bit)
- os/kernel: 5.15.43-Unraid (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.2
- go/linking: static
- go/tags: none
root@xxx:/mnt/cache/appdata/rclone# rclone lsf all: --config /mnt/cache/appdata/rclone/rclone.conf
diynez-1/
diynez-2/
root@xxx:/mnt/cache/appdata/rclone#

ncw · 2022-05-31T16:41:38Z

Working, What commands can I use to get automatic updates?

If you run rclone mount the polling should work automatically

showkubii · 2022-06-01T00:16:37Z

Working, What commands can I use to get automatic updates?

If you run rclone mount the polling should work automatically

Sorry i'm not clear I want to update the rclone fix-5600-combine version. automatically if there is a new version coming out.

TanukiAI · 2022-06-01T12:59:20Z

Looks like the update broke the CLI for the combine:
https://forum.rclone.org/t/polling-for-changes-in-a-shortcut-in-the-same-drive/31029/12?u=tanuki

Fixes #5600

ncw · 2022-06-01T15:44:52Z

Should be fixed here

v1.59.0-beta.6174.9aa5ce515.fix-5600-combine on branch fix-5600-combine (uploaded in 15-30 mins)

ivandeex added the new backend label Sep 14, 2021

ivandeex self-assigned this Sep 14, 2021

ivandeex mentioned this issue Oct 26, 2021

Support using google team drives (all not one) as a backend #4506

Open

ivandeex assigned nkitsaini and unassigned ivandeex Nov 5, 2021

ncw added a commit that referenced this issue Apr 21, 2022

Add combine backend to combine multiple remotes in one directory tree…

e62b9d0

… - FIXME WIP Needs - docs - integration tests Fixes #5600

ncw added a commit that referenced this issue May 31, 2022

Add combine backend to combine multiple remotes in one directory tree…

fb94d1f

… - FIXME WIP Needs - docs - integration tests Fixes #5600

ncw added a commit that referenced this issue May 31, 2022

combine backend to combine multiple remotes in one directory tree

910778b

Fixes #5600

ncw added a commit that referenced this issue Jun 1, 2022

combine: backend to combine multiple remotes in one directory tree

9aa5ce5

Fixes #5600

rclone / rclone Public

Add new `Combine` Remote #5600

Add new `Combine` Remote #5600

nkitsaini commented Sep 14, 2021

ivandeex commented Sep 14, 2021

ncw commented Sep 14, 2021

ivandeex commented Sep 15, 2021 •

edited

ncw commented Sep 15, 2021

ivandeex commented Oct 4, 2021

ncw commented Oct 5, 2021

ivandeex commented Oct 6, 2021

nkitsaini commented Nov 4, 2021 •

edited

dandar commented Apr 10, 2022 •

edited

ncw commented Apr 21, 2022

showkubii commented May 31, 2022 •

edited

ncw commented May 31, 2022

showkubii commented May 31, 2022 •

edited

ncw commented May 31, 2022

showkubii commented Jun 1, 2022

TanukiAI commented Jun 1, 2022

ncw commented Jun 1, 2022

May	JUN	Jul
	02
2021	2022	2023

rclone / rclone Public

Add new Combine Remote #5600

Add new Combine Remote #5600

Comments

nkitsaini commented Sep 14, 2021

What is your current rclone version (output from rclone version)?

What problem are you are trying to solve?

How do you think rclone should be changed to solve that?

How to use GitHub

ivandeex commented Sep 14, 2021

ncw commented Sep 14, 2021

ivandeex commented Sep 15, 2021 • edited

ncw commented Sep 15, 2021

ivandeex commented Oct 4, 2021

ncw commented Oct 5, 2021

ivandeex commented Oct 6, 2021

nkitsaini commented Nov 4, 2021 • edited

dandar commented Apr 10, 2022 • edited

ncw commented Apr 21, 2022

showkubii commented May 31, 2022 • edited

ncw commented May 31, 2022

showkubii commented May 31, 2022 • edited

ncw commented May 31, 2022

showkubii commented Jun 1, 2022

TanukiAI commented Jun 1, 2022

ncw commented Jun 1, 2022

Add new `Combine` Remote #5600

Add new `Combine` Remote #5600

What is your current rclone version (output from `rclone version`)?

ivandeex commented Sep 15, 2021 •

edited

nkitsaini commented Nov 4, 2021 •

edited

dandar commented Apr 10, 2022 •

edited

showkubii commented May 31, 2022 •

edited

showkubii commented May 31, 2022 •

edited