The Wayback Machine - https://web.archive.org/web/20230607221252/https://github.com/rclone/rclone/issues/6160
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preserve hard links #6160

Open
brianjmurrell opened this issue May 9, 2022 · 4 comments
Open

Preserve hard links #6160

brianjmurrell opened this issue May 9, 2022 · 4 comments

Comments

@brianjmurrell
Copy link

What problem are you are trying to solve?

Space efficiency with hard links.

How do you think rclone should be changed to solve that?

It should recognize, say, when doing rclone sync, when files are actually hard links and preserve those on the remote. This not only saves space on the remote but also saves the bandwidth of transferring all of the links to that one file as separate files.

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.
@animosity22
Copy link
Collaborator

I'm sure what you what to happen.

If you have a file that's a hard link, it would get fully copied to the remote.

Remotes have no concept of hard links so if you have:

/dira/file.txt
/dirb/file.txt

On the same physical volume and want to copy it to a cloud remote, you get two copies as the cloud remotes have no idea/concept of hard links as those are physical file constraints.

In the scenario above, what would you want to happen on the remote?

@brianjmurrell
Copy link
Author

If you have a file that's a hard link, it would get fully copied to the remote.

Yes, understood. Wastefully as many times as there are hard links.

Remotes have no concept of hard links

Is that universally true? AFAICT, SFTP supports hard links.

so if you have:

/dira/file.txt /dirb/file.txt

On the same physical volume and want to copy it to a cloud remote, you get two copies as the cloud remotes have no idea/concept of hard links as those are physical file constraints.

Again, I believe SFTP understands hard links. So when a remote is an SFTP server on a POSIX filesystem, there is no reason to not support hard links.

In the scenario above, what would you want to happen on the remote?

I would want the remote, which is an SFTP remote, to create hard links on it's POSIX file system.

So while, yes, many "cloud" providers don't support hard links, it is not universally true of all rclone remote types and where they are supported by a remote, they should be used.

@animosity22
Copy link
Collaborator

Is that universally true? AFAICT, SFTP supports hard links.

So let me be specific, generally rclone uses cloud based remotes and most/many do not support hard links. Are the exceptions? Sure. I was speaking generally so thanks for clarifying as I didn't say all but I do see it was implied.

I would want the remote, which is an SFTP remote, to create hard links on it's POSIX file system.

As above, there are so many SFTP servers out there as some support / some don't. It's very dependent on what the SFTP does as if it is supported or not.

It's a very narrow use case in terms of rclone so it can be logged as an enhancement, but realistically unless you plan to implement, it will most likely sit unless someone else picks it up, sponsor someone to make the change or it'll just collect dust.

@brianjmurrell
Copy link
Author

For anyone else who comes across this looking for hard-link preservation (and indeed, all of the rest of the rsync goodness, I was able to get rsync working over CryFS to have an encrypted remote such as rclone can do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants