The Wayback Machine - https://web.archive.org/web/20220602112504/https://github.com/rclone/rclone/issues/6162
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move to Azure Blob without read permission fails on first try #6162

Open
tomasgiden opened this issue May 11, 2022 · 3 comments
Open

Move to Azure Blob without read permission fails on first try #6162

tomasgiden opened this issue May 11, 2022 · 3 comments

Comments

@tomasgiden
Copy link

@tomasgiden tomasgiden commented May 11, 2022 •
edited by ncw

What is the problem you are having with rclone?

I have set up an Azure Blob storage with a SAS key with Write but WITHOUT Read permission. I then try to move local files to that storage. rClone gives an error message that it failed, but then it says afterwards that it worked and deletes the local file. I've checked and the data is correctly uploaded so the issue more along the lines of that the error message creates confusion.

If changing move to copy, there is no error message.

What is your rclone version (output from rclone version)

rclone v1.58.0

  • os/version: Microsoft Windows 11 Pro 21H2 (64 bit)
  • os/kernel: 10.0.22000.613 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.8
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (e.g. Google Drive)

Azure Blob Storage

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone -vv move C:\Users\Tomas\dev\test ulftest_write://tomas-test

Note, folder C:\Users\Tomas\dev\test contains one text file "test C.txt" containing a small dummy string.

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

2022/05/11 12:05:37 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "-vv" "move" "C:\\Users\\Tomas\\dev\\test" "ulftest_write://tomas-test"]
2022/05/11 12:05:37 DEBUG : Creating backend with remote "C:\\Users\\Tomas\\dev\\test"
2022/05/11 12:05:37 DEBUG : Using config file from "C:\\Users\\Tomas\\AppData\\Roaming\\rclone\\rclone.conf"
2022/05/11 12:05:37 DEBUG : fs cache: renaming cache item "C:\\Users\\Tomas\\dev\\test" to be canonical "//?/C:/Users/Tomas/dev/test"
2022/05/11 12:05:37 DEBUG : Creating backend with remote "ulftest_write://tomas-test"
2022/05/11 12:05:37 DEBUG : fs cache: renaming cache item "ulftest_write://tomas-test" to be canonical "ulftest_write:tomas-test"
2022/05/11 12:05:38 DEBUG : Azure container tomas-test: Waiting for checks to finish
2022/05/11 12:05:38 DEBUG : Azure container tomas-test: Waiting for transfers to finish
2022/05/11 12:05:38 ERROR : test C.txt: Failed to copy: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/[email protected]/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationPermissionMismatch) =====
Description=403 This request is not authorized to perform this operation using this permission., Details: (none)
   HEAD https://exciscopeulftest.blob.core.windows.net/tomas-test/test%20C.txt?se=2023-05-10t11%3A19%3A00z&sig=REDACTED&sp=acwl&sr=c&st=2022-05-09t11%3A19%3A42z&sv=2020-04-08&timeout=31536001
   User-Agent: [rclone/v1.58.0]
   X-Ms-Client-Request-Id: [e71ec017-a9d6-4237-7eee-aee8e1c8ab42]
   X-Ms-Version: [2020-04-08]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
   Date: [Wed, 11 May 2022 10:05:38 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   Vary: [Origin]
   X-Ms-Client-Request-Id: [e71ec017-a9d6-4237-7eee-aee8e1c8ab42]
   X-Ms-Error-Code: [AuthorizationPermissionMismatch]
   X-Ms-Request-Id: [f8a22b6e-501e-00a0-701e-655bd6000000]
   X-Ms-Version: [2020-04-08]


2022/05/11 12:05:38 ERROR : test C.txt: Not deleting source as copy failed: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/[email protected]/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationPermissionMismatch) =====
Description=403 This request is not authorized to perform this operation using this permission., Details: (none)
   HEAD https://exciscopeulftest.blob.core.windows.net/tomas-test/test%20C.txt?se=2023-05-10t11%3A19%3A00z&sig=REDACTED&sp=acwl&sr=c&st=2022-05-09t11%3A19%3A42z&sv=2020-04-08&timeout=31536001
   User-Agent: [rclone/v1.58.0]
   X-Ms-Client-Request-Id: [e71ec017-a9d6-4237-7eee-aee8e1c8ab42]
   X-Ms-Version: [2020-04-08]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
   Date: [Wed, 11 May 2022 10:05:38 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   Vary: [Origin]
   X-Ms-Client-Request-Id: [e71ec017-a9d6-4237-7eee-aee8e1c8ab42]
   X-Ms-Error-Code: [AuthorizationPermissionMismatch]
   X-Ms-Request-Id: [f8a22b6e-501e-00a0-701e-655bd6000000]
   X-Ms-Version: [2020-04-08]


2022/05/11 12:05:38 ERROR : Attempt 1/3 failed with 1 errors and: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/[email protected]/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationPermissionMismatch) =====
Description=403 This request is not authorized to perform this operation using this permission., Details: (none)
   HEAD https://exciscopeulftest.blob.core.windows.net/tomas-test/test%20C.txt?se=2023-05-10t11%3A19%3A00z&sig=REDACTED&sp=acwl&sr=c&st=2022-05-09t11%3A19%3A42z&sv=2020-04-08&timeout=31536001
   User-Agent: [rclone/v1.58.0]
   X-Ms-Client-Request-Id: [e71ec017-a9d6-4237-7eee-aee8e1c8ab42]
   X-Ms-Version: [2020-04-08]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation using this permission.
   Date: [Wed, 11 May 2022 10:05:38 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   Vary: [Origin]
   X-Ms-Client-Request-Id: [e71ec017-a9d6-4237-7eee-aee8e1c8ab42]
   X-Ms-Error-Code: [AuthorizationPermissionMismatch]
   X-Ms-Request-Id: [f8a22b6e-501e-00a0-701e-655bd6000000]
   X-Ms-Version: [2020-04-08]


2022/05/11 12:05:38 DEBUG : test C.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2022/05/11 12:05:38 DEBUG : test C.txt: Unchanged skipping
2022/05/11 12:05:38 DEBUG : Azure container tomas-test: Waiting for checks to finish
2022/05/11 12:05:38 INFO  : test C.txt: Deleted
2022/05/11 12:05:38 DEBUG : Azure container tomas-test: Waiting for transfers to finish
2022/05/11 12:05:38 INFO  : There was nothing to transfer
2022/05/11 12:05:38 ERROR : Attempt 2/3 succeeded
2022/05/11 12:05:38 INFO  :
Transferred:             10 B / 10 B, 100%, 0 B/s, ETA -
Checks:                 3 / 3, 100%
Deleted:                1 (files), 0 (dirs)
Elapsed time:         0.4s

2022/05/11 12:05:38 DEBUG : 5 go routines active

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.
@ncw
Copy link
Member

@ncw ncw commented May 12, 2022

I think what is happening here is that the post copy check is failing

HEAD https://exciscopeulftest.blob.core.windows.net/tomas-test/test%20C.txt?se=2023-05-10t11%3A19%3A00z&sig=REDACTED&sp=acwl&sr=c&st=2022-05-09t11%3A19%3A42z&sv=2020-04-08&timeout=31536001

Here rclone is attempting to read the MD5 sum etc off the object to check it got transferred OK.

There isn't currently a workaround for this as rclone will attempt to re-read the metadata regardless of the setting of --ignore-checksum.

Would it be possible to add HEAD objects permission to your SAS URL? That will let rclone check the objects got uploaded correctly.

@tomasgiden
Copy link
Author

@tomasgiden tomasgiden commented May 12, 2022

There is no HEAD permission in Azure Blob. Though there is a List permission and when listing blobs you can get the MD5 hash (according to this: https://stackoverflow.com/questions/67969357/azure-blob-content-md5-property-is-null-while-reading-it-using-azure-blob-python). The above result is actually with Write + List permission. (Without List permission the operation fails at an even earlier stage and does not complete at all).

The follow up question is then, if rClone cannot read the MD5 sum etc to check it got transferred OK, how come it succeeds at attempt 2/3?

"
2022/05/11 12:05:38 DEBUG : test C.txt: Size and modification time the same (differ by 0s, within tolerance 100ns)
2022/05/11 12:05:38 DEBUG : test C.txt: Unchanged skipping
2022/05/11 12:05:38 DEBUG : Azure container tomas-test: Waiting for checks to finish
2022/05/11 12:05:38 INFO : test C.txt: Deleted
"

Is it so that it at the start of the second attempt, rClone does a LIST action to retrieve all metadata (md5, size, modification time etc) and therefore because of the List permission it works?

If not, how can it be sure enough that it is transferred OK that it removes the local file?

@ncw
Copy link
Member

@ncw ncw commented May 19, 2022

Is it so that it at the start of the second attempt, rClone does a LIST action to retrieve all metadata (md5, size, modification time etc) and therefore because of the List permission it works?

That is correct, yes. As you noted above, the md5sums are in the listing, so when rclone retries, it discovers the file is there with the correct checksum so it doesn't need to copy it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tomasgiden @ncw