B2: Dynamically adjust --b2-chunk-size when necessary #6138
Conversation
Slugger
changed the title
Looks like a nice change :-)
I put some comments inline.
In order to accept the new package I'd like you to add a commit which uses it for s3 and another ideally for azureblob - I don't like abstractions which don't get used more than once :-)
I'll run the CI on this too so you can see if it passes or not.
|
Thanks for the feedback. I've addressed all the comments except the implementation in additional backends. I'll do that next time I can put some time in -- some evening hopefully later this week. Just dipping my toes into the project here with this relatively simple feature request I found in the backlog. Got to say, so far so good. It's always nice to get a super quick, constructive response when starting out in a new project. Looking forward to contributing (I have a big idea, which is what brought me here, but learning to crawl before I try to climb Everest! |
|
Refactored |
Changes look great - thank you :-)
I'll run the CI now.
Can you squash into 4 commits, one for the chunksize lib, then one for each backend then I'll run the CI one more time and merge - thank you :-)
Slugger
force-pushed
the
feature/4643-b2-dynamic-chunk-size
branch
2 times, most recently
from
749faf6
to
1e977fb
Compare
Slugger
force-pushed
the
feature/4643-b2-dynamic-chunk-size
branch
from
1e977fb
to
0778535
Compare
|
Linter issues fixed, commits squashed. Everything should be ready to go. |
What is the purpose of this change?
Add support for dynamic modification of
--b2-chunk-sizewhen necessary as discussed in #4643.Created a new package,
chunksizethat implements a chunk size calculator, which is then called by B2 to dynamically adjust the chunk size when needed. This package was created with the intent to eventually leverage this calculator in all backends that need to do dynamic chunk size adjustments (S3 at a minimum).Was the change discussed in an issue or in the forum before?
Yes. #4643
Checklist