The Wayback Machine - https://web.archive.org/web/20220531004520/https://github.com/go-gitea/gitea/pull/15928
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pause queues #15928

Merged
merged 27 commits into from Jan 22, 2022
Merged

Pause queues #15928

merged 27 commits into from Jan 22, 2022

Conversation

zeripath
Copy link
Contributor

@zeripath zeripath commented May 19, 2021

Add functionality to pause and resume queues with the ability to hand back unhandled data.

This also now alerts if a queue configuration will result in no workers and no worker scalling (as in #18189) - and will allow no-worker configurations on redis to not result in loss from the queue.

@zeripath zeripath added this to the 1.15.0 milestone May 19, 2021
@zeripath
Copy link
Contributor Author

@zeripath zeripath commented May 19, 2021

FAO @lafriks

@lafriks
Copy link
Member

@lafriks lafriks commented May 20, 2021

From code point of view looks good to me, will test it on weekend together with my WIP PR

@codecov-commenter

This comment has been minimized.

@zeripath zeripath removed this from the 1.15.0 milestone Jun 23, 2021
@zeripath zeripath added this to the 1.16.0 milestone Jun 23, 2021
@zeripath zeripath mentioned this pull request Jul 14, 2021
@lafriks
Copy link
Member

@lafriks lafriks commented Jul 23, 2021

Build seems to be failing now

@zeripath
Copy link
Contributor Author

@zeripath zeripath commented Jul 23, 2021

yeah I must have made a broken merge when I fixed the last conflict.

Signed-off-by: Andrew Thornton <[email protected]>
@zeripath
Copy link
Contributor Author

@zeripath zeripath commented Jul 23, 2021

fixed

@zeripath
Copy link
Contributor Author

@zeripath zeripath commented Jan 21, 2022

As an additional thing we can now add warnings for no workers and finally safely handle the case of no worker configurations for redis.

modules/queue/workerpool.go Outdated Show resolved Hide resolved
modules/queue/queue_bytefifo.go Outdated Show resolved Hide resolved
Copy link
Contributor

@wxiaoguang wxiaoguang left a comment

LGTM, while maybe util.StopTimer(timer) can be used to reduce some code.

@lafriks
Copy link
Member

@lafriks lafriks commented Jan 22, 2022

🚀

@zeripath
Copy link
Contributor Author

@zeripath zeripath commented Jan 22, 2022

Make lgtm work

@zeripath zeripath merged commit a82fd98 into go-gitea:main Jan 22, 2022
2 checks passed
@zeripath zeripath deleted the pause-queues branch Jan 22, 2022
zjjhot pushed a commit to zjjhot/gitea that referenced this issue Jan 23, 2022
* 'main' of https://github.com/go-gitea/gitea:
  Add packagist webhook (go-gitea#18224)
  Fix mime-type detection for HTTP server (go-gitea#18370)
  Always use git command but not os.Command (go-gitea#18363)
  Add deprecated for LFS_CONTENT_PATH on zh-cn docs (go-gitea#18362)
  [skip ci] Updated translations via Crowdin
  Make gitea, gitea-vet future-proof (go-gitea#18361)
  Pause queues (go-gitea#15928)
  Disable content sniffing on `PlainTextBytes` (go-gitea#18359)
  Update github.com/duo-labs/webauthn (go-gitea#18357)
zeripath added a commit that referenced this issue Jan 24, 2022
Implements the Pausable interface on WrappedQueues and PersistableChannelUniqueQueues

Reference #15928

Signed-off-by: Andrew Thornton [email protected]
Chianina pushed a commit to Chianina/gitea that referenced this issue Mar 28, 2022
* Start adding mechanism to return unhandled data

Signed-off-by: Andrew Thornton <[email protected]>

* Create pushback interface

Signed-off-by: Andrew Thornton <[email protected]>

* Add Pausable interface to WorkerPool and Manager

Signed-off-by: Andrew Thornton <[email protected]>

* Implement Pausable and PushBack for the bytefifos

Signed-off-by: Andrew Thornton <[email protected]>

* Implement Pausable and Pushback for ChannelQueues and ChannelUniqueQueues

Signed-off-by: Andrew Thornton <[email protected]>

* Wire in UI for pausing

Signed-off-by: Andrew Thornton <[email protected]>

* add testcases and fix a few issues

Signed-off-by: Andrew Thornton <[email protected]>

* fix build

Signed-off-by: Andrew Thornton <[email protected]>

* prevent "race" in the test

Signed-off-by: Andrew Thornton <[email protected]>

* fix jsoniter mismerge

Signed-off-by: Andrew Thornton <[email protected]>

* fix conflicts

Signed-off-by: Andrew Thornton <[email protected]>

* fix format

Signed-off-by: Andrew Thornton <[email protected]>

* Add warnings for no worker configurations and prevent data-loss with redis/levelqueue

Signed-off-by: Andrew Thornton <[email protected]>

* Use StopTimer

Signed-off-by: Andrew Thornton <[email protected]>

Co-authored-by: Lauris BH <[email protected]>
Co-authored-by: 6543 <[email protected]>
Co-authored-by: techknowlogick <[email protected]>
Co-authored-by: wxiaoguang <[email protected]>
Chianina pushed a commit to Chianina/gitea that referenced this issue Mar 28, 2022
…tea#18393)

Implements the Pausable interface on WrappedQueues and PersistableChannelUniqueQueues

Reference go-gitea#15928

Signed-off-by: Andrew Thornton [email protected]
@go-gitea go-gitea locked and limited conversation to collaborators Apr 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

Successfully merging this pull request may close these issues.

None yet

7 participants