The Wayback Machine - https://web.archive.org/web/20220503004212/https://github.com/go-gitea/gitea/pull/12244
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dump github/gitlab/gitea repository data to a local directory and restore to gitea #12244

Merged
merged 47 commits into from Dec 27, 2020
Merged

Dump github/gitlab/gitea repository data to a local directory and restore to gitea #12244

merged 47 commits into from Dec 27, 2020
+1,484 −225

Conversation

lunny
Copy link
Member

@lunny lunny commented Jul 14, 2020 •
edited

Currently online migrating is possible but in fact, it depends on the internet network status what's a mess. This PR makes it possible to dump a repository on GitHub/GitLab/Gitea to a disk directory and then transfer it to another Gitea instance via the possible transfer method. And then restore the repository on the target Gitea instance.

i.e. Gitea repository itself in GitHub spent about 80 GB but transfer 80GB online is almost impossible. Then we can dump the repository to a host nearby Github's host and then compress it and transfer it to another host.

A new Gitea command has been added to do the things, you can use it as below:

./gitea dump-repo --repo_dir ./dumps --clone_addr https://github.com/go-xorm/builder.git --auth_username <token> --owner_name xorm --repo_name builder

A dumped repository(go-gitea/test_repo) directory structure like this.

图片

Another command is restore-repo added from this PR which could restore the repository to a Gitea instance from the dumped folder.

./gitea restore-repo --repo_dir ./dumps/xorm/builder --owner_name xorm --repo_name builder

And this could be also used as a backup tool to backup repositories in Github/Gitlab/Gitea .

  • Add a command gitea dump-repo
  • Add a command gitea restore-repo

@lunny lunny added this to the 1.13.0 milestone Jul 14, 2020
@lunny lunny changed the title WIP: Dump github/gitlab repository data to a local directory Dump github/gitlab repository data to a local directory Jul 18, 2020
@lunny lunny mentioned this pull request Jul 18, 2020
Closed
@lunny lunny mentioned this pull request Jul 21, 2020
Closed
@6543
Copy link

@6543 6543 commented Jul 27, 2020

@lunny one thing left: we should also save the git-service type and source url somewhere

@6543
Copy link

@6543 6543 commented Jul 27, 2020

... to be able to migrate issue/pull ownership to existing users as we can do with the normal github/gitlab migration now

@codecov-commenter
Copy link

@codecov-commenter codecov-commenter commented Aug 21, 2020 •
edited

Codecov Report

Merging #12244 into master will increase coverage by 0.66%.
The diff coverage is 1.49%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #12244      +/-   ##
==========================================
+ Coverage   42.59%   43.25%   +0.66%     
==========================================
  Files         671      645      -26     
  Lines       73635    71468    -2167     
==========================================
- Hits        31365    30917     -448     
+ Misses      37184    35543    -1641     
+ Partials     5086     5008      -78
Impacted Files Coverage Δ
cmd/dump_repo.go 0.00% <0.00%> (ø)
modules/migrations/base/pullrequest.go 0.00% <ø> (ø)
modules/migrations/dump.go 0.00% <0.00%> (ø)
modules/structs/repo.go 54.54% <ø> (+4.54%) ⬆️
modules/migrations/migrate.go 22.47% <41.66%> (+0.91%) ⬆️
routers/api/v1/settings/settings.go 44.44% <0.00%> (-55.56%) ⬇️
modules/storage/local.go 0.00% <0.00%> (-38.10%) ⬇️
modules/queue/queue.go 24.39% <0.00%> (-7.87%) ⬇️
models/attachment.go 54.86% <0.00%> (-6.25%) ⬇️
modules/webhook/matrix.go 30.30% <0.00%> (-4.67%) ⬇️
... and 208 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54dd28f...5342029. Read the comment docs.

@techknowlogick
Copy link

@techknowlogick techknowlogick commented Aug 28, 2020

please resolve conflict

@lunny lunny force-pushed the lunny/dump_repository branch 2 times, most recently from 4ffbe89 to af69f37 Compare Sep 9, 2020
@techknowlogick
Copy link

@techknowlogick techknowlogick commented Sep 9, 2020

CI fail is related:

level=warning msg="[runner] Can't run linter goanalysis_metalinter: S1008: failed prerequisites: [([email protected]/gitea/modules/cron, [email protected]/gitea/modules/cron): analysis skipped: errors in package: [/go/src/code.gitea.io/gitea/modules/cron/tasks_basic.go:12:2: could not import code.gitea.io/gitea/modules/migrations (/go/src/code.gitea.io/gitea/modules/migrations/dump.go:514:29: cannot use downloader (variable of type *RepositoryRestorer) as base.Downloader value in argument to DoMigrateRepository: wrong type for method GetAsset)]]"
level=warning msg="[runner] Can't run linter unused: buildir: analysis skipped: errors in package: [/go/src/code.gitea.io/gitea/modules/migrations/dump.go:514:29: cannot use downloader (variable of type *RepositoryRestorer) as base.Downloader value in argument to DoMigrateRepository: wrong type for method GetAsset -: could not load export data: no export data for \"code.gitea.io/gitea/modules/migrations\"]"
level=error msg="Running error: buildir: analysis skipped: errors in package: [/go/src/code.gitea.io/gitea/modules/migrations/dump.go:514:29: cannot use downloader (variable of type *RepositoryRestorer) as base.Downloader value in argument to DoMigrateRepository: wrong type for method GetAsset -: could not load export data: no export data for \"code.gitea.io/gitea/modules/migrations\"]"

@6543
Copy link

@6543 6543 commented Sep 10, 2020

I think we should stave CloneURL to yaml too
why not move Topics into a generell repo.yml ?
so the local "import" still has the right urls ...

@lunny lunny changed the title Dump github/gitlab repository data to a local directory WIP: Dump/restore github/gitlab repository data to a local directory Sep 11, 2020
@lunny lunny removed this from the 1.13.0 milestone Sep 11, 2020
@lunny lunny added this to the 1.14.0 milestone Sep 11, 2020
@lunny lunny mentioned this pull request Sep 17, 2020
Closed
@lunny lunny changed the title WIP: Dump/restore github/gitlab repository data to a local directory WIP: Dump/restore github/gitlab repository data to a local directory and restore to gitea Sep 17, 2020
@lunny lunny mentioned this pull request Sep 17, 2020
Open
20 tasks
@codecov-io
Copy link

@codecov-io codecov-io commented Oct 19, 2020 •
edited

Codecov Report

Merging #12244 (695d23e) into master (cf9d471) will decrease coverage by 0.29%.
The diff coverage is 3.25%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #12244      +/-   ##
==========================================
- Coverage   42.36%   42.06%   -0.30%     
==========================================
  Files         728      733       +5     
  Lines       78015    78666     +651     
==========================================
+ Hits        33049    33093      +44     
- Misses      39540    40159     +619     
+ Partials     5426     5414      -12
Impacted Files Coverage Δ
cmd/dump_repo.go 0.00% <0.00%> (ø)
cmd/restore_repo.go 0.00% <0.00%> (ø)
models/admin.go 60.31% <0.00%> (-8.78%) ⬇️
models/task.go 34.95% <ø> (+0.04%) ⬆️
modules/migrations/base/downloader.go 17.51% <ø> (ø)
modules/migrations/base/pullrequest.go 0.00% <ø> (ø)
modules/migrations/dump.go 0.00% <0.00%> (ø)
modules/migrations/error.go 100.00% <ø> (ø)
modules/migrations/git.go 46.42% <ø> (+3.09%) ⬆️
modules/migrations/gitea_downloader.go 0.91% <0.00%> (+<0.01%) ⬆️
... and 29 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 212fa34...695d23e. Read the comment docs.

@6543
Copy link

@6543 6543 commented Oct 19, 2020

@lunny "migrations/restore.go exported methods should have comment or be unexported"

6543
6543 reviewed Oct 19, 2020
View changes
cmd/dump_repo.go Outdated Show resolved Hide resolved
@lunny lunny force-pushed the lunny/dump_repository branch 2 times, most recently from f9b17ae to 13717d8 Compare Oct 21, 2020
@lunny
Copy link
Author

@lunny lunny commented Dec 27, 2020

@techknowlogick done.

@lunny lunny merged commit dd08853 into go-gitea:master Dec 27, 2020
2 checks passed
@lunny lunny deleted the lunny/dump_repository branch Dec 27, 2020
@delvh delvh mentioned this pull request Jan 28, 2021
Closed
12 tasks
@go-gitea go-gitea locked and limited conversation to collaborators Feb 11, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@a1012112796 a1012112796

@techknowlogick techknowlogick

@6543 6543

Assignees
No one assigned
Labels
kind/feature lgtm/done theme/migration
Milestone
1.14.0
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@lunny @6543 @codecov-commenter @techknowlogick @codecov-io @a1012112796 @GiteaBot