The Wayback Machine - https://web.archive.org/web/20210928143604/https://github.com/Homebrew/homebrew-core/issues/60686
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

brew postgresql-upgrade-database fails with initb #60686

Open
1 task
obromios opened this issue Sep 5, 2020 · 26 comments
Open
1 task

brew postgresql-upgrade-database fails with initb #60686

obromios opened this issue Sep 5, 2020 · 26 comments

Comments

@obromios
Copy link

@obromios obromios commented Sep 5, 2020

Bug report

I can now reproduce the problem. Please see post below to see problem in a reproducible format.

  • [ x] ran brew update and can still reproduce the problem?
    On the other hand, brew update did not update any postgresl formulae. I have also asked this [question]https://stackoverflow.com/questions/63749141/initb-fails-for-brew-postgresql-upgrade-database) on SO but received no answer.
  • [ x] ran brew doctor, fixed all issues and can still reproduce the problem?
    I have run brew doctor and fixed a number of issues, but cannot reproduce the problem (see above).
  • [ x] ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
    When I run brew gist-logs postgresql-upgrade-database I get this output:
    Error: No available formula with the name "postgresql-upgrade-database"
  • if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?
    Included brew_config.txt and brew_doctor.txt and brew_doctor1.txt (output of brew doctor after trying to fix issues outlined initial run of brew_doctor
    brew_config.txt
    temp_brew_doctor.txt
    brew_doctor1.txt
    temp_brew_update.txt

What you were trying to do (and why)

Running brew postgresql-upgrade-database to upgrade database and data from 11.8 to 12

What happened (include command output)

Command output

Upgrading postgresql data from 11 to 12...
Stopping postgresql@11... (might take a while)
==> Successfully stopped postgresql@11 (label: homebrew.mxcl.postgresql@11)
waiting for server to shut down.... done
server stopped
==> Moving postgresql data from /usr/local/var/postgres to /usr/local/var/postgres.old...
==> Creating database...
Error: Upgrading postgresql data from 11 to 12 failed!
==> Moving postgresql data back from /usr/local/var/postgres.old to /usr/local/var/postgres...
Error: Failure while executing; /usr/local/Cellar/postgresql/12.4/bin/initdb --lc-collate=en_US.UTF-8 --lc-ctype=en_US.UTF-8 --lc-messages=en_US.UTF-8 --lc-monetary=en_US.UTF-8 --lc-numeric=en_US.UTF-8 --lc-time=en_US.UTF-8 -E\ UTF8 /usr/local/var/postgres exited with 1.

What you expected to happen

I expected the version of posgres to be updated to the most recent version 12 and my data migrated from the version 11 format to the version 12 format.

Step-by-step reproduction instructions (by running brew install commands)

Just run brew postgresql-upgrade-database. This issue looks very similar to this issue.

@obromios
Copy link
Author

@obromios obromios commented Sep 13, 2020

  • [ x] ran brew update and can still reproduce the problem?
    Yes.

  • [x ] ran brew doctor, fixed all issues and can still reproduce the problem?

  • [ x] ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
    When I run brew gist-logs postgresql-upgrade-database I get this output:
    Error: No available formula with the name "postgresql-upgrade-database"

  • [ x] if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?
    The output from brew doctor is

Warning: Some installed formulae were deleted!
You should find replacements for the following formulae:
  python@2

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  python@2
  python

However I have [email protected] linked.

Output from brew_config was

HOMEBREW_VERSION: 2.5.0
ORIGIN: https://github.com/Homebrew/brew.git
HEAD: 2ec8266697b2c9838cef86f8f97b5f392d0f926d
Last commit: 5 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 06a6ac51e2ed2930a21942ecfae1f8edcd5bfd67
Core tap last commit: 26 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: quad-core 64-bit kabylake
Clang: 11.0 build 1103
Git: 2.24.3 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 7.64.1 => /usr/bin/curl
macOS: 10.15.5-x86_64
CLT: 11.5.0.0.1.1588476445
Xcode: N/A

What you were trying to do (and why)

Running brew postgresql-upgrade-database to upgrade database and data from 11.8 to 12

Command output
==> Upgrading postgresql data from 11 to 12...
waiting for server to start....2020-09-13 07:31:24.237 AEST [22171] LOG:  listening on IPv6 address "::1", port 5432
2020-09-13 07:31:24.237 AEST [22171] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2020-09-13 07:31:24.238 AEST [22171] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-09-13 07:31:24.247 AEST [22172] LOG:  database system was shut down at 2020-09-13 07:31:09 AEST
2020-09-13 07:31:24.251 AEST [22171] LOG:  database system is ready to accept connections
 done
server started
waiting for server to shut down...2020-09-13 07:31:24.464 AEST [22171] LOG:  received fast shutdown request
.2020-09-13 07:31:24.466 AEST [22171] LOG:  aborting any active transactions
2020-09-13 07:31:24.467 AEST [22171] LOG:  background worker "logical replication launcher" (PID 22178) exited with exit code 1
2020-09-13 07:31:24.467 AEST [22173] LOG:  shutting down
2020-09-13 07:31:24.475 AEST [22171] LOG:  database system is shut down
 done
server stopped
==> Moving postgresql data from /usr/local/var/postgres to /usr/local/var/postgres.old...
==> Creating database...
Error: Upgrading postgresql data from 11 to 12 failed!
==> Moving postgresql data back from /usr/local/var/postgres.old to /usr/local/var/postgres...
Error: Failure while executing; `/usr/local/Cellar/postgresql/12.4/bin/initdb --lc-collate=en_AU.UTF-8 --lc-ctype=en_AU.UTF-8 --lc-messages=en_AU.UTF-8 --lc-monetary=en_AU.UTF-8 --lc-numeric=en_AU.UTF-8 --lc-time=en_AU.UTF-8 -E\ UTF8 /usr/local/var/postgres` exited with 1.
  

What you expected to happen

I expected the version of posgres to be updated to the most recent version 12 and my data migrated from the version 11 format to the version 12 format.

Step-by-step reproduction instructions (by running brew install commands)

These steps reproduce the problem:

  • brew install postgres
  • cd /usr/local/var/
  • rm -rf postgres
  • initdb /usr/local/var/postgres
  • brew postgresql-upgrade-database

This issue looks very similar to this issue.

@dtrodrigues
Copy link
Member

@dtrodrigues dtrodrigues commented Sep 14, 2020

Can you do a brew update to pull in #60694 and see if that resolves your issue?

@obromios
Copy link
Author

@obromios obromios commented Sep 14, 2020

I did a brew update and it did update taps related to homebrew cask and core, but still had the same problem. I then updated the operating system from 10.15.5 to 10.15.6, did another brew update which updated a tap related to homebrew core, but still have the same problem.

@obromios
Copy link
Author

@obromios obromios commented Sep 27, 2020

This is on a spare development laptop, so I have the option of fixing it by uninstalling version 11 and installing version 12, and recreating my databases from scratch. I am holding off doing this as it would mean I am not able to reproduce the problem for you. The spare machine is for travel, so just to let you know that I plan to do this next weekend, unless I hear from someone.

@obromios
Copy link
Author

@obromios obromios commented Oct 18, 2020

As foreshadowed, my work around for this was to

brew uninstall --force postgresql
 brew install postgresql@12
cd /usr/local/var/
rm -rf postgres
initdb /usr/local/var/postgres

and all was right with the world. Clearly this means I had to reinitialise all my databases, so is only good for a development machine.

@mzagaja
Copy link

@mzagaja mzagaja commented Nov 16, 2020

I have not been able to successfully update my Postgres since 2015. I have lazily just reinitialized my database but know this is a problem vexing another developer on my team so have dug more into it since I currently have a reproducible issue on my machine migrating from 12 --> 13.

Shell output of brew postgresql-upgrade-database
mzagaja@Matthew-Zagaja-MacBook-Pro /u/l/v/p/pg_snapshots> brew postgresql-upgrade-database
==> Upgrading postgresql data from 12 to 13...
waiting for server to start....2020-11-15 18:34:52.883 EST [76559] LOG:  starting PostgreSQL 12.4 on x86_64-apple-darwin20.1.0, compiled by Apple clang version 12.0.0 (clang-1200.0.32.27), 64-bit
2020-11-15 18:34:52.885 EST [76559] LOG:  listening on IPv6 address "::1", port 5432
2020-11-15 18:34:52.885 EST [76559] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2020-11-15 18:34:52.886 EST [76559] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2020-11-15 18:34:52.925 EST [76560] LOG:  database system was shut down at 2020-11-15 18:15:22 EST
2020-11-15 18:34:52.933 EST [76559] LOG:  database system is ready to accept connections
 done
server started
waiting for server to shut down....2020-11-15 18:34:53.167 EST [76559] LOG:  received fast shutdown request
2020-11-15 18:34:53.169 EST [76559] LOG:  aborting any active transactions
2020-11-15 18:34:53.170 EST [76559] LOG:  background worker "logical replication launcher" (PID 76566) exited with exit code 1
2020-11-15 18:34:53.171 EST [76561] LOG:  shutting down
2020-11-15 18:34:53.186 EST [76559] LOG:  database system is shut down
 done
server stopped
==> Moving postgresql data from /usr/local/var/postgres to /usr/local/var/postgres.old...
==> Creating database...
The files belonging to this database system will be owned by user "mzagaja".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /usr/local/var/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/New_York
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/local/opt/postgresql/bin/pg_ctl -D /usr/local/var/postgres -l logfile start

==> Migrating and upgrading data...
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Creating dump of global objects                             ok
Creating dump of database schemas
  dedham_master_plan_survey_development
*failure*

Consult the last few lines of "pg_upgrade_dump_53187.log" for
the probable cause of the failure.
Failure, exiting

*failure*

Consult the last few lines of "pg_upgrade_dump_54190.log" for
the probable cause of the failure.
Failure, exiting

*failure*

Consult the last few lines of "pg_upgrade_dump_18625.log" for
the probable cause of the failure.
Failure, exiting

*failure*

Consult the last few lines of "pg_upgrade_dump_18214.log" for
the probable cause of the failure.
Failure, exiting

child process exited abnormally: status 256
Failure, exiting
Error: Upgrading postgresql data from 12 to 13 failed!
==> Removing empty postgresql initdb database...
==> Moving postgresql data back from /usr/local/var/postgres.old to /usr/local/var/postgres...
Error: Failure while executing; `/usr/local/opt/postgresql/bin/pg_upgrade -r -b /usr/local/Cellar/postgresql@12/12.4_1/bin -B /usr/local/opt/postgresql/bin -d /usr/local/var/postgres.old -D /usr/local/var/postgres -j 4` exited with 1.

The error messages in the log files are mostly the same.

Log output from pg_dump
command: "/usr/local/opt/postgresql/bin/pg_dump" --host /usr/local/var/log --port 50432 --username mzagaja --schema-only --quote-all-identifiers --binary-upgrade --format=custom  --file="pg_upgrade_dump_18625.custom" 'dbname='"'"'ctnj-vote_test'"'"'' >> "pg_upgrade_dump_18625.log" 2>&1
pg_dump: error: query failed: FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
pg_dump: error: query was: SELECT oid, tableoid, pol.polname, pol.polcmd, pol.polpermissive, CASE WHEN pol.polroles = '{0}' THEN NULL ELSE    pg_catalog.array_to_string(ARRAY(SELECT pg_catalog.quote_ident(rolname) from pg_catalog.pg_roles WHERE oid = ANY(pol.polroles)), ', ') END AS polroles, pg_catalog.pg_get_expr(pol.polqual, pol.polrelid) AS polqual, pg_catalog.pg_get_expr(pol.polwithcheck, pol.polrelid) AS polwithcheck FROM pg_catalog.pg_policy pol WHERE polrelid = '18626'
command: "/usr/local/opt/postgresql/bin/pg_dump" --host /usr/local/var/log --port 50432 --username mzagaja --schema-only --quote-all-identifiers --binary-upgrade --format=custom  --file="pg_upgrade_dump_18625.custom" 'dbname='"'"'ctnj-vote_test'"'"'' >> "pg_upgrade_dump_18625.log" 2>&1
pg_dump: error: query failed: FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
pg_dump: error: query was: SELECT t.tableoid, t.oid, t.relname AS indexname, inh.inhparent AS parentidx, pg_catalog.pg_get_indexdef(i.indexrelid) AS indexdef, i.indnkeyatts AS indnkeyatts, i.indnatts AS indnatts, i.indkey, i.indisclustered, i.indisreplident, c.contype, c.conname, c.condeferrable, c.condeferred, c.tableoid AS contableoid, c.oid AS conoid, pg_catalog.pg_get_constraintdef(c.oid, false) AS condef, (SELECT spcname FROM pg_catalog.pg_tablespace s WHERE s.oid = t.reltablespace) AS tablespace, t.reloptions AS indreloptions, (SELECT pg_catalog.array_agg(attnum ORDER BY attnum)   FROM pg_catalog.pg_attribute   WHERE attrelid = i.indexrelid AND     attstattarget >= 0) AS indstatcols,(SELECT pg_catalog.array_agg(attstattarget ORDER BY attnum)   FROM pg_catalog.pg_attribute   WHERE attrelid = i.indexrelid AND     attstattarget >= 0) AS indstatvals FROM pg_catalog.pg_index i JOIN pg_catalog.pg_class t ON (t.oid = i.indexrelid) JOIN pg_catalog.pg_class t2 ON (t2.oid = i.indrelid) LEFT JOIN pg_catalog.pg_constraint c ON (i.indrelid = c.conrelid AND i.indexrelid = c.conindid AND c.contype IN ('p','u','x')) LEFT JOIN pg_catalog.pg_inherits inh ON (inh.inhrelid = indexrelid) WHERE i.indrelid = '18653'::pg_catalog.oid AND (i.indisvalid OR t2.relkind = 'p') AND i.indisready ORDER BY indexname

One of the pg_dump commands also fails due to PostGIS.

Log output from pg_dump
command: "/usr/local/opt/postgresql/bin/pg_dump" --host /usr/local/var/log --port 50432 --username mzagaja --schema-only --quote-all-identifiers --binary-upgrade --format=custom  --file="pg_upgrade_dump_53187.custom" 'dbname=art_spaces_development' >> "pg_upgrade_dump_53187.log" 2>&1
pg_dump: error: query failed: FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
pg_dump: error: query was: SELECT tableoid, oid, collname, collnamespace, (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = collowner) AS rolname FROM pg_collation
command: "/usr/local/opt/postgresql/bin/pg_dump" --host /usr/local/var/log --port 50432 --username mzagaja --schema-only --quote-all-identifiers --binary-upgrade --format=custom  --file="pg_upgrade_dump_53187.custom" 'dbname=art_spaces_development' >> "pg_upgrade_dump_53187.log" 2>&1
pg_dump: error: query failed: ERROR:  could not access file "$libdir/postgis-3": No such file or directory
pg_dump: error: query was: SELECT
a.attnum,
a.attname,
a.atttypmod,
a.attstattarget,
a.attstorage,
t.typstorage,
a.attnotnull,
a.atthasdef,
a.attisdropped,
a.attlen,
a.attalign,
a.attislocal,
pg_catalog.format_type(t.oid, a.atttypmod) AS atttypname,
a.attgenerated,
CASE WHEN a.atthasmissing AND NOT a.attisdropped THEN a.attmissingval ELSE null END AS attmissingval,
a.attidentity,
pg_catalog.array_to_string(ARRAY(SELECT pg_catalog.quote_ident(option_name) || ' ' || pg_catalog.quote_literal(option_value) FROM pg_catalog.pg_options_to_table(attfdwoptions) ORDER BY option_name), E',
    ') AS attfdwoptions,
CASE WHEN a.attcollation <> t.typcollation THEN a.attcollation ELSE 0 END AS attcollation,
array_to_string(a.attoptions, ', ') AS attoptions
FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_type t ON a.atttypid = t.oid
WHERE a.attrelid = '55202'::pg_catalog.oid AND a.attnum > 0::pg_catalog.int2
ORDER BY a.attnum

Looking at /usr/local/var/log/postgresql.log after doing a brew unlink postgresql && brew link postgresql@12 && brew services start postgresql@12 I can see

2020-11-15 23:51:27.459 GMT [88090] LOG:  could not open directory "/usr/local/share/postgresql/timezone": No such file or directory
2020-11-15 23:51:27.459 GMT [88090] LOG:  invalid value for parameter "log_timezone": "America/New_York"
2020-11-15 23:51:27.460 GMT [88090] LOG:  could not open directory "/usr/local/share/postgresql/timezone": No such file or directory
2020-11-15 23:51:27.460 GMT [88090] LOG:  invalid value for parameter "TimeZone": "America/New_York"
2020-11-15 23:51:27.460 GMT [88090] FATAL:  configuration file "/usr/local/var/postgres/postgresql.conf" contains errors

Googling did not reveal any ready to try fixes or culprits for the terminating connection log messages from Postgres. The culprit for me may or may not be PostGIS but I would expect more than one log file to include that error if it were blocking. I am open to next steps suggestions for debugging here.

@gromgit
Copy link
Member

@gromgit gromgit commented Nov 16, 2020

2020-11-15 23:51:27.459 GMT [88090] LOG: could not open directory "/usr/local/share/postgresql/timezone": No such file or directory

What's the output of ls -l /usr/local/share/postgresql/?

@MikeMcQuaid
Copy link
Member

@MikeMcQuaid MikeMcQuaid commented Nov 16, 2020

I have not been able to successfully update my Postgres since 2015.

This predates the existence of this script which means we're unlikely to fix your issue.

Closing this out as it's specific to a few users. This script is provided as a helper and is a shallow wrapper around postgres commands. If it doesn't work for you: we'll need PRs or you'll need to intervene directly.

@arbourd
Copy link
Contributor

@arbourd arbourd commented Nov 16, 2020

I don't think it's specific to a few users. I think any pre-existing Postgres install is affected by this. Simply reinstalling doesn't work either.

This is breaking for Postgres 13 if you had 12 in before.

@MikeMcQuaid
Copy link
Member

@MikeMcQuaid MikeMcQuaid commented Nov 16, 2020

Reopening but marking help wanted to make clear no maintainers are likely to fix this. I wrote this script already but don't use PostgreSQL in my development any more.

@arbourd
Copy link
Contributor

@arbourd arbourd commented Nov 16, 2020

Thanks @MikeMcQuaid. Understood. I'll try to take a look at it this week.

@mzagaja
Copy link

@mzagaja mzagaja commented Nov 17, 2020

I am happy to help debug this as well. I may have some bandwidth Wednesday and Thursday night.

@mzagaja
Copy link

@mzagaja mzagaja commented Nov 24, 2020

I attempted to put some work into this but am blocked by the fact that I accidentally lost my old Postgres data directory during other development work. I attempted to recreate a blank postgres + PostGIS database using postesql@12 but there doesn't appear to be a way to link postgis to postgresql@12 in order to create a basic/blank test database for this use case. If there is a trick anyone knows for this, that would help me create a reproducible test case.

I can confirm databases that are blank without PostGIS ever installed/enabled work fine with the script, however. If nobody has suggestions then I will likely be able to surface a usable test case again when Postgres 14 is released and can re-visit this.

@obromios
Copy link
Author

@obromios obromios commented Nov 24, 2020

So “brew install postgis does not link it in?

Just to be clear, are you saying that this issue only occurs if at some point PostGis has been used?

@shuiqingliu
Copy link

@shuiqingliu shuiqingliu commented Dec 21, 2020

same problem when I upgrade pg from 12 to 13,
pg_dump: error: query failed: ERROR: could not access file "$libdir/postgis-3": No such file or directory

@vdaubry
Copy link

@vdaubry vdaubry commented Dec 26, 2020

I have the exact same issue when I upgrade pg from 12 to 13

When migrating the existing DB, it fails at the step "Creating dump of database schemas"

$ tail /usr/local/var/log/pg_upgrade_dump_3317880.log
command: "/usr/local/opt/postgresql/bin/pg_dump" --host /usr/local/var/log --port 50432 --username vxxx --schema-only --quote-all-identifiers --binary-upgrade --format=custom  --file="pg_upgrade_dump_3317880.custom" 'dbname=company_development' >> "pg_upgrade_dump_3317880.log" 2>&1

pg_dump: error: query failed: ERROR:  could not access file "$libdir/postgis-3": No such file or directory

I uninstalled / reinstalled postgis without success

Thanks for your help !

@mzagaja
Copy link

@mzagaja mzagaja commented Dec 28, 2020

@obromios Apologies for the tardy reply, but yes brew install postgis will only link to 13 currently, and the issue is only recurring for me on a PostGIS enabled DB. Based on feedback from others I think we at least have found our "culprit" but either need someone that has this current state that can debug and test further, or a procedure to link PostGIS to the previous version so I can do it, or for next Postgres version to be released.

@kolten
Copy link

@kolten kolten commented Dec 28, 2020

Seeing the same issue as @vdaubry on upgrade from Postgres 12 to 13 and running brew postgresql-upgrade-database.

pg_dump: error: query failed: ERROR: could not access file "$libdir/postgis-3": No such file or directory

@mzagaja
Copy link

@mzagaja mzagaja commented Dec 29, 2020

I now have a working theory about this which is that the old database binary doesn't have a working PostGIS install when the pg_dump command is run. This may or may not be a function of the fact that homebrew uninstalls old versions of postgresql after running brew upgrade but before brew postgresql-upgrade-database. The postgresql-upgrade-database formula surmounts this by reinstalling the old Postgres version temporarily, but unfortunately for us, it no longer has PostGIS installed. Thus the pg_upgrade or pg_dump of old data fails.

Potential solution paths:

  1. Change the order of operations to run postgresql-upgrade-database before the old database install is uninstalled.
  2. Figure out a way to check/link PostGIS to the old version of postgresql that is installed in the script.

@bram-atmire
Copy link

@bram-atmire bram-atmire commented Feb 14, 2021

I just want to chime in that I have experienced similar problems: postgresql-upgrade-database from 12 to 13 didn't work for me, and after that, my postgresql@12 also got the error "could not access file "$libdir/postgis-3": No such file or directory" and it's unclear to me how I can fix the installation of postgis, for postgreqsl@12

Another similar issue that wasn't linked earlier in this thread: #47077

@shuiqingliu
Copy link

@shuiqingliu shuiqingliu commented Feb 20, 2021

we can downgrade pg to 12.4 and install postgis from source to solve this problem temporally. The solution steps are as follows.

  • downgrade pg to 12.4
    First edit postgresql formula and then modify url and hash
brew edit postgresql
// git diff
diff --git a/Formula/postgresql.rb b/Formula/postgresql.rb
index 863e35f6c14a..f4a641d306da 100644
--- a/Formula/postgresql.rb
+++ b/Formula/postgresql.rb
@@ -1,8 +1,8 @@
 class Postgresql < Formula
   desc "Object-relational database system"
   homepage "https://www.postgresql.org/"
-  url "https://ftp.postgresql.org/pub/source/v13.1/postgresql-13.1.tar.bz2"
-  sha256 "12345c83b89aa29808568977f5200d6da00f88a035517f925293355432ffe61f"
+  url "https://ftp.postgresql.org/pub/source/v12.4/postgresql-12.4.tar.bz2"
+  sha256 "bee93fbe2c32f59419cb162bcc0145c58da9a8644ee154a30b9a5ce47de606cc"
   license "PostgreSQL"
   head "https://github.com/postgres/postgres.git"
  • install postgresql
brew install postgresql
  • build postgis
//download postgis
wget https://download.osgeo.org/postgis/source/postgis-3.1.1.tar.gz && tar -zxvf postgis-3.1.1.tar.gz && cd postgis-3.1.1
//generate GUNMakefile
./configure --with-pgconfig=/usr/local/opt/postgresql/bin/pg_config --with-projdir=/usr/local/Cellar/proj/7.2.0 --with-gettext=/usr/local/Cellar/gettext/0.21 --disable-nls --with-jsondir=/usr/local/Cellar/json-c/0.15/ --with-protobufdir=/usr/local/Cellar/protobuf-c/1.3.3_3/ --with-pcredir=/usr/local/Cellar/pcre/8.44
//install
make && make install

ps: pay attention to the software version in configure

  • verify
pg_dump databasename > backup.sql
  • wait for the official to completely fix the problem

@mnutt
Copy link

@mnutt mnutt commented Feb 23, 2021

A failing postgresql-upgrade-database will leave a copy of postgres 12 in /usr/local/Cellar/postgresql@12/12.6. I was able to download and build postgis like the above, but instead of make install I just copied postgis/postgis-3.so to /usr/local/Cellar/postgresql@12/12.6/lib/postgresql and that was sufficient to be able to start postgres with /usr/local/opt/postgresql@12/bin/postgres -D /usr/local/var/postgres

At least for me, after the postgresql-upgrade-database script failed it attempted to move my postgres datadir from /usr/local/var/postgres.old back to /usr/local/var/postgres, but it had already created the directory so I ended up with my datadir stuck in /usr/local/var/postgres/postgres.old and had to manually move it back to /usr/local/var/postgres.

@jacoBeastman
Copy link

@jacoBeastman jacoBeastman commented Mar 25, 2021

I tried @mnutt approach and built the postgis from my 12 pg_config using modified config options from @shuiqingliu to allow me to copy it to my /usr/local/Cellar/postgresql@12/12.6/lib/postgresql/ directory. The 12 pg_config pointed INCLUDEDIR-SERVER = /usr/local/opt/postgresql@12/include/server which didn't exist any longer and postgis wouldn't compile due to missing .h files and the 13 pg_config pointed to the INCLUDEDIR-SERVER = /usr/local/include/postgresql/server which was NOT symlinked to the Cellar. At this point I gave up but I suspect if I had brew linked postgres 12 again this would have fixed the dependency issues to allow me to build postgis for postgres 12, copy it to the 12 directory, and maybe even successfully upgrade my data with postgresql-upgrade-database. Or maybe it would have broken postgres 13 and the upgrade would still fail.

In the end I wasted too much time trying to save my local test data and created a new data directory.

@smnorris
Copy link

@smnorris smnorris commented Mar 26, 2021

Thank you above for the hints.
I could not compile postgis either: fatal error: 'postgres.h' file not found.
Culprit is pg_config points to a folder that does not exist:

$ pg_config
...
INCLUDEDIR-SERVER = /usr/local/opt/postgresql@12/include/server
...

I tried uninstalling and re-installing postgresql@12, but the config still pointed to the wrong folder.
Creating a symbolic link worked:

brew uninstall postgresql@12
brew uninstall postgis 
brew uninstall postgresql 
brew install postgresql@12
brew link postgresql@12
# create link to missing folder
ln -s /usr/local/opt/postgresql@12/include/postgis/server /usr/local/opt/postgresql@12/include/postgis/server
# get postgis and build (check your versions as noted above)
wget https://download.osgeo.org/postgis/source/postgis-3.0.3.tar.gz && tar -zxvf postgis-3.0.3.tar.gz && cd postgis-3.0.3
./configure --with-pgconfig=/usr/local/opt/postgresql@12/bin/pg_config --with-projdir=/usr/local/Cellar/proj/7.2.1 --with-gettext=/usr/local/Cellar/gettext/0.21 --disable-nls --with-jsondir=/usr/local/Cellar/json-c/0.15/ --with-protobufdir=/usr/local/Cellar/protobuf-c/1.3.3_4/ --with-pcredir=/usr/local/Cellar/pcre/8.44
make
cp postgis/postgis-3.so /usr/local/Cellar/postgresql@12/12.6/lib/postgresql
/usr/local/opt/postgresql@12/bin/postgres -D /usr/local/var/postgres

Proceed to dump and restore to the new pg13 db.
I should have known better than trying to do a major version upgrade with a single command.

@Atixx
Copy link

@Atixx Atixx commented Mar 31, 2021

So following the comments, was able to make postgis work, thanks everybody, especially @smnorris. But needed to point out a slight change I had to do, might be a typo:
instead of:
ln -s /usr/local/opt/postgresql@12/include/postgis/server /usr/local/opt/postgresql@12/include/postgis/server
replace with:
ln -s /usr/local/opt/postgresql@12/include/postgresql/server /usr/local/opt/postgresql@12/include/server

i noticed I had no postgis directory out the get go, and started to try to understand what I was missing, but took me a while to notice that slight difference in the directory tree, to create the symlink on

Edit that might make restoring easier:

using brew to install posgres and postgis (from core tap) I was able to do a simple restore using pg_upgrade like so:

pg_upgrade --old-bindir /usr/local/Cellar/postgresql@12/12.6_1/bin --new-bindir /usr/local/Cellar/postgresql/13.2_1/bin --old-datadir /usr/local/var/postgres12.backup --new-datadir /usr/local/var/postgres

Where /usr/local/var/postgres12.backup was the old directory with my data files (incompatible with V13).
and /usr/local/var/postgres was fresh after initdb /usr/local/var/postgres execution

restoring like that was inspired by: @olivierlacan post

@smnorris
Copy link

@smnorris smnorris commented Mar 31, 2021

That does look like a typo, sorry for the sabotage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet