[postgis-users] Help to upgrade postgresql10 with postgis 2.5
Regina Obe
lr at pcorp.us
Thu Mar 23 14:02:22 PDT 2023
Well if you drop cascade postgis, you may be dropping table columns as well. So that is not a good idea.
You could try upgrading your PostgreSQL 10 to 3.something first and then upgrade that.
From: postgis-users [mailto:postgis-users-bounces at lists.osgeo.org] On Behalf Of pham lan
Sent: Thursday, March 23, 2023 7:41 AM
To: PostGIS Users Discussion <postgis-users at lists.osgeo.org>
Subject: Re: [postgis-users] Help to upgrade postgresql10 with postgis 2.5
Hi,
So my last test was to drop extension postgis cascade (from one test system) which drops more than 200 materialized views. Only afterward the pg_upgrade works without being crashed.
So it seems the problem is the postgis extension, though the version on postgres10 and postgres 12 are both postgis 2.5. I don't understand why it caused the problem then. Do you have any idea?
If so, in this case, there is no other way for us to upgrade postgres besides dropping those views and recreate them again, right?
Thanks,
Lan
On Sat, Mar 18, 2023 at 2:44 PM Regina Obe <lr at pcorp.us <mailto:lr at pcorp.us> > wrote:
I should add that ultimately it isn’t the vacuuming crashing, but something else running at the same time.
This article details how to find the root cause
https://pganalyze.com/docs/log-insights/server/S1
I have heard on occasion postgis raster crashing because of a bad library dependency (but this usually happens when on create extension of postgis (or postgis_raster) or running a raster process. I suppose it could happen on backend load of postgis libraries.
If you don’t need postgis raster, then the easy fix for that is just to
After you upgrade each database in your PG12 that has postgis installed with :
SELECT postgis_extensions_upgrade();
SELECT postgis_extensions_upgrade();
Do
DROP EXTENSION postgis_raster;
Dropping will fail if you have raster tables, so no need for concern accidentally dropping data.
From: Regina Obe [mailto:lr at pcorp.us <mailto:lr at pcorp.us> ]
Sent: Saturday, March 18, 2023 9:29 AM
To: 'PostGIS Users Discussion' <postgis-users at lists.osgeo.org <mailto:postgis-users at lists.osgeo.org> >
Subject: RE: [postgis-users] Help to upgrade postgresql10 with postgis 2.5
Don’t see anything out of the ordinary.
You don’t have a postgresql.auto.conf file or is it empty? The custom settings might be in that one. The auto is set when you use
ALTER SYSTEM
and that would generally be copied over from your old instance.
Also are all your databases on PG12 and full as I would expect, since I think the failure happened after all the data was copied over already.
The fact vacuum(full, analyze) didn’t fail is good and perhaps suggests nothing to worry about.
Whatever killed vacuumdb sounds like another process that was running at the same time. The fact it happened even on the other server you are testing might be just something you consistently install. I would try the vacuum on each of your databases to see if any cause a crash.
From: postgis-users [mailto:postgis-users-bounces at lists.osgeo.org] On Behalf Of pham lan
Sent: Saturday, March 18, 2023 3:37 AM
To: PostGIS Users Discussion <postgis-users at lists.osgeo.org <mailto:postgis-users at lists.osgeo.org> >
Subject: Re: [postgis-users] Help to upgrade postgresql10 with postgis 2.5
Hi Regina,
The postgresql.conf and pg_hba.conf for postgres12 are the default one. I just changed the port to 5433 and all authentication to trust. Nothing special.
Please find in attachments those files.
Please find the output:
postgres at host:~> psql -p 5433 -d template1 -c "SELECT * FROM pg_extension;"
oid | extname | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition
-------+---------+----------+--------------+----------------+------------+-----------+--------------
14385 | plpgsql | 10 | 11 | f | 1.0 | |
(1 row)
0 postgres at host:~> free -h
total used free shared buff/cache available
Mem: 15Gi 610Mi 3.5Gi 13Mi 11Gi 14Gi
Swap: 2.0Gi 95Mi 1.9Gi
Let me know if you see something wrong.
Thanks,
Lan
On Sat, Mar 18, 2023 at 7:04 AM Regina Obe <lr at pcorp.us <mailto:lr at pcorp.us> > wrote:
I’ve never seen that error before.
It might be some OOM killer getting in the way as it’s failing on vacuuming of pg_catalog.pg_proc in template1, which is a really strange place to be failing since the template database have pretty much nothing, unless you installed some extension in them.
What does output – is it low?
free –h
It might be something in your postgresql.conf or postgresql.auto.conf like your maintenance_work_mem is set too high for your server.
Since you are doing non-link mode pg_upgrade, you should have 2 instances off pg now. I think if it got to the point of vacuumdb it’s already loaded all the data in the PG12 instance and you can check if everything is there.
Can you connect to your PostgreSQL 12, you might have to manually start it up
And then see if you can run below without it crashing. That is essentially what vacuumdb does:
psql -p 5433 -d template1 -c "vacuum (analyze,verbose);"
Also check what extensions you have installed in template1.
psql -p 5433 -d template1 -c "SELECT * FROM pg_extension;"
For template1, there is usually only one extension plpgsql.
If you have more, you might want to drop the others to rule out it’s some extension getting in the way.
From: postgis-users [mailto:postgis-users-bounces at lists.osgeo.org <mailto:postgis-users-bounces at lists.osgeo.org> ] On Behalf Of pham lan
Sent: Saturday, March 18, 2023 1:21 AM
To: PostGIS Users Discussion <postgis-users at lists.osgeo.org <mailto:postgis-users at lists.osgeo.org> >
Subject: Re: [postgis-users] Help to upgrade postgresql10 with postgis 2.5
Hi Regina,
So I tried to pg_upgrade at least from postgresql 10 (postgis 2.5) to postgresql 12 (postgis 2.5) and get the following error. Do you have any idea why? (I tried this several times, I even pg_basebackup to another server with postgresql 10 and pg_upgrade to postgres12 but got the same error)
> /usr/pgsql-12/bin/pg_upgrade -b /usr/pgsql-10/bin/ -B /usr/pgsql-12/bin/ -d /var/lib/pgsql/10/data/ -D /var/lib/pgsql/12/data/ -p 5432 -P 5433
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for tables WITH OIDS ok
Checking for invalid "sql_identifier" user columns ok
Creating dump of global objects ok
Creating dump of database schemas
ok
Checking for presence of required libraries ok
Checking database user is the install user ok
Checking for prepared transactions ok
Checking for new cluster tablespace directories ok
If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.
Performing Upgrade
------------------
Analyzing all rows in the new cluster
*failure*
Consult the last few lines of "pg_upgrade_utility.log" for
the probable cause of the failure.
Failure, exiting
1 postgres at host:~> cat pg_upgrade_utility.log
-----------------------------------------------------------------
pg_upgrade run on Sat Mar 18 06:16:00 2023
-----------------------------------------------------------------
command: "/usr/pgsql-12/bin/pg_dumpall" --host /var/lib/pgsql --port 5432 --username postgres --globals-only --quote-all-identifiers --binary-upgrade -f pg_upgrade_dump_globals.sql >> "pg_upgrade_utility.log" 2>&1
command: "/usr/pgsql-12/bin/vacuumdb" --host /var/lib/pgsql --port 5433 --username postgres --all --analyze >> "pg_upgrade_utility.log" 2>&1
vacuumdb: vacuuming database "postgres"
vacuumdb: vacuuming database "template1"
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
vacuumdb: error: vacuuming of table "pg_catalog.pg_proc" in database "template1" failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Thanks,
Lan
On Fri, Mar 17, 2023 at 10:27 AM pham lan <phamlankt at gmail.com <mailto:phamlankt at gmail.com> > wrote:
Yes, actually I followed your instructions.
After drop extension postgis_sfcgal whichs required me to drop cascade 13 materialized views, the upgrade to postgres15 went well.
Afterward, I wanted to update the postgis extension:
ALTER EXTENSION postgis UPDATE;
it requests me to drop cascade around 200 materialized views. For the test DB, I did that, the update to postgis 3.3 went well but afterward, the vacuum step failed.
In the end, I talked to the teams and they can not afford to have those materialized views deleted and then being created again by them.
That's why I wanna ask if there is a simple way of migrating the materialized views? I am not familiar with this.
Thanks,
Lan
On Fri, Mar 17, 2023 at 9:26 AM Regina Obe <lr at pcorp.us <mailto:lr at pcorp.us> > wrote:
You’d probably want to upgrade PostgreSQL too since PostgreSQL 10 is EOL’d.
Below are some instructions I had written for Centos. Should be more or less the same for Redhat.
https://www.bostongis.com/blog/index.php?/archives/278-Using-pg_upgrade-to-upgrade-PostgreSQL-9.6-PostGIS-2.4-to-PostgreSQL-15-3.3-on-Yum.html
From: postgis-users [mailto:postgis-users-bounces at lists.osgeo.org <mailto:postgis-users-bounces at lists.osgeo.org> ] On Behalf Of pham lan
Sent: Friday, March 17, 2023 3:56 AM
To: PostGIS Users Discussion <postgis-users at lists.osgeo.org <mailto:postgis-users at lists.osgeo.org> >
Subject: Re: [postgis-users] Help to upgrade postgresql10 with postgis 2.5
Hi Regina,
Thank you for your reply. Below is the output of the command. Do you know a good way to migrate all the definitions of materialized views because we have a lot of them and they are quite complex?
# SELECT postgis_full_version(), version();
postgis_full_version
| version
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------+----------------------------------------------------------------------------------------------------------
POSTGIS="2.5.5" [EXTENSION] PGSQL="100" GEOS="3.9.2-CAPI-1.14.3" SFCGAL="1.4.1" PROJ="Rel. 7.2.1, January 1st, 2021" GDAL="GDAL 3.2.3, released 2021/04/27" LIBXML="2.9.7" LIBJSON="0.13.1" LIBPROTOBUF="
1.3.0" TOPOLOGY RASTER | PostgreSQL 10.23 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15), 64-bit
Thanks,
Lan
On Thu, Mar 16, 2023 at 2:15 PM Regina Obe <lr at pcorp.us <mailto:lr at pcorp.us> > wrote:
It depends on what platform you are on.
Can you output the following from your postgis enabled databases:
SELECT postgis_full_version(), version();
And send us the output of what each says. That will give a clue of at least what platform you are running.
You’ll need to run this on each database you have with postgis installed on, as it is possible to have 2 versions of postgis installed in separate databases.
As to whether you’d need to drop and recreate any materialized views, it would depend on what functions they are using. If they are using deprecated or removed functions, then eventually you will need to drop and recreate, but you could do that at a later time. The PostGIS 3+ upgrade will notify you of those issues, but generally can just rename the functions in use, so you can drop and recreate at a more convenient time.
For materialized views that take a long time to build, I generally build them under a new name, and do a swap after the new one has been built. That would reduce the downtime from hours to 1-2 minutes.
Thanks,
Regina
From: postgis-users [mailto:postgis-users-bounces at lists.osgeo.org <mailto:postgis-users-bounces at lists.osgeo.org> ] On Behalf Of pham lan
Sent: Wednesday, March 15, 2023 5:00 PM
To: PostGIS Users Discussion <postgis-users at lists.osgeo.org <mailto:postgis-users at lists.osgeo.org> >
Subject: [postgis-users] Help to upgrade postgresql10 with postgis 2.5
Hello,
I have very less experience with postgres and postgis. However i receive a task to upgrade an old database which has postgresql10 with postgis 2.5. The database has some extensions: postgis, postgis_sfcgal, postgis_topology, raster and have a lot of materialized views which depends on functions on postgis and sfcgal libraries. Could someone please instruct me to upgrade my postgres DB to a newer postgres and postgis version without having to drop all those materialized views? Idealy to postgis 3.x?
Thanks in advance.
Best regards
Lan Pham
_______________________________________________
postgis-users mailing list
postgis-users at lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/postgis-users
_______________________________________________
postgis-users mailing list
postgis-users at lists.osgeo.org <mailto:postgis-users at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/postgis-users
_______________________________________________
postgis-users mailing list
postgis-users at lists.osgeo.org <mailto:postgis-users at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/postgis-users
_______________________________________________
postgis-users mailing list
postgis-users at lists.osgeo.org <mailto:postgis-users at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/postgis-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-users/attachments/20230323/ee615b36/attachment.htm>
More information about the postgis-users
mailing list