comments about meeting agenda

[Regina Obe]

> I typically don't attend this sort of thing, and I can't predict schedule.
I'm
> addressing the packager items in this note.
> 
> * splitting addresss_standardizer and postgis_tiger_geocoder
> 
> This is going to cause effort, basically creating two new packages.  If
they have
> either a normal build system (or the same as postgis or other pgsql
> extensions), that will probably not be that much effort.  It's not clear
if
> "postgis_tiger_geocoder" depends (at build time) on postgis, or just on
pgsql,
> and idk if there is prior art for extensions that depend on extensions,
and if
> not, if there are issues.
> 

There is prior art.

address_standardizer does not depend on PostGIS at all and uses the
PostgreSQL general build system.  So a bit simpler than postgis tool chain,
which was one reason to break it from postgis core.

postgis_tiger_geocoder is all plgsql code so it will build without postgis,
but the install-check will fail unless postgis is installed. 
I think this is pretty similar to other extensions that rely on postgis not
packaged with postgis  -- e.g. pgrouting
https://github.com/pgRouting/pgrouting 
Again postgis_tiger_geocoder used the PostgreSQL install-check build chain,
so the simplicity was reduced a bit by separating it out of postgis.

It has a  conditional dependence on address_standardizer.
The address_standardizer part is exercised in tests only if you choose it as
a dependency.  Ultimately whether you test with or without doesn't matter.
It doesn't impact the built package.

This conditional dependency on address_standardizer is similar to the
conditional dependency that https://github.com/pramsey/pgsql-ogr-fdw has on
postgis.

> Users are going to have to add the new packages because when they just
"apt
> upgrade" or "pkgin up", they are going to get new postgis which lacks the
split
> components.
> 
> We are going to have to sequence all this, so we go from
> 
>   - postgis with the extras in the package, to
> 
>   - postgis without
>   - addresss_standardizer 1.0
>   - postgis_tiger_geocoder 1.0
> 

Ah good point.  I completely forgot about the user experience.  So we should
note that in the PostGIS 3.7.0 release notes once we have an idea what
packagers will do.
A good portion of our users now are on DbaaS, so most likely assuming the
DbaaS packagers do their job right, this change should be transparent to
them because they'll only have to worry about  CREATE EXTENSION
address_standardizer;  etc which they had to do already anyway.

> This is a much bigger deal than a regular release.  IMHO it calls for
alpha/beta
> not just rc, and a substantial time (a month from alpha to release), more
than
> the normal week from rc to release, because it's not just doing a build
that is
> widely expected to have zero problems.
> 

I was debating that. Okay will change to have an alpha first.


> Note that I'm saying alpha, not "you can try the repo".  Packaging systems
> package releases, not git master branches, and thus the right thing to
work on
> and test is an alpha which is structurally like a release, in the same way
an rc
> is.  *What is being tested is the entire flow from master to binary
package,
> including the process of creating a release tarball.*
> 
> As always, rushing is not ok, so if the split misses the boat, so be it,
because
> there are no terrible problems and no urgency.
> 
Well I feel otherwise cause missing the boat means having an additional year
to support old versions.


> It's good to separate things if
> 
>   - They need to have separate release cycles.
> 
>   - They have separate schedules for ABI and API breaks (sql is ABI
>     because it's a runtime interface, not a compile-against .h
>     interface).  Packaging systems need to rebuild depending packages
>     when there are ABI breaks.  That's mostly routine, but it must be
>     clear when breaks happen, from NEWS.
> 
>   - They have different dependencies such that one might want to avoid a
>     secondary extension because it has a far bigger requirement
> 
> I looked at my package and for the to-be-split extensions I have an
> addresss_standardizer shlib (and now tiger geocoder shlib), a README, a
> bunch of sql file, and a few .control files.
> 
>   184140612	total bytes for postigs and *every pkg it depends on*
>    44275036	total bytes for postgis package
>     2504225	bytes for standardizer
>     2078946	bytes for tiger geocoder
> 
> So it's hard to get excited about 4.5 MB in the context of 184 MB, even
though
> I used Sixth Edition with dual RK05s, I think 5 MB total system storage.
> 
> I can see how people not dealing with US addresses would want to not
install
> these.  I can see how these are judged "no reason to be part of postgis".
>
Yah the main reason is not because of size, but because
a) the release cycle is very different - address_standardizer rarely changes
except for security fixes and postgis_tiger_geocoder - aside from security
fixes only changes around September when US Census comes out with a new data
set.
b) Both have much simpler tooling than postgis proper.  They both just
follow the more simplistic PostgreSQL install-make check
So this means we can probably ship these on pgxn as well since their
make/install tooling is much simpler.

c) the other reason is postgis_tiger_geocoder is US centric, so really
doesn't belong in a package designed for a global user base.
I expect in future we may have other geocoders that are for example
openstreetmap centric that you can use with just PostgreSQL (no middleware).
In the case of address_standardizer true it is pretty US but in theory with
additional data dictionaries it can handle other countries and there is an
address_standardizer-2 that Steve Woodbrdige was working on before he
retired that does have dictionaries for Germany and had much more
flexibility.  That one had a dependency on boost regex so I didn't want it
part of postgis. That one I may resuscitate and pull into PostGIS org.

> (The real benefit is splitting things with heavier or difficult
dependencie
> (sfcgal?), so that people can install postgis plain and later install
those.  I'm
> unclear on how that collides with reality (and that's ok).)
> 
They can do that already by just doing --without-sfcgal  and postgis_sfcgal
does have an intimate dependency with liblwgeom and rest of postgis geometry
structures and does closely follow the postgis release cycle, so I think
that is a much trickier proposition and less desirable to do.
We already did split sfcgal function from the postgis lib file since having
another extension postgis_sfcgal that shared the same postgis lib was
confusing and would break should you decide next time not to build with
postgis_sfcgal support.

It also made the set of live functions in our postgis lib file  be
consistent regardless if you choose to compile with or without sfcgal.

> All in all, it's not clear to me that the benefits are there, because I'm
not sure
> how much pain there is now.  I do agree that logically it would be a good
> move.  I do not agree that there is urgency, or any justification for
deviating
> from adequately long alpha cycles.
> 
I expect there will be slight pain in the beginning but I think in the long
run it's best for all extensions involved.

Both postgis_tiger_geocoder and address_standardizer do not have to suffer
the same more frequent release cycle.
Also since their dependencies are much simpler, it may encourage others to
contribute who wouldn't have bothered because postgis repo is such an
intimidating beast.


> The first step is to get alpha releases for the split repos, probably
leaving
> postgis master alone (with an unmerged branch to remove them).
We already removed these from master and we aren't turning back on that.

Yes we can have an alpha I'm in agreement on that now I've heard your
concerns.

> However, I see they are gone from master for months, and I haven't seen
> announcements of new repos, and ther are definitely no releases or alphas.
> 
I recall we made mention of this on postgis-devel a couple of times and Paul
did just last week.
I don't expect packagers to pay much attention to this especially until we
release.

> There are no releases at
>   https://github.com/postgis/address_standardizer

> 
> There are apparently no releases at
> 
>   https://gitea.osgeo.org/postgis/postgis_tiger_geocoder
>   (and it looks like pgsql 16+, vs 14+ for postgis)
> 
Exactly postgis_tiger_geocoder requires 16+ because it's using functionality
I authored for PostgreSQL 16 and above to allow schema qualification of
dependent extensions -
https://www.postgresql.org/message-id/flat/20230118214223.2crj7hh3i5amzkji%4
0c19#c4f1683db08df216eedb91ad9de2b12b

I debated supporting 15 but it will make the build slightly more complicated
since I'd have to put in conditionals.

PG14 is going to be EOL soon so there is no point supporting it.  I also
figure most packagers aren't going to be building 3.7 for PostgreSQL 15 that
they'll just stick with 3.6 or lower.

I'm planning to release a postgis_tiger_geocoder this month and I believe
Paul is planning to release address_standardizer as well this  month,
probably a little before we do the PostGIS 3.7.0alpha1.


> If you're not ok with getting alphas out and having 30 days from that to
release
> of new postgis, I think the removal from master should be reverted, and
then
> alphas and then re-remove from master once the alphas have survived
> testing.
I expect PostgreSQL 19 to come out in September, so I think we can do the
30day thing.  Perhaps slightly shorter.

> 
> * h3-pg
> 
> I was unaware of this, and as far as I can tell it's just another library
one could
> use (and thus one could package), but I don't see how it relates to
postgis or
> postgis releases.

The only difference is that h3-pg moved from the original maintainer to
postgis ORG on github.  The original maintainer didn't have time to work on
it anymore and since it's a fairly popular extension, we adopted it.

It's much the same story why https://github.com/postgis/docker-postgis lives
under the PostGIS org.  We adopted it from the original maintainer.

Thanks for your input,
Regina