[pycsw-devel] MS4W installer containing pycsw-dev for Windows users

[Jeff McKenna]

Hi Tom, thanks for the responses, comments inline below:

On 2019-01-21 5:07 PM, Tom Kralidis wrote:
> Hi Jeff: comments interleaved:
> On Mon, Jan 21, 2019 at 10:43 AM Jeff McKenna
> <jmckenna at gatewaygeomatics.com> wrote:
>>
>> Thanks for your support on this Tom, comments inline below:
>>
>>
>> On 2019-01-19 3:27 PM, Tom Kralidis wrote:
>>> On Wed, Jan 16, 2019 at 1:48 PM Jeff McKenna
>>> <jmckenna at gatewaygeomatics.com> wrote:
>>>>
>>>> Hi pycsw community!
>>>>
>>>> Short summary
>>>> -------------
>>>>
>>>> Windows users might be interested that MS4W (the popular installer for
>>>> MapServer https://ms4w.com) is almost ready to be released, with pycsw
>>>> running (!!!).  It will be named "MS4W 4.0", and there is now an alpha
>>>> package containing pycsw ready to test at
>>>> https://ms4w.com/release/experimental/ms4w_4.0.0_alpha19.zip  (not
>>>> familiar with running MS4W or how to install it?  You can wait for the
>>>> setup.exe installer which will come shortly and install pycsw, Apache,
>>>> for you)
>>>>
>>>
>>> Wow this is great news/work Jeff!  Thanks for this valuable addition
>>> to ms4w.
>>
>> Thanks!!!
>>
>>>
>>>> Longer explanation
>>>> ------------------
>>>>
>>>> I've been working very hard on a big MS4W release, containing PHP 7 and
>>>> Python, Java, CSharp mapscripts, and also its own included Python
>>>> instance (hundreds of libraries in total, all compiled from source with
>>>> Visual Studio 2017).  I've also now included pycsw (master), running on
>>>> Python 3.7.0  (some of you devs are definitely cringing already I bet,
>>>> seeing that version, but please hear me out)
>>>>
>>>> Note that such an installer, for PHP7 and Python, GDAL, MapServer,
>>>> Apache, mod_wsgi (pycsw is running through mod_wsgi) demands a recent
>>>> compiler that can handle building all of that together, which is why
>>>> Visual Studio 2017 was chosen.
>>>>
>>>> Of course, to install pycsw I realize the hardcoded requirements (in
>>>> requirements.txt) for lxml, pyproj, Shapely, but to make all of this
>>>> work in the new 2017 compiler, I've used newer library versions for
>>>> those (lxml 4.3.0, pyproj 1.9.6, Shapely 1.6.4.post2).
>>>>
>>>
>>> Are the newer versions the latest stable versions?  I'm also wondering
>>> whether we should review our current requirements/dependencies (it's
>>> been awhile) and whether we should add 3.7 to our Travis CI setup.
>>
>> Yes those are the latest stable versions for lxml, pyproj, Shapely.  It
>> would be great if you could add those (and Python 3.7) to your TravisCI
>> setup.
>>
>>>
>>>> The only change I had to make was to use "WKTReadingError" in
>>>> pycsw/core/repository.py (line#41), because I am using a much newer
>>>> version of Shapely and GEOS.  For the record, I couldn't find anyone
>>>> else reporting that change.
>>>>
>>>
>>> I've filed an issue in https://github.com/geopython/pycsw/issues/584)
>>> to safeguard
>>> across Shapely versions.
>>
>> Great, I have expanded the explanation there (with exact error and
>> changes made).
>>
>>>
>>>> I guess my question is that if anyone sees any show stoppers for using
>>>> these updated library versions.  I know, it is not recommended, but, out
>>>> of this is an actual Windows installer for the many Windows users (MS4W
>>>> gets between 4,000 to 6,000 installs a month, I monitor it closely).
>>>>
>>>
>>> IHMO I don't see any issues.  If we do find issues we are committed to
>>> fixing them accordingly.
>>
>> Ok thanks, this is good to know.
>>
>>>
>>>> The Alpha now includes a running catalogue service with just the SQLite
>>>> cite.db records, but I am sure we can expand that later (note that I
>>>> discovered that most if not all records as part of the 'gisdata' package
>>>> are outdated/broken, which you can see for yourself in the pycsw viewer
>>>> at http://demo.pycsw.org/viewer/ (that viewer leverages the 'gisdata'
>>>> package for its records)
>>>>
>>>
>>> The view also has a 'service' endpoint which has a selected set of
>>> harvested OGC web services.  I've filed an issue in
>>> https://github.com/geopython/demo.pycsw.org/issues/2
>>> to update the endpoints on demo.pycsw.org
>>
>> Ok thanks.  I think having some working sample services is very useful
>> to new users.
>>
>>>
>>>> It would be great to have a lighter-weight viewer to demonstrate the
>>>> power of pycsw for MS4W users (I realize pycsw is headless, but it is
>>>> nice for new users to see it in action).  For sure I will push MS4W
>>>> users to test their CSW services with the QGIS MetaSearch plugin (great
>>>> work thanks!).
>>>>
>>>
>>> Can we start with the viewer on demo.pycsw.org? We can definitely
>>> implement updates to enhance its useability.
>>
>> I honestly originally had more written here about the existing viewer,
>> but deleted it.
>>
>> The issue is how heavy it is (requires GeoExt, OL, leaflet, gxp, ...).
>>
> 
> Note we carry 2 viewers at demo.pycsw.org:
> 
> 1. https://github.com/geopython/demo.pycsw.org/blob/master/www/viewer/index.html
> (Leaflet, jQuery, bootstrap)
> 2. https://github.com/geopython/demo.pycsw.org/blob/master/www/viewer/index2.html
> (GeoExt, gxp, etc.)
> 
> Viewer 1 is a lightweight approach (we can't do much without
> Leaflet/jQuery/bootstrap) and
> is infact a single page of HTML/JavaScript. With the dependencies the
> whole thing is around
> 400KB if you decide to store dependencies locally.

ah, perfect, thanks for this explanation, I must have installed the 
second one only.  The first will be perfect.  thanks for this!

> 
>> But, 'beggars cannot be choosers' so I deleted my email text on it.
>>
>> It's about 200MB, and I was not sure to include it inside MS4W, without
>> confirmation from pycsw devs that the source of demo.pycsw.org will be
>> actively maintained (it hasn't been touched in several years, and
>> leverages broken data services).  So you can see my concern as a packager.
>>
>> I didn't want to appear here as complaining, so I deleted my email text
>> before.  But it is just about my concern as a packager (of MS4W): should
>> the demo be distributed (even though it is based on old software
>> versions, many old software packages, and is quite heavy/lots of
>> packages and space required, and points to broken data).  As a packager,
>> it is a lot to commit to, before adding it to MS4W.
>>
>> I feel that more important is having about 10 working catalogue services
>> in a maintained endpoint, that is easy for new pycsw users to get (how
>> about inside the pycsw source at /pycsw/data/sample.db ?), so then new
>> pycsw users can goto something like the QGIS metasearch plugin and
>> access their new pycsw service easily.
>>
> 
> Per https://github.com/geopython/demo.pycsw.org/issues/2 can you suggest
> some WMS/WFS/WCS endpoints we could use?

willdo.  Will come with some and share back here/there.

> 
>> Thoughts?
>>
>>>
>>>> Well, I hope this news is accepted well, even if it pushes the limits of
>>>> 'recommended/required' software versions.  I hope this big effort helps
>>>> the whole pycsw community.
>>>>
>>>

thanks again!

-jeff