[QGIS-Developer] GeoSeer ogc services data harvesting

[Andreas Neumann]

Hi Jonathan, 

Thanks for sharing this information. I don't know anything better. 

While looking at some services that I know personally, I also found out
that others services are listed twice, because a machine might have a
DNS alias. That is also something to consider - perhaps sort out
machines that have identical GetCapabilities responses and just the DNS
name varies. 

I agree, the numbers probably wouldn't change significantly. 

Thanks and greetings, 

Andreas 

On 2020-06-09 13:14, Jonathan Moules wrote:

> Hi Andreas,
> Sure, happy to share.
> There's a little on the About page: https://www.geoseer.net/about.php and then scattered around blog posts (the ones with the "GeoSeer" tag are probably best for that: https://www.geoseer.net/blog/?t=GeoSeer ), but put simply - We scrape a lot of different sources and metadata catalogs and get the services from them. Then we request not only the GetCapabilities that was declared, but also make educated guesses as to what else might be on the box and request those too.
> 
> It's not perfect, but to the best of my knowledge it's by far the largest such index in the world, and more importantly, it's *current*. Everything in there responded with a valid GetCapabilities document with at least one meaningful named dataset when it was last scraped within the last few weeks.
> 
> Pertaining to your given services, GeoSeer has:
> http://geoweb.so.ch/wms/sogis_natgef.wms? and a few others on that sub-domain, as well as some on the subdomain: http://www.sogis1.so.ch/cgi-bin/sogis/sogis_natgef.wms? - both are now defunct I see which is why they're not in the database.
> 
> Thanks for the URL, I've added it for scraping.
> 
>> So I wonder how many other QGIS server installations may not be in your database?
> Alas that's a "unknown unknown"; there's no way to know (I can't think of a way to find out anyway; suggestions welcome). However the vast majority of the time when I come across a new service manually (i.e. from following various mailing lists like this), it turns out it's already in the index, so I think it's reasonably comprehensive at this point.
> 
> While missing servers may change the absolute number of QGIS Installations, they're very unlikely to change the proportions. For a sample-size this large I'd expect the proportions to remain largely the same, certainly for deployments.
> 
> Hope that's of interest and answers the question,
> Cheers,
> Jonathan
> 
> On 2020-06-09 10:45, Andreas Neumann wrote: 
> 
>> Hi Jonathan,
>> 
>> Can you share with us how you harvest your information on available public OGC services? You probably have that information published somewhere - so if you could point me towards this URL, it would help.
>> 
>> I noticed that all of the services of our province (my employer) can't be found, as an example.
>> 
>> Here is the start point:
>> 
>> https://so.ch/verwaltung/bau-und-justizdepartement/amt-fuer-geoinformation/geoportal/geodienste/wms-web-map-service/
>> 
>> and the GetCapabilities link:
>> 
>> https://geo.so.ch/api/wms?SERVICE=WMS&REQUEST=GetCapabilities&VERSION=1.3.0
>> 
>> So I wonder how many other QGIS server installations may not be in your database? Of course I know you don't claim full coverage, but it would still be good to know how you harvest your data.
>> 
>> Thanks for clarifying and greetings,
>> 
>> Andreas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20200609/ffa6b411/attachment-0001.html>