[Qgis-developer] Error connecting QGIS 1.9.0 to Ortho Refresh WCS
Bruce, Bob (CON)
Bob.Bruce at gov.mb.ca
Wed Jan 23 13:19:52 PST 2013
Radim,
Would you be willing to add an option to "Disable max extent verification" - I believe that would solve
all of our problems with connecting to this very large image store. If you do so then please let me know and I will
be happy to test it.
Thanks,
Bob Bruce
-----Original Message-----
From: qgis-developer-bounces at lists.osgeo.org [mailto:qgis-developer-bounces at lists.osgeo.org] On Behalf Of Radim Blazek
Sent: January-23-13 3:07 AM
To: ekeighan at cubewerx.com
Cc: Qgis-Developer; Bob Bruce
Subject: Re: [Qgis-developer] Error connecting QGIS 1.9.0 to Ortho Refresh WCS
On Tue, Jan 22, 2013 at 5:48 PM, ekeighan <ekeighan at cubewerx.com> wrote:
> Radim,
>
> Our WCS was designed to be used as per the intent of the OGC WCS
> specification for extracting data, not for browsing it. The OGC WMS or WMTS
> is intended to be used for browsing.
Just for curiosity, can you point me to an OGC document supporting
this interpretation? In any case, my aim is not to defend how QGIS
works pointing to WCS specification but to get it working in most
cases. If you insist on that interpretation we have to take it as
reality.
> But we admit that the WCS spec is deficient in that it does not describe the
> raster-data type sufficiently nor in a standard way. E.g., it doesn't tell
> you that the raster data has five channels of 16-bit unsigned integers
> representing Red, Green, Blue, Infrared and Alpha channels. This is why WCS
> client applications tend to request a sample image.
Indeed, also no data values ...
> GDAL does this in a more friendly way by requesting a small 2x2-pixel sample
> image at the resolution of the source data. This causes no problems for a
> WCS organized to extract data at the source resolution. We suggest that QGIS
> use the same method to avoid unnecessary problems with our and other
> servers.
QGIS does something similar, i.e. 6x3 pixels on original resolution.
As I already wrote twice, the 10x10 pixels request for the whole
extent is done to verify a real extent of the coverage. Some servers
may report wrong larger extent in capabilities and requests outside
the real extent fail. If I disable this additional check, we get it
working with CubeWerx, but break it for other servers and that is not
the way ahead.
> Our WCS is organized to always access the raw source data in order to
> produce the best image possible for analysis. However, this approach breaks
> down catastrophically in the QGIS case where your request translates into
> accessing tens of thousands of source images covering multiple terabytes of
> data that needs to be processed to fulfill the 10x10 request. Many
> developers are familiar with this particularity of the WCS (for data
> extraction at source resolution) and use it in tandem with a WMS or WMTS for
> views and only access the WCS when a download at source resolution is
> required, typically for analysis.
This interpretation of the WCS standard degrades it to a richer
download application. I believe that WCS has much more potential.
Maybe the possibilities shifted a bit during the years with broadband
connections and mobile applications?
> In order to support a more flexible access to the WCS we were thinking of
> modifying our access method when zoomed out sufficiently. Source-overview
> files and tile data are available which make WMS and WMTS access very
> efficient. CubeWerx have these products and our clients are of course
> deploying them in addition to a WCS but there can be a problem with both
> approaches in that the secondary data may not match the exact raster data
> type of the source data. E.g., the tiles are rendered for common maps which
> use 3-channel 8-bit RGB, but the source data might be 4-channel 16-bit
> RGB+Infrared.
But it should not be a problem (apart computational time) to create
lower resolution overviews also for WCS with exactly the same data
types and bands as original data. Is it?
As I said, I would like to fix QGIS to work smoothly also with your
services. An easy workaround is to allow the max extent check to fail
but it will still wait for server timeout. Well, if the server does
not support full extent, it could return WCS error response
immediately instead of timeout error.
Another possibility would be to add an option "Disable max extent
verification" but that is user unfriendly.
Radim
> Regards,
>
> Edric Keighan
> President & CEO
> CubeWerx Inc.
>
> Tel: 613-447-8306 Mobile
>
>
> On 22/01/2013 3:43 AM, Radim Blazek wrote:
>>
>> On Sun, Jan 20, 2013 at 5:14 AM, Bob Bruce <bob.bruce at shaw.ca> wrote:
>>>
>>> Radim,
>>> This dataset is a huge collection of imagery encompassing an
>>> area approximately 350 kilometres square. There are over 5000 images in the
>>> dataset, with each image having a .5m pixel size and a file size of 293Mb
>>> and I doubt that it is feasible to request a small sample image from the
>>> entire range of the imagery. Is there some way that we can get QGIS to
>>> request an image from a much smaller area (like the extent of the viewport)
>>> to establish the validity of the dataset?
>>
>> Currently not. It would be necessary to disable the check for max
>> extent. I am willing to add workarounds for various server
>> particularities (the check for true max extent is one of them) but in
>> this case I am not quite convinced. It is obviously server problem.
>> The server offers capabilities which it cannot fulfill.
>>
>> I'll consider to allow to pass the extent check without error.
>> However, the request will be done and it will wait timeout until it
>> fails.
>>
>> I think that GDAL does not do that check, so you can create GDAL WCS
>> XML file and open it as raster file.
>>
>> Radim
>>
>>> Ultimately we want to work with a dataset of .2m sized pixels and
>>> individual images of 2.94Gb and four bands.
>>>
>>> Thanks for your help,
>>>
>>> Bob Bruce
>>>
>>> Sent from my iPhone
>>>
>>> On 2013-01-18, at 10:08 PM, Radim Blazek <radim.blazek at gmail.com> wrote:
>>>
>>> Frank,
>>>
>>> here it is QGIS doing the request 10x10. Not sure about GDAL but IIRC,
>>> in QGIS it is used to get true extent. It may happen that a server is
>>> misconfigured and reports larger extent in
>>> GetCapabilities/DescribeCoverage and successive request inside that
>>> extent but outside the real coverage extent would fail. Requesting the
>>> whole extent and getting
>>> its real extent is the only way AFAIK to get real extent.
>>>
>>> Radim
>>>
>>> On Sat, Jan 19, 2013 at 2:23 AM, Frank Warmerdam <warmerdam at pobox.com>
>>> wrote:
>>>>
>>>> Bruce,
>>>>
>>>> I think the 10x10 request is being made by GDAL's WCS driver in order
>>>> to establish some details about the underlying WCS and it defaults to
>>>> the whole extents. It sucks fairly badly that such a request can't be
>>>> done. It might be possible to dummy in the information in the .wcs
>>>> config file to avoid having to do that query. Investigating.
>>>>
>>>> Best regards,
>>>> Frank
>>>>
>>>>
>>>>
>>>> On Fri, Jan 18, 2013 at 4:11 PM, Bruce, Bob (CON) <Bob.Bruce at gov.mb.ca>
>>>> wrote:
>>>>>
>>>>> Radim,
>>>>> I have been looking at this again with two people from Cubewerx
>>>>> and we find that QGIS is issuing this request:
>>>>>
>>>>>
>>>>> http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG=OIM_Ortho_Refresh_2010&DATASTORE=OIM_Ortho_Refresh_2010_store&&SERVICE=WCS&VERSION=1.0.0&REQUEST=GetCoverage&FORMAT=GeoTIFF&COVERAGE=image_set&BBOX=315000.2500000000000000,5425000.2500000000000000,779999.7500000000000000,5804999.7500000000000000&CRS=EPSG:26914&RESPONSE_CRS=EPSG:26914&WIDTH=10&HEIGHT=10
>>>>>
>>>>> while the bounding box that I currently have displayed in QGIS is
>>>>> roughly:
>>>>>
>>>>>
>>>>> BBOX=632450.0000000000000000,5527900.0000000000000000,634150.0000000000000000,5529000.000000000000
>>>>>
>>>>> I need to get QGIS to restrict its request to this extent to work. Do
>>>>> you know how I can do that?
>>>>>
>>>>> If I issue this request from my web browser:
>>>>>
>>>>>
>>>>> http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG=OIM_Ortho_Refresh_2010&DATASTORE=OIM_Ortho_Refresh_2010_store&&SERVICE=WCS&VERSION=1.0.0&REQUEST=GetCoverage&FORMAT=GeoTIFF&COVERAGE=image_set&BBOX=632450.0000000000000000,5527900.0000000000000000,634150.0000000000000000,5529000.000000000000&CRS=EPSG:26914&RESPONSE_CRS=EPSG:26914&WIDTH=1700&HEIGHT=1100
>>>>>
>>>>> I actually get a tiff image that I can view that looks good and is
>>>>> similar to what I would expect to see in QGIS. Now I need to get QGIS to
>>>>> actually issue that request.
>>>>>
>>>>> Thanks,
>>>>> Bob Bruce
>>>>>
>>>>> -----Original Message-----
>>>>> From: Radim Blazek [mailto:radim.blazek at gmail.com]
>>>>> Sent: January-18-13 3:36 AM
>>>>> To: Bruce, Bob (CON)
>>>>> Cc: Qgis-Developer; ekeighan at cubewerx.com
>>>>> Subject: Re: [Qgis-developer] Error connecting QGIS 1.9.0 to Ortho
>>>>> Refresh WCS
>>>>>
>>>>> Hi again,
>>>>>
>>>>> On Fri, Jan 18, 2013 at 10:28 AM, Radim Blazek <radim.blazek at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> I don't see the same problem here. GetCapabilities and
>>>>>> DescribeCoverage are not mixed together here. Could it be that you
>>>>>> accidentally used GetCapabilities request URL as "URL" parameter in
>>>>>> connection details dialog and it was logged on server but that is
>>>>>> another issue?
>>>>>>
>>>>>> I have got another error however. DescribeCoverage is successfully
>>>>>> received and parsed, then QGIS sends test GetCoverage request (10x10
>>>>>> raster to get more info):
>>>>>>
>>>>>>
>>>>>> http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG=OIM_Ortho_Refresh_2010&DATASTORE=OIM_Ortho_Refresh_2010_store&&SERVICE=WCS&VERSION=1.0.0&REQUEST=GetCoverage&FORMAT=GeoTIFF&COVERAGE=image_set&BBOX=-101.9598305476503981,48.7910076027865429,-94.6376374194891667,52.5230584889113814&CRS=EPSG:4326&RESPONSE_CRS=EPSG:4326&WIDTH=10&HEIGHT=10
>>>>>>
>>>>>> and it fails on server side with:
>>>>>>
>>>>>> "Gateway Time-out
>>>>>> The gateway did not receive a timely response from the upstream server
>>>>>> or application."
>>>>>>
>>>>>> Because the request covers the whole extent, it could be that the
>>>>>> raster is quite large and there are no overviews/pyramids ready on
>>>>>> server, it starts to do resampling of the whole extent but it takes to
>>>>>> much time?
>>>>>
>>>>> It really seems to be the problem, I have tried with smaller extent
>>>>> and server sent a correct TIFF in response. In any case it takes very
>>>>> long time.
>>>>>
>>>>> Radim
>>>>>
>>>>>> BTW, you can find some interesting info in QGIS Settings > Panels >
>>>>>> Log Messages - WCS tab or in debug messages. In log messages are
>>>>>> usually printed requests which failed and in debug messages are
>>>>>> printed all requests.
>>>>>>
>>>>>> Radim
>>>>>>
>>>>>> On Thu, Jan 17, 2013 at 11:07 PM, Bruce, Bob (CON)
>>>>>> <Bob.Bruce at gov.mb.ca> wrote:
>>>>>>>
>>>>>>> I am having trouble using the WCS capability in version 1.9.0 code
>>>>>>> revision f210668
>>>>>>> Here are the full details of the QGIS version from the Help listing:
>>>>>>>
>>>>>>> QGIS version 1.9.0-Master QGIS code revision
>>>>>>> f210668
>>>>>>> Compiled against Qt 4.7.1 Running
>>>>>>> against Qt 4.7.1
>>>>>>> Compiled against GDAL/OGR 1.9.2 Running
>>>>>>> against GDAL/OGR 1.9.2
>>>>>>> GEOS Version 3.3.5 PostgreSQL
>>>>>>> Client Version 8.3.10
>>>>>>> SpatiaLite Version 3.0.1 QWT Version
>>>>>>> 5.2.1
>>>>>>> PROJ.4 Version 480 QScintilla2
>>>>>>> Version 2.6.2
>>>>>>> This copy of QGIS writes debugging output.
>>>>>>>
>>>>>>> The URL that I am connecting to is:
>>>>>>> http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG=OIM_Ortho_Refresh_2010&DATASTORE=OIM_Ortho_Refresh_2010_store
>>>>>>> The URL of the GetCapabilities request is:
>>>>>>> http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG=OIM_Ortho_Refresh_2010&VERSION=1.0.0&SERVICE=WCS&DATASTORE=OIM_Ortho_Refresh_2010_store&REQUEST=GetCapabilities
>>>>>>>
>>>>>>> According to Cubewerx, who is our imagery OGC WCS service provider,
>>>>>>> their log shows both the GetCapabilities and GetCoverage requests present in
>>>>>>> the same HTTP request while they should be done separately.
>>>>>>>
>>>>>>> This is the request that they are finding in their server log:
>>>>>>> 2012-11-15 19:06:00.980 10527 received HTTP GET request from
>>>>>>> 205.200.189.2 to URL
>>>>>>> "http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG=OIM_Ortho_Refresh_2010&DATASTORE=OIM_Ortho_Refresh_2010_store&service=wcs&version=1.1.1&REQUEST=DescribeCoverage&COVERAGE=OIM_Ortho_Refresh_2010_store&SERVICE=WCS&REQUEST=GetCapabilities&AcceptVersions=1.1.0,1.0.0"
>>>>>>>
>>>>>>> Is there a fix to this problem available in a later version of QGIS?
>>>>>>>
>>>>>>> Here is the error report that I am seeing in QGIS when I try and add
>>>>>>> this data source to the project:
>>>>>>> WCS provider: Cannot calculate extent
>>>>>>> (d:\src\qgis\src\providers\wcs\qgswcsprovider.cpp : 188 :
>>>>>>> QgsWcsProvider::QgsWcsProvider)
>>>>>>> Raster layer: Provider is not valid (provider: wcs, URI:
>>>>>>> cache=AlwaysCache&crs=EPSG:26914&format=GeoTIFF&identifier=image_set&url=http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG%
>>>>>>> 3DOIM_Ortho_Refresh_2010%26DATASTORE%3DOIM_Ortho_Refresh_2010_store
>>>>>>> (d:\src\qgis\src\core\raster\qgsrasterlayer.cpp : 1609 :
>>>>>>> QgsRasterLayer::setDataProvider)
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bob Bruce
>>>>>>>
>>>>>>> From: ekeighan [mailto:ekeighan at cubewerx.com]
>>>>>>> Sent: January-17-13 3:33 PM
>>>>>>> To: Bruce, Bob (CON)
>>>>>>> Subject: Re: Error connecting QGIS 1.9.0 to Ortho Refresh WCS
>>>>>>>
>>>>>>> Bob,
>>>>>>>
>>>>>>> FYI...
>>>>>>>
>>>>>>> ekeighan at cubewerx.com wrote:
>>>>>>>
>>>>>>> *Raster layer:* Provider is not valid (provider: wcs, URI:
>>>>>>>
>>>>>>> cache=AlwaysCache&crs=EPSG:26914&format=GeoTIFF&identifier=image_set&url=http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG%3DOIM_Ortho_Refresh_2010%26DATASTORE%3DOIM_Ortho_Refresh_2010_store
>>>>>>> (d:\src\qgis\src\core\raster\qgsrasterlayer.cpp : 1609 :
>>>>>>> QgsRasterLayer::setDataProvider)
>>>>>>>
>>>>>>> Is there something at your end that could be changed to allow this to
>>>>>>> calculate the extent?
>>>>>>> Looking through the server log, we see only the following related
>>>>>>> request
>>>>>>> present twice:
>>>>>>>
>>>>>>> 2012-11-15 19:06:00.980 10527 received HTTP GET request from
>>>>>>> 205.200.189.2 to URL
>>>>>>> "http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG=OIM_Ortho_Refresh_2010&DATASTORE=OIM_Ortho_Refresh_2010_store&service=wcs&version=1.1.1&REQUEST=DescribeCoverage&COVERAGE=OIM_Ortho_Refresh_2010_store&SERVICE=WCS&REQUEST=GetCapabilities&AcceptVersions=1.1.0,1.0.0"
>>>>>>>
>>>>>>> ---------------------------------------------------------------
>>>>>>>
>>>>>>> This doesn't make any sense, since it contains two different REQUEST
>>>>>>> arguments, one for DescribeCoverage and another for GetCapabilities.
>>>>>>> The Qgs error suggests that the base URL it's using is:
>>>>>>>
>>>>>>>
>>>>>>> http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG%3DOIM_Ortho_Refresh_2010%26DATASTORE%3DOIM_Ortho_Refresh_2010_store
>>>>>>>
>>>>>>> but it almost seems as though it was entered improperly as:
>>>>>>>
>>>>>>>
>>>>>>> http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG=OIM_Ortho_Refresh_2010&DATASTORE=OIM_Ortho_Refresh_2010_store&service=wcs&version=1.1.1&REQUEST=DescribeCoverage&COVERAGE=OIM_Ortho_Refresh_2010_store
>>>>>>>
>>>>>>> The WCS request received from the client produces an error return,
>>>>>>> since
>>>>>>> the server has problems parsing the broken request:
>>>>>>>
>>>>>>> <ExceptionReport version="1.1.0" xml:lang="en"
>>>>>>> xsi:schemaLocation="http://www.opengis.net/ows/1.1
>>>>>>> http://schemas.cubewerx.com/schemas/ows/1.1.0/owsExceptionReport.xsd">
>>>>>>> <Exception exceptionCode="MissingParameterValue"
>>>>>>> locator="identifiers">
>>>>>>> <ExceptionText>CubeSERV: Cannot execute WCS request, raised in
>>>>>>> Wcs_Execute() in file "wcs.c" line 162</ExceptionText>
>>>>>>> <ExceptionText>CubeSERV: Missing parameter "identifiers", raised
>>>>>>> in throwMissingParamError() in file "cw_cgiparams.c" line
>>>>>>> 797</ExceptionText>
>>>>>>> </Exception>
>>>>>>> </ExceptionReport>
>>>>>>>
>>>>>>> The first request from the client should be just a proper
>>>>>>> GetCapabilities
>>>>>>> request:
>>>>>>>
>>>>>>>
>>>>>>> http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG=OIM_Ortho_Refresh_2010&DATASTORE=OIM_Ortho_Refresh_2010_store&SERVICE=WCS&REQUEST=GetCapabilities&AcceptVersions=1.1.0,1.0.0
>>>>>>>
>>>>>>> which returns the expected data.
>>>>>>> -------------------------------------------------
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Edric
>>>>>>>
>>>>>>> On 17/01/2013 3:35 PM, Bruce, Bob (CON) wrote:
>>>>>>> Edric,
>>>>>>> I took another look at connecting QGIS to your WCS, using
>>>>>>> a newer beta test version. It provided a mechanism to connect to the WCS and
>>>>>>> it allowed me to try and add it to my map. When I tried to add it I got the
>>>>>>> following message:
>>>>>>>
>>>>>>> WCS provider: Cannot calculate extent
>>>>>>> (d:\src\qgis\src\providers\wcs\qgswcsprovider.cpp : 188 :
>>>>>>> QgsWcsProvider::QgsWcsProvider)
>>>>>>> Raster layer: Provider is not valid (provider: wcs, URI:
>>>>>>> cache=AlwaysCache&crs=EPSG:26914&format=GeoTIFF&identifier=image_set&url=http://nis.cubewerx.com/cubewerx/cubeserv.cgi?CONFIG%3DOIM_Ortho_Refresh_2010%26DATASTORE%3DOIM_Ortho_Refresh_2010_store
>>>>>>> (d:\src\qgis\src\core\raster\qgsrasterlayer.cpp : 1609 :
>>>>>>> QgsRasterLayer::setDataProvider)
>>>>>>>
>>>>>>> Is there something at your end that could be changed to allow this to
>>>>>>> calculate the extent?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bob
>>>>>>>
>>>>>>>
>>>>>>> *******************************************************************************************************************
>>>>>>> ** Bob Bruce, FEC, P.Eng. Geomatics Support Engineer
>>>>>>> ** Bob.Bruce at gov.mb.ca Geospatial Technology Services
>>>>>>> Management
>>>>>>> ** work # (204) 945-6636 GeoManitoba, Manitoba
>>>>>>> Conservation and Water Stewardship
>>>>>>> ** FAX # (204) 945-1365 1007 Century Street,
>>>>>>> ** Winnipeg,
>>>>>>> Manitoba, Canada, R3H 0W4
>>>>>>> **
>>>>>>> 49-54-31.85N,97-12-16.08W
>>>>>>> **
>>>>>>> ** Check out our digital maps at: http://mli2.gov.mb.ca/
>>>>>>> ** and WMS: http://mlidata.gov.mb.ca/wms/request.aspx
>>>>>>>
>>>>>>> *******************************************************************************************************************
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Qgis-developer mailing list
>>>>>>> Qgis-developer at lists.osgeo.org
>>>>>>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>>>>>
>>>>> _______________________________________________
>>>>> Qgis-developer mailing list
>>>>> Qgis-developer at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> ---------------------------------------+--------------------------------------
>>>> I set the clouds in motion - turn up | Frank Warmerdam,
>>>> warmerdam at pobox.com
>>>> light and sound - activate the windows | http://pobox.com/~warmerdam
>>>> and watch the world go round - Rush | Geospatial Software Developer
>>>
>>> _______________________________________________
>>> Qgis-developer mailing list
>>> Qgis-developer at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
_______________________________________________
Qgis-developer mailing list
Qgis-developer at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer
More information about the Qgis-developer
mailing list