[pycsw-devel] SOS Harvesting Error

[dan at inlet.geol.sc.edu]

Tom,

I'll do that. It turns out to be the first result from NDBC which is
something like All Stations and is the only entry that has an entry for
operateson. It has every station listed in the form of station-<name>, so
it is a large entry.
I'll enter a bug entry tomorrow.


Dan
> Dan: thanks for the info.  I'm guessing the index value is so big it's
> causing a PostgreSQL.  If you comment out
> https://github.com/geopython/pycsw/blob/master/pycsw/metadata.py#L678,
> this will suppress the problem for the time being.
>
> Can you open a GitHub issue
> (https://github.com/geopython/pycsw/issues/new)? There are many big
> SOS endpoints out there, and I think it would be valuable to assess
> this more closely.
>
> On Mon, Oct 27, 2014 at 12:48 PM,  <dan at inlet.geol.sc.edu> wrote:
>> More info: The database backend when this is happening is PostGRES 9.2.
>>
>> When I switch over to SQLITE, the processing appears to work.
>> I'm assuming SQLITE does not have the index limitation.
>>
>> Dan
>>
>>
>>> The exception traceback on the local server:
>>>
>>> Exception happened during processing of request from ('127.0.0.1',
>>> 45798)
>>> Traceback (most recent call last):
>>>   File "/usr/local/src/python/lib/python2.7/SocketServer.py", line 295,
>>> in
>>> _handle_request_noblock
>>>     self.process_request(request, client_address)
>>>   File "/usr/local/src/python/lib/python2.7/SocketServer.py", line 321,
>>> in
>>> process_request
>>>     self.finish_request(request, client_address)
>>>   File "/usr/local/src/python/lib/python2.7/SocketServer.py", line 334,
>>> in
>>> finish_request
>>>     self.RequestHandlerClass(request, client_address, self)
>>>   File "/usr/local/src/python/lib/python2.7/SocketServer.py", line 653,
>>> in
>>> __init__
>>>     self.finish()
>>>   File "/usr/local/src/python/lib/python2.7/SocketServer.py", line 712,
>>> in
>>> finish
>>>     self.wfile.close()
>>>   File "/usr/local/src/python/lib/python2.7/socket.py", line 279, in
>>> close
>>>     self.flush()
>>>   File "/usr/local/src/python/lib/python2.7/socket.py", line 303, in
>>> flush
>>>     self._sock.sendall(view[write_offset:write_offset+buffer_size])
>>> error: [Errno 32] Broken pipe
>>>
>>>
>>>> I am attempting to populate a test pycsw server with the offerings
>>>> from
>>>> NDBC's SOS server. I am getting an exception of:
>>>> <ows:Exception exceptionCode="NoApplicableCode" locator="source">
>>>>     <ows:ExceptionText>Harvest (insert) failed: ERROR: index row size
>>>> 3956
>>>> exceeds maximum 2712 for index "ix_records_operateson"
>>>> HINT:  Values larger than 1/3 of a buffer page cannot be indexed.
>>>> Consider a function index of an MD5 hash of the value, or use full
>>>> text
>>>> indexing.
>>>> .</ows:ExceptionText>
>>>>   </ows:Exception>
>>>> </ows:ExceptionReport>
>>>>
>>>> I am not sure what this is telling me. The xml file I am using to
>>>> direct
>>>> the pycsw-admin is:
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <Harvest xmlns="http://www.opengis.net/cat/csw/2.0.2"
>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>> xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2
>>>> http://schemas.opengis.net/csw/2.0.2/CSW-publication.xsd"
>>>> service="CSW"
>>>> version="2.0.2">
>>>>   <Source>http://sdf.ndbc.noaa.gov/sos/server.php</Source>
>>>>   <ResourceType>http://www.opengis.net/sos/1.0</ResourceType>
>>>> </Harvest>
>>>>
>>>> Any pointers would be great.
>>>>
>>>> Dan
>>>>
>>>> _______________________________________________
>>>> pycsw-devel mailing list
>>>> pycsw-devel at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/pycsw-devel
>>>>
>>>
>>>
>>> _______________________________________________
>>> pycsw-devel mailing list
>>> pycsw-devel at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/pycsw-devel
>>>
>>
>>
>> _______________________________________________
>> pycsw-devel mailing list
>> pycsw-devel at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/pycsw-devel
>