<div dir="ltr">Hi Tom. <div><br></div><div><div>I do this having trouble accessing external urls in my server. Fix, and did the test again pointing to the xml you stated:</div><div><br></div><div><a href="https://github.com/geopython/pycsw/blob/master/tests/suites/harvesting/post/Harvest-csw-run1.xml">https://github.com/geopython/pycsw/blob/master/tests/suites/harvesting/post/Harvest-csw-run1.xml</a></div><div><br></div><div>.. But the error was the same:</div><div><br></div><div>pycsw-admin.py -c -u post_xml http: //localhost/pycsw/csw.py -x / var / www / html / pycsw / bin / request .xml</div><div><br></div><div>Initializing static context</div><div>Executing HTTP POST request /var/www/html/pycsw/bin/request.xml on server http: //localhost/pycsw/csw.py</div><div><? Xml version = "1.0" encoding = "UTF-8" standalone = "no"?></div><div><! - Pycsw 1.10.0 -></div><div><Ows: ExceptionReport xmlns: dc = "<a href="http://purl.org/dc/elements/1.1/">http://purl.org/dc/elements/1.1/</a>" xmlns: inspire_common = "<a href="http://inspire.ec.europa.eu/schemas/common/1.0">http://inspire.ec.europa.eu/schemas/common/1.0</a>" xmlns: atom = "<a href="http://www.w3.org/2005/Atom">http://www.w3.org/2005/Atom</a>" xmlns: xs = "<a href="http://www.w3.org/2001/XMLSchema">http://www.w3.org/2001/XMLSchema</a>" xmlns: dct = "<a href="http://purl.org/dc/">http://purl.org/dc/</a> terms / "xmlns: ows =" <a href="http://www.opengis.net/ows">http://www.opengis.net/ows</a> "xmlns: apiso =" <a href="http://www.opengis.net/cat/csw/apiso/1.0">http://www.opengis.net/cat/csw/apiso/1.0</a> "xmlns: gml =" http: //<a href="http://www.opengis.net/gml">www.opengis.net/gml</a> "xmlns: diff =" <a href="http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/">http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/</a> "xmlns: xlink =" <a href="http://www.w3.org/1999">http://www.w3.org/1999</a> / xlink "xmlns: gco =" <a href="http://www.isotc211.org/2005/gco">http://www.isotc211.org/2005/gco</a> "xmlns: gmd =" ​​<a href="http://www.isotc211.org/2005/gmd">http://www.isotc211.org/2005/gmd</a> "xmlns: rdf =" http: // <a href="http://www.w3.org/1999/02/22-rdf-syntax-ns#">www.w3.org/1999/02/22-rdf-syntax-ns#</a> "xmlns: srv =" <a href="http://www.isotc211.org/2005/srv">http://www.isotc211.org/2005/srv</a> "xmlns: ogc =" http: //www.opengis .net / ogc "xmlns: FGDC =" <a href="http://www.opengis.net/cat/csw/csdgm">http://www.opengis.net/cat/csw/csdgm</a> "xmlns: inspire_ds =" <a href="http://inspire.ec.europa.eu/schemas/inspire_ds/1.0">http://inspire.ec.europa.eu/schemas/inspire_ds/1.0</a> "xmlns : csw = "<a href="http://www.opengis.net/cat/csw/2.0.2">http://www.opengis.net/cat/csw/2.0.2</a>" xmlns: xsi = "<a href="http://www.w3.org/2001/XMLSchema-instance">http://www.w3.org/2001/XMLSchema-instance</a>" xmlns: the = "http: //<a href="http://a9.com/-/spec/opensearch/1.1/">a9.com/-/spec/opensearch/1.1/</a> "xmlns: soapenv =" <a href="http://www.w3.org/2003/05/soap-envelope">http://www.w3.org/2003/05/soap-envelope</a> "xmlns: sitemap =" http: //www.sitemaps .org / schemas / sitemap / 0.9 "language =" en-US "version =" 1.2.0 "xsi: schemaLocation =" <a href="http://www.opengis.net/ows">http://www.opengis.net/ows</a> <a href="http://schemas.opengis.net/ows">http://schemas.opengis.net/ows</a> /1.0.0/owsExceptionReport.xsd"><ows:Exception ExceptionCode = "NoApplicableCode" locator = "source"> <ows: ExceptionText> Harvest (insert) failed: ERROR: null value in column "identifier" violates not-null constraint</div><div>DETAIL: Failing row contains (null, csw: Record, <a href="http://www.opengis.net/cat/csw/2.0.2">http://www.opengis.net/cat/csw/2.0.2</a>, local, 2015-06-22T11: 23: 47Z & lt; ows: ExceptionReport xmlns: dc = "<a href="http://purl.org/dc/elements/1.1/">http://purl.org/dc/elements/1.1/</a>" ... Missing keyword: service, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, <a href="http://demo.geonode.org/catalogue/csw">http://demo.geonode.org/catalogue/csw</a>, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 'keyword': 2 'miss': 1 'servic': 3, null).</div><div>. </ Ows: ExceptionText> </ ows: Exception> </ ows: ExceptionReport></div><div>Done</div><div><br></div><div>Attached, I am sending the response for </div><div><br></div><div><a href="http://aguai.cnpm.embrapa.br/catalogue/csw?request=GetRecords&service=CSW&version=2.0.2&resultType=results&outputSchema=http://www.isotc211.org/2005/gmd&typeNames=csw:Record&elementSetName=summary&maxrecords=1000">http://aguai.cnpm.embrapa.br/catalogue/csw?request=GetRecords&service=CSW&version=2.0.2&resultType=results&outputSchema=http://www.isotc211.org/2005/gmd&typeNames=csw:Record&elementSetName=summary&maxrecords=1000</a></div><div><br></div><div>in which you suggested.</div><div><br></div><div>I am still using the 1.10.0 version. Could this being resolved by upgrading to 1.10.1? I'm using an Ubuntu 4.14 server. I tried to update the pycsw using apt-get update and then apt-get upgrade, but the pycsw remained in version 1.10.0. How do I upgrade using the Ubuntu packages?</div><div><br></div><div>Thanks</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 19, 2015 at 4:28 PM, Tom Kralidis <span dir="ltr"><<a href="mailto:tomkralidis@gmail.com" target="_blank">tomkralidis@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Davi: thanks for moving this discussion here from geonode-users.<br>
Comments interleaved.<br>
<div><div class="h5"><br>
On Fri, Jun 19, 2015 at 3:09 PM, Davi Custodio <<a href="mailto:davicustodio@gmail.com">davicustodio@gmail.com</a>> wrote:<br>
> Hello. I have a scenario consists of 8 GeoNode servers with their respective<br>
> PyCSW assets.<br>
> To test, Getrecords use as:<br>
><br>
> <a href="http://localhost/catalogue/csw?request=GetRecords&service=CSW&version=2.0.2&resultType=results&outputSchema=http://www.isotc211.org/2005/gmd&typeNames=csw:Record&elementSetName=summary" rel="noreferrer" target="_blank">http://localhost/catalogue/csw?request=GetRecords&service=CSW&version=2.0.2&resultType=results&outputSchema=http://www.isotc211.org/2005/gmd&typeNames=csw:Record&elementSetName=summary</a><br>
><br>
> and receive the corresponding xml correctly.<br>
><br>
> You can also configure the harvest within the Geonetwork using:<br>
><br>
> http: // localhost / catalog / csw version = 2.0.2 & request =<br>
> GetCapabilities & service = CSW<br>
><br>
> and also the harvest runs properly.<br>
><br>
> I created a dedicated server with the default installation of PyCSW 1.10.0,<br>
> and I'm trying to get this server collect all the metadata records of 8<br>
> servers GeoNode.<br>
><br>
> My intention is to have a PyCSW to gather all the metadata of my<br>
> organization. I'm avoiding using "federatedcatalogues" because I do not want<br>
> the user who will rep csw need to specify additional parameters.<br>
><br>
> In a first test, I used the command:<br>
><br>
> pycsw-admin.py -c -u post_xml http: -x //localhost/pycsw/csw.py<br>
> /var/www/html/pycsw/bin/request.xml<br>
><br>
> where request.xml =<br>
><br>
> <? Xml version = "1.0" encoding = "UTF-8"?><br>
> <Harvest xmlns = "<a href="http://www.opengis.net/cat/csw/2.0.2" rel="noreferrer" target="_blank">http://www.opengis.net/cat/csw/2.0.2</a>" xmlns: ogc =<br>
> "<a href="http://www.opengis.net/ogc" rel="noreferrer" target="_blank">http://www.opengis.net/ogc</a>" xmlns: gmd = "http: // www .<a href="http://isotc211.org" rel="noreferrer" target="_blank">isotc211.org</a> / 2005<br>
> / gmd "xmlns: ows =" <a href="http://www.opengis.net/ows" rel="noreferrer" target="_blank">http://www.opengis.net/ows</a> "xmlns: xsd<br>
> ="<a href="http://www.w3.org/2001/XMLSchema" rel="noreferrer" target="_blank">http://www.w3.org/2001/XMLSchema</a> "xmlns: dc ="<br>
> <a href="http://purl.org/dc/elements/1.1/" rel="noreferrer" target="_blank">http://purl.org/dc/elements/1.1/</a> "xmlns: dct =" <a href="http://purl.org/dc/terms/" rel="noreferrer" target="_blank">http://purl.org/dc/terms/</a><br>
> "xmlns: gml =" <a href="http://www.opengis.net/gml" rel="noreferrer" target="_blank">http://www.opengis.net/gml</a> " xmlns: xsi =<br>
> "<a href="http://www.w3.org/2001/XMLSchema-instance" rel="noreferrer" target="_blank">http://www.w3.org/2001/XMLSchema-instance</a>" xsi: schemaLocation =<br>
> "<a href="http://www.opengis.net/cat/csw/2.0.2" rel="noreferrer" target="_blank">http://www.opengis.net/cat/csw/2.0.2</a> http: //schemas.opengis .net / csw /<br>
> 2.0.2 / CSW-publication.xsd "service =" CSW "version =" 2.0.2 "><br>
>   <Source><a href="http://demo.geonode.org//catalogue/csw" rel="noreferrer" target="_blank">http://demo.geonode.org//catalogue/csw</a> </ Source><br>
>   <ResourceType> <a href="http://www.opengis.net/cat/csw/2.0.2" rel="noreferrer" target="_blank">http://www.opengis.net/cat/csw/2.0.2</a> </ ResourceType><br>
>   <ResourceFormat> application / xml </ ResourceFormat><br>
> </ Harvest><br>
><br>
> pointing to <a href="http://demo.geonode.org//catalogue/csw" rel="noreferrer" target="_blank">http://demo.geonode.org//catalogue/csw</a> ..<br>
><br>
> I get the following error:<br>
><br>
> Initializing static context<br>
> Executing HTTP POST request /var/www/html/pycsw/bin/request.xml on server<br>
> http: //localhost/pycsw/csw.py<br>
> Traceback (most recent call last):<br>
>    File "/usr/bin/pycsw-admin.py", line 246, in <module><br>
>      print admin.post_xml (CSW_URL, XML, TIMEOUT)<br>
>    File "/usr/lib/python2.7/dist-packages/pycsw/admin.py", line 495, in<br>
> post_xml<br>
>      raise RuntimeError (err)<br>
> RuntimeError: timed out<br>
<br>
</div></div>I'm not getting any timeout when running here (using 1.10.1 -- any<br>
chance you can upgrade your single non-GeoNode pycsw instance to<br>
1.10.1?) using the Harvest XML request like<br>
<a href="https://github.com/geopython/pycsw/blob/master/tests/suites/harvesting/post/Harvest-csw-run1.xml" rel="noreferrer" target="_blank">https://github.com/geopython/pycsw/blob/master/tests/suites/harvesting/post/Harvest-csw-run1.xml</a>,<br>
but substituting the Source with <a href="http://demo.geonode.org/catalogue/csw" rel="noreferrer" target="_blank">http://demo.geonode.org/catalogue/csw</a><br>
<br>
pycsw harvests 36 records (1 for the service and 35 metadata records).<br>
Are you able to see <a href="http://demo.geonode.org/catalogue/csw" rel="noreferrer" target="_blank">http://demo.geonode.org/catalogue/csw</a> from the box<br>
on which you are running your harvests from?<br>
<div><div class="h5"><br>
> when I point to one of my GeoNode-PyCSW servers using request.xml as:<br>
><br>
> <? Xml version = "1.0" encoding = "UTF-8"?><br>
> <Harvest xmlns = "<a href="http://www.opengis.net/cat/csw/2.0.2" rel="noreferrer" target="_blank">http://www.opengis.net/cat/csw/2.0.2</a>" xmlns: ogc =<br>
> "<a href="http://www.opengis.net/ogc" rel="noreferrer" target="_blank">http://www.opengis.net/ogc</a>" xmlns: gmd = "http: // www .<a href="http://isotc211.org" rel="noreferrer" target="_blank">isotc211.org</a> / 2005<br>
> / gmd "xmlns: ows =" <a href="http://www.opengis.net/ows" rel="noreferrer" target="_blank">http://www.opengis.net/ows</a> "xmlns: xsd<br>
> ="<a href="http://www.w3.org/2001/XMLSchema" rel="noreferrer" target="_blank">http://www.w3.org/2001/XMLSchema</a> "xmlns: dc ="<br>
> <a href="http://purl.org/dc/elements/1.1/" rel="noreferrer" target="_blank">http://purl.org/dc/elements/1.1/</a> "xmlns: dct =" <a href="http://purl.org/dc/terms/" rel="noreferrer" target="_blank">http://purl.org/dc/terms/</a><br>
> "xmlns: gml =" <a href="http://www.opengis.net/gml" rel="noreferrer" target="_blank">http://www.opengis.net/gml</a> " xmlns: xsi =<br>
> "<a href="http://www.w3.org/2001/XMLSchema-instance" rel="noreferrer" target="_blank">http://www.w3.org/2001/XMLSchema-instance</a>" xsi: schemaLocation =<br>
> "<a href="http://www.opengis.net/cat/csw/2.0.2" rel="noreferrer" target="_blank">http://www.opengis.net/cat/csw/2.0.2</a> http: //schemas.opengis .net / csw /<br>
> 2.0.2 / CSW-publication.xsd "service =" CSW "version =" 2.0.2 "><br>
>   <Source> <a href="http://aguai.cnpm.embrapa.br/catalogue/csw" rel="noreferrer" target="_blank">http://aguai.cnpm.embrapa.br/catalogue/csw</a> </ Source><br>
>   <ResourceType> <a href="http://www.opengis.net/cat/csw/2.0.2" rel="noreferrer" target="_blank">http://www.opengis.net/cat/csw/2.0.2</a> </ ResourceType><br>
>   <ResourceFormat> application / xml </ ResourceFormat><br>
> </ Harvest><br>
><br>
> pointing to my server: <a href="http://aguai.cnpm.embrapa.br/catalogue/csw" rel="noreferrer" target="_blank">http://aguai.cnpm.embrapa.br/catalogue/csw</a><br>
><br>
> I get the error:<br>
><br>
> Initializing static context<br>
> Executing HTTP POST request /var/www/html/pycsw/bin/request.xml on server<br>
> http: //localhost/pycsw/csw.py<br>
> <? Xml version = "1.0" encoding = "UTF-8" standalone = "no"?><br>
> <! - Pycsw 1.10.0 -><br>
> <Ows: ExceptionReport xmlns: dc = "<a href="http://purl.org/dc/elements/1.1/" rel="noreferrer" target="_blank">http://purl.org/dc/elements/1.1/</a>" xmlns:<br>
> inspire_common = "<a href="http://inspire.ec.europa.eu/schemas/common/1.0" rel="noreferrer" target="_blank">http://inspire.ec.europa.eu/schemas/common/1.0</a>" xmlns:<br>
> atom = "<a href="http://www.w3.org/2005/Atom" rel="noreferrer" target="_blank">http://www.w3.org/2005/Atom</a>" xmlns: xs =<br>
> "<a href="http://www.w3.org/2001/XMLSchema" rel="noreferrer" target="_blank">http://www.w3.org/2001/XMLSchema</a>" xmlns: dct = "<a href="http://purl.org/dc/" rel="noreferrer" target="_blank">http://purl.org/dc/</a> terms /<br>
> "xmlns: ows =" <a href="http://www.opengis.net/ows" rel="noreferrer" target="_blank">http://www.opengis.net/ows</a> "xmlns: apiso ="<br>
> <a href="http://www.opengis.net/cat/csw/apiso/1.0" rel="noreferrer" target="_blank">http://www.opengis.net/cat/csw/apiso/1.0</a> "xmlns: gml =" http:<br>
> //<a href="http://www.opengis.net/gml" rel="noreferrer" target="_blank">www.opengis.net/gml</a> "xmlns: diff<br>
> ="<a href="http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/" rel="noreferrer" target="_blank">http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/</a> "xmlns: xlink ="<br>
> <a href="http://www.w3.org/1999" rel="noreferrer" target="_blank">http://www.w3.org/1999</a> / xlink "xmlns: gco ="<br>
> <a href="http://www.isotc211.org/2005/gco" rel="noreferrer" target="_blank">http://www.isotc211.org/2005/gco</a> "xmlns: gmd ="<br>
> <a href="http://www.isotc211.org/2005/gmd" rel="noreferrer" target="_blank">http://www.isotc211.org/2005/gmd</a> "xmlns: rdf =" http:<br>
> //<a href="http://www.w3.org/1999/02/22-rdf-syntax-ns#" rel="noreferrer" target="_blank">www.w3.org/1999/02/22-rdf-syntax-ns#</a> "xmlns: srv ="<br>
> <a href="http://www.isotc211.org/2005/srv" rel="noreferrer" target="_blank">http://www.isotc211.org/2005/srv</a> "xmlns: ogc =" http: //www.opengis .net /<br>
> ogc "xmlns: FGDC =" <a href="http://www.opengis.net/cat/csw/csdgm" rel="noreferrer" target="_blank">http://www.opengis.net/cat/csw/csdgm</a> "xmlns: inspire_ds<br>
> ="<a href="http://inspire.ec.europa.eu/schemas/inspire_ds/1.0" rel="noreferrer" target="_blank">http://inspire.ec.europa.eu/schemas/inspire_ds/1.0</a> "xmlns : csw =<br>
> "<a href="http://www.opengis.net/cat/csw/2.0.2" rel="noreferrer" target="_blank">http://www.opengis.net/cat/csw/2.0.2</a>" xmlns: xsi =<br>
> "<a href="http://www.w3.org/2001/XMLSchema-instance" rel="noreferrer" target="_blank">http://www.w3.org/2001/XMLSchema-instance</a>" xmlns: the = "http:<br>
> //<a href="http://a9.com/-/spec/opensearch/1.1/" rel="noreferrer" target="_blank">a9.com/-/spec/opensearch/1.1/</a> "xmlns: soapenv ="<br>
> <a href="http://www.w3.org/2003/05/soap-envelope" rel="noreferrer" target="_blank">http://www.w3.org/2003/05/soap-envelope</a> "xmlns: sitemap =" http:<br>
> //www.sitemaps .org / schemas / sitemap / 0.9 "language =" en-US "version ="<br>
> 1.2.0 "xsi: schemaLocation ="<br>
> <a href="http://www.opengis.net/owshttp://schemas.opengis.net/ows" rel="noreferrer" target="_blank">http://www.opengis.net/owshttp://schemas.opengis.net/ows</a><br>
> /1.0.0/owsExceptionReport.xsd"><ows:Exception ExceptionCode =<br>
> "NoApplicableCode" locator = "source"> <ows: ExceptionText> Harvest (insert)<br>
> failed: ERROR: null value in column "identifier" violates not-null<br>
> constraint<br>
> DETAIL: Failing row contains (null, csw: Record,<br>
> <a href="http://www.opengis.net/cat/csw/2.0.2" rel="noreferrer" target="_blank">http://www.opengis.net/cat/csw/2.0.2</a>, local, 2015-06-19T15: 19: 28Z & lt;<br>
> ows: ExceptionReport xmlns: dc = "<a href="http://purl.org/dc/elements/1.1/" rel="noreferrer" target="_blank">http://purl.org/dc/elements/1.1/</a>" ...<br>
> Missing keyword: service, null, null, null, null, null, null, null, null,<br>
> null, null, null, null, null, null, null, null, null, null, null, null,<br>
> null, null, null, null, null, null,<br>
> <a href="http://aguai.cnpm.embrapa.br/catalogue/csw" rel="noreferrer" target="_blank">http://aguai.cnpm.embrapa.br/catalogue/csw</a>, null, null, null, null, null,<br>
> null, null, null, null, null, null, null, null, null, null, null, null,<br>
> null, null, null, null, null, 'keyword': 2 'miss': 1' servic '3, null).<br>
> . </ Ows: ExceptionText> </ ows: Exception> </ ows: ExceptionReport><br>
> Done<br>
><br>
><br>
> Can anyone help me explaining how to best implement it? And because of<br>
> errors?<br>
><br>
<br>
</div></div>It looks like one of the layers in your pycsw serving GeoNode CSW does<br>
not have an identifier?  Can you turn on pycsw logging from your<br>
single pycsw to DEBUG and run the harvest again?  There might be<br>
valuable information in the log that can help.  Feel free to send me<br>
the log output offline.<br>
<br>
As well, wow many records does<br>
<a href="http://aguai.cnpm.embrapa.br/catalogue/csw" rel="noreferrer" target="_blank">http://aguai.cnpm.embrapa.br/catalogue/csw</a> have?  Might be valuable to<br>
do a full GetRecords request, i.e.:<br>
<br>
<a href="http://aguai.cnpm.embrapa.br/catalogue/csw?request=GetRecords&service=CSW&version=2.0.2&resultType=results&outputSchema=http://www.isotc211.org/2005/gmd&typeNames=csw:Record&elementSetName=summary&maxrecords=FOO" rel="noreferrer" target="_blank">http://aguai.cnpm.embrapa.br/catalogue/csw?request=GetRecords&service=CSW&version=2.0.2&resultType=results&outputSchema=http://www.isotc211.org/2005/gmd&typeNames=csw:Record&elementSetName=summary&maxrecords=FOO</a><br>
<br>
to inspect the actual CSW output that pycsw is trying to harvest.<br>
Feel free to send me the result offline.<br>
<div class="HOEnZb"><div class="h5"><br>
> --<br>
> Davi de O. Custódio<br>
><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Davi de O. Custódio</div>
</div>