[gdal-dev] Python-OGR not looping over more than 500 PGSQL features

[Chaitanya kumar CH]

Matthieu,

I looked at the OGR's pg driver code. It is retrieving the data 500
records at a time using cursors. Can you provide some more lines from
the log files, before and after the given two lines?

In the mean time here is a quick and dirty solution. Recompile gdal
after setting CURSOR_PAGE to some value greater than the records in
your tables. http://trac.osgeo.org/gdal/browser/branches/1.6/gdal/ogr/ogrsf_frmts/pg/ogrpglayer.cpp#L68


On Fri, Oct 9, 2009 at 7:19 PM, Matthieu Rigal <rigal at rapideye.de> wrote:
> Chaitanya,
>
> I looked into the log files and could see the following lines :
> 2009-10-09 15:23:56 CEST lua_db lua_admin STATEMENT:  FETCH 500 in
> OGRPGResultLayerReader
> 2009-10-09 15:23:56 CEST lua_db lua_admin STATEMENT:  CLOSE
> OGRPGResultLayerReader
>
> So it is OGR which is requesting only 500 features.
> I may understand the problems to not have a limit but shouldn't there be some
> kind of a loop to be able to go automatically through all the features ?
> If not, this should be mentioned on the PG driver page, isn't it ?
>
> And furthermore, I never noticed this behaviour when previously working with
> older GDAL versions. Was I blind or did something changed ?
>
> Best regards,
> Matthieu
>
>
> On Friday 09 October 2009 15:21:05 Chaitanya kumar CH wrote:
>> Matthieu,
>>
>> GetFeatureCount in postgres driver is implemented with the COUNT()
>> function. (
>> http://trac.osgeo.org/gdal/browser/trunk/gdal/ogr/ogrsf_frmts/pg/ogrpgtable
>>layer.cpp#L1895 ) So it was never returning more than 500 features. IMHO,
>> the number 500 looks like an intentional limitation, probably set on the
>> server connection.
>>
>> On Fri, Oct 9, 2009 at 4:11 PM, Matthieu Rigal <rigal at rapideye.de> wrote:
>> > Hi people,
>> >
>> > I recently discovered a strange behaviour which was not happening before,
>> > I think...
>> > I'm using Python 2.5 with GDAL/OGR 1.6 and PGsql 8.3 on Linux.
>> >
>> > I am reading a table containing 858 features with several columns and
>> > geometry.
>> > I first do a SELECT on the table and the GetFeatureCount tells me 858
>> > lines are being readen.
>> > Then I am looping on the OGR object Feature by feature, to make some
>> > calculation and I send an UPDATE for this particular line through another
>> > OGR object.
>> > I have a counter after every UPDATE.
>> >
>> > Whereas it should go through all the 858 features, it stops after having
>> > gone through 500 features, my counter at the end shows 500 and looking to
>> > the database, it clearly stops after 500.
>> > I've done a test on a second database and it also stopped at 500...
>> >
>> > Is there some recently introduced limitations on OGR side ? I have
>> > searched all the website without finding something about it... May it
>> > come from some Python memory limitation ( I doubt) ? Or from some PGSQL
>> > server limitation (I doubt also because 858 features are returned at the
>> > beginning ?
>> >
>> > Thanks for your help,
>> > Best Regards,
>> > Matthieu Rigal
>> >
>
> RapidEye AG
> Molkenmarkt 30
> 14776 Brandenburg an der Havel
> Germany
>
> Follow us on Twitter! www.twitter.com/rapideye_ag
>
> Head Office/Sitz der Gesellschaft: Brandenburg an der Havel
> Management Board/Vorstand: Wolfgang G. Biedermann
> Chairman of Supervisory Board/Vorsitzender des Aufsichtsrates:
> Juergen Breitkopf
> Commercial Register/Handelsregister Potsdam HRB 17 796
> Tax Number/Steuernummer: 048/100/00053
> VAT-Ident-Number/Ust.-ID: DE 199331235
> DIN EN ISO 9001 certified
>
> *************************************************************************
> Diese E-Mail enthaelt vertrauliche und/oder rechtlich geschuetzte
> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese
> E-Mail irrtuemlich erhalten haben, informieren Sie bitte sofort den
> Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie
> die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.
>
> The information in this e-mail is intended for the named recipients
> only. It may contain privileged and confidential information. If you
> have received this communication in error, any use, copying or
> dissemination of its contents is strictly prohibited. Please erase all
> copies of the message along with any included attachments and notify
> RapidEye AG or the sender immediately by telephone at the number
> indicated on this page.
>



-- 
Best regards,
Chaitanya kumar CH.