[Gdal-dev] Max no of layers in OGR shapefile driver

[Dave Robertson]

Hi Mateusz and Frank,

Thanks for the help with the following 2 issues.

Max open files:

OSX does indeed have a default limit of 256 open files per process  
although ulimit reports "unlimited" which is of course the hard  
(upper) limit.  I'm now using setrlimit() to allow 2048 and all is well.


PostGIS driver vs shapefile driver:


I made some measurements and would be interested to hear your thoughts.
The code reads a data via OGR and renders this to screen.

Typical time to read and render PostGIS: 1,209,100,332 nanoseconds
Typical time to read and render shapefiles:  225,924,592 nanoseconds

These timings are rendering exactly the same data at the same  
resolution using the same code. I am using SetSpatialFilterRect in  
both cases. PostGIS is running on the same machine, is GiST indexed,  
and is using its default driver options.

PG appears to me to be 6 times slower than shapefiles: is this what  
you'd expect,  or should I be able to tune the PG query in some way  
(perhaps using binary cursors?)

Regards,
Dave Robertson



On 18/05/2007, at 1:27 AM, Mateusz Loskot wrote:

> Dave Robertson wrote:
>> Hi,
>>
>> I'm working with OGR and a set of 280 shapefiles each representing a
>> layer. I find I can open the first 84 shapefiles and all the rest  
>> return
>> error code 4 Error opening file). It doesn't seem to matter which
>> particular 84 files - I can't open any more than that. The shapefiles
>> are all one directory, along with their .dbf, .shx and .qix files.  
>> I've
>> looked at the source for the shapefile driver and I can't see any  
>> limits
>> built in there, and as far as I'm aware the operating system's  
>> limit is
>> 12500 open files (per process). The OS is OS X.
>
> Dave,
>
> ADAIR, OSX9 used very low number of open files per process (64).
> See this FAQ, error code 24:
>
> http://www.osxfaq.com/man/2/intro.ws
>
> When you open 280 shapefiles, you get (not counting .prj) 280 * 3  
> number
> of open files, so I suppose you hit beyond the limit
> which I *suppose* is ~255 per process, in newer OSX versions.
>
>> I have loaded all the shapefiles into postgis as an experiment and  
>> I can
>> read all the layers from there just fine, albeit slowly.
>
> What means slowly?
> Do you use spatial index, GiST ?
>
> Cheers
> -- 
> Mateusz Loskot
> http://mateusz.loskot.net