[gdal-dev] Possible bug: hard limit to the number of Datasets creatable in a Python process?

[Michal Migurski]

On Aug 14, 2012, at 12:37 PM, Even Rouault wrote:

> Le mardi 14 août 2012 21:00:20, Michal Migurski a écrit :
>> I'm experiencing a reproducible bug.
>> 
>> I create new 258x258 pixel datasets like this:
>> 
>> 	driver = gdal.GetDriverByName('GTiff')
>> 	handle, filename = mkstemp(dir=tmpdir,
>> prefix='dem-tools-hillup-data-render-', suffix='.tif') ds =
>> driver.Create(filename, width, height, 1, gdal.GDT_Float32)
>> 
>> I also open a number of datasets in the same process using gdal.Open() and
>> warp them together with gdal.ReprojectImage().
>> 
>> When I have created 1,020 new datasets / opened 2,041 total datasets, I get
>> a None back from driver.Create(). This number is stable, and appears
>> unrelated to the specific datasets that I'm creating. Datasets are being
>> flushed and falling out of scope, so I believe that they're correctly
>> being closed when they should. Do these numbers sound like known, hard
>> internal limits to anyone?
> 
> I don't think it is a bug in GDAL, but an issue in your script. This sounds 
> like you hit the default limit of 1024 files opened simultaneously by a Linux 
> process. So I suspect that your datasets are not properly closed.
> 
> Calling ds.FlushCache() is not sufficient. In principle, if the dataset handle 
> goes out of scope, it should be destroyed, so I suspect that it is not the 
> case. Assigning None to the dataset will force its closing.


Great, thank you Even.

I have a place to look now. When I watch the output of lsof as the script runs, I can see that the failure occurs near the moment where the number of open file handles approaches 1024.

I've added `ds = None` to the relevant parts of my script, but I don't think the file handles are actually being closed. Here's an excerpt of lsof output for the specific pid, which shows correctly that the TIF files have each been deleted but handles remain open:

	python  4719 migurski  127u   REG    8,1   267898  65148403 /tmp/dem-tools-hillup-data-render-KvUA8d.tif (deleted)
	python  4719 migurski  128u   REG    8,1   267898  65148404 /tmp/dem-tools-hillup-data-render-MAwCVw.tif (deleted)
	python  4719 migurski  129u   REG    8,1   267898  65148405 /tmp/dem-tools-hillup-data-render-M_NvT_.tif (deleted)
	python  4719 migurski  130u   REG    8,1   267898  65148406 /tmp/dem-tools-hillup-data-render-dV1p7h.tif (deleted)
	python  4719 migurski  131u   REG    8,1   267898  65148407 /tmp/dem-tools-hillup-data-render-rmuD4b.tif (deleted)
	python  4719 migurski  132u   REG    8,1   267898  65148408 /tmp/dem-tools-hillup-data-render-KGtcFD.tif (deleted)
	python  4719 migurski  133u   REG    8,1   267898  65148409 /tmp/dem-tools-hillup-data-render-EI4Jkz.tif (deleted)
	python  4719 migurski  134u   REG    8,1   267898  65148410 /tmp/dem-tools-hillup-data-render-MnHVKl.tif (deleted)
	python  4719 migurski  135u   REG    8,1   267898  65148411 /tmp/dem-tools-hillup-data-render-oGd6_b.tif (deleted)
	python  4719 migurski  136u   REG    8,1   267898  65148412 /tmp/dem-tools-hillup-data-render-8p0cew.tif (deleted)
	python  4719 migurski  137u   REG    8,1   267898  65148413 /tmp/dem-tools-hillup-data-render-CZTPWc.tif (deleted)

I'm going to try switching to vsimem, maybe that will help.

-mike.

----------------------------------------------------------------
michal migurski- mike at stamen.com
                 415.558.1610