<div dir="ltr">Hey,<div><br></div><div>just to let you know, I found the issue.</div><div>So I think it has to do with the fact that I keep some handles opened on the original TIFF In the parent thread. Even setting the variable to None does not change anything, but I guess it's because the garbage collector is asynchronous.</div><div>So if I do my initial code in a child process, return the data through a pipe and then spawn a pool of child processes doing tiles one at a time, I don't get this concurrency issue.</div><div><br></div><div>Now I "just" need to write it properly which is going to take a while :)</div><div><br></div><div>Cheers</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><font color="#999999"><br></font></div><div><font color="#999999">---</font></div><div><font color="#999999">Gregory Bataille</font><br></div></div></div></div>
<br><div class="gmail_quote">On Tue, Mar 21, 2017 at 9:38 AM, Grégory Bataille <span dir="ltr"><<a href="mailto:gregory.bataille@gmail.com" target="_blank">gregory.bataille@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">hum... debugging in the C layers, that'll be interesting...<div>Ok, I'll continue to dig in, thanks</div></div><div class="gmail_extra"><br clear="all"><div><div class="m_-3736654439809518951gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><font color="#999999"><br></font></div><div><font color="#999999">---</font></div><div><font color="#999999">Gregory Bataille</font><br></div></div></div></div><div><div class="h5">
<br><div class="gmail_quote">On Tue, Mar 21, 2017 at 8:44 AM, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div style="font-family:'Sans Serif';font-size:9pt;font-weight:400;font-style:normal"><span>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">On mardi 21 mars 2017 05:51:18 CET Grégory Bataille wrote:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Hey Even,</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> reaching out again, because I just don't know...</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> so here is gdal2tiles_parallel strategy (as far as I understand it)</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> - in each process, open the input file, compute the autowarped vrt (in the</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> case of the dataset I use at least) loop over tiles to generate and work</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> only depending on the process number (so that this work is made in</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> parallel). We therefore end up with, in each process</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> - the input file opened as a dataset (but not touched after the</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> autowarped vrt is generated)</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> - a different autowarped vrt in memory (for each process) but pointing</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> to the same source (as far as I can see)</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> My strategy:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> - open the input file.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> - generate the autowarped vrt and save it to disk</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> - compute all the tile details and store them in a data structure.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Then in each process:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">I'm confused. You're talking about process, but is it a real operating system process or a thread ? I guess the later.</p><span>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> - take one tile detail</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> - open the vrt (same file for each process, pointing to the same source</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> TIFF)</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> - read the vrt and write the tile</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> And that's the read that is failing as mentioned in my first email.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> I can't see much difference. To try and be complete, I tried to (or think I</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> tried to):</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> - open the input file in each process</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> - generate a different vrt file for each thread</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> I even copy/pasted the code that gdal2tiles_parallel uses in their</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> processes, but with no success, always the same error. The</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> VRT_SHARED_SOURCE option</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> does not seem to change anything (but looking at the wiki, it's not clear</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> whether it applies to autowarped vrt too)</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">VRT_SHARED_SOURCE indeed only works for "regular" mosaicing VRTs. So not to warped VRT.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Warped VRT opens the source dataset as a shared dataset in</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><a href="https://github.com/OSGeo/gdal/blob/trunk/gdal/alg/gdalwarper.cpp#L1675" target="_blank">https://github.com/OSGeo/gdal/<wbr>blob/trunk/gdal/alg/gdalwarper<wbr>.cpp#L1675</a></p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">So you could have an issue if you open the VRT twice in the same thread, and then use each handle in a different thread, since both VRT handles would point to the same shared dataset. But it doesn't look that's what you are doing.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">As a workaround to investigate, you could try to remove the GDAL_OF_SHARED | flag in the line I mentionned and see if that makes a difference (but we could potentially - not sure - have issues at closing time since different call sites might make an assumption on the dataset being opened with shared semantics)</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">What you could do too is run the script through your prefered debugger and set a breakpoint in GDALOpenEx and look at the calling sites. If you are using 2 threads, then if we want things to work correctly, GDALOpenEx() should be called twice on the TIFF dataset and return a different handle.</p><span>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Even</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">-- </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Spatialys - Geospatial professional services</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a></p></span></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>