<div dir="ltr">Hey Even,<div><br></div><div>reaching out again, because I just don't know...</div><div><br></div><div>so here is gdal2tiles_parallel strategy (as far as I understand it)</div><div>- in each process, open the input file, compute the autowarped vrt (in the case of the dataset I use at least) loop over tiles to generate and work only depending on the process number (so that this work is made in parallel). We therefore end up with, in each process</div><div>    - the input file opened as a dataset (but not touched after the autowarped vrt is generated)</div><div>    - a different autowarped vrt in memory (for each process) but pointing to the same source (as far as I can see)</div><div><br></div><div><br></div><div>My strategy:</div><div>- open the input file.</div><div>- generate the autowarped vrt and save it to disk</div><div>- compute all the tile details and store them in a data structure.</div><div>Then in each process:</div><div>- take one tile detail</div><div>- open the vrt (same file for each process, pointing to the same source TIFF)</div><div>- read the vrt and write the tile</div><div>And that's the read that is failing as mentioned in my first email.</div><div><br></div><div>I can't see much difference. To try and be complete, I tried to (or think I tried to):</div><div>- open the input file in each process</div><div>- generate a different vrt file for each thread</div><div><br></div><div>I even copy/pasted the code that gdal2tiles_parallel uses in their processes, but with no success, always the same error. The <span style="font-family:"sans serif";font-size:12px;white-space:pre-wrap">VRT_</span><wbr style="font-family:"sans serif";font-size:12px;white-space:pre-wrap"><span style="font-family:"sans serif";font-size:12px;white-space:pre-wrap">SHARED_SOURCE</span> option does not seem to change anything (but looking at the wiki, it's not clear whether it applies to autowarped vrt too)</div><div><br></div><div>If you have any idea, I'll take it, I have been stuck for a long time without making any progress :(</div><div><br></div><div>Cheers</div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><font color="#999999"><br></font></div><div><font color="#999999">---</font></div><div><font color="#999999">Gregory Bataille</font><br></div></div></div></div>
<br><div class="gmail_quote">On Fri, Mar 17, 2017 at 6:15 AM, Grégory Bataille <span dir="ltr"><<a href="mailto:gregory.bataille@gmail.com" target="_blank">gregory.bataille@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">no luck.<div>I tried this config.</div><div>I also tried (with this config) to do a copy of the vrt file (with shell utility, not gdal) in each thread before opening it (still goes to the same TIFF file though), but no luck.</div><div><br></div><div>But at least now that you confirm that it's related to concurrent reads, I have something precise to search for. I'll try to investigate how the gdal2tiles_parallel does it because it did not look different than what I was doing, but likely I missed something.</div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br clear="all"><div><div class="m_-5446053997064309388gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><font color="#999999"><br></font></div><div><font color="#999999">---</font></div><div><font color="#999999">Gregory Bataille</font><br></div></div></div></div><div><div class="h5">
<br><div class="gmail_quote">On Thu, Mar 16, 2017 at 7:58 PM, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div style="font-family:'Sans Serif';font-size:9pt;font-weight:400;font-style:normal"><span>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">On jeudi 16 mars 2017 18:45:03 CET Grégory Bataille wrote:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> Does that mean a different vrt file or simply reopen the file to create a</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> gdal object in each process. Because I'm reopening the vrt file in each</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> thread</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
</span><p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Ah, that must be issue of</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><a href="http://gdal.org/gdal_vrttut.html#gdal_vrttut_mt" target="_blank">http://gdal.org/gdal_vrttut.ht<wbr>ml#gdal_vrttut_mt</a></p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Try defining</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">gdal.SetConfigOption('VRT_SHAR<wbr>ED_SOURCE', '0')</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">before opening the VRTs</p><div><div class="m_-5446053997064309388h5">
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> On Thu, 16 Mar 2017 at 17:23, Even Rouault <<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>></p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> wrote:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > On jeudi 16 mars 2017 17:16:20 CET Grégory Bataille wrote:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > Hello all,</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > Reaching out to the community caused I have failed for the past few</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > days.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > *short version*, I'm trying to multithread the gdal2tiles utility, and</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > I'm</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > getting this</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > Generating Base Tiles:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > ERROR 1: LZWDecode:Wrong length of decoded string: data probably</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > corrupted</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > at scanline 256</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > ERROR 1: TIFFReadEncodedTile() failed.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > ERROR 1:</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > /Users/gbataille/Downloads/Pro<wbr>ject_58704_transparent_mosaic_<wbr>group1.tif,</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > band 1: IReadBlock failed at X offset 1, Y offset 0</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > ERROR 1: GetBlockRef failed at X block offset 1, Y block offset 0</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > ERROR 1: gba.vrt, band 1: IReadBlock failed at X offset 0, Y offset 0</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > > Any idea?</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > Yes, you need one dataset object per thread . Dataset objects cannot be</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > used simultaneously from several threads.</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > Even</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > --</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > Spatialys - Geospatial professional services</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">> > <a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a></p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"> </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">-- </p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px">Spatialys - Geospatial professional services</p>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px;text-indent:0px"><a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a></p></div></div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>