<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I took a closer look at the files. It looks like they maxed out the byte size and were truncated. </div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Several NITF images have different rows * columns, but the same byte size.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
May not be a GDAL issue.</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> gdal-dev <gdal-dev-bounces@lists.osgeo.org> on behalf of ni hao <ni_hao88@hotmail.com><br>
<b>Sent:</b> April 7, 2021 10:52 AM<br>
<b>To:</b> Even Rouault <even.rouault@spatialys.com>; gdal-dev@lists.osgeo.org <gdal-dev@lists.osgeo.org><br>
<b>Subject:</b> Re: [gdal-dev] NITF Int16 image size limit</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Even,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
It was CSA using their own software. <span style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">They said that it was fine access the last 2000 lines.</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">But I could not confirm.</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Even Rouault <even.rouault@spatialys.com><br>
<b>Sent:</b> April 7, 2021 10:41 AM<br>
<b>To:</b> ni hao <ni_hao88@hotmail.com>; gdal-dev@lists.osgeo.org <gdal-dev@lists.osgeo.org><br>
<b>Subject:</b> Re: [gdal-dev] NITF Int16 image size limit</font>
<div> </div>
</div>
<div>
<p>Hum, 4294967295 = 0xffffffff = max uint32 value. This really a suspicious file size that looks like truncation at the 4 GB limit. Are you really really sure that you tested the non-GDAL implementation with that exact file, and not another version that might
have been truncated later ?<br>
</p>
<div class="x_x_moz-cite-prefix">Le 07/04/2021 à 14:59, ni hao a écrit :<br>
</div>
<blockquote type="cite">
<div>Hi Even,</div>
<div><br>
</div>
<div>I am using GDAL 2.4.4. I may try the latest version as you suggested.</div>
<div><span><br>
</span></div>
<div><span>The NITF file size is 4,294,967,295 bytes </span></div>
<div><br>
</div>
<div><span>The vendor used a non-GDAL implementation, and was able to display all image.</span><br>
</div>
<div><span><br>
</span></div>
<div><span><span>Metadata:</span>
<div> NITF_ABPP=16</div>
<div> NITF_BLOCKA_BLOCK_COUNT=01</div>
<div> NITF_BLOCKA_BLOCK_INSTANCE_01=01</div>
<div> NITF_BLOCKA_FRFC_LOC_01=</div>
<div> NITF_BLOCKA_FRLC_LOC_01=</div>
<div> NITF_BLOCKA_LAYOVER_ANGLE_01=000</div>
<div> NITF_BLOCKA_LRFC_LOC_01=</div>
<div> NITF_BLOCKA_LRLC_LOC_01=</div>
<div> NITF_BLOCKA_L_LINES_01=19694</div>
<div> NITF_BLOCKA_N_GRAY_01=00000</div>
<div> NITF_BLOCKA_SHADOW_ANGLE_01=180</div>
<div> NITF_CCS_COLUMN=0</div>
<div> NITF_CCS_ROW=0</div>
<div> NITF_CLEVEL=07</div>
<div> NITF_ENCRYP=0</div>
<div> NITF_FBKGC= 0, 0, 0</div>
<div> NITF_FDT=</div>
<div> NITF_FHDR=NITF02.10</div>
<div> NITF_FSCATP=</div>
<div> NITF_FSCAUT=</div>
<div> NITF_FSCLAS=U</div>
<div> NITF_FSCLSY=CA</div>
<div> NITF_FSCLTX=</div>
<div> NITF_FSCODE=</div>
<div> NITF_FSCOP=00000</div>
<div> NITF_FSCPYS=00000</div>
<div> NITF_FSCRSN=</div>
<div> NITF_FSCTLH=</div>
<div> NITF_FSCTLN=</div>
<div> NITF_FSDCDT=</div>
<div> NITF_FSDCTP=</div>
<div> NITF_FSDCXM=</div>
<div> NITF_FSDG=</div>
<div> NITF_FSDGDT=</div>
<div> NITF_FSREL=</div>
<div> NITF_FSSRDT=</div>
<div> NITF_FTITLE=</div>
<div> NITF_IALVL=0</div>
<div> NITF_IC=NC</div>
<div> NITF_ICAT=SAR</div>
<div> NITF_ICORDS=G</div>
<div> NITF_IDATIM=</div>
<div> NITF_IDLVL=</div>
<div> NITF_IGEOLO=</div>
<div> NITF_IID1=P1</div>
<div> NITF_IID2=</div>
<div> NITF_ILOC_COLUMN=0</div>
<div> NITF_ILOC_ROW=0</div>
<div> NITF_IMAG=1.0</div>
<div> NITF_IMODE=P</div>
<div> NITF_IREP=MULTI</div>
<div> NITF_ISCATP=</div>
<div> NITF_ISCAUT=</div>
<div> NITF_ISCLAS=U</div>
<div> NITF_ISCLSY=CA</div>
<div> NITF_ISCLTX=</div>
<div> NITF_ISCODE=</div>
<div> NITF_ISCRSN=</div>
<div> NITF_ISCTLH=</div>
<div> NITF_ISCTLN=</div>
<div> NITF_ISDCDT=</div>
<div> NITF_ISDCTP=</div>
<div> NITF_ISDCXM=</div>
<div> NITF_ISDG=</div>
<div> NITF_ISDGDT=</div>
<div> NITF_ISORCE=</div>
<div> NITF_ISREL=</div>
<div> NITF_ISSRDT=</div>
<div> NITF_ONAME=</div>
<div> NITF_OPHONE=</div>
<div> NITF_OSTAID=CSA</div>
<div> NITF_PJUST=R</div>
<div> NITF_PVTYPE=INT</div>
<div> NITF_STYPE=BF01</div>
<span> NITF_TGTID=</span><br>
</span></div>
<hr tabindex="-1">
<div id="x_x_divRplyFwdMsg" dir="ltr"><b>From:</b> Even Rouault <a class="x_x_moz-txt-link-rfc2396E" href="mailto:even.rouault@spatialys.com">
<even.rouault@spatialys.com></a><br>
<b>Sent:</b> April 7, 2021 5:23 AM<br>
<b>To:</b> ni hao <a class="x_x_moz-txt-link-rfc2396E" href="mailto:ni_hao88@hotmail.com">
<ni_hao88@hotmail.com></a>; <a class="x_x_moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">
gdal-dev@lists.osgeo.org</a> <a class="x_x_moz-txt-link-rfc2396E" href="mailto:gdal-dev@lists.osgeo.org">
<gdal-dev@lists.osgeo.org></a><br>
<b>Subject:</b> Re: [gdal-dev] NITF Int16 image size limit
<div> </div>
</div>
<div>
<p>Hi,</p>
<p>Actually the error message doesn't look like there's a uint32 overflow as it displays a file offset that is beyond the 4 GB limit.</p>
<p>Did you check that your file size is at least 4324329780 + 4194302 = 4 328 524 082 bytes ? I guess it is not, or then there's some mysterious error in the low level GDAL I/O layer.</p>
<p>So if your file is smaller than that,</p>
<p>- either it is corrupted (would be good if you could check that with a non-GDAL implementation)</p>
<p>- or there's a bug in the NITF driver in how it computes offsets. Could you report the values of the NITF_ABPP, NITF_IC and NITF_IMODE metadata reported by gdalinfo ?<br>
</p>
<p>Even<br>
</p>
<div class="x_x_x_moz-cite-prefix">Le 07/04/2021 à 02:09, ni hao a écrit :<br>
</div>
<blockquote type="cite">
<div><span>gdalinfo output i:</span><br>
</div>
<div><span>Driver: NITF/National Imagery Transmission Format
<div>Files: ##CH_CV_GRD\imagery\12499.ntf</div>
<div>Size is 61180, 19694</div>
<div>Coordinate System is `'</div>
<div>GCP Projection =</div>
GEOGCS["WGS 84", ... ]<br>
</span></div>
<div><span>Band 1 Block=1024x1024 Type=UInt16, ColorInterp=Gray</span></div>
<div><span>Band 2 Block=1024x1024 Type=UInt16, ColorInterp=Gray</span><br>
</div>
<div><br>
</div>
<div>Also I am sure it is the UInt16 overflow, as the message is:</div>
<div>ERROR 3: Unable to read 4194302 byte block from 4324329780<br>
</div>
<div>and 2**32 = 4294967296</div>
<div><br>
</div>
<hr tabindex="-1">
<div id="x_x_x_divRplyFwdMsg" dir="ltr"><b>From:</b> <a class="x_x_x_moz-txt-link-abbreviated" href="mailto:bradh@frogmouth.net">
bradh@frogmouth.net</a> <a class="x_x_x_moz-txt-link-rfc2396E" href="mailto:bradh@frogmouth.net">
<bradh@frogmouth.net></a><br>
<b>Sent:</b> April 6, 2021 8:17 PM<br>
<b>To:</b> Even Rouault <a class="x_x_x_moz-txt-link-rfc2396E" href="mailto:even.rouault@spatialys.com">
<even.rouault@spatialys.com></a>; ni hao <a class="x_x_x_moz-txt-link-rfc2396E" href="mailto:ni_hao88@hotmail.com">
<ni_hao88@hotmail.com></a><br>
<b>Cc:</b> <a class="x_x_x_moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">
gdal-dev@lists.osgeo.org</a> <a class="x_x_x_moz-txt-link-rfc2396E" href="mailto:gdal-dev@lists.osgeo.org">
<gdal-dev@lists.osgeo.org></a><br>
<b>Subject:</b> Re: [gdal-dev] NITF Int16 image size limit
<div> </div>
</div>
<div>
<div dir="auto">Also if you can show the headers (e.g. from gdalinfo), that might provide indication of exactly which value is the issue.
<div dir="auto"><br>
</div>
<div dir="auto">Brad</div>
</div>
<div class="x_x_x_x_gmail_extra"><br>
<div class="x_x_x_x_gmail_quote">On 7 Apr. 2021 04:00, Even Rouault <a class="x_x_x_moz-txt-link-rfc2396E" href="mailto:even.rouault@spatialys.com">
<even.rouault@spatialys.com></a> wrote:<br type="attribution">
<blockquote class="x_x_x_x_quote">
<div>
<p>Please keep the mailing list copied (I re-added it)</p>
<p>This is not the most recent one (3.2.2 is), but skimming quickly through release notes, I can't see anything directly related, so it might still be current with the recent versions. Altough I see that a bunch of "avoid unsigned integer overflow" type of
fixes have been done since 2.4, so some of them might have accidentally fixed the issue.</p>
<p>If you can try with the latest version ( Docker images mentioned at <a href="https://github.com/OSGeo/gdal/tree/master/gdal/docker">
https://github.com/OSGeo/gdal/tree/master/gdal/docker</a>, Conda, etc .), that could be interesting.<br>
</p>
<p><br>
</p>
<div>Le 06/04/2021 à 19:51, ni hao a écrit :<br>
</div>
<blockquote>
<div>Hi Even,</div>
<div><br>
</div>
<div>The version is 2.4.4.</div>
<hr>
<div dir="ltr"><b>From:</b> Even Rouault <a href="mailto:even.rouault@spatialys.com">
<even.rouault@spatialys.com></a><br>
<b>Sent:</b> April 6, 2021 2:46 PM<br>
<b>To:</b> ni hao <a href="mailto:ni_hao88@hotmail.com"><ni_hao88@hotmail.com></a>;
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a> <a href="mailto:gdal-dev@lists.osgeo.org">
<gdal-dev@lists.osgeo.org></a><br>
<b>Subject:</b> Re: [gdal-dev] NITF Int16 image size limit
<div> </div>
</div>
<div>
<p>Shawn,</p>
<p>This sounds more like a unwanted integer overflow somewhere in the NITF driver. Is this with a recent GDAL ? If so, please file a bug at
<a href="https://github.com/OSGeo/gdal/issues/new">https://github.com/OSGeo/gdal/issues/new</a> with all the details needed. A link to the dataset would be ideal, but otherwise please provide in the ticket description the output of "gdalinfo your.nitf"</p>
<p>Even<br>
</p>
<div>Le 06/04/2021 à 19:35, ni hao a écrit :<br>
</div>
<blockquote>
<div>
<p>Hi list,</p>
<p><br>
</p>
<p>I encountered problem ingesting a large NITF Int16 image with GDAL:</p>
<p>NITF 2-band image in Int16 format, 4.3 GB. It has 19690 lines x 61180 pixels. The upper 90% of the image looks fine. But the last 2000 lines fail to load by GDAL. That coincides with the 32-bit boundary. </p>
<p><br>
</p>
<p>Note that another larger NITF in 32-bit float, 10.1 GB file loads fine. </p>
<p><br>
</p>
<p>Is there a hard limit on NITF Int16 file size?</p>
<div>
<div>
<div>
<div> </div>
<div>Thanks,<br>
Shawn</div>
</div>
</div>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre>--
<a href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote>
<pre>--
<a href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="x_x_x_mimeAttachmentHeader"></fieldset>
<pre class="x_x_x_moz-quote-pre">_______________________________________________
gdal-dev mailing list
<a class="x_x_x_moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>
<a class="x_x_x_moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre class="x_x_x_moz-signature" cols="72">--
<a class="x_x_x_moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote>
<pre class="x_x_moz-signature" cols="72">--
<a class="x_x_moz-txt-link-freetext" href="http://www.spatialys.com">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</div>
</body>
</html>