[gdal-dev] NITF Int16 image size limit
Even Rouault
even.rouault at spatialys.com
Wed Apr 7 06:41:22 PDT 2021
Hum, 4294967295 = 0xffffffff = max uint32 value. This really a
suspicious file size that looks like truncation at the 4 GB limit. Are
you really really sure that you tested the non-GDAL implementation with
that exact file, and not another version that might have been truncated
later ?
Le 07/04/2021 à 14:59, ni hao a écrit :
> Hi Even,
>
> I am using GDAL 2.4.4. I may try the latest version as you suggested.
>
> The NITF file size is 4,294,967,295 bytes
>
> The vendor used a non-GDAL implementation, and was able to display all
> image.
>
> Metadata:
> NITF_ABPP=16
> NITF_BLOCKA_BLOCK_COUNT=01
> NITF_BLOCKA_BLOCK_INSTANCE_01=01
> NITF_BLOCKA_FRFC_LOC_01=
> NITF_BLOCKA_FRLC_LOC_01=
> NITF_BLOCKA_LAYOVER_ANGLE_01=000
> NITF_BLOCKA_LRFC_LOC_01=
> NITF_BLOCKA_LRLC_LOC_01=
> NITF_BLOCKA_L_LINES_01=19694
> NITF_BLOCKA_N_GRAY_01=00000
> NITF_BLOCKA_SHADOW_ANGLE_01=180
> NITF_CCS_COLUMN=0
> NITF_CCS_ROW=0
> NITF_CLEVEL=07
> NITF_ENCRYP=0
> NITF_FBKGC= 0, 0, 0
> NITF_FDT=
> NITF_FHDR=NITF02.10
> NITF_FSCATP=
> NITF_FSCAUT=
> NITF_FSCLAS=U
> NITF_FSCLSY=CA
> NITF_FSCLTX=
> NITF_FSCODE=
> NITF_FSCOP=00000
> NITF_FSCPYS=00000
> NITF_FSCRSN=
> NITF_FSCTLH=
> NITF_FSCTLN=
> NITF_FSDCDT=
> NITF_FSDCTP=
> NITF_FSDCXM=
> NITF_FSDG=
> NITF_FSDGDT=
> NITF_FSREL=
> NITF_FSSRDT=
> NITF_FTITLE=
> NITF_IALVL=0
> NITF_IC=NC
> NITF_ICAT=SAR
> NITF_ICORDS=G
> NITF_IDATIM=
> NITF_IDLVL=
> NITF_IGEOLO=
> NITF_IID1=P1
> NITF_IID2=
> NITF_ILOC_COLUMN=0
> NITF_ILOC_ROW=0
> NITF_IMAG=1.0
> NITF_IMODE=P
> NITF_IREP=MULTI
> NITF_ISCATP=
> NITF_ISCAUT=
> NITF_ISCLAS=U
> NITF_ISCLSY=CA
> NITF_ISCLTX=
> NITF_ISCODE=
> NITF_ISCRSN=
> NITF_ISCTLH=
> NITF_ISCTLN=
> NITF_ISDCDT=
> NITF_ISDCTP=
> NITF_ISDCXM=
> NITF_ISDG=
> NITF_ISDGDT=
> NITF_ISORCE=
> NITF_ISREL=
> NITF_ISSRDT=
> NITF_ONAME=
> NITF_OPHONE=
> NITF_OSTAID=CSA
> NITF_PJUST=R
> NITF_PVTYPE=INT
> NITF_STYPE=BF01
> NITF_TGTID=
> ------------------------------------------------------------------------
> *From:* Even Rouault <even.rouault at spatialys.com>
> *Sent:* April 7, 2021 5:23 AM
> *To:* ni hao <ni_hao88 at hotmail.com>; gdal-dev at lists.osgeo.org
> <gdal-dev at lists.osgeo.org>
> *Subject:* Re: [gdal-dev] NITF Int16 image size limit
>
> Hi,
>
> Actually the error message doesn't look like there's a uint32 overflow
> as it displays a file offset that is beyond the 4 GB limit.
>
> Did you check that your file size is at least 4324329780 + 4194302 = 4
> 328 524 082 bytes ? I guess it is not, or then there's some mysterious
> error in the low level GDAL I/O layer.
>
> So if your file is smaller than that,
>
> - either it is corrupted (would be good if you could check that with a
> non-GDAL implementation)
>
> - or there's a bug in the NITF driver in how it computes offsets.
> Could you report the values of the NITF_ABPP, NITF_IC and NITF_IMODE
> metadata reported by gdalinfo ?
>
> Even
>
> Le 07/04/2021 à 02:09, ni hao a écrit :
>> gdalinfo output i:
>> Driver: NITF/National Imagery Transmission Format
>> Files: ##CH_CV_GRD\imagery\12499.ntf
>> Size is 61180, 19694
>> Coordinate System is `'
>> GCP Projection =
>> GEOGCS["WGS 84", ... ]
>> Band 1 Block=1024x1024 Type=UInt16, ColorInterp=Gray
>> Band 2 Block=1024x1024 Type=UInt16, ColorInterp=Gray
>>
>> Also I am sure it is the UInt16 overflow, as the message is:
>> ERROR 3: Unable to read 4194302 byte block from 4324329780
>> and 2**32 = 4294967296
>>
>> ------------------------------------------------------------------------
>> *From:* bradh at frogmouth.net <mailto:bradh at frogmouth.net>
>> <bradh at frogmouth.net> <mailto:bradh at frogmouth.net>
>> *Sent:* April 6, 2021 8:17 PM
>> *To:* Even Rouault <even.rouault at spatialys.com>
>> <mailto:even.rouault at spatialys.com>; ni hao <ni_hao88 at hotmail.com>
>> <mailto:ni_hao88 at hotmail.com>
>> *Cc:* gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
>> <gdal-dev at lists.osgeo.org> <mailto:gdal-dev at lists.osgeo.org>
>> *Subject:* Re: [gdal-dev] NITF Int16 image size limit
>> Also if you can show the headers (e.g. from gdalinfo), that might
>> provide indication of exactly which value is the issue.
>>
>> Brad
>>
>> On 7 Apr. 2021 04:00, Even Rouault <even.rouault at spatialys.com>
>> <mailto:even.rouault at spatialys.com> wrote:
>>
>> Please keep the mailing list copied (I re-added it)
>>
>> This is not the most recent one (3.2.2 is), but skimming quickly
>> through release notes, I can't see anything directly related, so
>> it might still be current with the recent versions. Altough I see
>> that a bunch of "avoid unsigned integer overflow" type of fixes
>> have been done since 2.4, so some of them might have accidentally
>> fixed the issue.
>>
>> If you can try with the latest version ( Docker images mentioned
>> at https://github.com/OSGeo/gdal/tree/master/gdal/docker
>> <https://github.com/OSGeo/gdal/tree/master/gdal/docker>, Conda,
>> etc .), that could be interesting.
>>
>>
>> Le 06/04/2021 à 19:51, ni hao a écrit :
>>
>> Hi Even,
>>
>> The version is 2.4.4.
>> ------------------------------------------------------------------------
>> *From:* Even Rouault <even.rouault at spatialys.com>
>> <mailto:even.rouault at spatialys.com>
>> *Sent:* April 6, 2021 2:46 PM
>> *To:* ni hao <ni_hao88 at hotmail.com>
>> <mailto:ni_hao88 at hotmail.com>; gdal-dev at lists.osgeo.org
>> <mailto:gdal-dev at lists.osgeo.org> <gdal-dev at lists.osgeo.org>
>> <mailto:gdal-dev at lists.osgeo.org>
>> *Subject:* Re: [gdal-dev] NITF Int16 image size limit
>>
>> Shawn,
>>
>> This sounds more like a unwanted integer overflow somewhere
>> in the NITF driver. Is this with a recent GDAL ? If so,
>> please file a bug at https://github.com/OSGeo/gdal/issues/new
>> <https://github.com/OSGeo/gdal/issues/new> with all the
>> details needed. A link to the dataset would be ideal, but
>> otherwise please provide in the ticket description the output
>> of "gdalinfo your.nitf"
>>
>> Even
>>
>> Le 06/04/2021 à 19:35, ni hao a écrit :
>>
>> Hi list,
>>
>>
>> I encountered problem ingesting a large NITF Int16 image
>> with GDAL:
>>
>> NITF 2-band image in Int16 format, 4.3 GB. It has 19690
>> lines x 61180 pixels. The upper 90% of the image looks
>> fine. But the last 2000 lines fail to load by GDAL.
>> That coincides with the 32-bit boundary.
>>
>>
>> Note that another larger NITF in 32-bit float, 10.1 GB
>> file loads fine.
>>
>>
>> Is there a hard limit on NITF Int16 file size?
>>
>> Thanks,
>> Shawn
>>
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev <https://lists.osgeo.org/mailman/listinfo/gdal-dev>
>>
>> --
>> http://www.spatialys.com <http://www.spatialys.com>
>> My software is free, but my time generally not.
>>
>> --
>> http://www.spatialys.com <http://www.spatialys.com>
>> My software is free, but my time generally not.
>>
>>
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev <https://lists.osgeo.org/mailman/listinfo/gdal-dev>
> --
> http://www.spatialys.com <http://www.spatialys.com>
> My software is free, but my time generally not.
--
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210407/8b9a6862/attachment-0001.html>
More information about the gdal-dev
mailing list