[gdal-dev] NITF Int16 image size limit

ni hao ni_hao88 at hotmail.com
Wed Apr 7 05:59:09 PDT 2021 

Hi Even,

I am using GDAL 2.4.4. I may try the latest version as you suggested.

The NITF file size is 4,294,967,295 bytes

The vendor used a non-GDAL implementation, and was able to display all image.

Metadata:
  NITF_ABPP=16
  NITF_BLOCKA_BLOCK_COUNT=01
  NITF_BLOCKA_BLOCK_INSTANCE_01=01
  NITF_BLOCKA_FRFC_LOC_01=
  NITF_BLOCKA_FRLC_LOC_01=
  NITF_BLOCKA_LAYOVER_ANGLE_01=000
  NITF_BLOCKA_LRFC_LOC_01=
  NITF_BLOCKA_LRLC_LOC_01=
  NITF_BLOCKA_L_LINES_01=19694
  NITF_BLOCKA_N_GRAY_01=00000
  NITF_BLOCKA_SHADOW_ANGLE_01=180
  NITF_CCS_COLUMN=0
  NITF_CCS_ROW=0
  NITF_CLEVEL=07
  NITF_ENCRYP=0
  NITF_FBKGC=  0,  0,  0
  NITF_FDT=
  NITF_FHDR=NITF02.10
  NITF_FSCATP=
  NITF_FSCAUT=
  NITF_FSCLAS=U
  NITF_FSCLSY=CA
  NITF_FSCLTX=
  NITF_FSCODE=
  NITF_FSCOP=00000
  NITF_FSCPYS=00000
  NITF_FSCRSN=
  NITF_FSCTLH=
  NITF_FSCTLN=
  NITF_FSDCDT=
  NITF_FSDCTP=
  NITF_FSDCXM=
  NITF_FSDG=
  NITF_FSDGDT=
  NITF_FSREL=
  NITF_FSSRDT=
  NITF_FTITLE=
  NITF_IALVL=0
  NITF_IC=NC
  NITF_ICAT=SAR
  NITF_ICORDS=G
  NITF_IDATIM=
  NITF_IDLVL=
  NITF_IGEOLO=
  NITF_IID1=P1
  NITF_IID2=
  NITF_ILOC_COLUMN=0
  NITF_ILOC_ROW=0
  NITF_IMAG=1.0
  NITF_IMODE=P
  NITF_IREP=MULTI
  NITF_ISCATP=
  NITF_ISCAUT=
  NITF_ISCLAS=U
  NITF_ISCLSY=CA
  NITF_ISCLTX=
  NITF_ISCODE=
  NITF_ISCRSN=
  NITF_ISCTLH=
  NITF_ISCTLN=
  NITF_ISDCDT=
  NITF_ISDCTP=
  NITF_ISDCXM=
  NITF_ISDG=
  NITF_ISDGDT=
  NITF_ISORCE=
  NITF_ISREL=
  NITF_ISSRDT=
  NITF_ONAME=
  NITF_OPHONE=
  NITF_OSTAID=CSA
  NITF_PJUST=R
  NITF_PVTYPE=INT
  NITF_STYPE=BF01
  NITF_TGTID=
________________________________
From: Even Rouault <even.rouault at spatialys.com>
Sent: April 7, 2021 5:23 AM
To: ni hao <ni_hao88 at hotmail.com>; gdal-dev at lists.osgeo.org <gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] NITF Int16 image size limit


Hi,

Actually the error message doesn't look like there's a uint32 overflow as it displays a file offset that is beyond the 4 GB limit.

Did you check that your file size is at least 4324329780 + 4194302 = 4 328 524 082 bytes ? I guess it is not, or then there's some mysterious error in the low level GDAL I/O layer.

So if your file is smaller than that,

- either it is corrupted (would be good if you could check that with a non-GDAL implementation)

- or there's a bug in the NITF driver in how it computes offsets. Could you report the values of the NITF_ABPP, NITF_IC and NITF_IMODE metadata reported by gdalinfo ?

Even

Le 07/04/2021 à 02:09, ni hao a écrit :
gdalinfo output i:
Driver: NITF/National Imagery Transmission Format
Files: ##CH_CV_GRD\imagery\12499.ntf
Size is 61180, 19694
Coordinate System is `'
GCP Projection =
GEOGCS["WGS 84",   ... ]
Band 1 Block=1024x1024 Type=UInt16, ColorInterp=Gray
Band 2 Block=1024x1024 Type=UInt16, ColorInterp=Gray

Also I am sure it is the UInt16 overflow, as the message is:
ERROR 3: Unable to read 4194302 byte block from 4324329780
and 2**32 = 4294967296

________________________________
From: bradh at frogmouth.net<mailto:bradh at frogmouth.net> <bradh at frogmouth.net><mailto:bradh at frogmouth.net>
Sent: April 6, 2021 8:17 PM
To: Even Rouault <even.rouault at spatialys.com><mailto:even.rouault at spatialys.com>; ni hao <ni_hao88 at hotmail.com><mailto:ni_hao88 at hotmail.com>
Cc: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org> <gdal-dev at lists.osgeo.org><mailto:gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] NITF Int16 image size limit

Also if you can show the headers (e.g. from gdalinfo), that might provide indication of exactly which value is the issue.

Brad

On 7 Apr. 2021 04:00, Even Rouault <even.rouault at spatialys.com><mailto:even.rouault at spatialys.com> wrote:

Please keep the mailing list copied (I re-added it)

This is not the most recent one (3.2.2 is), but skimming quickly through release notes, I can't see anything directly related, so it might still be current with the recent versions. Altough I see that a bunch of "avoid unsigned integer overflow" type of fixes have been done since 2.4, so some of them might have accidentally fixed the issue.

If you can try with the latest version ( Docker images mentioned at https://github.com/OSGeo/gdal/tree/master/gdal/docker, Conda, etc .), that could be interesting.


Le 06/04/2021 à 19:51, ni hao a écrit :
Hi Even,

The version is 2.4.4.
________________________________
From: Even Rouault <even.rouault at spatialys.com><mailto:even.rouault at spatialys.com>
Sent: April 6, 2021 2:46 PM
To: ni hao <ni_hao88 at hotmail.com><mailto:ni_hao88 at hotmail.com>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org> <gdal-dev at lists.osgeo.org><mailto:gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] NITF Int16 image size limit


Shawn,

This sounds more like a unwanted integer overflow somewhere in the NITF driver.  Is this with a recent GDAL ? If so, please file a bug at https://github.com/OSGeo/gdal/issues/new with all the details needed. A link to the dataset would be ideal, but otherwise please provide in the ticket description the output of "gdalinfo your.nitf"

Even

Le 06/04/2021 à 19:35, ni hao a écrit :

Hi list,


I encountered problem ingesting a large NITF Int16 image with GDAL:

NITF 2-band image in Int16 format, 4.3 GB. It has 19690 lines x 61180 pixels.   The upper 90% of the image looks fine.   But the last 2000 lines fail to load by GDAL.   That coincides with the 32-bit boundary.


Note that another larger NITF in 32-bit float, 10.1 GB file loads fine.


Is there a hard limit on NITF Int16 file size?


Thanks,
Shawn




_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

--
http://www.spatialys.com
My software is free, but my time generally not.




_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210407/0947346b/attachment-0001.html>

More information about the gdal-dev mailing list