HI Martin,<div><br></div><div>Our application renders the data and then does some processing on it after it is loaded based on user request. Our application can typically load and display very large files in a few seconds and navigate them interactively very quickly. After doing some tests, I don't believe there is a problem with NITF read performance is slow using GDAL because I could render and load some very big files on my end with no problem.</div>
<div><br></div><div>Thanks for everyone's help, I appreciate it!</div><div><br></div><div>Jason </div><div><br><div class="gmail_quote">On Wed, May 14, 2008 at 7:08 PM, Martin Chapman <<a href="mailto:mchapman@texelinc.com">mchapman@texelinc.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p><span style="font-size:11.0pt;color:#1F497D">Jason,</span></p>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">Is your program rendering the data to the screen or just processing
it somehow? As a bench mark for you, I can render nitf files that are 1.8GB
and bigger in one to two seconds on my machine using gdal over a samba network connection
to a linux machine (100MB/sec Ethernet). The trick is resampling the image on
the fly using rasterio() to fit the screen size. There is a HUGE performance
difference when you do this. </span></p>
<p><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D"> </span></p>
<p><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">If you are just making copies of the image then I would look at
your physical disk data subsystem. If you do not have a write cache enabler
then it may be your customers disk that is slow and not the software. I have
run into that situation before with HP Smart Arrays. You can validate that
this is the case by comparing your CPU usage to the average write disk queue
length. If the CPU usage is less than your write queue it means your data
subsystem is not able to keep up with the CPU…basically GDAL is faster
than your hardware. Your CPU usage should be somewhere between 70 and 100 % is
everything is working fine would be my guess.</span></p>
<p><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D"> </span></p>
<p><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">Martin</span></p>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p><b><span style="font-size:10.0pt">From:</span></b><span style="font-size:10.0pt">
<a href="mailto:gdal-dev-bounces@lists.osgeo.org" target="_blank">gdal-dev-bounces@lists.osgeo.org</a> [mailto:<a href="mailto:gdal-dev-bounces@lists.osgeo.org" target="_blank">gdal-dev-bounces@lists.osgeo.org</a>] <b>On
Behalf Of </b>Jason Beverage<br>
<b>Sent:</b> Wednesday, May 14, 2008 12:24 PM<br>
<b>To:</b> Even Rouault<br>
<b>Cc:</b> <a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<b>Subject:</b> Re: [gdal-dev] Large NITF file performance problems</span></p>
</div><div><div></div><div class="Wj3C7c">
<p> </p>
<p style="margin-bottom:12.0pt">Hi Even,<br>
<br>
Thanks for your sample file, it really helped. It loads quickly in our
application and in OpenEV so I'll have to look down another route to find out
what's wrong with the customer's data.<br>
<br>
However, it does appear that there is a problem somewhere when doing a
CreateCopy() as Ivan pointed out. I can translate that file to a GeoTiff
in about 40seconds, but writing NITF has been going for about 5 minutes now and
its still on "0" with no progress reported.<br>
<br>
Thanks,<br>
<br>
Jason</p>
<div>
<p>On Tue, May 13, 2008 at 2:45 PM, Even Rouault <<a href="mailto:even.rouault@mines-paris.org" target="_blank">even.rouault@mines-paris.org</a>>
wrote:</p>
<p>Hi,<br>
<br>
I don't really understand why they would have issues with large file sizes<br>
(below the 4 GB of course)<br>
<br>
I've tried the following small python script to generate a 1.2 GB NITF file<br>
filled with '1' as a value :<br>
<br>
#!/usr/bin/env python<br>
<br>
import gdal<br>
<br>
new_ds = gdal.GetDriverByName( 'NITF' ).Create( 'largentf.ntf', 100000, 12000,<br>
1 )<br>
new_ds.GetRasterBand(1).Fill(1)<br>
new_ds = None<br>
<br>
<br>
It runs in less than one minute on my slow machine. I can open the resulting<br>
file with OpenEV and scroll through it quite smoothly. gdalinfo -checksum<br>
largetntf.ntf also runs in about 3 minutes, which seems reasonnable.<br>
<br>
a gdalinfo on the file shows that it is automatically tiled in blocks of<br>
256x256 and by looking at the code I can see that 256x256 tiling is<br>
automatically activated when either the number of lines or columns of the<br>
file is larger than 9999.<br>
<br>
So I don't see any obvious reason why you get poor performance. You could<br>
probably break with a debugger to see where it idles during gdal_translate ?<br>
<br>
I email you the bzip2 result of the 1.2 GB file that is only 1KB... so you can<br>
test on the same file as me.<br>
<br>
As far as your customer is concerned, maybe there is an issue with compression<br>
(for example a very large mono-block JPEG image ?). A 'gdalinfo' on the files<br>
could maybe give some hints ?<br>
<br>
Le Tuesday 13 May 2008 18:21:15 Jason Beverage, vous avez écrit :</p>
<div>
<div>
<p style="margin-bottom:12.0pt">> Hi all,<br>
><br>
> I've got a customer who is using large NITF files (~1.5 GB) and is seeing<br>
> ridiculously slow load times in our application. I don't have access
to<br>
> his data, so I can't test directly, but it seems like the NITF driver may<br>
> have some issues with large file sizes (> 1GB).<br>
><br>
> To test on my end, I created a few different GeoTiff files (600 MB, 800
MB,<br>
> and 1.2 GB) and tried to convert them to NITF using gdal_translate.<br>
> Converting the 600 and 800 MB files worked just fine and had very<br>
> reasonable speed (few minutes). However, when I tried to use<br>
> gdal_translate on the 1.2 GB file, the process hung at 0% for forever and
I<br>
> had to kill it after waiting for a very long time. It seems as if
there is<br>
> something magical about this 1 GB boundary.<br>
><br>
> Does anyone have any ideas or suggestions as to what could be causing this<br>
> issue?<br>
><br>
> Thanks!<br>
><br>
> Jason<br>
<br>
</p>
</div>
</div>
</div>
<p> </p>
</div></div></div>
</div>
</blockquote></div><br></div>