[mapserver-users] Best way to import 4.5TB of imagery?

[Stephen Woodbridge]

On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote:
> Hi Stephen,
> Thanks, for the reply.  I previously got 4 sample images from the USDA, and
> was able to get them to work just fine.  There was no processing required.
> The sample images I got were all from Utah, and they are NAD83, UTM zone 12.
> I added the 4 sample images to a shape file using gdaltindex.   I used UPSG
> 26912 and mapserver served them up very quickly for such large files.
>
> Now I have this entire data set, and it stretches from UTM zone 10, to UTM
> zone 19.  The data is divided into directories by two letter state
> abbreviations, and under that into subdirectories.  I'm just wondering how
> to add this to my mapfile.  Do I need a different entry for each UTM Zone?
> How is it possible to get a single layer entry that includes multiple
> projections?  This is looking like a huge job and I just want to know the
> best strategy for getting this done.

So now you have a problem. You data is in UTM spread over 10 different 
projections. What do you plane to do when have your image is zone 10 and 
half is in zone 11 or if you zoom out and you images has 3-4 zones 
displayed?

All data in an image must be rendered in the same projection. While I 
don't doubt that your test with 4 images worked fine, did you you test 
this a multiple zoom levels and at some point you will probably want to 
create a super overlay image so you do not have to open multiple files 
to just pull a tiny overlay out of each one.

Your use cases will determine how you want to deal with the data. For 
example does it HAVE to be in a UTM projection, or can you work with a 
Spherical Mercator or geographic projection? The end solution will be 
much easier if you can work with one common projection over your whole 
data set. Otherwise, you will have to deal with the transitions from one 
zone to the next or maybe set up 10 separate servers that only serve one 
zone.

Having pushed larges amounts (4-25TB) of imagery data more than once it 
is important to make these decisions up front and and prototype up 
something like a 4-10 degree square across a UTM boundary and make sure 
that the results are going to be what you expect before you process all 
the data.

-Steve

> Thanks,
> James
>
>
>
>
>
> -----Original Message-----
> From: mapserver-users-bounces at lists.osgeo.org
> [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Stephen
> Woodbridge
> Sent: Friday, June 07, 2013 8:41 AM
> To: mapserver-users at lists.osgeo.org
> Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?
>
> On 6/7/2013 10:31 AM, James_in_Utah wrote:
>> Hi,
>> We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all of
>> CONUS.  I think there's a total of about 400,000 jpgs.  The data is in
>> directories, by states.  Under each state, there are subfolders,
>> probably reference by longitude.  Other than going through folder by
>> folder, adding each image to a shape file using gdaltindex, what's the
>> best strategy for loading a couple of hundred thousand files up to our
>> server and making the imagery available via our mapserver?  Should I
>> maintain the current directory structure when I copy the imagery to
>> the server, or just dump all of it into a single directory?  Do I want
>> to stay with 1 shape file, or break it up by state?  We eventually
>> want a contiguous layer for all of CONUS to be served up to our users.
>
> James,
>
> Since imagery data is served via gdal, you might want to also ask this
> question on the gdal list.
>
> There are issues with jpg related to the fact that if you only want a small
> part of the image you still have to uncompress the whole image. So part of
> the answer might be that you need to pre-process all the imagery into
> something like a jpg compress tiled geotif or something else.
>
> You also need to consider what projection your imagery is in and what
> projection you want to display it in. Because if you need to preprocess the
> data, that would also be a good time to reproject it.
>
> Anyway the gdal list can probably ask additional questions to help sort all
> that out.
>
> -Steve W
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>