[mapserver-users] Best way to import 4.5TB of imagery?

[Stephen Woodbridge]

On 6/7/2013 10:31 AM, James_in_Utah wrote:
> Hi,
> We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all of
> CONUS.  I think there's a total of about 400,000 jpgs.  The data is in
> directories, by states.  Under each state, there are subfolders, probably
> reference by longitude.  Other than going through folder by folder, adding
> each image to a shape file using gdaltindex, what's the best strategy for
> loading a couple of hundred thousand files up to our server and making the
> imagery available via our mapserver?  Should I maintain the current
> directory structure when I copy the imagery to the server, or just dump all
> of it into a single directory?  Do I want to stay with 1 shape file, or
> break it up by state?  We eventually want a contiguous layer for all of
> CONUS to be served up to our users.

James,

Since imagery data is served via gdal, you might want to also ask this 
question on the gdal list.

There are issues with jpg related to the fact that if you only want a 
small part of the image you still have to uncompress the whole image. So 
part of the answer might be that you need to pre-process all the imagery 
into something like a jpg compress tiled geotif or something else.

You also need to consider what projection your imagery is in and what 
projection you want to display it in. Because if you need to preprocess 
the data, that would also be a good time to reproject it.

Anyway the gdal list can probably ask additional questions to help sort 
all that out.

-Steve W