[mapserver-users] Best way to import 4.5TB of imagery?

Mon Jun 10 15:48:22 PDT 2013

So, I'm guessing there's no easy way to automate this?  Even looking at the
states, some of the states are in two zones, and Texas is across 3 zones.
At least the naming convention of the files indicate the UTM zone.  For
instance:  m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17.  As far
as I can tell, all the files in a particular directory are all in the same
UTM zone.  I could create a layer for each UTM zone across CONUS, but that's
not going to be particularly useful to my users.  I'm thinking of making a
layer for each state.  For the stats that cross zones, there will probably
be two layers.  For Texas, there would be Texas_east, Texas_middle, and
Texas_west.  I will probably limit visibility until zoomed in sufficiently
to see the whole state on the screen anyway, since the continental view of
this data is pretty crappy anyway.  So now it seems like it will be a lot of
grunt work just copying these directories up to the server, and going
through and creating a shape file index for each state.  For states in more
than one UTM, there would be more than one shape.  Then I'll have to add a
layer for to my mapfile for each shapefile, using the correct projection.
Is there an easier way?  I'm starting with Oklahoma, which is also in three
UTM zones.  I'll get that working before moving on.  Any suggestions on
making this pretty would be welcomed.  :-)

-----Original Message-----
From: Stephen Woodbridge [mailto:woodbri at swoodbridge.com] 
Sent: Monday, June 10, 2013 12:34 PM
To: Evans, James R Civ USAF ACC 84 RADES/SCZE
Cc: mapserver-users at lists.osgeo.org
Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?

On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote:
> Hi Stephen,
> Thanks, for the reply.  I previously got 4 sample images from the 
> USDA, and was able to get them to work just fine.  There was no processing
required.
> The sample images I got were all from Utah, and they are NAD83, UTM zone
12.
> I added the 4 sample images to a shape file using gdaltindex.   I used
UPSG
> 26912 and mapserver served them up very quickly for such large files.
>
> Now I have this entire data set, and it stretches from UTM zone 10, to 
> UTM zone 19.  The data is divided into directories by two letter state 
> abbreviations, and under that into subdirectories.  I'm just wondering 
> how to add this to my mapfile.  Do I need a different entry for each UTM
Zone?
> How is it possible to get a single layer entry that includes multiple 
> projections?  This is looking like a huge job and I just want to know 
> the best strategy for getting this done.

So now you have a problem. You data is in UTM spread over 10 different
projections. What do you plane to do when have your image is zone 10 and
half is in zone 11 or if you zoom out and you images has 3-4 zones
displayed?

All data in an image must be rendered in the same projection. While I don't
doubt that your test with 4 images worked fine, did you you test this a
multiple zoom levels and at some point you will probably want to create a
super overlay image so you do not have to open multiple files to just pull a
tiny overlay out of each one.

Your use cases will determine how you want to deal with the data. For
example does it HAVE to be in a UTM projection, or can you work with a
Spherical Mercator or geographic projection? The end solution will be much
easier if you can work with one common projection over your whole data set.
Otherwise, you will have to deal with the transitions from one zone to the
next or maybe set up 10 separate servers that only serve one zone.

Having pushed larges amounts (4-25TB) of imagery data more than once it is
important to make these decisions up front and and prototype up something
like a 4-10 degree square across a UTM boundary and make sure that the
results are going to be what you expect before you process all the data.

-Steve

> Thanks,
> James
>
>
>
>
>
> -----Original Message-----
> From: mapserver-users-bounces at lists.osgeo.org
> [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Stephen 
> Woodbridge
> Sent: Friday, June 07, 2013 8:41 AM
> To: mapserver-users at lists.osgeo.org
> Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?
>
> On 6/7/2013 10:31 AM, James_in_Utah wrote:
>> Hi,
>> We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all 
>> of CONUS.  I think there's a total of about 400,000 jpgs.  The data 
>> is in directories, by states.  Under each state, there are 
>> subfolders, probably reference by longitude.  Other than going 
>> through folder by folder, adding each image to a shape file using 
>> gdaltindex, what's the best strategy for loading a couple of hundred 
>> thousand files up to our server and making the imagery available via 
>> our mapserver?  Should I maintain the current directory structure 
>> when I copy the imagery to the server, or just dump all of it into a 
>> single directory?  Do I want to stay with 1 shape file, or break it 
>> up by state?  We eventually want a contiguous layer for all of CONUS to
be served up to our users.
>
> James,
>
> Since imagery data is served via gdal, you might want to also ask this 
> question on the gdal list.
>
> There are issues with jpg related to the fact that if you only want a 
> small part of the image you still have to uncompress the whole image. 
> So part of the answer might be that you need to pre-process all the 
> imagery into something like a jpg compress tiled geotif or something else.
>
> You also need to consider what projection your imagery is in and what 
> projection you want to display it in. Because if you need to 
> preprocess the data, that would also be a good time to reproject it.
>
> Anyway the gdal list can probably ask additional questions to help 
> sort all that out.
>
> -Steve W
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6912 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20130610/a9707a3d/attachment.bin>