[mapserver-users] Best way to import 4.5TB of imagery?

[Stephen Woodbridge]

On 6/10/2013 6:48 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote:
> So, I'm guessing there's no easy way to automate this?  Even looking at the
> states, some of the states are in two zones, and Texas is across 3 zones.
> At least the naming convention of the files indicate the UTM zone.  For
> instance:  m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17.  As far
> as I can tell, all the files in a particular directory are all in the same
> UTM zone.  I could create a layer for each UTM zone across CONUS, but that's
> not going to be particularly useful to my users.  I'm thinking of making a
> layer for each state.  For the stats that cross zones, there will probably
> be two layers.  For Texas, there would be Texas_east, Texas_middle, and
> Texas_west.  I will probably limit visibility until zoomed in sufficiently
> to see the whole state on the screen anyway, since the continental view of
> this data is pretty crappy anyway.  So now it seems like it will be a lot of
> grunt work just copying these directories up to the server, and going
> through and creating a shape file index for each state.  For states in more
> than one UTM, there would be more than one shape.  Then I'll have to add a
> layer for to my mapfile for each shapefile, using the correct projection.
> Is there an easier way?  I'm starting with Oklahoma, which is also in three
> UTM zones.  I'll get that working before moving on.  Any suggestions on
> making this pretty would be welcomed.  :-)

If you are using something like OpenLayers for your client you can code 
smarts into that to change mapfiles based on the zone. I think I would 
start with a Perl script that takes every image file and creates a 
record in a postgresql/postgis database that represents the extents of 
the image.

path/to/image/file.jpg | wgs84_geom | utm_geom | zone

or something like that. Then you can create mapfiles that displays these 
rectangles in wgs84, or work out your logic for display the polygons in 
shifting zones. This is quick and easy to setup and lets you proceed 
with the disgn of that with waiting on imagery and it wil be faster.

Beyond that, pushing around 4.5 TB of imagery is not going to be fast.

Good luck,
   -Steve W

> -----Original Message-----
> From: Stephen Woodbridge [mailto:woodbri at swoodbridge.com]
> Sent: Monday, June 10, 2013 12:34 PM
> To: Evans, James R Civ USAF ACC 84 RADES/SCZE
> Cc: mapserver-users at lists.osgeo.org
> Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?
>
> On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote:
>> Hi Stephen,
>> Thanks, for the reply.  I previously got 4 sample images from the
>> USDA, and was able to get them to work just fine.  There was no processing
> required.
>> The sample images I got were all from Utah, and they are NAD83, UTM zone
> 12.
>> I added the 4 sample images to a shape file using gdaltindex.   I used
> UPSG
>> 26912 and mapserver served them up very quickly for such large files.
>>
>> Now I have this entire data set, and it stretches from UTM zone 10, to
>> UTM zone 19.  The data is divided into directories by two letter state
>> abbreviations, and under that into subdirectories.  I'm just wondering
>> how to add this to my mapfile.  Do I need a different entry for each UTM
> Zone?
>> How is it possible to get a single layer entry that includes multiple
>> projections?  This is looking like a huge job and I just want to know
>> the best strategy for getting this done.
>
> So now you have a problem. You data is in UTM spread over 10 different
> projections. What do you plane to do when have your image is zone 10 and
> half is in zone 11 or if you zoom out and you images has 3-4 zones
> displayed?
>
> All data in an image must be rendered in the same projection. While I don't
> doubt that your test with 4 images worked fine, did you you test this a
> multiple zoom levels and at some point you will probably want to create a
> super overlay image so you do not have to open multiple files to just pull a
> tiny overlay out of each one.
>
> Your use cases will determine how you want to deal with the data. For
> example does it HAVE to be in a UTM projection, or can you work with a
> Spherical Mercator or geographic projection? The end solution will be much
> easier if you can work with one common projection over your whole data set.
> Otherwise, you will have to deal with the transitions from one zone to the
> next or maybe set up 10 separate servers that only serve one zone.
>
> Having pushed larges amounts (4-25TB) of imagery data more than once it is
> important to make these decisions up front and and prototype up something
> like a 4-10 degree square across a UTM boundary and make sure that the
> results are going to be what you expect before you process all the data.
>
> -Steve
>
>> Thanks,
>> James
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: mapserver-users-bounces at lists.osgeo.org
>> [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Stephen
>> Woodbridge
>> Sent: Friday, June 07, 2013 8:41 AM
>> To: mapserver-users at lists.osgeo.org
>> Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?
>>
>> On 6/7/2013 10:31 AM, James_in_Utah wrote:
>>> Hi,
>>> We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all
>>> of CONUS.  I think there's a total of about 400,000 jpgs.  The data
>>> is in directories, by states.  Under each state, there are
>>> subfolders, probably reference by longitude.  Other than going
>>> through folder by folder, adding each image to a shape file using
>>> gdaltindex, what's the best strategy for loading a couple of hundred
>>> thousand files up to our server and making the imagery available via
>>> our mapserver?  Should I maintain the current directory structure
>>> when I copy the imagery to the server, or just dump all of it into a
>>> single directory?  Do I want to stay with 1 shape file, or break it
>>> up by state?  We eventually want a contiguous layer for all of CONUS to
> be served up to our users.
>>
>> James,
>>
>> Since imagery data is served via gdal, you might want to also ask this
>> question on the gdal list.
>>
>> There are issues with jpg related to the fact that if you only want a
>> small part of the image you still have to uncompress the whole image.
>> So part of the answer might be that you need to pre-process all the
>> imagery into something like a jpg compress tiled geotif or something else.
>>
>> You also need to consider what projection your imagery is in and what
>> projection you want to display it in. Because if you need to
>> preprocess the data, that would also be a good time to reproject it.
>>
>> Anyway the gdal list can probably ask additional questions to help
>> sort all that out.
>>
>> -Steve W
>>
>> _______________________________________________
>> mapserver-users mailing list
>> mapserver-users at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>