[mapserver-users] Best way to import 4.5TB of imagery?

Mon Jun 10 19:42:11 PDT 2013

I have used gdalwarp to do this in the past and find or a perl script. I 
have not used global mapper for anything so I can't compare.

-Steve

On 6/10/2013 10:00 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote:
>
> For some reason I didn't see the reply from Robert Sanson.  Anyway,
> we have no requirement to stay with NAD83.  I have global mapper, and
> I think it will do a bulk reprojection, from my local hard drive to a
> network drive.  Maybe I will take a look at that tomorrow.  Is there
> a better tool for that?  Some GDAL utility?  Anyway, thanks for all
> the great suggestions. James
>
>
> -----Original Message----- From:
> mapserver-users-bounces at lists.osgeo.org on behalf of Stephen
> Woodbridge Sent: Mon 6/10/2013 6:36 PM To:
> mapserver-users at lists.osgeo.org Subject: Re: [mapserver-users] Best
> way to import 4.5TB of imagery?
>
> On 6/10/2013 7:07 PM, Robert Sanson wrote:
>> Sounds all very complicated. I would advise that you choose a
>> single projection that will work across your entire are and then
>> re-project your imagery first, and then build your image datastore
>> around that.
>
> I would agree with this that if htere are not other contraints on
> the problem this is often the best way to go. But that will not owrk
> if his users HAVE to have the data in UTM projection for some
> reason.
>
> -Steve
>
>>>>> "Evans, James R Civ USAF ACC 84 RADES/SCZE"
>>>>> <James.Evans at hill.af.mil> 11/06/2013 10:48 a.m. >>>
>> So, I'm guessing there's no easy way to automate this?  Even
>> looking at the states, some of the states are in two zones, and
>> Texas is across 3 zones. At least the naming convention of the
>> files indicate the UTM zone.  For instance:
>> m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17.  As far as
>> I can tell, all the files in a particular directory are all in the
>> same UTM zone.  I could create a layer for each UTM zone across
>> CONUS, but that's not going to be particularly useful to my users.
>> I'm thinking of making a layer for each state.  For the stats that
>> cross zones, there will probably be two layers.  For Texas, there
>> would be Texas_east, Texas_middle, and Texas_west.  I will
>> probably limit visibility until zoomed in sufficiently to see the
>> whole state on the screen anyway, since the continental view of
>> this data is pretty crappy anyway.  So now it seems like it will be
>> a lot of grunt work just copying these directories up to the
>> server, and going through and creating a shape file index for each
>> state.  For states in more than one UTM, there would be more than
>> one shape.  Then I'll have to add a layer for to my mapfile for
>> each shapefile, using the correct projection. Is there an easier
>> way?  I'm starting with Oklahoma, which is also in three UTM zones.
>> I'll get that working before moving on.  Any suggestions on making
>> this pretty would be welcomed.  :-)
>>
>>
>>
>>
>> -----Original Message----- From: Stephen Woodbridge
>> [mailto:woodbri at swoodbridge.com] Sent: Monday, June 10, 2013 12:34
>> PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc:
>> mapserver-users at lists.osgeo.org Subject: Re: [mapserver-users]
>> Best way to import 4.5TB of imagery?
>>
>> On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE
>> wrote:
>>> Hi Stephen, Thanks, for the reply.  I previously got 4 sample
>>> images from the USDA, and was able to get them to work just
>>> fine. There was no processing
>> required.
>>> The sample images I got were all from Utah, and they are NAD83,
>>> UTM zone
>> 12.
>>> I added the 4 sample images to a shape file using gdaltindex.
>>> I used
>> UPSG
>>> 26912 and mapserver served them up very quickly for such large
>>> files.
>>>
>>> Now I have this entire data set, and it stretches from UTM zone
>>> 10, to UTM zone 19.  The data is divided into directories by two
>>> letter state abbreviations, and under that into subdirectories.
>>> I'm just wondering how to add this to my mapfile.  Do I need a
>>> different entry for each UTM
>> Zone?
>>> How is it possible to get a single layer entry that includes
>>> multiple projections?  This is looking like a huge job and I
>>> just want to know the best strategy for getting this done.
>>
>> So now you have a problem. You data is in UTM spread over 10
>> different projections. What do you plane to do when have your
>> image is zone 10 and half is in zone 11 or if you zoom out and you
>> images has 3-4 zones displayed?
>>
>> All data in an image must be rendered in the same projection. While
>> I don't doubt that your test with 4 images worked fine, did you
>> you test this a multiple zoom levels and at some point you will
>> probably want to create a super overlay image so you do not have to
>> open multiple files to just pull a tiny overlay out of each one.
>>
>> Your use cases will determine how you want to deal with the data.
>> For example does it HAVE to be in a UTM projection, or can you
>> work with a Spherical Mercator or geographic projection? The end
>> solution will be much easier if you can work with one common
>> projection over your whole data set. Otherwise, you will have to
>> deal with the transitions from one zone to the next or maybe set up
>> 10 separate servers that only serve one zone.
>>
>> Having pushed larges amounts (4-25TB) of imagery data more than
>> once it is important to make these decisions up front and and
>> prototype up something like a 4-10 degree square across a UTM
>> boundary and make sure that the results are going to be what you
>> expect before you process all the data.
>>
>> -Steve
>>
>>> Thanks, James
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message----- From:
>>> mapserver-users-bounces at lists.osgeo.org
>>> [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of
>>> Stephen Woodbridge Sent: Friday, June 07, 2013 8:41 AM To:
>>> mapserver-users at lists.osgeo.org Subject: Re: [mapserver-users]
>>> Best way to import 4.5TB of imagery?
>>>
>>> On 6/7/2013 10:31 AM, James_in_Utah wrote:
>>>> Hi, We just got 3 hard drive, loaded with 4.5TB of NAIP
>>>> imagery for all of CONUS.  I think there's a total of about
>>>> 400,000 jpgs. The data is in directories, by states.  Under
>>>> each state, there are subfolders, probably reference by
>>>> longitude.  Other than going through folder by folder, adding
>>>> each image to a shape file using gdaltindex, what's the best
>>>> strategy for loading a couple of hundred thousand files up to
>>>> our server and making the imagery available via our mapserver?
>>>> Should I maintain the current directory structure when I copy
>>>> the imagery to the server, or just dump all of it into a single
>>>> directory?  Do I want to stay with 1 shape file, or break it up
>>>> by state?  We eventually want a contiguous layer for all of
>>>> CONUS to
>> be served up to our users.
>>>
>>> James,
>>>
>>> Since imagery data is served via gdal, you might want to also
>>> ask this question on the gdal list.
>>>
>>> There are issues with jpg related to the fact that if you only
>>> want a small part of the image you still have to uncompress the
>>> whole image. So part of the answer might be that you need to
>>> pre-process all the imagery into something like a jpg compress
>>> tiled geotif or something else.
>>>
>>> You also need to consider what projection your imagery is in and
>>> what projection you want to display it in. Because if you need
>>> to preprocess the data, that would also be a good time to
>>> reproject it.
>>>
>>> Anyway the gdal list can probably ask additional questions to
>>> help sort all that out.
>>>
>>> -Steve W
>>>
>>> _______________________________________________ mapserver-users
>>> mailing list mapserver-users at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>>
>>
>>
>>
>> This email and any attachments are confidential and intended
>> solely for the addressee(s). If you are not the intended recipient,
>> please notify us immediately and then delete this email from your
>> system.
>>
>> This message has been scanned for Malware and Viruses by Websense
>> Hosted Security. www.websense.com
>> _______________________________________________ mapserver-users
>> mailing list mapserver-users at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>
>
> _______________________________________________ mapserver-users
> mailing list mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>