[Landsat-pds] Converting MTL.txt to JSON

Tue Mar 3 00:29:08 PST 2015

Jeff,

This looks promising.  Can you prepare a pull request incorporate this into
the landsat ingestor (https://github.com/landsat-pds/landsat_ingestor)?
Ideally it would be a module that can be called from l8_process_scene.py
and reprocess_scene.py in a manner similar to how the scene_index_maker
code is invoked.

Best regards,
Frank

-

On Thu, Feb 26, 2015 at 9:51 AM, Layton, Jeff <jefflay at amazon.com> wrote:

>  I think I finally have it. I’m attaching the original MTL file, the JSON
> output (testfile4) and the code I used to create it (test7.py).
>
>
>
> Jeff
>
>
>
>
>
>
>
> *From:* landsat-pds-bounces at lists.osgeo.org [mailto:
> landsat-pds-bounces at lists.osgeo.org] *On Behalf Of *Layton, Jeff
> *Sent:* Thursday, February 26, 2015 12:44 PM
> *To:* Frank Warmerdam
>
> *Cc:* landsat-pds at lists.osgeo.org
> *Subject:* Re: [Landsat-pds] Converting MTL.txt to JSON
>
>
>
> Doh! Good catch – silly typo in the code. Give me a minute…
>
>
>
>
>
> *From:* fwarmerdam at gmail.com [mailto:fwarmerdam at gmail.com
> <fwarmerdam at gmail.com>] *On Behalf Of *Frank Warmerdam
> *Sent:* Thursday, February 26, 2015 12:35 PM
> *To:* Layton, Jeff
> *Cc:* Sundwall, Jed; landsat-pds at lists.osgeo.org
> *Subject:* Re: [Landsat-pds] Converting MTL.txt to JSON
>
>
>
> Jeff,
>
>
>
> I'm not sure what scene that was generated for, but I don't see any values
> that are strings. For instance, I expect the PRODUCT_METADATA group to have
> an item called DATA_TYPE with a string value (ie. "L1T").
>
>
>
> Also, the whole METADATA_FILE_INFO section appears to be missing.
>
>
>
> I am suspecting you lost everything that failed to translate to numbers?
>
>
>
> Also, I see now that you have an content-type set *within* the json.  I'm
> not sure what the utility of that is as opposed to setting it on the S3
> object as Jed was discussing.
>
>
>
> Best regards,
>
> Frank
>
>
>
>
>
> On Thu, Feb 26, 2015 at 9:20 AM, Layton, Jeff <jefflay at amazon.com> wrote:
>
>  Everyone,
>
>
>
> Try the attached JSON file. It should have everything numeric that can be
> numerical J  Let me know if you find any errors (my cursory examination
> didn’t find any). If it passes muster, I’ll fire off the code.
>
>
>
> Thanks!
>
>
>
> Jeff
>
>
>
>
>
>
>
> *From:* Sundwall, Jed
> *Sent:* Thursday, February 26, 2015 11:25 AM
> *To:* Frank Warmerdam
> *Cc:* Layton, Jeff; landsat-pds at lists.osgeo.org
>
>
> *Subject:* Re: [Landsat-pds] Converting MTL.txt to JSON
>
>
>
> I’ve uploaded the file to
> https://s3-us-west-2.amazonaws.com/awsopendata/test/testfile2.json
>
>
>
> The mime type is “application/json” which should be appropriate.
>
>
>
> I’m not sure where this would fit within the landsat_ingestor scripts, but
> setting the mime type with boto is as simple as:
>
>
>
>  if s3_path.endswith(‘.json'):
>
>          key.content_type = ‘application/json'
>
>
>
> Re: converting the numbers, Jeff, could you share the script you’ve
> written to create the json files? Perhaps someone will be able to see a
> simple way to make the conversion.
>
>
>
> Jed.
>
>
>
>
>
>
>
>  On Feb 26, 2015, at 10:55 AM, Frank Warmerdam <warmerdam at pobox.com>
> wrote:
>
>
>
> Jeff,
>
>
>
> From an attachment I (don't think) I can know if mime types are getting
> properly attached in S3.  If you can point me to an object in S3 I can try
> and check.  I see the AWS S3 browser shows the mimetype under the
> "Metadata" properties section if you have access to that for our bucket.
>
>
>
> I'll second Charlie's suggestion to preserve numeric types for numeric
> fields.  I don't know this gets obscured by mtlutils.
>
>
>
> Best regards,
>
> Frank
>
>
>
> On Thu, Feb 26, 2015 at 7:38 AM, Layton, Jeff <jefflay at amazon.com> wrote:
>
>  Frank,
>
>
>
> I think I’ve added the mimetype correctly.Can you look at the attached
> file and let me know if I did?
>
>
>
> Thanks!
>
>
>
> Jeff
>
>
>
>
>
>
>
> *From:* fwarmerdam at gmail.com [mailto:fwarmerdam at gmail.com] *On Behalf Of *Frank
> Warmerdam
> *Sent:* Wednesday, February 25, 2015 5:20 PM
> *To:* Layton, Jeff
> *Cc:* landsat-pds at lists.osgeo.org
> *Subject:* Re: [Landsat-pds] Converting MTL.txt to JSON
>
>
>
> Jeff,
>
>
>
> This looks good to me.  I'd suggest we drop these in the same directory as
> the MTL txt files, and just name them .json.
>
>
>
> eg.
>
> LC80150252015005LGN00_B4.TIF
>
> LC80150252015005LGN00_B10.TIF
>
> LC80150252015005LGN00_B5.TIF
>
> LC80150252015005LGN00_B2.TIF
>
> LC80150252015005LGN00_B9.TIF
>
> LC80150252015005LGN00_B1.TIF
>
> LC80150252015005LGN00_B7.TIF
>
> LC80150252015005LGN00_B8.TIF
>
> LC80150252015005LGN00_B11.TIF
>
> LC80150252015005LGN00_B3.TIF
>
> LC80150252015005LGN00_BQA.TIF
>
> LC80150252015005LGN00_B6.TIF
>
> LC80150252015005LGN00_MTL.txt
>
> LC80150252015005LGN00_MTL.json
>
>
>
> It might be nice to have the mimetype set properly for json too.
>
>
>
> With regard to putting them all in one place, the downside I see to that
> is they aren't as likely to be noticed by folks who have maneuvered to a
> scene.  The scene_list.gz file already has the links to each scene
> directory for folks who need a directory-in-a-file.
>
>
>
> Best regards,
>
> Frank
>
>
>
>
>
> On Wed, Feb 25, 2015 at 1:13 PM, Layton, Jeff <jefflay at amazon.com> wrote:
>
>  Hello everyone,
>
>
>
> I just joined the mailing list and I’m working with Jed from AWS (I work
> for AWS as well). The first project I did was write a simple Python tool to
> convert the MTL.txt files to JSON. I’m attaching a sample MTL file and the
> resulting JSON. Let me know what you think!
>
>
>
> Thanks!
>
>
> Jeff
>
>
>
>
>
>
>
> Dr. Jeff Layton
>
> Solutions Architect – HPC
>
> Amazon Web Services
>
> jefflay at amazon.com
>
> 678-310-3685
>
>
>
>
> _______________________________________________
> Landsat-pds mailing list
> Landsat-pds at lists.osgeo.org
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds
>
>
>
>
>
> --
>
>
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | Geospatial Software Developer
>
>
>
>
>
> --
>
>
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | Geospatial Software Developer
>
> _______________________________________________
> Landsat-pds mailing list
> Landsat-pds at lists.osgeo.org
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds
>
>
>
>
>
>
>
> --
>
>
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | Geospatial Software Developer
>

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150303/f71f793b/attachment-0001.html>