[GRASS-dev] which files are mandatory for raster maps

[Stefan Blumentrath]

Hi,

The small auxiliary files are also causing trouble on the NAS filers (NFS3 mounted network attached storage with SSD or Hybrid disk systems) where my IT department wants us to store (and process) GRASS data.
Especially for time series data with literally thousands of maps performance decreases significantly.

Would it be possible to reduce the number of files for native GRASS raster format in GRASS 8 (e.g.writing color tables or other metadata into a kind of header?

Cheers,
Stefan
________________________________
From: grass-dev <grass-dev-bounces at lists.osgeo.org> on behalf of Moritz Lennert <mlennert at club.worldonline.be>
Sent: Wednesday, April 17, 2019 2:03:50 PM
To: Markus Neteler
Cc: GRASS developers list
Subject: Re: [GRASS-dev] which files are mandatory for raster maps

On 16/04/19 18:10, Markus Neteler wrote:
> Hi Moritz,
>
> On Tue, Apr 16, 2019 at 2:56 PM Moritz Lennert
> <mlennert at club.worldonline.be> wrote:
>>
>> Hello,
>>
>> I'm working with a large batch of raster maps (4550 * 4 bands) and
>> including all the auxiliary files this explodes my inode quota on our
>> university's HPC system.
>
> Out of curiosity, which file system is that?


GPFS AFAIK.

But it is not an intrinsic limit to the system. It's them limiting
individual access in order to be sure to be able to provide services to
the potentionally thousands of users.

>
>> While waiting to see if they can increase this,
>> I was wondering which of the auxiliary files are absolutely mandatory.
>>
>> Can I just erase colr/* and hist/* for example ?
>
> I just tried:
>
> r.info elevation
> WARNING: Unable to get history information for <elevation@>
> ...
> [rest as usual]
>
> So, it should not be a problem.
>
> # idea r.external:
> I just checked, it seems that r.external generates the same amount of
> files, hence not helpful in this regard.
>
> # idea find directories with most inode consumption (source [1]):
> for i in `ls -1A | grep -v "\.\./" | grep -v "\./"`; do echo "`find $i
> | sort -u | wc -l` $i"; done | sort -rn | head -10

Well, most are clearly in cell_misc as there are three files per raster
map. But I do think these are necessary.

>
> Maybe you can still delete something?

At this stage they've raised my quota, so I can go on working for now,
but I'll explore this when I have the time.

>
> Markus
>
> PS: it happened to me in the past when exporting XFS over NFS (while
> it worked ok inside the HPC system). In the end I had to reformat the
> disk array...

I have to admit that I don't have the knowledge of exactly the different
parts of the HPC system are connected.

Thanks for the help !

Moritz
_______________________________________________
grass-dev mailing list
grass-dev at lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20190417/54111d60/attachment.html>