[GRASS-dev] Number of files in GRASS mapsets

[Markus Neteler]

Hi,

On Mon, Nov 11, 2019 at 8:57 AM Stefan Blumentrath
<Stefan.Blumentrath at nina.no> wrote:
>
> Hi,
>
> Currently, I am working on a system (GPFS parallel file system) with quotas not only on the amount of data (5 TB), but also on the number of files stored in my project area (100,000).
>
> I did hit the number of files limit before i even got close to the quota on the amount of data (in fact at ~ 1TB).

While I can kind of understand a size limit, do they motivate the
limit on open files? Say, when explaining to the administrators how
GRASS GIS maps work, might they increase the quota?

> I saw that some changes in the GRASS raster format are planned for GRASS 8. So, my question is, would it be possible and of interest to reduce the number of files per map in GRASS 8 file format? E.g. writing color table, metadata, NoData mask, and similar into one file?
>
> A large number of files can also reduce performance when working against NFS...

It will be a major work to change that.
An option might be to aggregate the metadata files into one or at
least less file (JSON format?) incl. color table, range, history etc.
To embed NULL into the raster files was also discussed in the past.

However, all ideas which can only be implemented in a major release
(so, GRASS GIS 8, as you already said [1]).

Markus

PS:
Funny idea: could you add "user land" file systems on top of GPFS to
let GPSF just see a few blobs while you having access to the data on
FUSE level?

[1] https://trac.osgeo.org/grass/wiki/Grass8Planning