[GRASS-dev] vector large file support

[Glynn Clements]

Markus Metz wrote:

> >> 3) solve the fseek/fseeko and ftell/ftello problem. Get inspiration from 
> >> libgis and LFS-safe modules? Or as suggested in the grass wiki on LFS, add
> >> extern off_t G_ftell(FILE *fp);
> >> extern int G_fseek(FILE *stream, off_t offset, int whence);
> >> for global use?
> >>     
> >
> > That would be my preference.
> >   
> How about
> 
> extern off_t G_ftell(FILE *fp)
> {
> #ifdef HAVE_LARGEFILES
>     return (ftello(fp);
> #else
>     return (ftell(fp);
> #endif     
> }

Yep, other than the extraneous open parenthesis (2 open, 1 close).

G_fseek() is slightly more tricky, as you need to check for an
out-of-range value rather than silently truncating.

> >> 4) figure out if coor file size really needs to be stored in coor and 
> >> topo. coor file size doesn't say a lot about the number of features 
> >> because coor can contain a high proportion of dead lines (a problem in 
> >> itself, vector TODO). If if does not need to be stored in coor and topo, 
> >> how does removing coor file size info affect reading and writing of coor 
> >> and topo? Are there hard-coded offsets for reading these files?
> >>     
> >
> > No idea here.
> 
> I think coor file size is stored as safety check but not needed to read 
> features. That safety check is probably there for a good reason.
> The problem is that the offset of a given feature in the coor file is 
> stored in topo and must be properly retrieved, i.e. the number of bytes 
> used to store a given feature offset must match the off_t size of the 
> current library. A topo file written with LFS support will thus be only 
> readable by old libraries or grass compiled without LFS support after 
> rebuilding topology. The other way around, a vector that was written 
> without LFS support can be opened with new LFS-enabled libraries only 
> after rebuilding topology. The off_t size used to write the vector must 
> be stored somewhere in the header info of topo (most important) and 
> coor. If this size does not match the current off_t size, topology must 
> be rebuild.
> Letting older libraries know that topology needs to be rebuilt is the 
> easiest part...
> 
> Nonsense or going into the right direction?

I think that the code which reads these files needs functions to
read/write off_t values at the size used by the file, not the size
used by the code.

I.e. if the code is built for 64-bit off_t, it should still be able to
directly read/write files using a 32-bit off_t. Code built for 32-bit
off_t should also directly read/write files which use a 64-bit off_t,
subject to the constraint that only 31 bits are non-zero (if you have
a 32-bit off_t, attempting to open a file >=2GiB will fail, as will
attempting to enlarge a file beyond that size).

-- 
Glynn Clements <glynn at gclements.plus.com>