[GRASS-dev] Re: [GRASS-user] r.in.xyz: Could open text file ~ 2.5GB

[Glynn Clements]

Hamish wrote:

> > I forgot that the --enable-largefile switch only enables LFS in
> > libgis, not anywhere else.
> > 
> > [Although config.h includes definitions to enable LFS automatically,
> > those definitions are currently inactive. This is probably a good
> > thing; a lot of GRASS' code isn't LFS-aware, and explicit failure is
> > preferable to silently corrupting data.]
> > 
> > To enable LFS elsewhere, you need to manually add
> > -D_FILE_OFFSET_BITS=64 to the compilation flags. The simplest approach
> > is to add:
> > 
> > ifneq ($(USE_LARGEFILES),)
> > 	EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
> > endif
> > 
> > to raster/r.in.xyz/Makefile, then repeat the "make" and "make install"
> > steps.
> 
> 
> I have added this knowledge (and some other tips posted to the mailing
> list by Glynn & Brad) to a page in the Development section of the Wiki.
> Please correct, update, and expand as needed.
> 
> http://grass.gdf-hannover.de/wiki/Large_File_Support
> 
> how hard will it be to get a HAVE_FTELLO (or whatever) switch tested &
> set by ./configure?

The trick is deciding what to test, and what to indicate. Do we want
to know:

1. Whether ftello() exists with the default CFLAGS/CPPFLAGS?

2. Whether ftello() exists with the default CFLAGS/CPPFLAGS plus a
fixed selection of additional switches (e.g. -D_LARGEFILE_SOURCE)?

3. Whether ftello() exists with the default CFLAGS/CPPFLAGS plus a
variable selection of additional switches (e.g. -D_LARGEFILE_SOURCE)
with those switches being communicated via an additional variable?

> If we get that, should every LFS compliant module be
> written like:
> 
> #ifdef HAVE_FTELLO
>     off_t filesize;
> #else
>     long filesize;
> #endif

No. You may as well just use "off_t filesize" unconditionally. An
"off_t" will always be large enough to hold a "long".

> ...
> 
> #ifdef HAVE_FTELLO
>     fseeko(in_fd, 0L, SEEK_END);
>     filesize = ftello(in_fd);
> #else
>     fseek(in_fd, 0L, SEEK_END);
>     filesize = ftell(in_fd);
> #endif

The test would need to be:

	#if defined(HAVE_FTELLO) && defined(HAVE_FSEEKO)

as you are using both of those.

Also, if the existence of FSEEKO/FTELLO is conditional upon certain
macros (e.g. _LARGEFILE_SOURCE), you have to ensure that those macros
are defined before the first inclusion of <stdio.h>, including any
"hidden" inclusion from other headers, which essentially means that
you have to ensure that the macros are defined before you include any
other headers (except for <grass/config.h>).

> Or is that better handled in G_fseek() etc fns etc as suggested by Brad?
> (still should try and match variable type in that case (how?))

My inclination would be to add:

	extern off_t G_ftell(FILE *fp);
	extern int G_fseek(FILE *stream, off_t offset, int whence);

These would be implemented using fseeko/ftello where available, and
fseek/ftell otherwise. That eliminates the need to perform checks in
individual source files. However, we would need to take care not to
use them in code which can't handle large offsets (i.e. code which
will truncate off_t values to int/long).

> fopen() and fclose() are ok in both cases? (dependant on finding
> -D_FILE_OFFSET_BITS=64 in the Makefile and including grass/config.h ?)

If you use -D_FILE_OFFSET_BITS=64, fopen() should be redirected to
fopen64(). There are reports that the MacOSX fopen() is equivalent to
fopen64() even without additional switches.

-- 
Glynn Clements <glynn at gclements.plus.com>