[GRASS-dev] Raster format and dual function module

[Glynn Clements]

Joel Pitt wrote:

> > > This is due to another frustration I've had with speed of execution.
> > > While running long chains of commands in simulations I can't help
> > > realising that every command has to reload a map from disk and then
> > > write it back. As we all know, disk access is REALLY slow in
> > > comparison to memory etc. so if GRASS modules were compiled as
> > > libraries the last N (N being configurable) number of loaded maps
> > > could remain in memory for quick processing and display.
> >
> > Your OS should cache the data in non-reserved memory space and reuse it
> > if possible, or dump it if something else wants the space.
> >
> > e.g. grep for something in the grass source code, then when it is done
> > try it again. The second time will finish in 5% of the time as it wasn't
> > reading from the disk. The more memory you have, the more potential for
> > cache space.
> 
> Okay, good point about reading maps, however the writing still takes
> time and writing to disk is the only simple way for passing a map from
> one command to the other.

Actually writing to disk doesn't take much CPU time, and it won't
cause delays in applications unless disk bandwidth is saturated.

> Unless the OS doesn't actually write the file when it is closed. Which
> I'm guessing would be a Bad Thing (TM)
> 
> BTW, do you know whether a newly written file also changes the OS
> cached version? Or in other words, is the cache invalidated after a
> write changes the disk? Just curious...

No. When you write to a file, the OS writes to the cache. Any
subsequent reads from that file will read from the cache.

Each cached block has a flag to indicate whether it is "clean" (is
identical to the copy on disk) or "dirty" (differs from the disk
copy). If something reads a block which isn't cached, it is read into
the cache from disk, and the cache block is marked as clean. Whenever
a block is modified it will be marked as dirty.

The OS will write dirty blocks to the disk lazily (or when explicitly
requested via sync() or fsync() or when a device is unmounted). 
Afterwards, the cached block will be marked as clean and retained.

Essentially, almost anything which accesses a disk reads or writes the
cached copy; the physical disk is merely backing storage.

-- 
Glynn Clements <glynn at gclements.plus.com>