[SAC] [OSGeo] #2958: woodie-server issue with disk space

[OSGeo]

#2958: woodie-server issue with disk space
---------------------------+---------------------------------------
 Reporter:  robe           |       Owner:  sac@…
     Type:  task           |      Status:  closed
 Priority:  normal         |   Milestone:  Sysadmin Contract 2023-I
Component:  Systems Admin  |  Resolution:  fixed
 Keywords:                 |
---------------------------+---------------------------------------
Changes (by robe):

 * status:  new => closed
 * resolution:   => fixed

Comment:

 I ended up taking the harsh route of

 # this took a couple of ours and I stopped it once disk usage went from
 260GB to 2 GB
 {{{
 cd /var/lib/docker/vfs/dir
 rm -r *
 }}}

 The above, while it preserved the docker volumes of data the woodie server
 writes to, I think damaged the agent as it tried ot use a file that was in
 that folder.  After rebooting the container I did:


 {{{
 su woodie
 cd ~/
 docker compose pull #which fixed up the images and pulled woodpecker-
 server 1.0 (so yah the screen looks quite different now)
 docker compose up -d
 }}}

 Things seem to be now back to normal except with a hiccup I had logging in
 and authenticating, but after clearing out my browser cache and relogging
 in, things seemed fine and I see all the run logs from before in the
 interface, so db must be fine too.

 I did reduce in the docker-compose.yml (in /home/woodie)  the number of
 procs we run on this server to 2 from 5.

 I'm going to setup another docker image, first on osgeo8 that has 4 procs.

 I suspect what happened why :/var/lib/docker/vfs/dir was not being cleared
 by docker system prune was because somewhere along the line, with all that
 shaking we've been doing in postgis lately, it ran out of space before it
 could figure out what to delete.

 I set in cron to do a docker system prune nightly, so that should keep
 things clean.  Also having agent on a separate container will allow us to
 be more reckless since agents can be thrown away without risking damaging
 history.
-- 
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2958#comment:2>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.