[SAC] [OSGeo] #2958: woodie-server issue with disk space
OSGeo
trac_osgeo at osgeo.org
Sat Jul 29 23:45:00 PDT 2023
#2958: woodie-server issue with disk space
---------------------------+---------------------------------------
Reporter: robe | Owner: sac@…
Type: task | Status: closed
Priority: normal | Milestone: Sysadmin Contract 2023-I
Component: Systems Admin | Resolution: fixed
Keywords: |
---------------------------+---------------------------------------
Changes (by robe):
* status: new => closed
* resolution: => fixed
Comment:
I ended up taking the harsh route of
# this took a couple of ours and I stopped it once disk usage went from
260GB to 2 GB
{{{
cd /var/lib/docker/vfs/dir
rm -r *
}}}
The above, while it preserved the docker volumes of data the woodie server
writes to, I think damaged the agent as it tried ot use a file that was in
that folder. After rebooting the container I did:
{{{
su woodie
cd ~/
docker compose pull #which fixed up the images and pulled woodpecker-
server 1.0 (so yah the screen looks quite different now)
docker compose up -d
}}}
Things seem to be now back to normal except with a hiccup I had logging in
and authenticating, but after clearing out my browser cache and relogging
in, things seemed fine and I see all the run logs from before in the
interface, so db must be fine too.
I did reduce in the docker-compose.yml (in /home/woodie) the number of
procs we run on this server to 2 from 5.
I'm going to setup another docker image, first on osgeo8 that has 4 procs.
I suspect what happened why :/var/lib/docker/vfs/dir was not being cleared
by docker system prune was because somewhere along the line, with all that
shaking we've been doing in postgis lately, it ran out of space before it
could figure out what to delete.
I set in cron to do a docker system prune nightly, so that should keep
things clean. Also having agent on a separate container will allow us to
be more reckless since agents can be thrown away without risking damaging
history.
--
Ticket URL: <https://trac.osgeo.org/osgeo/ticket/2958#comment:2>
OSGeo <https://osgeo.org/>
OSGeo committee and general foundation issue tracker.
More information about the Sac
mailing list