Bamboo build-dir excessive space can it be cleaned up with a cron job?

Question

We use Bamboo CI. There are multiple bamboo local agents and parallel builds across many plans. The build-dir in bamboo-home is many hundreds of gigabytes, and analysis shows that it just continually grows as new feature branches are added. Plans seem to be duplicated in each local agent directory, and also directly in build-dir.

Unlike expiring artifacts, Bamboo does not seem to clean this up by itself. For example, if a local agent is removed then the local agents build directory sits there forever taking up a significant amount of space.

Plans can be set to clean up at the end of a build, however this impacts problem analysis in the event of needing to do a post-mortem on the build.

Due to the directory running out of space I have just added a daily cron task to periodically remove files and directories that haven't been accessed for more than 21 days. When I first ran this manually I reclaimed 300GB from a 600GB partition. I want to know if others have encountered this same issue, and if it is safe to externally clean the build-dir in the long term. Could it impact bamboo builds? Is there some bamboo option that I have missed that would do this for me?

Searching on the Atlassian site has not been helpful and yields no answers... what are others doing to tame this space hog?

Sirois

I have similar setup and I delete all artifacts which are more than one month old, which frees huge chunk of space. I have written a script which runs at the end of every month. I don't think anyone uses data which is older than one week. – Kamala

I wasn't so much worried about artifacts, they do take up a lot of space but bamboo will clean them up itself based on the plan configuration and whether they are in the past 3 deployments to any environment. My question was more about the space consumed in the bamboo_home/xml-data/build-dir tree. – Sirois

If running bamboo on EC2 you can safely place the bamboo build directory on an ephemeral data store as long as the artifacts and bamboo home are on EBS and retained between restarts. – Sirois

Answer 1

The cron job has been running for a while now without any issues, and it is keeping the space usage under control.

I have reduced the parameter to 15 days.

My crontab looks like this:

# clean up old files from working directory
0 20 * * * find /<path_to>/bamboo-home/xml-data/build-dir/ -depth -not -path *repositories-cache* -atime +15 -delete

# clean up old backups every Sunday
0 21 * * 0 find /<path_to>/bamboo-home/backups -type f -mtime +28 -delete

# remove any old logs from install directory after 15 days
0 22 * * * find /<path_to>/bamboo/logs/ -type f -mtime +15 -delete

# quick and dirty truncate catalina.out to stop it growing too large (or better still use logrotate) 
0 23 * * * cat /dev/null > /<path_to>/bamboo/logs/catalina.out

I hope this is useful for others trying to tame bamboo's diskspace usage. The first job is the important one, the last three are just housekeeping.

N.B. logrotate is not used on catalina.out due to unique circumstances in my companies outsourced linux environment. I would generally recommend logrotate if possible rather than my quick and dirty truncate method - see answer by Jon V.

Sirois

This takes care of the xml-data/build-dir/ alright, but what about the builds/ directory right next to it? I just found out, that ours has 78Gb worth of data in it -- in addition to 118Gb in build-dir/. Can they both be cleaned up the same way -- removing everything older than a certain age? (BTW, searching under build-dir/ I find nothing matching *repositories-cache*.) – Personification

there is no repository cache under builds, things under builds can be safely removed without impacting Bamboo. – Frogman

Answer 2

While the cron idea works well - the thing that I've also done in the past with Bamboo is to "Clean working directory after each build" options. Basically, for any given job, there's a config option that will clean up the appropriate build-dir/<build_plan_job> directory for a given plan/job:

Actions -> Configure Plan -> click the Job -> Miscelaneous Tab -> first checkbox

While that makes sure that future build scratch areas are cleaned up, it does not help for already existing and/or old builds. Given the normal git style workflow where you have lots of branches (and each branch creates a specific job ID for it (like PLAN-JOB_WITH_BRANCH_NUMBER-BUILD_NUMBER or similar) that gets old/large fast. I just did a quick check, and we're now cleaning up the build areas for most builds (the large ones at least), but we have over 100Gig of build cruft from branches that have been merged loooong ago.

Thanks for the cron example, though, that should work OK for the future.

Unrelated: the more I use Bamboo, the more I love/hate it.

EDIT: as a general comment, I'd try really hard to work with an SA to get a logrotate rule set up/implemented for the catalina.out - overwriting with /dev/null seems like a really bad idea, unless you're already slurping them up with something like ELK or Splunk.

My /etc/logrotate.d/bamboo_catalina_out looks like (using your paths):

/<path_to>/bamboo/logs/catalina.out {
  create 0660 bamboo bamboo
  compress
  copytruncate
  missingok
  rotate 10
  size 100M
}

Finally - is there a reason why you have both the third and fourth cron scripts?

Marika

Yes I agree.. an outsourcing arrangement here makes root administrator requests expen$ive hence my quick and dirty truncate. The Clean Working Directory after build option in Bamboo goes some way towards cleaning up, but it also makes troubleshooting / post-mortems more difficult if a job fails. So I leave it unchecked and "force a clean build" on the source code checkout. This cleans up the directory before a subsequent job starts. Later cron picks up the all old merged feature branches and cleans them up. – Sirois

Yeah - it's not ideal - plus you still get a lot of directories floating around that never get cleaned up (some of our builds have had 900+ branches created for them - given that build has 4 jobs that run, that's 3600+ directories for stale builds). If it had a "clean up if build successful only", and a check box for "this branch was deleted in source, so delete all build-dir's associated with it", I think that would be a good solution. – Marika

Answer 3

You can follow

1. Login with an Admin account
2. Go to Administration (the cog icon in the top right corner)
3. Select Expiry from the left hand side menu
4. Click Edit and configure (if you haven't already) the Global expiry settings and set up a schedule for executing it
5. Click Save

If you want to execute it immediately click the Run Now button

C