ansible - delete unmanaged files from directory?

Question

I want to recursively copy over a directory and render all .j2 files in there as templates. For this I am currently using the following lines:

- template: >
            src=/src/conf.d/{{ item }}
            dest=/dest/conf.d/{{ item|replace('.j2','') }}
  with_lines: find /src/conf.d/ -type f -printf "%P\n"

Now I'm looking for a way to remove unmanaged files from this directory. For example if I remove a file/template from /src/conf.d/ I want Ansible to remove it from /dest/conf.d/ as well.

Is there some way to do this? I tried fiddling around with rsync --delete, but there I got a problem with the templates which get their suffix .j2 removed.

Menstruate

Answer 1

I'd do it like this, assuming a variable defined as 'managed_files' up top that is a list.

- shell: ls -1 /some/dir
  register: contents

- file: path=/some/dir/{{ item }} state=absent
  with_items: contents.stdout_lines
  when: item not in managed_files

Enucleate

Thanks. Works great. Now, any advise on how I would remove files if my managed_files do not have filename extentions but my files do? Perhaps this should be another question entirely. – Mixie

It actually works. Saved me a from finishing my hair. – Sheba

@Mixie you can use jinja filter, like this: when: item|regex_replace('^(.*)\\.ext$', '\\1') not in managed_files – Disreputable

To determine managed_files based on a local directory, use set_fact: managed_files="{{lookup('fileglob', 'conf.d/*').split(',') | map('basename') | list}}" – Oxfordshire

You may want to use Ansible's find task instead of shell which will make it work in --check mode, too. I'll add an answer with example code. – Vicennial

Answer 2

We do this with our nginx files, since we want them to be in a special order, come from templates, but remove unmanaged ones this works:

# loop through the nginx sites array and create a conf for each file in order 
# file will be name 01_file.conf, 02_file.conf etc
- name: nginx_sites conf
  template: >
    src=templates/nginx/{{ item.1.template }}
    dest={{ nginx_conf_dir }}/{{ '%02d' % item.0 }}_{{ item.1.conf_name|default(item.1.template) }}
    owner={{ user }}
    group={{ group }}
    mode=0660
  with_indexed_items: nginx_sites
  notify:
    - restart nginx 
  register: nginx_sites_confs

# flatten and map the results into simple list
# unchanged files have attribute dest, changed have attribute path
- set_fact:
    nginx_confs: "{{ nginx_sites_confs.results|selectattr('dest', 'string')|map(attribute='dest')|list + nginx_sites_confs.results|selectattr('path', 'string')|map(attribute='path')|select|list }}"
  when: nginx_sites

# get contents of conf dir
- shell: ls -1 {{ nginx_conf_dir }}/*.conf
  register: contents
  when: nginx_sites

# so we can delete the ones we don't manage
- name: empty old confs 
  file: path="{{ item }}" state=absent
  with_items: contents.stdout_lines
  when: nginx_sites and item not in nginx_confs

The trick (as you can see) is that template and with_items have different attributes in the register results. Then you turn them into a list of files you manage and then get a list of the the directory and removed the ones not in that list.

Could be done with less code if you already have a list of files. But in this case I'm creating an indexed list so need to create the list as well with map.

Tempietempla

This is very cool - upvoted. I struggled with it though: if you're not specifying the wildcard, it will compare the full path and filename to just the filename so it will delete everything. Instead of ls -l try find {{ nginx_conf_dir }} -type f instead in that scenario – Lenette

Answer 3

I want to share my experience with this case.

Ansible from 2.2 is had with_filetree loop provides simple way to upload dirs, links, static files and even (!) templates. It's best way to keep my config dir synchronized.

- name: etc config - Create directories
  file:
    path: "{{ nginx_conf_dir }}/{{ item.path }}"
    state: directory
    mode: 0755
  with_filetree: etc/nginx
  when: item.state == 'directory'

- name: etc config - Creating configuration files from templates
  template:
    src: "{{ item.src }}"
    dest: "{{ nginx_conf_dir }}/{{ item.path | regex_replace('\\.j2$', '') }}"
    mode: 0644
  with_filetree: etc/nginx
  when:
    - item.state == "file"
    - item.path | match('.+\.j2$') | bool

- name: etc config - Creating staic configuration files
  copy:
    src: "{{ item.src }}"
    dest: "{{ nginx_conf_dir }}/{{ item.path }}"
    mode: 0644
  with_filetree: etc/nginx
  when:
    - item.state == "file"
    - not (item.path | match('.+\.j2$') | bool)

- name: etc config - Recreate symlinks
  file:
    src: "{{ item.src }}"
    dest: "{{ nginx_conf_dir }}/{{ item.path }}"
    state: link
    force: yes
    mode: "{{ item.mode }}"
  with_filetree: etc/nginx
  when: item.state == "link"

Next we may want delete unused files from config dir. It's simple. We gather list of uploaded files and files exist on remote server, next remove diffrence.

But we may want to have unmanaged files in config dir. I've used -prune functionality of find to avoid clearing folders with unmanaged files.

PS _(Y)_ sure after I have deleted some unmanaged files

- name: etc config - Gathering managed files
  set_fact:
    __managed_file_path: "{{ nginx_conf_dir }}/{{ item.path | regex_replace('\\.j2$', '') }}"
  with_filetree: etc/nginx
  register: __managed_files

- name: etc config - Convert managed files to list
  set_fact: managed_files="{{ __managed_files.results | map(attribute='ansible_facts.__managed_file_path') | list }}"

- name: etc config - Gathering exist files (excluding .ansible_keep-content dirs)
  shell: find /etc/nginx -mindepth 1 -type d -exec test -e '{}/.ansible_keep-content' \; -prune -o -print
  register: exist_files
  changed_when: False

- name: etc config - Delete unmanaged files
  file: path="{{ item }}" state=absent
  with_items: "{{ exist_files.stdout_lines }}"
  when:
    - item not in managed_files

Collide

Answer 4

Here's something I came up with:

- template: src=/source/directory{{ item }}.j2 dest=/target/directory/{{ item }}
  register: template_results
  with_items:
    - a_list.txt
    - of_all.txt
    - templates.txt
- set_fact:
    managed_files: "{{ template_results.results|selectattr('invocation', 'defined')|map(attribute='invocation.module_args.dest')|list }}"

- debug:
    var: managed_files
    verbosity: 0

- find:
    paths: "/target/directory/"
    patterns: "*.txt"
  register: all_files
- set_fact:
    files_to_delete: "{{ all_files.files|map(attribute='path')|difference(managed_files) }}"

- debug:
    var: all_files
    verbosity: 0
- debug:
    var: files_to_delete
    verbosity: 0

- file: path={{ item }} state=absent
  with_items: "{{ files_to_delete }}"

• This generates the templates (however which way you want), and records the results in 'template_results'
• The the results are mangled to get a simple list of the "dest" of each template. Skipped templates (due to a when condition, not shown) have no "invocation" attribute, so they're filtered out.
• "find" is then used to get a list of all files that should be absent unless specifically written.
• this is then mangled to get a raw list of files present, and then the "supposed to be there" files are removed.
• The remaining "files_to_delete" are then removed.

Pros: You avoid multiple 'skipped' entries showing up during deletes.

Cons: You'll need to concatenate each template_results.results if you want to do multiple template tasks before doing the find/delete.

Swinge

Answer 5

There might be a couple of ways to handle this, but would it be possible to entirely empty the target directory in a task before the template step? Or maybe drop the templated files into a temporary directory and then delete+rename in a subsequent step?

Zacharia

Entirely emptying the target directory before copying would mean that the play would always 'change'. Even if there were no changes made to the source directory. When locally renaming the files (e.g. rsync->render->rename->rsync) there is always the problem that ansible will report changes (the renaming) when there really are no changes. – Menstruate

@keks changed_when: false will fix that – Leilaleilah

But you want to know when/if a config file has changed to kick off handlers – Tempietempla

@Leilaleilah No it won't, reporting never changed is just as bad as reporting always changed, if not worse. – Llamas

Answer 6

Usually I do not remove files but I add -unmanaged suffix to its name. Sample ansible tasks:

- name: Get sources.list.d files
  shell: grep -r --include=\*.list -L '^# Ansible' /etc/apt/sources.list.d || true
  register: grep_unmanaged
  changed_when: grep_unmanaged.stdout_lines

- name: Add '-unmanaged' suffix
  shell: rename 's/$/-unmanaged/' {{ item }}
  with_items: grep_unmanaged.stdout_lines

EXPLANATION

Grep command uses:

• -r to do recursive search
• --include=\*.list - only take files with .list extension during recursive search
• -L '^# Ansible' - display file names that are not having line starting with '# Ansible'
• || true - this is used to ignore errors. Ansible's ignore_errors also works but before ignoring the error ansible will show it in red color during ansible-playbook run which is undesired (at least for me).

Then I register output of grep command as a variable. When grep displays any output I set this task as changed (the line changed_when is responsible for this).

In next task I iterate grep output (i.e. file names returned by grep) and run rename command to add suffix to each file.

That's all. Next time you run the command first task should be green and second skipped.

Coffey

Thanks! This is my preferred way of doing it now, because the top voted comment doesn't handle "stuff hasn't changed" very well. – Ploughshare

Answer 7

I am using Ansible version 2.9.20

---
# tasks file for delete_unmanaged_files
- name: list files in dest
  shell: ls -1 dest/conf.d
  register: files_in_dest

- name: list files in src
  shell: ls -1 src/conf.d
  register: files_in_src

- name: Managed files - dest
  command: echo {{ item|replace('.j2','') }}
  with_items: "{{ files_in_dest.stdout_lines }}"
  register: managed_files_dest

- name: Managed files - src
  command: echo {{ item|replace('.j2','') }}
  with_items: "{{ files_in_src.stdout_lines }}"
  register: managed_files_src

- name: Convert src managed files to list
  set_fact: managed_files_src_list="{{ managed_files_src.results | map(attribute='stdout') | list }}"

- name: Delete unmanaged files in dest
  file: path=dest/conf.d/{{ item.stdout }} state=absent
  with_items: "{{ managed_files_dest.results }}"
  when: item.stdout not in managed_files_src_list

I think depending on the usecase of this issue, I found above solution might help you. Here, I have created 6 tasks.

Explanation:

• Task-1 & Task-2 will help storing file names in variable "files_in_dest" and "files_in_src" in it.
• Task-3 & Task-4 will inherit the output coming from Task-1 & Task-2 and then replace the j2 file (required for the usecase). Then these tasks will store the output in "managed_files_dest" and "managed_files_src" variables.
• Task-5 will convert the output of "managed_files_src" to list, so that we can have all the present files (at current state) stored in src directory in a proper or single list and then we can use this list in next task to know the unmanaged files in dest directory.
• Task-6 will delete the unmanaged files in dest.

Inscription

Answer 8

Building on @user2645850's answer I came up with this improved version, in this case it manages the vhost configuration of Apache. It doesn't use shell and thus works also in --check mode.

# Remove unmanged vhost configs left over from renaming or removing apps
# all managed configs need to be added to "managed_sites" in advance
- find:
    paths: /etc/apache2/sites-available
    patterns: '*.conf'
  register: sites_available_contents

- name: Remove unmanaged vhost config files
  file:
    path: /etc/apache2/sites-available/{{ item }}
    state: absent
  with_items: "{{ sites_available_contents.files | map(attribute='path') | map('basename') | list }}"
  when: item not in managed_sites
  
# links may differ from files, therefore we need our own find task for them
- find:
    paths: /etc/apache2/sites-enabled
    file_type: any 
  register: sites_enabled_contents

- name: Remove unmanaged vhost config links
  file:
    path: /etc/apache2/sites-enabled/{{ item }}
    state: absent
  with_items: "{{ sites_enabled_contents.files | map(attribute='path') | map('basename') | list }}"
  when: item not in managed_sites

Examples on how to build managed_sites:

# Add single conf and handle managed_sites being unset
- set_fact:
    managed_sites: "{{ (managed_sites | default([])) + [ '000-default.conf' ] }}"

# Add a list of vhosts appending ".conf" to each entry of vhosts
- set_fact:
    managed_sites: "{{ managed_sites + ( vhosts | map(attribute='app') | product(['.conf']) | map('join') | list ) }}"

Vicennial

Answer 9

Apparently this isn't possible with ansible at the moment. I had a conversation with mdehaan on IRC and it boils down to ansible not having a directed acyclic graph for resources, making things like this very hard.

Asking mdehaan for an example e.g. authoritatively managing a sudoers.d directory he came up with these things:

14:17 < mdehaan> Robe: http://pastebin.com/yrdCZB0y
14:19 < Robe> mdehaan: HM
14:19 < Robe> mdehaan: that actually looks relatively sane
14:19 < mdehaan> thanks :)
14:19 < Robe> the problem I'm seeing is that I'd have to gather the managed files myself
14:19 < mdehaan> you would yes
14:19 < mdehaan> ALMOST
14:20 < mdehaan> you could do a fileglob and ... well, it would be a little gross
[..]
14:32 < mdehaan> eh, theoretical syntax, nm
14:33 < mdehaan> I could do it by writing a lookup plugin that filtered a list
14:34 < mdehaan> http://pastebin.com/rjF7QR24
14:34 < mdehaan> if that plugin existed, for instance, and iterated across lists in A that were also in B

Coryphaeus

Answer 10

I had the same issue and found another solution that seems cleaner to me:

vars:
  dir: /tmp/foo
  managed_files:
    - 'bar'
    - 'bar/titi'

tasks:
  - name: Capture files to delete
    find:
      paths: "{{ dir }}"
      recurse: true
      file_type: any
    register: all_files

- name: Delete files
  file:
    path: "{{ dir }}/{{ item }}"
    state: absent
  with_items: "{{ all_files.files | map(attribute='path') | map('regex_replace',dir+'/','') | list }}"
  when: item not in managed_files | map('dirname') | select() + managed_files

This also ensure cleaning the empty directories.

Lindblad