How to wait for server restart using Ansible?
Asked Answered
Z

11

47

I'm trying to restart the server and then wait, using this:

- name: Restart server
  shell: reboot

- name: Wait for server to restart
  wait_for:
    port=22
    delay=1
    timeout=300

But I get this error:

TASK: [iptables | Wait for server to restart] ********************************* 
fatal: [example.com] => failed to transfer file to /root/.ansible/tmp/ansible-tmp-1401138291.69-222045017562709/wait_for:
sftp> put /tmp/tmpApPR8k /root/.ansible/tmp/ansible-tmp-1401138291.69-222045017562709/wait_for

Connected to example.com.
Connection closed
Zeralda answered 26/5, 2014 at 21:11 Comment(2)
wait_for is running remotely (on the rebooting server). You need to run it on the local machine.Alius
Note that there is a proper reboot action in the works: github.com/ansible/ansible/issues/16186Foreword
I
37

You should change the wait_for task to run as local_action, and specify the host you're waiting for. For example:

- name: Wait for server to restart
  local_action:
    module: wait_for
      host=192.168.50.4
      port=22
      delay=1
      timeout=300
Indecipherable answered 27/5, 2014 at 11:35 Comment(5)
"sudo: false" might be required also.Angarsk
Also Ansible has this blog post on the topic also thats helpful: support.ansible.com/hc/en-us/articles/…Angarsk
@Angarsk In the blog he mention the setting for version 1.9.4. So for the non-1.9.4 case, is it pre-1.9.4, or post-1.9.4?Supereminent
@chachra: the URL you provided is not valid anymore as of Jan 2018.Mysterious
this method returns: fatal: [rpi]: FAILED! => {"changed": false, "module_stderr": "", "module_stdout": "", "msg": "MODULE FAILURE", "rc": -1}Satanic
M
67

Ansible >= 2.7 (released in Oct 2018)

Use the built-in reboot module:

- name: Wait for server to restart
  reboot:
    reboot_timeout: 3600

Ansible < 2.7

Restart as a task

- name: restart server
  shell: 'sleep 1 && shutdown -r now "Reboot triggered by Ansible" && sleep 1'
  async: 1
  poll: 0
  become: true

This runs the shell command as an asynchronous task, so Ansible will not wait for end of the command. Usually async param gives maximum time for the task but as poll is set to 0, Ansible will never poll if the command has finished - it will make this command a "fire and forget". Sleeps before and after shutdown are to prevent breaking the SSH connection during restart while Ansible is still connected to your remote host.

Wait as a task

You could just use:

- name: Wait for server to restart
  local_action:
    module: wait_for
      host={{ inventory_hostname }}
      port=22
      delay=10
    become: false

..but you may prefer to use {{ ansible_ssh_host }} variable as the hostname and/or {{ ansible_ssh_port }} as the SSH host and port if you use entries like:

hostname         ansible_ssh_host=some.other.name.com ansible_ssh_port=2222 

..in your inventory (Ansible hosts file).

This will run the wait_for task on the machine running Ansible. This task will wait for port 22 to become open on your remote host, starting after 10 seconds delay.

Restart and wait as handlers

But I suggest to use both of these as handlers, not tasks.

There are 2 main reason to do this:

  • code reuse - you can use a handler for many tasks. Example: trigger server restart after changing the timezone and after changing the kernel,

  • trigger only once - if you use a handler for a few tasks, and more than 1 of them will make some change => trigger the handler, then the thing that handler does will happen only once. Example: if you have a httpd restart handler attached to httpd config change and SSL certificate update, then in case both config and SSL certificate changes httpd will be restarted only once.

Read more about handlers here.

Restarting and waiting for the restart as handlers:

  handlers:

    - name: Restart server
      command: 'sleep 1 && shutdown -r now "Reboot triggered by Ansible" && sleep 1'
      async: 1
      poll: 0
      ignore_errors: true
      become: true

    - name: Wait for server to restart
      local_action:
        module: wait_for
          host={{ inventory_hostname }}
          port=22
          delay=10
        become: false

..and use it in your task in a sequence, like this, here paired with rebooting the server handler:

  tasks:
    - name: Set hostname
        hostname: name=somename
        notify:
          - Restart server
          - Wait for server to restart

Note that handlers are run in the order they are defined, not the order they are listed in notify!

Mysterious answered 21/1, 2015 at 15:24 Comment(4)
Handlers don't necessarily execute in sequence. Please check: Handlers or maybe I misunderstood?Perineurium
Thanks for pointing that out, @Joze. It's the order in which handlers are defined, not the order they are listed in notify that matters. I will clarify my answer regarding this.Mysterious
Why do you think it is better as a handler?Monthly
I have replied you in the edited answer, @JonathanHartley . Thanks for asking!Mysterious
I
37

You should change the wait_for task to run as local_action, and specify the host you're waiting for. For example:

- name: Wait for server to restart
  local_action:
    module: wait_for
      host=192.168.50.4
      port=22
      delay=1
      timeout=300
Indecipherable answered 27/5, 2014 at 11:35 Comment(5)
"sudo: false" might be required also.Angarsk
Also Ansible has this blog post on the topic also thats helpful: support.ansible.com/hc/en-us/articles/…Angarsk
@Angarsk In the blog he mention the setting for version 1.9.4. So for the non-1.9.4 case, is it pre-1.9.4, or post-1.9.4?Supereminent
@chachra: the URL you provided is not valid anymore as of Jan 2018.Mysterious
this method returns: fatal: [rpi]: FAILED! => {"changed": false, "module_stderr": "", "module_stdout": "", "msg": "MODULE FAILURE", "rc": -1}Satanic
V
10

Most reliable I've with 1.9.4 got is (this is updated, original version is at the bottom):

- name: Example ansible play that requires reboot
  sudo: yes
  gather_facts: no
  hosts:
    - myhosts
  tasks:
    - name: example task that requires reboot
      yum: name=* state=latest
      notify: reboot sequence
  handlers:
    - name: reboot sequence
      changed_when: "true"
      debug: msg='trigger machine reboot sequence'
      notify:
        - get current time
        - reboot system
        - waiting for server to come back
        - verify a reboot was actually initiated
    - name: get current time
      command: /bin/date +%s
      register: before_reboot
      sudo: false
    - name: reboot system
      shell: sleep 2 && shutdown -r now "Ansible package updates triggered"
      async: 1
      poll: 0
      ignore_errors: true
    - name: waiting for server to come back
      local_action: wait_for host={{ inventory_hostname }} state=started delay=30 timeout=220
      sudo: false
    - name: verify a reboot was actually initiated
      # machine should have started after it has been rebooted
      shell: (( `date +%s` - `awk -F . '{print $1}' /proc/uptime` > {{ before_reboot.stdout }} ))
      sudo: false

Note the async option. 1.8 and 2.0 may live with 0 but 1.9 wants it 1. The above also checks if machine has actually been rebooted. This is good because once I had a typo that failed reboot and no indication of the failure.

The big issue is waiting for machine to be up. This version just sits there for 330 seconds and never tries to access host earlier. Some other answers suggest using port 22. This is good if both of these are true:

  • you have direct access to the machines
  • your machine is accessible immediately after port 22 is open

These are not always true so I decided to waste 5 minutes compute time.. I hope ansible extend the wait_for module to actually check host state to avoid wasting time.

btw the answer suggesting to use handlers is nice. +1 for handlers from me (and I updated answer to use handlers).

Here's original version but it it not so good and not so reliable:

- name: Reboot
  sudo: yes
  gather_facts: no
  hosts:
    - OSEv3:children
  tasks:
    - name: get current uptime
      shell: cat /proc/uptime | awk -F . '{print $1}'
      register: uptime
      sudo: false
    - name: reboot system
      shell: sleep 2 && shutdown -r now "Ansible package updates triggered"
      async: 1
      poll: 0
      ignore_errors: true
    - name: waiting for server to come back
      local_action: wait_for host={{ inventory_hostname }} state=started delay=30 timeout=300
      sudo: false
    - name: verify a reboot was actually initiated
      # uptime after reboot should be smaller than before reboot
      shell: (( `cat /proc/uptime | awk -F . '{print $1}'` < {{ uptime.stdout }} ))
      sudo: false
Vaseline answered 25/1, 2016 at 19:35 Comment(0)
P
8

2018 Update

As of 2.3, Ansible now ships with the wait_for_connection module, which can be used for exactly this purpose.

#
## Reboot
#

- name: (reboot) Reboot triggered
  command: /sbin/shutdown -r +1 "Ansible-triggered Reboot"
  async: 0
  poll: 0

- name: (reboot) Wait for server to restart
  wait_for_connection:
    delay: 75

The shutdown -r +1 prevents a return code of 1 to be returned and have ansible fail the task. The shutdown is run as an async task, so we have to delay the wait_for_connection task at least 60 seconds. 75 gives us a buffer for those snowflake cases.

wait_for_connection - Waits until remote system is reachable/usable

Punchdrunk answered 8/2, 2018 at 17:58 Comment(0)
A
6

I wanted to comment on Shahar post, that he is using a hardcoded host address better is to have it a variable to reference the current host ansible is configuring {{ inventory_hostname }}, so his code will be like that:

- name: Wait for server to restart
  local_action:
    module: wait_for
     host={{ inventory_hostname }}
     port=22
     delay=1
     timeout=300
Awash answered 25/10, 2014 at 6:30 Comment(0)
M
4

With newer versions of Ansible (i.e. 1.9.1 in my case), poll and async parameters set to 0 are sometimes not enough (may be depending on what distribution is set up ansible ?). As explained in https://github.com/ansible/ansible/issues/10616 one workaround is :

- name: Reboot
  shell: sleep 2 && shutdown -r now "Ansible updates triggered"
  async: 1
  poll: 0
  ignore_errors: true

And then, wait for reboot complete as explained in many answers of this page.

Maribelmaribelle answered 26/7, 2015 at 21:29 Comment(3)
And beware of not using the command module here but the shell moduleMaribelmaribelle
Why sleep 2 && instead of just shutdown -r 2?Supereminent
because shutdown would wait 2 minutes.Swain
Z
4

Through trial and error + a lot of reading this is what ultimately worked for me using the 2.0 version of Ansible:

$ ansible --version
ansible 2.0.0 (devel 974b69d236) last updated 2015/09/01 13:37:26 (GMT -400)
  lib/ansible/modules/core: (detached HEAD bbcfb1092a) last updated 2015/09/01 13:37:29 (GMT -400)
  lib/ansible/modules/extras: (detached HEAD b8803306d1) last updated 2015/09/01 13:37:29 (GMT -400)
  config file = /Users/sammingolelli/projects/git_repos/devops/ansible/playbooks/test-2/ansible.cfg
  configured module search path = None

My solution for disabling SELinux and rebooting a node when needed:

---
- name: disable SELinux
  selinux: state=disabled
  register: st

- name: reboot if SELinux changed
  shell: shutdown -r now "Ansible updates triggered"
  async: 0
  poll: 0
  ignore_errors: true
  when: st.changed

- name: waiting for server to reboot
  wait_for: host="{{ ansible_ssh_host | default(inventory_hostname) }}" port={{ ansible_ssh_port | default(22) }} search_regex=OpenSSH delay=30 timeout=120
  connection: local
  sudo: false
  when: st.changed

# vim:ft=ansible:
Zacharia answered 3/9, 2015 at 18:51 Comment(0)
R
2

I haven't seen a lot of visibility on this, but a recent change (https://github.com/ansible/ansible/pull/43857) added the "ignore_unreachable" keyword. This allows you to do something like this:

- name: restart server
  shell: reboot
  ignore_unreachable: true

- name: wait for server to come back
  wait_for_connection: 
      timeout: 120

- name: the next action
  ...
Raphaelraphaela answered 14/1, 2019 at 17:19 Comment(0)
H
1
- wait_for:
    port: 22
    host: "{{ inventory_hostname }}"
  delegate_to: 127.0.0.1
Hypopituitarism answered 10/4, 2015 at 4:53 Comment(1)
According to ansible-lint this seems to be the recommended way to do it with recent versions.Stalactite
D
1

In case you don't have DNS setup for the remote server yet, you can pass the IP address instead of a variable hostname:

- name: Restart server
  command: shutdown -r now

- name: Wait for server to restart successfully
  local_action:
    module: wait_for
      host={{ ansible_default_ipv4.address }}
      port=22
      delay=1
      timeout=120

These are the two tasks I added to the end of my ansible-swap playbook (to install 4GB of swap on new Digital Ocean droplets.

Drud answered 21/5, 2016 at 17:15 Comment(0)
W
1

I've created a reboot_server ansible role that can get dynamically called from other roles with:

- name: Reboot server if needed
  include_role:
    name: reboot_server
  vars:
    reboot_force: false

The role content is:

- name: Check if server restart is necessary
  stat:
    path: /var/run/reboot-required
  register: reboot_required

- name: Debug reboot_required
  debug: var=reboot_required

- name: Restart if it is needed
  shell: |
    sleep 2 && /sbin/shutdown -r now "Reboot triggered by Ansible"
  async: 1
  poll: 0
  ignore_errors: true
  when: reboot_required.stat.exists == true
  register: reboot
  become: true

- name: Force Restart
  shell: |
    sleep 2 && /sbin/shutdown -r now "Reboot triggered by Ansible"
  async: 1
  poll: 0
  ignore_errors: true
  when: reboot_force|default(false)|bool
  register: forced_reboot
  become: true

# # Debug reboot execution
# - name: Debug reboot var
#   debug: var=reboot

# - name: Debug forced_reboot var
#   debug: var=forced_reboot

# Don't assume the inventory_hostname is resolvable and delay 10 seconds at start
- name: Wait 300 seconds for port 22 to become open and contain "OpenSSH"
  wait_for:
    port: 22
    host: '{{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }}'
    search_regex: OpenSSH
    delay: 10
  connection: local
  when: reboot.changed or forced_reboot.changed

This was originally designed to work with Ubuntu OS.

Willwilla answered 4/12, 2018 at 14:28 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.