Ansible Maintenance Guide

This document explains the current patch-management setup running from itvxwebansible and the procedures for routine maintenance.

1. Overview

• Control node: itvxwebansible (Ansible Core 2.13.7).
• Primary playbook: /home/administrator/playbooks/update-upgrade.yml.
• Target hosts grouped under meta-group update_targets (children: beaulieu, cloud).
• Automation runs via cron: 0 8 * * 3 /usr/bin/ansible-playbook -e @/home/administrator/vault/vault.yml /home/administrator/playbooks/update-upgrade.yml.
• Slack notifications (success + failure) use vaulted credentials stored in /home/administrator/vault/vault.yml.

2. Inventory Layout

• File: /etc/ansible/hosts.
• Groups:
  • beaulieu: itvxwebansible (local), itvxweb, itvxwebdmz, itvxwebstagingdmz.
  • cloud: itwebcloudprod, itwebclouddatabase (accessed via bastion at 137.184.161.76).
  • update_targets: meta-group including both beaulieu and cloud.

2.1 Modify an Existing Host IP

1. Edit /etc/ansible/hosts (use sudo when necessary).

   sudo nano /etc/ansible/hosts

2. Update the ansible_host= value for the desired entry.
3. Test SSH access:

   ssh <host-alias>

4. Verify Ansible connectivity:

   ansible <group-or-host> -m ping -e @/home/administrator/vault/vault.yml

2.2 Update SSH Proxy or Keys

• Adjust ansible_ssh_args, ansible_user, or ansible_ssh_private_key in /etc/ansible/hosts.
• Store private keys in ~/.ssh/ and keep permissions restrictive (chmod 600 ~/.ssh/id_rsa) so SSH accepts them.

3. Managing Credentials

• Sensitive values are stored in /home/administrator/vault/vault.yml and protected by Ansible Vault (password file /etc/ansible/9d78b81a.txt).

3.1 Changing a Sudo Password

1. Decrypt to a temp file (Ansible will prompt if the vault password file has moved):

   ansible-vault decrypt /home/administrator/vault/vault.yml --output /tmp/vault_plain.yml --vault-password-file /etc/ansible/9d78b81a.txt

2. Update the relevant variable (e.g., master_password for beaulieu hosts) and save the file.
3. Re-encrypt:

   ansible-vault encrypt /tmp/vault_plain.yml --output /home/administrator/vault/vault.yml --vault-password-file /etc/ansible/9d78b81a.txt --encrypt-vault-id default
   rm /tmp/vault_plain.yml

4. Run a dry-run to confirm:

   ansible-playbook /home/administrator/playbooks/update-upgrade.yml -e @/home/administrator/vault/vault.yml --limit update_targets --check

3.2 Rotating Slack Tokens

1. Generate a new token at https://api.slack.com/apps (reinstall the app if Slack asks for authorization).
2. Update slack_bot_token in the vault (same process as above).
3. Optionally adjust slack_channel or slack_username.
4. Trigger a dry-run to validate Slack notifications.

4. Adding or Removing Hosts

4.1 Add a Host

1. Add inventory entry under the appropriate group in /etc/ansible/hosts.

   [beaulieu]
   newhost ansible_host=10.0.0.10 ansible_ssh_private_key=~/.ssh/id_rsa
    # set permission with: chmod 600 ~/.ssh/id_rsa

2. If the host needs a proxy jump, include ansible_ssh_args similar to existing cloud hosts.
3. Ensure required credentials are stored in the vault (e.g., add master_password or host-specific sudo variables).
4. Test access and Ansible connectivity (ssh and ansible -m ping).
5. Run the playbook in check mode limited to the new host:

   ansible-playbook /home/administrator/playbooks/update-upgrade.yml -e @/home/administrator/vault/vault.yml --limit newhost --check

4.2 Remove a Host

1. Comment or delete the host entry in /etc/ansible/hosts.
2. Remove any host-specific secrets from the vault if they are no longer needed.
3. Clean up SSH config entries (~/.ssh/config) if present.

5. Running the Playbook Manually

• Dry-run (safe preview):

  ansible-playbook /home/administrator/playbooks/update-upgrade.yml -e @/home/administrator/vault/vault.yml --limit update_targets --check

• Full execution:

  ansible-playbook /home/administrator/playbooks/update-upgrade.yml -e @/home/administrator/vault/vault.yml --limit update_targets

• Limit to a subset (e.g., only cloud hosts):

  ansible-playbook ... --limit cloud

6. Cron Automation

• Check the scheduled job:

  crontab -l

• To adjust schedule, edit the crontab entry (crontab -e).
• Monitor /var/log/syslog or Slack notifications for run status.

7. Troubleshooting & Maintenance Tips

• Host key changes: Remove or update entries in ~/.ssh/known_hosts if SSH warns about mismatched keys (ssh-keygen -R hostname).
• Proxy issues: Ensure the bastion (137.184.161.76) is reachable when managing cloud hosts.
• Slack errors: missing_scope or invalid_auth indicates the token needs updated permissions or rotation.
• Failed tasks: Review Slack failure alerts; they include the task name, action, and JSON error output.
• Backups: Previous copies of /etc/ansible/hosts may exist with .pre_*.backup naming—maintain or prune as needed.