Zim's Notes

Just work related notes.

New Ansible Module Azure_rm_hdinsightcluster

https://github.com/Azure-Samples/ansible-playbooks/blob/master/hdinsight-create.yml

This sample will:

  • create resource group
  • create storage account
  • obtain storage account keys required by the cluster
  • create HDInsight cluster
  • resize cluster
  • delete cluster

Create Random Prefix

1
2
3
4
5
6

- hosts: localhost
  tasks:
    - name: Prepare random prefix
      set_fact:
        rpfx: "{{ resource_group_name | hash('md5') | truncate(7, True, '') }}{{ 1000 | random }}"
      run_once: yes

Variables

1
2
3
4
5
6
7

  vars:
    resource_group: "{{ resource_group_name }}"
    location: eastus
    vnet_name: myVirtualNetwork
    subnet_name: mySubnet
    cluster_name: mycluster{{ rpfx }}
    storage_account_name: mystorage{{ rpfx }}

Create Resource Group

1
2
3
4

- name: Create a resource group
  azure_rm_resourcegroup:
    name: "{{ resource_group }}"
    location: "{{ location }}"

Create Storage Account and Obtain Keys

1
2
3
4
5
6

- name: Create storage account
  azure_rm_storageaccount:
      resource_group: "{{ resource_group }}"
      name: "{{ storage_account_name }}"
      account_type: Standard_LRS
      location: eastus2

1
2
3
4
5
6
7
8
9
10
11
12
13
14

- name: Get storage account keys
  azure_rm_resource:
    api_version: '2018-07-01'
    method: POST
    resource_group: "{{ resource_group }}"
    provider: storage
    resource_type: storageaccounts
    resource_name: "{{ storage_account_name }}"
    subresource:
      - type: listkeys
  register: storage_output

- debug:
    var: storage_output

Create Instance of HDInsight Cluster

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

- name: Create instance of Cluster
  azure_rm_hdinsightcluster:
    resource_group: "{{ resource_group }}"
    name: "{{ cluster_name }}"
    location: eastus2
    cluster_version: 3.6
    os_type: linux
    tier: standard
    cluster_definition:
      kind: spark
      gateway_rest_username: http-user
      gateway_rest_password: MuABCPassword!!@123
    storage_accounts:
      - name: "{{ storage_account_name }}.blob.core.windows.net"
        is_default: yes
        container: "{{ cluster_name }}"
        key: "{{ storage_output['response']['keys'][0]['value'] }}"
    compute_profile_roles:
      - name: headnode
        target_instance_count: 1
        vm_size: Standard_D3
        linux_profile:
          username: sshuser
          password: MuABCPassword!!@123
      - name: workernode
        target_instance_count: 1
        vm_size: Standard_D3
        linux_profile:
          username: sshuser
          password: MuABCPassword!!@123
      - name: zookeepernode
        target_instance_count: 3
        vm_size: Medium
        linux_profile:
          username: sshuser
          password: MuABCPassword!!@123

Resize Cluster

The only thing that can be changed after HDInsight cluster is created is the number of worker nodes. Below task will increment number of nodes from 1 to 2.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

- name: Resize cluster
  azure_rm_hdinsightcluster:
    resource_group: "{{ resource_group }}"
    name: "{{ cluster_name }}"
    location: eastus2
    cluster_version: 3.6
    os_type: linux
    tier: standard
    cluster_definition:
      kind: spark
      gateway_rest_username: http-user
      gateway_rest_password: MuABCPassword!!@123
    storage_accounts:
      - name: "{{ storage_account_name }}.blob.core.windows.net"
        is_default: yes
        container: "{{ cluster_name }}"
        key: "{{ storage_output['response']['keys'][0]['value'] }}"
    compute_profile_roles:
      - name: headnode
        target_instance_count: 1
        vm_size: Standard_D3
        linux_profile:
          username: sshuser
          password: MuABCPassword!!@123
      - name: workernode
        target_instance_count: 2
        vm_size: Standard_D3
        linux_profile:
          username: sshuser
          password: MuABCPassword!!@123
      - name: zookeepernode
        target_instance_count: 3
        vm_size: Medium
        linux_profile:
          username: sshuser
          password: MuABCPassword!!@123
    tags:
      aaa: bbb
  register: output

Clean Up

1
2
3
4
5

- name: Delete instance of Cluster
  azure_rm_hdinsightcluster:
    resource_group: "{{ resource_group }}"
    name: "{{ cluster_name }}"
    state: absent

New Ansible Module Azure_rm_cosmosdbaccount

Cosmos DB Account sample is available here:

https://github.com/Azure-Samples/ansible-playbooks/blob/master/cosmosdb-create.yml

This sample:

  • creates random postfix to use in CosmosDB account name
  • creates resource group
  • creates virtual network
  • creates subnet
  • creates CosmosDB account
  • queries and prints CosmosDB keys

Create Random Postfix

1
2
3
4
5
6

- hosts: localhost
  tasks:
    - name: Prepare random postfix
      set_fact:
        rpfx: "{{ 1000 | random }}"
      run_once: yes

Variables

1
2
3
4
5
6

  vars:
    resource_group: "{{ resource_group_name }}"
    location: eastus
    vnet_name: myVirtualNetwork
    subnet_name: mySubnet
    cosmosdbaccount_name: cosmos{{ rpfx }}

Following variables can be set in vars section of the playbook:

Variable Name Description Notes
resource_group resource group where resources will be created by default it’s using resource_group_name parameter passed when running the playbook
location location where resources should be created
vnet_name virtual network name
subnet subnet name
cosmosdbaccount_name CosmosDB account name should include only lowercase characters and be globally unique

Create Resource Group

This simple task creates a resource group if doesn’t exist yet.

1
2
3
4

- name: Create a resource group
  azure_rm_resourcegroup:
    name: "{{ resource_group }}"
    location: "{{ location }}"

Create Virtual Network and Subnet

This task created prerequisites for CosmosDB account:

  • virtual network
  • subnet
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

- name: Create virtual network
  azure_rm_virtualnetwork:
    resource_group: "{{ resource_group }}"
    name: "{{ vnet_name }}"
    address_prefixes_cidr:
      - 10.1.0.0/16
      - 172.100.0.0/16
    dns_servers:
      - 127.0.0.1
      - 127.0.0.3

- name: Add subnet
  azure_rm_subnet:
    name: "{{ subnet_name }}"
    virtual_network_name: "{{ vnet_name }}"
    resource_group: "{{ resource_group }}"
    address_prefix_cidr: "10.1.0.0/24"

Create CosmosDB Account

This task creates actual CosmosDB account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

- name: Create instance of Cosmos DB Account
  azure_rm_cosmosdbaccount:
    resource_group: "{{ resource_group }}"
    name: "{{ cosmosdbaccount_name }}"
    location: "{{ location }}"
    kind: global_document_db
    geo_rep_locations:
      - name: eastus
        failover_priority: 0
      - name: westus
        failover_priority: 1
    database_account_offer_type: Standard
    is_virtual_network_filter_enabled: yes
    virtual_network_rules:
      - subnet:
          resource_group: "{{ resource_group }}"
          virtual_network_name: "{{ vnet_name }}"
          subnet_name: "{{ subnet_name }}"
        ignore_missing_vnet_service_endpoint: yes
    enable_automatic_failover: yes

Retrieve Keys

This task shows how to retrieve CosmosDB account keys that should be used later by any application using the database:

1
2
3
4
5
6
7
8
9
10

- name: Get Cosmos DB Account facts with keys
  azure_rm_cosmosdbaccount_facts:
    resource_group: "{{ resource_group }}"
    name: "{{ cosmosdbaccount_name }}"
    retrieve_keys: all
  register: output

- name: Display Cosmos DB Acccount facts output
  debug:
    var: output

Use in your application

…

Note on Ansible Examples

When I started working with Ansible I wa

Let’s do following exercise. I want to create an Azure Virtual Machine using examples from documentation.

Let’s start with default virtual machine example (https://docs.ansible.com/ansible/latest/modules/azure_rm_virtualmachine_module.html)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

- name: Create Azure VM using default samples
  hosts: localhost
  connection: local
  tasks:
    - name: Create VM with defaults
      azure_rm_virtualmachine:
        resource_group: Testing
        name: testvm10
        admin_username: chouseknecht
        admin_password: <your password here>
        image:
          offer: CentOS
          publisher: OpenLogic
          sku: '7.1'
          version: latest

Now, Ansible will complain that resource group Testing is not present, so I add resource group from here https://docs.ansible.com/ansible/latest/modules/azure_rm_resourcegroup_module.html:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

- name: Create Azure VM using default samples
  hosts: localhost
  connection: local
  tasks:
    - name: Create a resource group
      azure_rm_resourcegroup:
        name: Testing
        location: westus
        tags:
          testing: testing
          delete: never
    - name: Create VM with defaults
      azure_rm_virtualmachine:
        resource_group: Testing
        name: testvm10
        admin_username: chouseknecht
        admin_password: <your password here>
        image:
          offer: CentOS
          publisher: OpenLogic
          sku: '7.1'
          version: latest

Removing Resource Locks Using Ansible and Azure REST API

This is one of the topic I had to look into. Removing locks appeared to be an issue while performing automatic clean up of the resources.

Currently there’s no support for locks in Ansible, but I have tried to use azure_rm_resource_facts module to list locks on both resource group and subscription level, and then delete locks using azure_rm_resource.

To list all the locks in the resource group:

1
2
3
4
5
6
7
8
9
10

  - name: List all the locks in the resource group
    azure_rm_resource_facts:
      api_version: '2016-09-01'
      resource_group: ""
      provider: authorization
      resource_type: locks
    register: output

  - debug:
      var: output

or all resources in the subscription, just remove resource_group parameter:

1
2
3
4
5
6
7
8
9

  - name: List all the locks in the resource group
    azure_rm_resource_facts:
      api_version: '2016-09-01'
      provider: authorization
      resource_type: locks
    register: output

  - debug:
      var: output

Output will look as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

{
    "output": {
        "changed": false,
        "failed": false,
        "response": [
            {
                "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimslockedrg/providers/Microsoft.Network/virtualNetworks/zimslockedvb/providers/Microsoft.Authorization/locks/justalock",
                "name": "justalock",
                "properties": {
                    "level": "CanNotDelete",
                    "notes": "blabla"
                },
                "type": "Microsoft.Authorization/locks"
            }
        ],
        "url": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimslockedrg/providers/Microsoft.authorization/locks",
        "warnings": [
            "Azure API profile latest does not define an entry for GenericRestClient"
        ]
    }
}

Based on this output you can just delete all the locks one by one using azure_rm_resource module:

1
2
3
4
5
6
7
8

  - name: Delete locks one by one
    azure_rm_resource:
      api_version: '2016-09-01'
      url: ""
      provider: authorization
      resource_type: locks
      state: absent
    with_items: ""

Please note that there’s a bug in the versions of Ansible earlier than 2.8, and the last playbook needs small modification:

1
2
3
4
5
6
7
8

  - name: Delete locks one by one
    azure_rm_resource:
      api_version: '2016-09-01'
      url: ""
      provider: authorization
      resource_type: locks
      state: absent
    with_items: ""

Ansible and Azure Function Apps

This is something I was planning to investigate for some time. We have azure_rm_functionapp module, but it seems to be very limited. Actually I don’t really know what it creates.

I decided to reverse engineer a very simple scenario - creating a container based function app through Azure Portal.

The best way to figure out what was created is to use following playbook to list all the resources in the resource group:

1
2
3
4
5
6
7
8

- name: List all the resources in the resource group
  azure_rm_resource_facts:
    api_version: '2018-05-01'
    resource_group: zimsfunctionapp
    resource_type: resources
  register: output
- debug:
    var: output

Inside response I found following list of resources:

1
2
3
4
5
6
7
8
9

[
    {
        "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsfunctionapp/providers/Microsoft.Web/sites/zimsfunctionapp",
        "kind": "functionapp,linux,container",
        "location": "westus",
        "name": "zimsfunctionapp",
        "type": "Microsoft.Web/sites"
    }
]

Now I can do another query to get more details:

1
2
3
4
5
6
7
8
9
10

- name: Get WebApp information
  azure_rm_resource_facts:
    api_version: '2016-08-01'
    provider: web
    resource_group: zimsfunctionapp
    resource_type: sites
    resource_name: zimsfunctionapp
  register: output
- debug:
    var: output

And that returs following large structure:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

[
    {
        "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsfunctionapp/providers/Microsoft.Web/sites/zimsfunctionapp",
        "kind": "functionapp,linux,container",
        "location": "West US",
        "name": "zimsfunctionapp",
        "properties": {
            "adminEnabled": true,
            "availabilityState": "Normal",
            "cers": null,
            "clientAffinityEnabled": true,
            "clientCertEnabled": false,
            "clientCertExclusionPaths": null,
            "cloningInfo": null,
            "computeMode": null,
            "containerSize": 1536,
            "contentAvailabilityState": "Normal",
            "csrs": [],
            "dailyMemoryTimeQuota": 0,
            "defaultHostName": "zimsfunctionapp.azurewebsites.net",
            "deploymentId": "zimsfunctionapp",
            "domainVerificationIdentifiers": null,
            "enabled": true,
            "enabledHostNames": [
                "zimsfunctionapp.azurewebsites.net",
                "zimsfunctionapp.scm.azurewebsites.net"
            ],
            "functionExecutionUnitsCache": null,
            "geoDistributions": null,
            "homeStamp": "waws-prod-bay-081",
            "hostNameSslStates": [
                {
                    "hostType": "Standard",
                    "ipBasedSslResult": null,
                    "ipBasedSslState": "NotConfigured",
                    "name": "zimsfunctionapp.azurewebsites.net",
                    "sslState": "Disabled",
                    "thumbprint": null,
                    "toUpdate": null,
                    "toUpdateIpBasedSsl": null,
                    "virtualIP": null
                },
                {
                    "hostType": "Repository",
                    "ipBasedSslResult": null,
                    "ipBasedSslState": "NotConfigured",
                    "name": "zimsfunctionapp.scm.azurewebsites.net",
                    "sslState": "Disabled",
                    "thumbprint": null,
                    "toUpdate": null,
                    "toUpdateIpBasedSsl": null,
                    "virtualIP": null
                }
            ],
            "hostNames": [
                "zimsfunctionapp.azurewebsites.net"
            ],
            "hostNamesDisabled": false,
            "hostingEnvironment": null,
            "hostingEnvironmentId": null,
            "hostingEnvironmentProfile": null,
            "httpsOnly": false,
            "hyperV": false,
            "inProgressOperationId": null,
            "isXenon": false,
            "kind": "functionapp,linux,container",
            "lastModifiedTimeUtc": "2019-03-07T13:52:14.7033333",
            "maxNumberOfWorkers": null,
            "name": "zimsfunctionapp",
            "outboundIpAddresses": "13.64.73.110,40.118.133.8,40.118.169.141,40.118.253.162,13.64.147.140",
            "owner": null,
            "possibleOutboundIpAddresses": "13.64.73.110,40.118.133.8,40.118.169.141,40.118.253.162,13.64.147.140,52.160.85.217,13.93.238.69",
            "redundancyMode": "None",
            "repositorySiteName": "zimsfunctionapp",
            "reserved": true,
            "resourceGroup": "zimsfunctionapp",
            "runtimeAvailabilityState": "Normal",
            "scmSiteAlsoStopped": false,
            "selfLink": "https://waws-prod-bay-081.api.azurewebsites.windows.net:454/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/webspaces/jw-webapp-linux-nginx-06-WestUSwebspace/sites/zimsfunctionapp",
            "serverFarm": null,
            "serverFarmId": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/jw-webapp-linux-nginx-06/providers/Microsoft.Web/serverfarms/jw-webapp-linux-plan-01",
            "siteConfig": null,
            "siteDisabledReason": 0,
            "siteMode": null,
            "siteProperties": {
                "appSettings": null,
                "metadata": null,
                "properties": [
                    {
                        "name": "LinuxFxVersion",
                        "value": "DOCKER|httpd"
                    },
                    {
                        "name": "WindowsFxVersion",
                        "value": null
                    }
                ]
            },
            "sku": "Standard",
            "slotSwapStatus": null,
            "sslCertificates": null,
            "state": "Running",
            "storageRecoveryDefaultState": "Running",
            "suspendedTill": null,
            "tags": null,
            "targetSwapSlot": null,
            "trafficManagerHostNames": null,
            "usageState": "Normal",
            "webSpace": "jw-webapp-linux-nginx-06-WestUSwebspace"
        },
        "type": "Microsoft.Web/sites"
    }
]

Diving Into Azure_rm_deployment

It’s time to refresh azure_rm_deployment.

Currently we have following problems with azure_rm_deployment

  • it’s a kind of VM oriented only
  • no proper facts module, no proper information about created resources, so it’s difficult to build on the top of resources created by ARM deployment
  • deleting deployment actually deletes entire resource group, so no convenient way to have any additional resources in the same resource group
  • default deployment name (user can optionally specify) is confusing and inconsistent with other resources

What we can fix, and it’s quite easy: - facts module that lists all the resource ids in dictionary by resource type - properly deleting resources created by deployment, enabling users to have more than one deployment in a resource group

Let’s start with a simple playbook. I have just taken it from integration tests:

How it currently works?

Just a simple playbook that creates a virtual machine:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

- name: Create Azure Deploy
    azure_rm_deployment:
    resource_group: ""
    location: eastus2
    template_link: 'https://raw.githubusercontent.com/Azure/azure-quickstart-templates/d01a5c06f4f1bc03a049ca17bbbd6e06d62657b3/101-vm-simple-linux/azuredeploy.json'
    deployment_name: ""
    parameters:
        adminUsername:
        value: chouseknecht
        adminPassword:
        value: password123!
        dnsLabelPrefix:
        value: ""
        ubuntuOSVersion:
        value: "16.04.0-LTS"
    register: output

- debug:
    var: output

and the output looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

{
    "output": {
        "changed": true,
        "deployment": {
            "group_name": "zimsdeployment",
            "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/zimsdeployment/providers/Microsoft.Resources/deployments/zimsdeployment",
            "instances": [
                {
                    "ips": [
                        {
                            "dns_settings": {
                                "domain_name_label": "zimsdeployment",
                                "fqdn": "zimsdeployment.eastus2.cloudapp.azure.com"
                            },
                            "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/zimsdeployment/providers/Microsoft.Network/publicIPAddresses/myPublicIP",
                            "name": "myPublicIP",
                            "public_ip": "23.100.65.195",
                            "public_ip_allocation_method": "Dynamic"
                        }
                    ],
                    "vm_name": "MyUbuntuVM"
                }
            ],
            "name": "zimsdeployment",
            "outputs": {
                "hostname": {
                    "type": "String",
                    "value": "zimsdeployment.eastus2.cloudapp.azure.com"
                },
                "sshCommand": {
                    "type": "String",
                    "value": "ssh chouseknecht@zimsdeployment.eastus2.cloudapp.azure.com"
                }
            }
        },
        "failed": false,
        "msg": "deployment succeeded"
    }
}

What is actually returned by Azure REST API?

I have created following task to find it out:

1
2
3
4
5
6
7
8
9
10

- name: Get deployment
    azure_rm_resource_facts:
    api_version: '2019-03-01'
    resource_group: ""
    provider: resources
    resource_type: deployments
    resource_name:  ""
    register: output
- debug:
    var: output

As you see below it actually returns quite a lot: - list of all created resource IDs - list of all dependencies between resources

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156

{
    "output": {
        "changed": false,
        "failed": false,
        "response": [
            {
                "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Resources/deployments/zimsdplxxxx",
                "name": "zimsdplxxxx",
                "properties": {
                    "correlationId": "0c26c003-f147-4f18-95c6-a28e43630dd1",
                    "dependencies": [
                        {
                            "dependsOn": [
                                {
                                    "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Network/publicIPAddresses/myPublicIP",
                                    "resourceName": "myPublicIP",
                                    "resourceType": "Microsoft.Network/publicIPAddresses"
                                },
                                {
                                    "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Network/virtualNetworks/MyVNET",
                                    "resourceName": "MyVNET",
                                    "resourceType": "Microsoft.Network/virtualNetworks"
                                }
                            ],
                            "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Network/networkInterfaces/myVMNic",
                            "resourceName": "myVMNic",
                            "resourceType": "Microsoft.Network/networkInterfaces"
                        },
                        {
                            "dependsOn": [
                                {
                                    "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Storage/storageAccounts/iovbjgl443l64sawinvm",
                                    "resourceName": "iovbjgl443l64sawinvm",
                                    "resourceType": "Microsoft.Storage/storageAccounts"
                                },
                                {
                                    "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Network/networkInterfaces/myVMNic",
                                    "resourceName": "myVMNic",
                                    "resourceType": "Microsoft.Network/networkInterfaces"
                                }
                            ],
                            "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Compute/virtualMachines/SimpleWinVM",
                            "resourceName": "SimpleWinVM",
                            "resourceType": "Microsoft.Compute/virtualMachines"
                        }
                    ],
                    "duration": "PT46.927468S",
                    "mode": "Incremental",
                    "outputResources": [
                        {
                            "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Compute/virtualMachines/SimpleWinVM"
                        },
                        {
                            "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Network/networkInterfaces/myVMNic"
                        },
                        {
                            "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Network/publicIPAddresses/myPublicIP"
                        },
                        {
                            "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Network/virtualNetworks/MyVNET"
                        },
                        {
                            "id": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.Storage/storageAccounts/iovbjgl443l64sawinvm"
                        }
                    ],
                    "outputs": {
                        "hostname": {
                            "type": "String",
                            "value": "zimsdplxxxx.eastus2.cloudapp.azure.com"
                        }
                    },
                    "parameters": {
                        "adminPassword": {
                            "type": "SecureString"
                        },
                        "adminUsername": {
                            "type": "String",
                            "value": "chouseknecht"
                        },
                        "dnsLabelPrefix": {
                            "type": "String",
                            "value": "zimsdplxxxx"
                        },
                        "location": {
                            "type": "String",
                            "value": "eastus2"
                        },
                        "windowsOSVersion": {
                            "type": "String",
                            "value": "2016-Datacenter"
                        }
                    },
                    "providers": [
                        {
                            "namespace": "Microsoft.Storage",
                            "resourceTypes": [
                                {
                                    "locations": [
                                        "eastus2"
                                    ],
                                    "resourceType": "storageAccounts"
                                }
                            ]
                        },
                        {
                            "namespace": "Microsoft.Network",
                            "resourceTypes": [
                                {
                                    "locations": [
                                        "eastus2"
                                    ],
                                    "resourceType": "publicIPAddresses"
                                },
                                {
                                    "locations": [
                                        "eastus2"
                                    ],
                                    "resourceType": "virtualNetworks"
                                },
                                {
                                    "locations": [
                                        "eastus2"
                                    ],
                                    "resourceType": "networkInterfaces"
                                }
                            ]
                        },
                        {
                            "namespace": "Microsoft.Compute",
                            "resourceTypes": [
                                {
                                    "locations": [
                                        "eastus2"
                                    ],
                                    "resourceType": "virtualMachines"
                                }
                            ]
                        }
                    ],
                    "provisioningState": "Succeeded",
                    "templateHash": "17224794003184781395",
                    "templateLink": {
                        "contentVersion": "1.0.0.0",
                        "uri": "https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/101-vm-simple-windows/azuredeploy.json"
                    },
                    "timestamp": "2019-03-07T05:49:05.3868647Z"
                },
                "type": "Microsoft.Resources/deployments"
            }
        ],
        "url": "/subscriptions/1c5b82ee-9294-4568-b0c0-b9c523bc0d86/resourceGroups/zimsanotherdeployment/providers/Microsoft.resources/deployments/zimsdplxxxx",
        "warnings": [
            "Azure API profile latest does not define an entry for GenericRestClient"
        ]
    }
}

Ansible Modules - Proposed Improvements

I have done a lot of experiments with Ansible module idempotence and here’s a summary and a proposal of generic solution.

Let’s start with problems we have: - every module has their own idempotency checks so the behaviour is not consistent - we agreed to display warning when change is detected but can’t be applied. This is not implemented yet in majority in the modules.

Manual Idempotence Check

  • error prone
  • time consuming
  • no consistency between modules

Generic Compare Function

  • quite effective
  • robust
  • difficult to handle exceptions

Comparison Types

Type Description
exact This kind of comparison should work for most of the properties.
case insensitive for instance resource id may need this type of comparison
location location may be returned in different formats, for instance east us or East US
ignore for properties that are not returned by GET, for instance passwords, secrets, or any other fields that we can’t perform comparison

Exceptions

  • sometimes (quite rarely) structure returned by GET is different from PUT
  • some properties are not returned by GET, so can’t compare
  • sometimes comparison method needs to be adjusted to particular scenario

Proposed Approach: Global Compare Function + Metadata in module_arg_spec

I have done appropriate tests and: - it’s possible to add custom metadata to module_arg_spec without disturbing Ansible sanity checks

Therefore we could have Azure specific metadata defined as follows, to provide auxilliary information for modules about: - updatability of the field - field relationship to structures obtained from Get and CreateOrUpdate - idempotence comparison method

1
2
3
4
5
6
7
8
9
10
11

self.module_arg_spec = dict(
    # ...
    option_a=dict(
        type='str',
        required=True,
        # Azure Module Specific Metadata
        updatable=False,
        disposition='parameters/option_a',
        comparison='insensitive'
    ),
)
Name Default Description
updatable True Set to False if property cannot be updated. Warning will be automatically generated if change is detected.
disposition - Different name or path if property in different location after expanding options
comparison exact Comparison method, use one of: exact, insensitive, location, ignore

Module base will provide function:

map_and_compare(old_properties)

old_properties is a dictionary returned by Get or empty dictionary {}.

Function will map new property values into existing dictionary or populate empty dictionary.

Function will return True if any change is detected.

Function will generate appropriate warnings if change of non-updatable fields is detected.

Ansible Module Anatomy

This post describes what and where needs to be added in order to create a new module.

File Locations

Let’s say you want to create modules for a new Azure resource. You will need two modules - main module and facts module. Following files should be created in Ansible directory:

1
2

~/lib/ansible/modules/cloud/azure/azure_rm_xxxxx.py
~/lib/ansible/modules/cloud/azure/azure_rm_xxxxx_facts.py

In addition to modules you need to create integration test. In order to do that you will have to create following files:

1
2
3

~/test/integration/targets/azure_rm_xxxx/meta/main.yml
~/test/integration/targets/azure_rm_xxxx/tasks/main.yml
~/test/integration/targets/azure_rm_xxxx/aliases

Please note that integration tests can be shared (and in many cases should be shared) by multiple modules.

Typical Module Dissected

Header

Module header is always the same. Only parts that are changed are: - copyright year - original contributor name - original contributor git handle

1
2
3
4
5
6
7
8
9
10
11
12
13

#!/usr/bin/python
#
# Copyright (c) 2019 John Doe, (@johndoe)
#
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

from __future__ import absolute_import, division, print_function
__metaclass__ = type


ANSIBLE_METADATA = {'metadata_version': '1.1',
                    'status': ['preview'],
                    'supported_by': 'community'}

Documentation

The next section is the documentation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

DOCUMENTATION = '''
---
module: azure_rm_devtestlabvirtualmachine
version_added: "2.8"
short_description: Manage Azure DevTest Lab Virtual Machine instance.
description:
    - Create, update and delete instance of Azure DevTest Lab Virtual Machine.

options:
    resource_group:
        description:
            - The name of the resource group.
        required: True
    lab_name:
        description:
            - The name of the lab.
        required: True
    name:
        description:
            - The name of the virtual machine.
        required: True
    notes:
        description:
            - The notes of the virtual machine.
    state:
      description:
        - Assert the state of the Virtual Machine.
        - Use 'present' to create or update an Virtual Machine and 'absent' to delete it.
      default: present
      choices:
        - absent
        - present

extends_documentation_fragment:
    - azure
    - azure_tags

author:
    - "Zim Kalinowski (@zikalino)"
'''

Examples

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

EXAMPLES = '''
  - name: Create (or update) Virtual Machine
    azure_rm_devtestlabvirtualmachine:
      resource_group: myrg
      lab_name: mylab
      name: myvm
      notes: Virtual machine notes....
      os_type: linux
      vm_size: Standard_A2_v2
      user_name: vmadmin
      password: ZSuppas$$21!
      lab_subnet:
        name: myvnSubnet
        virtual_network_name: myvn
      disallow_public_ip_address: no
      image:
        offer: UbuntuServer
        publisher: Canonical
        sku: 16.04-LTS
        os_type: Linux
        version: latest
      artifacts:
        - source_name: myartifact
          source_path: "/Artifacts/linux-install-mongodb"
      allow_claim: no
      expiration_date: "2019-02-22T01:49:12.117974Z"
'''

Return Value

Return value should contain detailed specification of return value in YAML format.

Please note that the main module should: - contain resource id - contain information that is essential for subsequent use of the resource (for instance FQDN, IP address, etc.) - shouldn’t contain information that is supplied via input parameters

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

RETURN = '''
id:
    description:
        - The identifier of the DTL Virtual Machine resource.
    returned: always
    type: str
    sample: /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/myrg/providers/microsoft.devtestlab/labs/mylab/virtualmachines/myvm
compute_id:
    description:
        - The identifier of the underlying Compute Virtual Machine resource.
    returned: always
    type: str
    sample: /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/myrg/providers/microsoft.devtestlab/labs/mylab/virtualmachines/myvm
fqdn:
    description:
        - Fully qualified domain name or IP Address of the virtual machine.
    returned: always
    type: str
    sample: myvm.eastus.cloudapp.azure.com
'''

Imports

Usually very similar, may contain additional imports related to particular Azure resource, etc.

1
2
3
4
5
6
7
8
9
10
11
12
13

import time
from ansible.module_utils.azure_rm_common import AzureRMModuleBase
from ansible.module_utils.common.dict_transformations import _snake_to_camel

try:
    from msrestazure.azure_exceptions import CloudError
    from msrest.polling import LROPoller
    from msrestazure.azure_operation import AzureOperationPoller
    from azure.mgmt.devtestlabs import DevTestLabsClient
    from msrest.serialization import Model
except ImportError:
    # This is handled in azure_rm_common
    pass

Class Definition and Init Function

Action types definition:

1
2

class Actions:
    NoAction, Create, Update, Delete = range(4)

1
2
3

class AzureRMVirtualMachine(AzureRMModuleBase):
    """Configuration class for an Azure RM Virtual Machine resource"""
    def __init__(self):

Module Argument Specification

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

    self.module_arg_spec = dict(
        resource_group=dict(
            type='str',
            required=True
        ),
        lab_name=dict(
            type='str',
            required=True
        ),
        name=dict(
            type='str',
            required=True
        ),
        notes=dict(
            type='str'
        )
    )

Conditional Required Options

1
2
3
4

    required_if = [
        ('state', 'present', [
         'image', 'lab_subnet', 'vm_size', 'os_type'])
    ]

Argument Properties

1
2
3
4

    self.resource_group = None
    self.lab_name = None
    self.name = None
    self.lab_virtual_machine = dict()

Module State

1
2
3
4

    self.results = dict(changed=False)
    self.mgmt_client = None
    self.state = None
    self.to_do = Actions.NoAction

Base Class Initialization

1
2
3
4

    super(AzureRMVirtualMachine, self).__init__(derived_arg_spec=self.module_arg_spec,
                                                supports_check_mode=True,
                                                supports_tags=True,
                                                required_if=required_if)

Main Module Execution Method

1
2

def exec_module(self, **kwargs):
    """Main module execution method"""

Mapping parameters to class properties:

1
2
3
4
5

    for key in list(self.module_arg_spec.keys()) + ['tags']:
        if hasattr(self, key):
            setattr(self, key, kwargs[key])
        elif kwargs[key] is not None:
            self.lab_virtual_machine[key] = kwargs[key]

Parameter transformations

Notes: - we have several approaches right now - in general here input parameters structure should be converted to match SDK call parameters

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

    self.lab_virtual_machine['gallery_image_reference'] = self.lab_virtual_machine.pop('image', None)

    if self.lab_virtual_machine.get('artifacts') is not None:
        for artifact in self.lab_virtual_machine.get('artifacts'):
            source_name = artifact.pop('source_name')
            source_path = artifact.pop('source_path')
            template = "/subscriptions/{0}/resourceGroups/{1}/providers/Microsoft.DevTestLab/labs/{2}/artifactsources/{3}{4}"
            artifact['artifact_id'] = template.format(self.subscription_id, self.resource_group, self.lab_name, source_name, source_path)

    self.lab_virtual_machine['size'] = self.lab_virtual_machine.pop('vm_size')
    self.lab_virtual_machine['os_type'] = _snake_to_camel(self.lab_virtual_machine['os_type'], True)

    if self.lab_virtual_machine.get('storage_type'):
        self.lab_virtual_machine['storage_type'] = _snake_to_camel(self.lab_virtual_machine['storage_type'], True)

    lab_subnet = self.lab_virtual_machine.pop('lab_subnet')

    if isinstance(lab_subnet, str):
        vn_and_subnet = lab_subnet.split('/subnets/')
        if (len(vn_and_subnet) == 2):
            self.lab_virtual_machine['lab_virtual_network_id'] = vn_and_subnet[0]
            self.lab_virtual_machine['lab_subnet_name'] = vn_and_subnet[1]
        else:
            self.fail("Invalid 'lab_subnet' resource id format")
    else:
        template = "/subscriptions/{0}/resourceGroups/{1}/providers/Microsoft.DevTestLab/labs/{2}/virtualnetworks/{3}"
        self.lab_virtual_machine['lab_virtual_network_id'] = template.format(self.subscription_id,
                                                                             self.resource_group,
                                                                             self.lab_name,
                                                                             lab_subnet.get('virtual_network_name'))
        self.lab_virtual_machine['lab_subnet_name'] = lab_subnet.get('name')

Create Mgmt Client Instance

1
2
3
4

    response = None

    self.mgmt_client = self.get_mgmt_svc_client(DevTestLabsClient,
                                                base_url=self._cloud_environment.endpoints.resource_manager)

Getting old response and idempotency check

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

    old_response = self.get_virtualmachine()

    if not old_response:
        self.log("Virtual Machine instance doesn't exist")
        if self.state == 'absent':
            self.log("Old instance didn't exist")
        else:
            self.to_do = Actions.Create
        # get location from the lab as it has to be the same and has to be specified (why??)
        lab = self.get_devtestlab()
        self.lab_virtual_machine['location'] = lab['location']
    else:
        self.log("Virtual Machine instance already exists")
        if self.state == 'absent':
            self.to_do = Actions.Delete
        elif self.state == 'present':
            self.lab_virtual_machine['location'] = old_response['location']

            if old_response['size'].lower() != self.lab_virtual_machine.get('size').lower():
                self.lab_virtual_machine['size'] = old_response['size']
                self.module.warn("Property 'size' cannot be changed")

            if self.lab_virtual_machine.get('storage_type') is not None and \
               old_response['storage_type'].lower() != self.lab_virtual_machine.get('storage_type').lower():
                self.lab_virtual_machine['storage_type'] = old_response['storage_type']
                self.module.warn("Property 'storage_type' cannot be changed")

            if old_response.get('gallery_image_reference', {}) != self.lab_virtual_machine.get('gallery_image_reference', {}):
                self.lab_virtual_machine['gallery_image_reference'] = old_response['gallery_image_reference']
                self.module.warn("Property 'image' cannot be changed")

            # currently artifacts can be only specified when vm is created
            # and in addition we don't have detailed information, just a number of "total artifacts"
            if len(self.lab_virtual_machine.get('artifacts', [])) != old_response['artifact_deployment_status']['total_artifacts']:
                self.module.warn("Property 'artifacts' cannot be changed")

            if self.lab_virtual_machine.get('disallow_public_ip_address') is not None:
                if old_response['disallow_public_ip_address'] != self.lab_virtual_machine.get('disallow_public_ip_address'):
                    self.module.warn("Property 'disallow_public_ip_address' cannot be changed")
            self.lab_virtual_machine['disallow_public_ip_address'] = old_response['disallow_public_ip_address']

            if self.lab_virtual_machine.get('allow_claim') is not None:
                if old_response['allow_claim'] != self.lab_virtual_machine.get('allow_claim'):
                    self.module.warn("Property 'allow_claim' cannot be changed")
            self.lab_virtual_machine['allow_claim'] = old_response['allow_claim']

            if self.lab_virtual_machine.get('notes') is not None:
                if old_response['notes'] != self.lab_virtual_machine.get('notes'):
                    self.to_do = Actions.Update
            else:
                self.lab_virtual_machine['notes'] = old_response['notes']

Performing Desired Action

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

    if (self.to_do == Actions.Create) or (self.to_do == Actions.Update):
        self.log("Need to Create / Update the Virtual Machine instance")

        self.results['changed'] = True
        if self.check_mode:
            return self.results

        response = self.create_update_virtualmachine()

        self.log("Creation / Update done")
    elif self.to_do == Actions.Delete:
        self.log("Virtual Machine instance deleted")
        self.results['changed'] = True

        if self.check_mode:
            return self.results

        self.delete_virtualmachine()
    else:
        self.log("Virtual Machine instance unchanged")
        self.results['changed'] = False
        response = old_response

Formatting Response

1
2
3
4
5
6
7

    if self.state == 'present':
        self.results.update({
            'id': response.get('id', None),
            'compute_id': response.get('compute_id', None),
            'fqdn': response.get('fqdn', None)
        })
    return self.results

Create / Update Function

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

def create_update_virtualmachine(self):
    '''
    Creates or updates Virtual Machine with the specified configuration.

    :return: deserialized Virtual Machine instance state dictionary
    '''
    self.log("Creating / Updating the Virtual Machine instance {0}".format(self.name))

    try:
        response = self.mgmt_client.virtual_machines.create_or_update(resource_group_name=self.resource_group,
                                                                      lab_name=self.lab_name,
                                                                      name=self.name,
                                                                      lab_virtual_machine=self.lab_virtual_machine)
        if isinstance(response, LROPoller) or isinstance(response, AzureOperationPoller):
            response = self.get_poller_result(response)

    except CloudError as exc:
        self.log('Error attempting to create the Virtual Machine instance.')
        self.fail("Error creating the Virtual Machine instance: {0}".format(str(exc)))
    return response.as_dict()

Delete Function

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

def delete_virtualmachine(self):
    '''
    Deletes specified Virtual Machine instance in the specified subscription and resource group.

    :return: True
    '''
    self.log("Deleting the Virtual Machine instance {0}".format(self.name))
    try:
        response = self.mgmt_client.virtual_machines.delete(resource_group_name=self.resource_group,
                                                            lab_name=self.lab_name,
                                                            name=self.name)
    except CloudError as e:
        self.log('Error attempting to delete the Virtual Machine instance.')
        self.fail("Error deleting the Virtual Machine instance: {0}".format(str(e)))

    if isinstance(response, LROPoller) or isinstance(response, AzureOperationPoller):
        response = self.get_poller_result(response)

    return True

Get Function

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

def get_virtualmachine(self):
    '''
    Gets the properties of the specified Virtual Machine.

    :return: deserialized Virtual Machine instance state dictionary
    '''
    self.log("Checking if the Virtual Machine instance {0} is present".format(self.name))
    found = False
    try:
        response = self.mgmt_client.virtual_machines.get(resource_group_name=self.resource_group,
                                                         lab_name=self.lab_name,
                                                         name=self.name)
        found = True
        self.log("Response : {0}".format(response))
        self.log("Virtual Machine instance : {0} found".format(response.name))
    except CloudError as e:
        self.log('Did not find the Virtual Machine instance.')
    if found is True:
        return response.as_dict()

    return False

Module Entry Point

Always the same and looks as follows:

1
2
3
4
5
6
7

def main():
    """Main execution"""
    AzureRMVirtualMachine()


if __name__ == '__main__':
    main()

Installing Ansible Using Cloud-init

Recently was playing with cloud-init a bit, and it seems like a great and consistent way of setting up Ansible VMs. Here’s an example of creating a virtual machine via Azure portal and installing Ansible at once.

What you need to do is to go to Guest Config tag and add appropriate cloud-config playbook:

Guest Config

In this example I am using following code:

1
2
3
4
5
6
7

#cloud-config
packages:
  - python-pip
runcmd:
  - sudo pip install ansible
  - sudo ansible-galaxy install azure.azure_preview_modules
  - sudo pip install -r ~/.ansible/roles/azure.azure_preview_modules/files/requirements-azure.txt

It can be even shorted if you don’t need Azure preview modules:

1
2
3
4
5

#cloud-config
packages:
  - python-pip
runcmd:
  - sudo pip install ansible[azure]

Waiting for Resources in Ansible

Some resources may be not in a desired state after running a playbook. Especially when using Azure REST API, resource may be in Creating or Provisioning state. Sometimes it’s necessary to wait unll provisioning state becomes Succeeded.

The best way to do this is to use facts module to periodically query the resource and exit the loop when desired state is achieved. Here’s how to do it.

After the task that creates your virtual machine (or later), put following tasks. Actually only Wait for Virtual Machine to be ready is essential here. All the other tasks are just for demo purposes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

- name: Get VM facts
  azure_rm_resource_facts:
    api_version: '2018-10-01'
    resource_group: ''
    provider: Compute
    resource_type: virtualMachines
    resource_name: ""
  register: output

- debug:
    var: output.response[0].properties.provisioningState

- name: Wait for Virtual Machine to be ready
  azure_rm_resource_facts:
    api_version: '2018-10-01'
    resource_group: ''
    provider: Compute
    resource_type: virtualMachines
    resource_name: ""
  register: output
  until: output.response[0].properties.provisioningState == 'Succeeded'
  delay: 60
  retries: 5

- debug:
    var: output.response[0].properties.provisioningState

You should see following output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

TASK [Create a vm with password authentication.] *******************************************
changed: [localhost]

TASK [Get VM facts] ************************************************************************
ok: [localhost]

TASK [debug] ******************************************************************************
ok: [localhost] => {
    "output.response[0].properties.provisioningState": "Creating"
}

TASK [Wait for Virtual Machine to be ready] *****************************************************
FAILED - RETRYING: Wait for Virtual Machine to be ready (5 retries left).
FAILED - RETRYING: Wait for Virtual Machine to be ready (4 retries left).
ok: [localhost]

TASK [debug] ******************************************************************************
ok: [localhost] => {
    "output.response[0].properties.provisioningState": "Succeeded"
}

PLAY RECAP ********************************************************************************
localhost                  : ok=18   changed=1    unreachable=0    failed=0    skipped=0

References:

Ansible documentation on do-until loops: https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#do-until-loops