Private cluster deployment fails - template error

[mlacko64]

Describe the bug
Private cluster deployment fails when masterClusterType=private or routerClusterType=private (or both are private)

To Reproduce

1. subnets and PrivateDNS are created in Azure
2. run following command with latest template

az group deployment create --name MCPOCP-PrivCL-12-DEP --template-file C:\Users\mlacko\azure\azuredeploy.json --parameters C:\Users\mlacko\azure\azuredeploy.parameters.json --parameters _artifactsLocation=https://mcpocpcodeaqs.blob.core.windows.net/mcpocpocp430911 enableCNS=false masterInstanceCount=3 nodeInstanceCount=3 cnsInstanceCount=3 masterVmSize=Standard_D4s_v3 infraInstanceCount=3 infraVmSize=Standard_D4s_v3 nodeVmSize=Standard_D4s_v3 openshiftClusterPrefix=MCPOCP11CN keyVaultResourceGroup=MCPOCP-PrivCL-12-RG keyVaultName=MCPOCP-PrivCL-12-KV rhsmPoolId=some.server.com rhsmBrokerPoolId=some.server.com rhsmUsernameOrOrgId=1234 sshPublicKey=1234 storageKind=managed minorVersion=82 adminUsername=1234 keyVaultSubscriptionId=1234 aadClientId=1234 domainName=private.pokus.xyz virtualNetworkName=openshiftvnet masterSubnetName=mastersubnet infraSubnetName=infrasubnet nodeSubnetName=nodesubnet enableLogging=false enableMetrics=false existingMasterSubnetReference=/subscriptions//resourceGroups/MCPOCP-PrivCL-12-RG/providers/Microsoft.Network/virtualNetworks/PokusAzurePrivate/subnets/PokusAzurePrivateMasterSubnet existingInfraSubnetReference=/subscriptions//resourceGroups/MCPOCP-PrivCL-12-RG/providers/Microsoft.Network/virtualNetworks/PokusAzurePrivate/subnets/PokusAzurePrivateInfraSubnet existingNodeSubnetReference=/subscriptions//resourceGroups/MCPOCP-PrivCL-12-RG/providers/Microsoft.Network/virtualNetworks/PokusAzurePrivate/subnets/PokusAzurePrivateNodeSubnet existingCnsSubnetReference=/subscriptions//resourceGroups/MCPOCP-PrivCL-12-RG/providers/Microsoft.Network/virtualNetworks/PokusAzurePrivate/subnets/PokusAzurePrivateInfraSubnet masterPrivateClusterIp=10.10.1.200 masterClusterDns=console.private.pokus.xyz routerPrivateClusterIp=10.10.2.200 routingSubDomain=apps.private.pokus.xyz masterClusterType=private masterClusterDnsType=custom routerClusterType=private routingSubDomainType=custom --resource-group MCPOCP-PrivCL-12-RG --verbose --debug

Expected behavior
Private cluster is deployed

Screenshots
no

stdout
when routerClusterType=private and masterClusterType=public:

msrest.http_logger : {"error":{"code":"InvalidTemplate","message":"Deployment template validation failed: 'The template resource 'Microsoft.Resources/deployments/OpenShiftDeployment' reference to 'Microsoft.Network/publicIPAddresses/infradnsqm25fqgo46jha' requires an API version. Please see https://aka.ms/arm-template for usage details.'.","additionalInfo":[{"type":"TemplateViolation","info":{"lineNumber":0,"positionNumber":0,"snippet":""}}]}}
msrest.exceptions : Unable to deserialize to object: type, KeyError: 'TypedErrorInfo'
msrest.exceptions : Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/****/resourcegroups/MCPOCP-PrivCL-12-RG/providers/Microsoft.Resources/deployments/MCPOCP-PrivCL-12-DEP?api-version=2018-05-01

when masterClusterType=private and/or routerClusterType=private :

msrest.http_logger : {"error":{"code":"InvalidTemplate","message":"Deployment template validation failed: 'The template resource 'Microsoft.Resources/deployments/OpenShiftDeployment' reference to 'Microsoft.Network/publicIPAddresses/masterdnsreymdmx3c62m4' requires an API version. Please see https://aka.ms/arm-template for usage details.'.","additionalInfo":[{"type":"TemplateViolation","info":{"lineNumber":0,"positionNumber":0,"snippet":""}}]}}
msrest.exceptions : Unable to deserialize to object: type, KeyError: 'TypedErrorInfo'
msrest.exceptions : Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/****/resourcegroups/MCPOCP-PrivCL-12-RG/providers/Microsoft.Resources/deployments/MCPOCP-PrivCL-12-DEP?api-version=2018-05-01

Template Information (please complete the following information):

• OS: RedHat Linux
• Branch: master

Additional context
when both masterClusterType and routerClusterType are set to public (or not set) , deployment works

[haroldwongms]

When masterClusterType and routerClusterType are set to private, then you must also provide a domainName. In your example, it would be private.pokus.xyz. You also need to have DNS configured properly to register hosts and resolve hosts in your domain.

[mlacko64]

Hello,
yes domain name is missing in info I have provided (because I have hardcoded it to my template and forget to paste here (sorry for that)). Anyway even with domain name and working private DNS resolver (I used native private azure DNS - currently in "tech preview" mode, but working) I was still receiving this error.

I was later able to fix it when I added following condition to some template lines. I believe issue is that when I specify private cluster , template still tries to use some resources which are but created only in case of public cluster - because of condition ""condition": "[equals(parameters('routerClusterType'), 'public')]","

so here is what I changed in azuredeploy.json :

"infraLbPublicIpDnsLabel": "[if(equals(parameters('routerClusterType'), 'public'),concat('infradns', uniqueString(concat(resourceGroup().id, 'infra'))),'no_infra_label')]",

"openshiftMasterPublicIpDnsLabel": "[if(equals(parameters('masterClusterType'), 'public'),concat('masterdns', uniqueString(concat(resourceGroup().id, 'master'))),'no_master_label')]",

"masterPublicIpAddressId": "[if(equals(parameters('masterClusterType'), 'public'),resourceId('Microsoft.Network/publicIPAddresses', variables('openshiftMasterPublicIpDnsLabel')),json('null'))]",

"infraPublicIpAddressId": "[if(equals(parameters('routerClusterType'), 'public'),resourceId('Microsoft.Network/publicIPAddresses', variables('infraLbPublicIpDnsLabel')),json('null'))]",

"nipioDomain": {
"value": "[if(equals(parameters('routerClusterType'), 'public'), concat(reference(variables('infraLbPublicIpDnsLabel')).ipAddress, '.nip.io'), '.nip.io')]"
},

"OpenshiftConsoleUrl": {
"type": "string",
"value": "[if(equals(parameters('masterClusterType'), 'public'), concat('https://', reference(variables('openshiftMasterPublicIpDnsLabel')).dnsSettings.fqdn, '/console'), concat('https://', variables('masterClusterDns'), '/console'))]"
}

[janegilring]

@mlacko64 Did you resolve this error? I think it is the same I just ran into in #185

[mlacko64]

@janegilring yes, my above mentioned steps helped, seems you are facing same issue; so all you need it to fix reference on non-existing resources related to public master and router IPs and labels (because in case of private cluster they simply do not exists).

[janegilring]

Ah, thanks!

I went ahead and performed the changes in azuredeploy.json, however, I get this error:

Azure Error: InvalidTemplate Message: Deployment template validation failed: 'The template resource 'Microsoft.Resources/deployments/OpenShiftDeployment' reference to 'Microsoft.Network/publicIPAddresses/no_master_label' requires an API version. Please see https://aka.ms/arm-template for usage details.'.

I suppose I need to double check and see if there are additional non-existing resources.

[mlacko64]

@janegilring
seems I forgot to mention this change:

defined variable:
"emptyString": "",

and then changed this line:

				"openshiftMasterPublicIpAddress": {
					"value": "[reference(variables('openshiftMasterPublicIpDnsLabel')).ipAddress]"

to following:

                "openshiftMasterPublicIpAddress": {
                    "value": "[if(equals(parameters('masterClusterType'), 'public'), reference(variables('openshiftMasterPublicIpDnsLabel')).ipAddress, variables('emptyString'))]"

P.S.
these lines can be fixed a bit "more nice" if you want (with null json):

    "masterPublicIpAddressId": "[if(equals(parameters('masterClusterType'), 'public'),resourceId('Microsoft.Network/publicIPAddresses', variables('openshiftMasterPublicIpDnsLabel')),json('null'))]",

    "infraPublicIpAddressId": "[if(equals(parameters('routerClusterType'), 'public'),resourceId('Microsoft.Network/publicIPAddresses', variables('infraLbPublicIpDnsLabel')),json('null'))]",

[janegilring]

@mlacko64 Thanks a lot, that did the trick. I suppose we should create a pull request for this. If you don't have time, I`m happy to do it and give you the credits in the PR comment.

[mlacko64]

@janegilring I am happy to hear that it help. I haven't created pull request when I was facing this issue as I was not sure if I am not doing something wrong (it was first time for me using Azure templates).

Yes, it seems that pull request should be created, if you have time to create it , I will be grateful to you. Thanks for cooperation :)

[mlacko64]

@janegilring I am happy to hear that it help. I haven't created pull request when I was facing this issue as I was not sure if I am not doing something wrong (it was first time for me using Azure templates).

Yes, it seems that pull request should be created, if you have time to create it , I will be grateful to you. Thanks for cooperation :)