Azure DevOps Build Pipeline- use keys and secrets from Key Vault

In Cybersecurity, Defense in Depth (DID) approach should be followed when available. DID is a concept in which multiple layers of security controls are placed throughout an information technology system. Build pipeline is critical infrastructure of the organization and it often requires elevated privilege to access other dependent systems. To elevate the privilege of a service principal, we need keys and/or secrets associated with the account. Hence, the use of Azure Key Vault!

As part of DID, Service Principal (SP) must pass multiple security requirements-

Service Principal (SP) must be a valid member of Azure AD.
SP must be configured with Key Vault Access Policy to list and get keys and secrets.
Key Vault (KV) can be accessed from selected virtual network, trusted azure services and from trusted internet ip’s.
KV must be configured (Diagnostics) to log/monitor Audit Events.

Challenges:

Azure DevOps Build Pipeline can’t get secrets from Key Vault when secured with vNET and Firewall!
You would run into similar challenge with Azure Storage Account when it’s protected by vNET and Firewall.
As of this writing, Azure DevOps is not one of the trusted services and it’s ip address span the entire region (i. e. US East 2).
Key Vault (KV) firewall and vNET limit allow up to 127 rules for each category but a region can have CIDR ranges exceed 300!
Whitelisting the IPs is not an efficient way, but its definitely a workaround. We are working with the Product team to get Azure Dev Ops as a Trusted service for Azure Key Vault and that would be a complete fix in all terms. Do allow us sometime and I will share my next updates with you. — This is the response I have received from Microsoft support team. It’s encouraging but we have to wait!

Solution:

You can add a step in the build definition to whitelist the agent IP address, then remove it from the whitelist at the end of the build. This is not a solution but a workaround until Azure product team adds Azure DevOps as a trusted service. Thanks to @Daniel Mann for providing the idea in response to my query at stackoverflow.com.

The solution is simple but I was not going to trust ipify.org as REST API endpoint to get my build agent’s ip address. Instead, I created my own (and trusted) service at Azure Function- GetClientIP. DevOps is not my day job and I was having hard time to figure out how to assign and use user defined variables and pass them on to next step/task/stage in pipeline! Microsoft documentation on variables usages was not helping me enough but I figured it out after lot of unsuccessful runs!

Let me share the summary from one of my build that creates a container image and pushes the image to Azure Container Registry. I need the secret to connect to ACR which I retrieve from Key Vault using workaround mentioned above.

Now that you know the concept, let’s see the actual build definition (yaml)-

# ASP.NET Core (.NET Framework)
# Build and test ASP.NET Core projects targeting the full .NET Framework.
# Add steps that publish symbols, save build artifacts, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

trigger:
- master

variables:
  buildConfiguration: 'Release'
  outputPath: './obj/Docker/publish'
  sourceFolder: $(Build.SourcesDirectory)
  buildId: $(Build.BuildId)
  imageName: 'azure.flowlogs.dotnetcore'
  imageUbantu: 'prodip:api-demo'
  dockerId: 'aspnet4you'
  ipAddr: ''
  kvName: 'aspnet4you-keyvault'
  kvRG: 'key-vaults'

jobs:
- job: 
  steps:
  - task: PowerShell@2
    name: GetClientIP
    displayName: Get Client IP from Trusted REST API Service
    inputs:
      targetType: 'inline'
      script: |
        $resp = Invoke-RestMethod -Uri 'https://afa-aspnet4you.azurewebsites.net/api/GetClientIP'
        Write-Host $resp
        # ##vso setvariable is required to set the value of ipAddr variable. 
        # $ipAddr = $resp only works local to the task! ** setvariable must be used to pass the variable down the pipeline.**
        Write-Host "##vso[task.setvariable variable=ipAddr]$resp"
        $ipAddr = $resp
        Write-Host ipAddr is: $ipAddr

  - task: AzureCLI@1
    name: AddFirewallRule
    displayName: Add Firewall Rule to Azure Key Vault (Format @ ip)
    inputs:
      azureSubscription: 'Build Pipeline Service Connection'
      scriptLocation: 'inlineScript'
      inlineScript: 'az keyvault network-rule add --resource-group $(kvRG) --name $(kvName) --ip-address $(ipAddr)'

  - task: AzureKeyVault@1
    name: KeyVaultSecrets
    displayName: Get Secret from Key Vault
    inputs:
      azureSubscription: 'Build Pipeline Service Connection'
      KeyVaultName: 'aspnet4you-keyvault'
      SecretsFilter: 'aspnet4you-acr'

  - task: AzureCLI@1
    name: RemoveFirewallRule
    displayName: Remove Firewall Rule from Azure Key Vault (Format @ ip/32)
    inputs:
      azureSubscription: 'Build Pipeline Service Connection'
      scriptLocation: 'inlineScript'
      inlineScript: 'az keyvault network-rule remove --resource-group $(kvRG) --name $(kvName) --ip-address $(ipAddr)/32'

  - script: docker build . -f $(sourceFolder)/AzureFlowLogs/Dockerfile-ubantu -t $(imageUbantu)
    displayName: docker build AzureFlowLogs/Dockerfile-ubantu

  - script: dotnet publish AzureFlowLogs --configuration $(buildConfiguration) -o $(outputPath)
    displayName: 'dotnet publish $(buildConfiguration)'

  - script: docker build . --build-arg source=$(outputPath) -f $(sourceFolder)/AzureFlowLogs/Dockerfile -t $(imageName)
    displayName: docker build AzureFlowLogs/Dockerfile

  - script: docker tag $(imageName):latest $(dockerId).azurecr.io/$(imageName):$(buildId) 
    displayName: docker tag as $(dockerId).azurecr.io/$(imageName):$(buildId)

  - script: docker login -u $(dockerId) -p $(aspnet4you-acr) $(dockerId).azurecr.io
    displayName: login to Azure Container Registry (ACR)

  - script: docker push $(dockerId).azurecr.io/$(imageName)
    displayName: push image to Azure Container Registry (ACR)

Hope you can make use of the whitelisting workaround to get secrets from Key Vault when secured with vnet and firewall until a permanent solution is available from Azure product team. :)-

Leave a Reply Cancel reply