In this post, I'll demonstrate how to determine whether a client service, such as an App Service, has access to an Azure Key Vault with Private Endpoint.

Private Endpoint

A Private Endpoint allows us to tie a PaaS service, such as a Key Vault, to a subnet that includes a NIC and a private IP address. Public connections to PaaS resources are disabled, and an Azure Private DNS Zone is utilised to redirect the PaaS resource's name resolution to the private IP address.

It is important to note that communications to the private endpoint are inbound (only response). A PaaS resource cannot establish outbound connections using a Private Endpoint.

Use Case

The customer has an App Service Plan with VNet Integration enabled, which allows the App Services to connect outbound from "random" IP addresses on this subnet; NSG/Firewall rules should allow access from the subnet prefix.

The App Services on the plan have Private Endpoints on a second subnet in the VNet. There is also a Key Vault, which also has a Private Endpoint. The “Private Endpoint subnet” has an NSG to deny everything except desired traffic, including allowing HTTPS from the VNet Integration subnet prefix to the Key Vault Private Endpoint.

A developer wanted to know if App Service connections were operating and if we could see this in the logs.

A Road-Block

The developer in this situation sought to confirm network connectivity. The obvious place to look was the network! Typically, this involves verifying that packets arrived at the destination NIC. You can do this (usually) with NSG Flow Logs. It can take up to 25 minutes (or longer during pandemic compute shortages) for a flow to emerge in Log Analytics. We checked the logs, but there was nothing there.

And that is because, (at this time) NSG Flow Logs cannot produce data destined to Private Endpoints.

Solution

The solution is to examine the logs of the target resource. We enable a lot of logging as a standard, including Key Vault logs. A little bit of Kql-Fu yielded the following query:

The resulting columns included the following:

• CallerIPAddress: In our case, the IP address of the client (the IP address that the App Service Plan VNet integration uses).
• OperationName: Things like SecretGet, Authentication, VaultGet, and SecretList
• requestUri_s: The URI of the secret being requested
• ResultType: Was it a success or not?
• identity_claim_xms_mirid_s: The resource ID of the requesting client (the resource ID of the App Service, in our case)

With the generated data, the developer had all they needed to establish that the App Service was connecting to the Key Vault.