Azure ILB ASE v2 Advanced Scenarios (PCI DSS, Application Gateway, CI/CD, Multisite hosting, SSL and more)

Intro

When dealing with the isolated hosting environment in Azure (known as ASE), devs could come across complicated subjects, but unfortunately, the official documentation doesn't help much: https://github.com/MicrosoftDocs/azure-docs/issues/10588

Knowing that ASE is a pretty advanced feature (pricy as well) in Azure, many enterprises choose it because of security reasons. One of the typical scenarios is the PCI DSS architecture as described on this page: https://docs.microsoft.com/en-us/azure/security/blueprints/pcidss-paaswa-overview

If you're dealing with the payment systems, you want to protect your services as much as possible. However, setting this whole infrastructure up can really be a handful, especially for beginners.

So, after weeks of searching and testing, I managed to provision the PCI DSS infrastructure in Azure with ILB ASE (App Service Environment v2 with Internal Load Balancer) behind the Application Gateway (WAF), that hosts multiple applications of TLS/SSL, with CI/CD pipeline enabled in Azure DevOps.

I hope this article could be helpful and eventually saves your time. Now let's cover the basics in order.

1. Add Virtual Network and Subnets

Add a new VNet in your subscription, configure subnets for ILB ASE, WAF App Gateway, Virtual Machine which will be used as both DNS server and build agent later. 

2. Create ASE v2 with ILB as VIP Type

2.1 For ASE created before May 2019

Now create your ASE v2 from the portal, make sure that you select the virtual network created in step 1, choose the "ilb" subnet and the Internal VIP Type, as for the Domain, use something unique as suggested by MS, ex. internal-mydomain.com if your services' domain name is mydomain.com.

Here the ownership of the domain name internal-mydomain.com is not required, as it will be used internally only.

The creation of the ASE could take a couple of hours, so just relax and take a break :)

Once it's deployed, you need to upload the ILB certificate, if you own the domain you can purchase an app certificate and use it directly, otherwise, you can also generate a self-signed certificate as described in the official documentation: https://docs.microsoft.com/en-us/azure/app-service/environment/create-ilb-ase#post-ilb-ase-creation-validation

Make sure it covers both the *.internal-mydomain.com and the *.scm.internal-mydomain.com

2.2 For ASE created after May 2019

You don't need to generate certificate anymore and ASE will have the default appserviceenvironment.net domain name.

3. Add Service Plan and Apps in your ASE

This step is pretty simple, you only need to make sure to create your service plan inside the ASE that you just added in step 2, and here in my example I have 4 Web App + 3 Slots, you should add apps to your ASE according to your own needs.

4. Create Application Gateway

Add new application gateway from Azure Portal, choose the WAF tier so that you'll have the application firewall feature, choose the VNet created in step 1 and also the waf subnet, create a new public IP address and give it a friendly DNS name such as appgw-mydomain.westeurope.cloudapp.azure.com, configure the listener to be HTTPS, then confirm the creation.

5. Configure Application Gateway

Here it comes the most important section! After the creation of your appgw, you need to configure it for multisite hosting (HTTPS), and I recommend you to follow the below steps:

5.1 Backend pool

Put the Internal Load Balancer IP address of your ILB ASE in the target of type IP address or FQDN, you can find it as shown here:

5.2 DNS names of your apps

Log in to your DNS provider portal, and add the CNAME records for your applications, such as

app-1.mydomain.com

app-2.mydomain.com

...

Point them all to your newly created application gateway DNS appgw-mydomain.westeurope.cloudapp.azure.com.

5.3 Health probes

Add new health probes for each of your app, select HTTPS and put the registered names of your app as Host

5.4 Http settings

Add http setting for each of the apps, select HTTPS as protocol, and upload the .CER certificate of your self-signed certificate of internal-mydomain.com

You can use 3rd party tools to export your certificate as .CER file such as DigiCert Utility

If you've purchased the Azure App Certificate for internal-mydomain.com, you may find this article very useful to manage the certificate locally: https://azure.github.io/AppService/2017/02/24/Creating-a-local-PFX-copy-of-App-Service-Certificate.html

Remember to select the custom probe that you've added in step 5.3

Latest update: For ASE created after May 2019, you need to get the .CER for the default appserviceenvironment.net domain, I found the official doc's description is a bit rough, however, a useful tip here for you is to add a VM inside the same VNET of the ASE, and navigate to a simple ASE hosted application, then you can export the website certificate in base 64 encoded .CER from your browser.

5.5 Listeners

You need to add Multi-site listener for your apps in this scenario, choose a pertinent name and select the default front IP configuration (which points to the public IP of your application gateway), and the port 443 for HTTPS protocol. Here you need to type the same hostname of your app as you did in step 5.3

5.6 Rules

Add Basic rule for each of your apps. You need to simply select the corresponded Listener and HTTP setting of the app from the dropdown list.

5.7 Application Custom Domain

If you managed to make it this far, congratulations!

This is the last configuration you need to do about the Application Gateway. Remember that all your apps are hosted inside the private ASE now and they're not accessible from the public network. Now you need to configure their custom domain so that they'll have the same hostname as you configured in your http listeners.

Navigate to the App Service, go to Settings -> Custom domains section, and add the app-1.mydomain.com etc. to your app. Once added, you also need to bind the SSL with your private certificate (.pfx) of your domain mydomain.com

5.8 Final test

Test your application gateway by simply navigate to https://app-1.mydomain.com

Check in the HTTP headers if your request hits well your public IP address and the port 443. I hope you get a beautiful 200 as response :)

If you don't get a successful code, go through the previous steps again and make sure you didn't misconfigure anything. From my personal experience, verify the health probe again of your app, because if your default URL returns 404, it might cause 502 error. If it's the case, either modify your probe matching condition or change the path to return a healthy response.

You can also refer to the official troubleshoot doc of Microsoft: https://docs.microsoft.com/en-us/azure/application-gateway/create-gateway-internal-load-balancer-app-service-environment

6. Set up CI/CD pipeline

Now that you have an ILB ASE living inside your VNet, it becomes impossible to use public Azure DevOps pipeline to build and deploy your websites.

Here's the official explanation: https://docs.microsoft.com/en-us/azure/app-service/environment/create-ilb-ase#publish-with-an-ilb-ase

Assuming that you already have an Azure DevOps account in your organization, let's set up the CI/CD pipeline for your protected application now.

6.1 Virtual machine

Create a virtual machine in the same VNet as your ILB ASE. Since we've already prepared the subnet for it now you just need to select the vm subnet of step 1.

6.2 DNS server

Once the VM is created, enable the DNS server feature on it.

You need to create A-records pointing to the App Service Environment's Internal Load Balancer IP address for the following hostnames (*, *.scm, ftp, publish).

Once added, set the VNet to use your VM as the DNS server (you may need to reboot your VM after the configuration)

Latest update: For ASE created after May 2019, configure the records for the *.appserviceenvironment.net domain.

6.3 Build agent

You could just use the same VM as your build agent since it's living inside the same VNet of the ILB ASE.

Simply follow this documentation for the provision: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops&viewFallbackFrom=vsts

6.4 Azure DevOps

Add the build agent to the Agent pools from your Azure DevOps portal.

Create a new build and release definition in VSTS pipeline by using the private build agent.

This article describes more in details: https://devopsandcloud.wordpress.com/2018/04/27/deploy-to-azure-ilb-ase-using-visual-studio-online-services/

A very important tip: if you ever encountered the certificate issue while publishing the app, you may want to set the variable VSTS_ARM_REST_IGNORE_SSL_ERRORS to true in your settings.

Summary

In this article, we covered the basics of an advanced scenario of Azure Application Gateway & ILB ASE. It helps reduce the overhead of the advanced infrastructure provision in Azure.

If you like this content, please feel free to leave a comment, share it to your network, or simply click the like button at the bottom of the article.

References

Below are the external resources that I used while working on this subject and I found them very useful as well:

https://docs.microsoft.com/en-us/azure/app-service/environment/integrate-with-application-gateway

https://sabbour.me/how-to-run-an-app-service-behind-a-waf-enabled-application-gateway/

https://github.com/Azure/fta-internalbusinessapps/blob/master/appmodernization/app-service-environment/ase-walkthrough.md

https://docs.microsoft.com/en-us/azure/app-service/environment/app-service-app-service-environment-geo-distributed-scale

https://azure.github.io/AppService/2017/02/24/Creating-a-local-PFX-copy-of-App-Service-Certificate.html

https://devopsandcloud.wordpress.com/2018/04/27/deploy-to-azure-ilb-ase-using-visual-studio-online-services/

https://medium.com/@jkudo/recommendation-of-azure-app-service-environment-f0e57c42b95a

https://docs.microsoft.com/en-us/azure/app-service/environment/network-info