Azure DevOps for CI and CD

I set up CI and CD for two of my applications using Azure DevOps. It was quite easy. Setting up the build pipeline is as simple as including a YAML file in your source repository. It then just comes down to knowing how the build YAML schema works. As far as the release (or deployment) pipeline is concerned, though, I could not find a similar method. I had to set it up through the Azure DevOps UI. I don’t know if there is some information I am missing, but that would seem to somewhat go against the DevOps principle – you know – infrastructure as code and all.

Anyway, my personal website and blog is a straight forward Razor app based on ASP.NET Core. The build YAML was then very simple:

pool:
  vmImage: 'VS2017-Win2016'

variables:
  buildConfiguration: 'Release'

steps:
- script: |
    dotnet build --configuration $(buildConfiguration)
    dotnet publish --configuration $(buildConfiguration) --output $(Build.BinariesDirectory)/publish
  displayName: 'Build Application'

- task: ArchiveFiles@2
  inputs:
    rootFolderOrFile: '$(Build.BinariesDirectory)/publish' 
    includeRootFolder: false
    archiveType: 'zip'
    archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip' 
    replaceExistingArchive: true 
  displayName: 'Zip Build Output'

- task: PublishBuildArtifacts@1
displayName: 'Publish Build Artifacts' 

Things to note here – at first, it was not obvious to me that I need to actually publish the build artifacts for it to be available further down the pipeline. Once I figured that out, though, the built-in task makes this easy. I did have to zip up my folder, though. It’s a good idea, anyway, as it’s easier than uploading a gazillion files, but I couldn’t get it to work without zipping it up. Second, even though I am running the build on a Windows machine, I could just as easily have picked a Linux box since this is .NET Core.

My release pipeline is pretty simple – there are no multiple stages, and it just comes down to one task – the Azure App Service deploy task.

I guess the only difference with my other application, Listor, is that it is a React based SPA with a .NET Core backend. So the build is a little more involved. I therefore found it easier to just write a custom PowerShell script and call that from the build YAML rather than messing around with it in YAML.

$ErrorActionPreference = "Stop"
Push-Location AK.Listor.WebClient
Write-Host "Installing web client dependencies..."
npm install
If ($LastExitCode -Ne 0) { Throw "npm install failed." }
Write-Host "Building web client..."
npm run build
If ($LastExitCode -Ne 0) { Throw "npm run build failed." }
Pop-Location
Write-Host "Deleting existing client files..."
[System.IO.Directory]::GetFiles("AK.Listor\Client", "*.*") | Where-Object {
	[System.IO.Path]::GetFileName($_) -Ne ".gitignore"
} | ForEach-Object {
	$File = [System.IO.Path]::GetFullPath($_)
	[System.IO.File]::Delete($File)
}
[System.IO.Directory]::GetDirectories("AK.Listor\Client") | ForEach-Object {
	$Directory = [System.IO.Path]::GetFullPath($_)
	[System.IO.Directory]::Delete($Directory, $True)
}
Write-Host "Copying new files..."
[System.IO.Directory]::GetFiles("AK.Listor.WebClient\build", "*.*", "AllDirectories") | Where-Object {
	-Not ($_.EndsWith("service-worker.js"))
} | ForEach-Object {
	$SourceFile = [System.IO.Path]::GetFullPath($_)
	$TargetFile = [System.IO.Path]::GetFullPath([System.IO.Path]::Combine("AK.Listor\Client", $_.Substring(26)))
	Write-Host "$SourceFile --> $TargetFile"
	$TargetDirectory = [System.IO.Path]::GetDirectoryName($TargetFile)
	If (-Not [System.IO.Directory]::Exists($TargetDirectory)) {
		[System.IO.Directory]::CreateDirectory($TargetDirectory) | Out-Null
	}
	[System.IO.File]::Copy($SourceFile, $TargetFile, $True) | Out-Null
}
Write-Host "Building application..."
dotnet build
If ($LastExitCode -Ne 0) { Throw "dotnet build failed." }

The build YAML, then is simple enough:

pool:
  vmImage: 'VS2017-Win2016'

variables:
  buildConfiguration: 'Release'

steps:
- script: |
    powershell -F build.ps1
    dotnet publish --configuration $(buildConfiguration) --output $(Build.BinariesDirectory)/publish	
  displayName: 'Run Build Script'

- task: ArchiveFiles@2
  inputs:
    rootFolderOrFile: '$(Build.BinariesDirectory)/publish' 
    includeRootFolder: false
    archiveType: 'zip'
    archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip' 
    replaceExistingArchive: true 
  displayName: 'Zip Build Output'

- task: PublishBuildArtifacts@1
displayName: 'Publish Build Artifacts'

The release pipeline, of course, is identical since both these applications are hosted as app services on Azure. Since deploying these applications to production, I have had to make a few changes and this CI/CD pipeline has proved very helpful.

Advertisements

On Service Fabric, Kubernetes and Docker

Let us get Docker out of the way first. Microservices and containers are quite the hype these days. With hype comes misinformation and hysteria. A lot of people conflate the two (fortunately there are wise people out there to set us all straight). If you have done your due diligence and decided to go with microservices, you don’t have to go with containers. In fact, one would argue that using containers for production might be a good crutch for applications that have too many tentacles and there is no appetite to port them or rewrite them to be “portable”. Containers do have other good use cases too. Docker being the leading container format (although starting to face some competition from rkt these days), all in all I am glad containers exist and I am glad that Docker exists. Just be aware of the fact that what you think you must use may not be what you need at all.

Service Fabric was written by Microsoft as a distributed process and state management system to run their own Azure cloud infrastructure. After running Azure on top of it for some years, Microsoft decided to release it as a product with the marketing pitch being geared towards microservices. Whether you are doing microservices or not, if you have any multitude of services or deployment units, Service Fabric is an excellent option. If you are running .NET workloads, then it is an even better option as you get to use its native features such as the Stateful model, among others. Since it is also a process orchestrator, and at the end of the day, a Docker container is just a Docker process, Service Fabric can also act as a decent container orchestrator. Kubernetes I am told has a better ingress configuration system, but Service Fabric has improvements coming related to that. The other big promise is around Service Fabric Mesh – which is a fully managed PaaS offering – so that may sway the decision somewhat.

If you have done your homework and find that containers are the way to go, then Kubernetes is a good choice for the orchestration system. Even Microsoft is all behind Kubernetes now with their Azure Kubernetes Service, which only makes sense given that it is in their best interest the more developer ecosystems adopt Azure. It also boasts having run years worth of production workloads at Google. If you are completely onboard with containers, then even if you are running .NET workloads (especially if you are running cross platform workloads such as .NET Core), perhaps you mitigate a lot of risk by sticking to Kubernetes. However, especially if you have a .NET workload, you have to decide whether you want to pick up what you have, put it in containers, and move into Kubernetes; or with some effort you can just make your application portable enough so you don’t need containers.

Where do I stand? For my purposes (which involves writing primarily .NET applications), I am partial to Service Fabric. However, take this fact within the context that I have been working with Service Fabric in production for almost a couple of years now, while I have only “played around” with Kubernetes. I do like the fact that Service Fabric can run on a set of homogenous nodes as opposed to Kubernetes that needs a master. However, as with stance on things, this one can change given new information and experience.

What should you do? Not listen to quick and easy decision making flowcharts. When you do listen to opinions, be aware of the fact that they are influenced by experience and also sometimes by prejudice. Finally, evaluate your own needs carefully rather than do something because “everybody is doing it”. I could be eating my words years from now, but based on their origins and use, I don’t think either Service Fabric or Kubernetes are at risk of being “abandoned” by their stewards (i.e. Microsoft and Google) at any point.

Reader/Writer Locking with Async/Await

Consider this another pitfall warning. If you are a frequent user of reader/writer locking (via the ReaderWriterLockSlim class) like I am, you will undoubtedly run into this situation. As more and more code we write these days are asynchronous with the use of async/await, it is easy to end up in the following situation (an oversimplification, but just imagine write locks in there as well):

async Task MyMethod()
{
	...
	myReaderWriterLockSlim.EnterReadLock();
	var thing = await ReadThingAsync();
	... 
	myReaderWriterLockSlim.ExitReadLock(); // This guy will choke.
}

This, of course, will not work. This is because reader/writer locks, at least the implementation in .NET, are thread-affine. This means the very same thread that acquired a lock must be the one to release it. As soon as you hit an await, you have dispatched the rest of the behavior to some other thread. So this cannot work.

This explains why other thread-synchronization classes such as SemaphoreSlim are not async/await savvy with methods like WaitAsync but not ReaderWriterLockSlim.

So, what are our options?

  1. Carefully write our code such that whatever happens between reader/writer locks is always synchronous.
  2. Relax some of the rules around reader/writer locking that requires it to be thread-affine and roll your own.
  3. Look for an already existing, widely adopted, mature library that handles this very scenario.

In the spirit of Option 3, Stephen Cleary has an AsyncEx library that includes this functionality and many others geared towards working efficiently with async/await. If that is too heavy-handed, may I suggest this post by Stephen Toub that lays out a basic implementation that you can build upon?

Listor- Showcasing React and .NET Core

For both React and for .NET Core (specifically ASP.NET Core and Entity Framework Core), I got sick of playing around with little prototypes and decided to build an application. Listor is my first proper application I have built using both these technologies. It is a simple list-maker application- nothing fancy. But I have been using it since I put it up and it has come in handy quite a bit.

I am quite impressed with .NET Core (or I should say “the new .NET” – to mean not just the .NET Core runtime, but .NET Standard, the new project system, ASP.NET Core, EF Core, and let’s say even the latest language improvements to C#). So much so, that it is going to suck a bit going back to writing traditional .NET Framework stuff for my day job.

My journey with React was much more convoluted. I started with it, took some time getting into the whole idea of JSX, but eventually was quite impressed by it. Then I got into the rabbit hole of setting up the environment, boilerplates, and then Flux and then Redux – and then I got jaded real fast. I had to then reset myself back into vanilla React to see if I could just get stuff done with it. As far as this little application is concerned, it did the job nicely. I used create-react-app to set it up.

EF Core worked out nicely for this application – I am not doing anything crazy anyway, so those basic use cases are already nicely covered. In terms of the build process, rather than treat the build output of the client application as content and simply use something like app.UseFileServer(), I opted to go with compiling them as embedded resources, and built an AssetServer middleware that would cache it in-memory and serve it out fast. Again, I didn’t have to, but it turned out okay.

I am using Let’s Encrypt for TLS, so the other middleware I put in place is a HttpsVerifier that I could use to easily set up an ACME verification endpoint for when I renew the certificate.

All in all, the application turned out to be useful, and I think it will serve as a nice playground for when I need to try out new technologies as they come out.

Listor the application is hosted at listor.aashishkoirala.com. The source code can be found on GitHub over here.

An Azure Service Fabric Restarter in F#

Trying to get beyond just writing quick scripts here and there in F#, I went through functional design patterns targeted at building mainstream applications. Railway-oriented programming specifically stuck with me. I decided to try it along with some of the other core functional concepts such as projecting to other domains with map and bind operations. My first foray into this was applying it to, surprise, surprise, yet another quick script I had in place. This one was something I had put together already using F# to recycle all code packages for a given application running on Azure Service Fabric.

Thus what was one end-to-end script became three distinct modules, as implemented in three different files:

The full source code is in this repository.

OAuth2 and OpenID Connect versus WS-* and SAML

I have mentioned how part of our replatforming project that saw us move to Azure was moving the security protocol from WS-Federation/WS-Trust to OAuth2 and OpenID Connect. I kept running into rumblings on the internet about how even though it was widely adopted, OAuth2/OpenID Connect were somehow less secure. Comparing a secure implementation of both side by side, I did not really see how this could be. Since our industry is not short on oversimplification and grand proclamations, I decided to pose this question to experts in the field.

I posted this question on the Information Security Stack Exchange site. The quality of the responses I got blew me away- carefully thought through and well articulated, to say the least.

I liked this answer by Karl McGuinness the best and thought it worthwhile to socialize it further through this blog post.

The key takeaway, though, is:

  • All these protocols are secure, but an implementation may be insecure if not properly done. In this spirit, the simpler the protocol, the better.
  • All these protocols use cryptographically signed tokens that support optional encryption.
  • There are some problems with OAuth2 by itself which are addressed by OpenID Connect.

I hope this can serve as a good resource to refute any other oversimplified statement to the contrary.

Moving to Azure PaaS and Service Fabric- Part 2

This is Part 2 of a two-part blog series:

  • Part 1 (Application- Services, Security and UI)
  • Part 2 (this one; Database, Configuration, Logging, Caching, Service Bus, Emails, Tooling, Rollout)

Database

We moved from our on-premises installation of SQL Server to the PaaS offering that is SQL on Azure. Other than the actual physical moving of the data, the additional challenge we had was that our system had a number of separate databases that were interconnected via synonyms. Since each SQL Database is an independent resource on Azure, this would not be possible without introducing external data sources which would still be performance prohibitive. We therefore had to remove the synonyms and rework some of our code to account for this. We opted to go with an Elastic Pool that was associated with all our databases. We also configured geo-replication for redundancy.

Additionally, part of our system includes a document storage mechanism that ends up storing the document BLOBs in SQL. We decided to take this opportunity to also move these BLOBs out of SQL and in to Azure BLOB Storage where they naturally belong. We wrote a tool to extract all BLOBs out of SQL and upload them into Azure BLOB Storage. We ran it a week before the actual rollout, with the plan being to upload the last week’s worth as part of the rollout. In order to move the SQL data, we provisioned empty databases on Azure and set up replication from the on-premises databases. This way, come rollout time, the on-premises databases and the ones on Azure would already be in sync, thus eliminating the riskiest step of massive data movement.

Configuration

We have a bespoke configuration system based around XML files. The major change here would be going from the files residing on some local share to them residing in BLOB storage. Code change needed for that was done, as well as the change to the settings deployment mechanism – which now went from simply copying files to the share to uploading the files to Azure BLOB storage.

Logging

We had two forms of logging: application level logging that uses log4net to write out to configured appenders. Since we went from file system to Azure Table Storage, this involved writing a new appender that would send the log entry off to Table Storage. The other piece was ETW-based tracing. We configured a VM extension as part of the Service Fabric cluster that would gather all ETW traces and upload them to Table Storage. The last part was redoing the log viewer tooling to read from Table Storage.

Caching

This was a significant change in going from AppFabric, which was being sunset, to Redis – which was the recommended distributed cache for Azure. We discovered a few leaks in our existing abstraction layer where some consuming code was dependent on AppFabric-specific features such as versioning. This needed some reworking of the code. Other than this, we quite extensively use the locking feature for distributed pessimistic locking. With Redis, no such feature natively exists. However, there is a published algorithm that states how it can be done. While our main Redis client library, StackExchange.Redis, did not have this implemented, we ended up using RedLock.net for this purpose.

Service Bus

I think this was the least work required, because we were already using Windows Service Bus – the almost out-of-support on-premises version of Azure Service Bus. So moving to Azure Service Bus was akin to a version upgrade without much rework needed on the code.

Emails

For all automated emails that the applications send out, we moved from an on-premises SMTP server to SendGrid. The initial setup was straightforward. Other considerations we had to work around was setting up SPF and DKIP with our DNS provider so that SendGrid could be designated as an authorized sender of emails from our domain. SendGrid has facilities to do all these so that step was easy. Additionally, we had to make sure DMARC setup was properly done on our side.

Tooling

Much of this has already been covered in other sections, but essentially boiling it down to a few bullet points, we had to work on the following aspects of tooling:

  • Deployment – we are now deploying to Service Fabric and not to IIS anymore, plus as part of deployment, we are also generating Fabric projects on the fly based on host model descriptors.
  • Logging – the tool we use to aggregate and view logs now talks to Table Storage.
  • Databases – tooling related to database backup/restore, export/import, etc. all have to work against Azure. Additionally, database maintenance scripts are now hosted using Azure Automation instead of SQL Server Agent.
  • Provisioning – we needed to write a brand new provisioning system using Azure Resource Manager APIs that would allow us to stand up and tear down platforms as needed on Azure.

Rollout

We stood up the entire environment a week prior to actual rollout, which gave us plenty of time to test and correct. Owing to the database migration strategy that involved replication, a lot of risk was mitigated. Thus, the actual rollout involved simply:

  • Start outage window.
  • Repoint DNS to new application.
  • Ensure replication has no transactions pending, and turn off replication.
  • Upload BLOBs that were created during the last week.
  • End outage window.

And with that, we were on Azure. Not a traditional lift-and-shift by any means. We were fully cloud-native on PaaS – wholesale- the entire system. Quite a feat if you ask me.