Tag Archives: aashish

Azure DevOps for CI and CD

I set up CI and CD for two of my applications using Azure DevOps. It was quite easy. Setting up the build pipeline is as simple as including a YAML file in your source repository. It then just comes down to knowing how the build YAML schema works. As far as the release (or deployment) pipeline is concerned, though, I could not find a similar method. I had to set it up through the Azure DevOps UI. I don’t know if there is some information I am missing, but that would seem to somewhat go against the DevOps principle – you know – infrastructure as code and all.

Anyway, my personal website and blog is a straight forward Razor app based on ASP.NET Core. The build YAML was then very simple:

pool:
  vmImage: 'VS2017-Win2016'

variables:
  buildConfiguration: 'Release'

steps:
- script: |
    dotnet build --configuration $(buildConfiguration)
    dotnet publish --configuration $(buildConfiguration) --output $(Build.BinariesDirectory)/publish
  displayName: 'Build Application'

- task: ArchiveFiles@2
  inputs:
    rootFolderOrFile: '$(Build.BinariesDirectory)/publish' 
    includeRootFolder: false
    archiveType: 'zip'
    archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip' 
    replaceExistingArchive: true 
  displayName: 'Zip Build Output'

- task: PublishBuildArtifacts@1
displayName: 'Publish Build Artifacts' 

Things to note here – at first, it was not obvious to me that I need to actually publish the build artifacts for it to be available further down the pipeline. Once I figured that out, though, the built-in task makes this easy. I did have to zip up my folder, though. It’s a good idea, anyway, as it’s easier than uploading a gazillion files, but I couldn’t get it to work without zipping it up. Second, even though I am running the build on a Windows machine, I could just as easily have picked a Linux box since this is .NET Core.

My release pipeline is pretty simple – there are no multiple stages, and it just comes down to one task – the Azure App Service deploy task.

I guess the only difference with my other application, Listor, is that it is a React based SPA with a .NET Core backend. So the build is a little more involved. I therefore found it easier to just write a custom PowerShell script and call that from the build YAML rather than messing around with it in YAML.

$ErrorActionPreference = "Stop"
Push-Location AK.Listor.WebClient
Write-Host "Installing web client dependencies..."
npm install
If ($LastExitCode -Ne 0) { Throw "npm install failed." }
Write-Host "Building web client..."
npm run build
If ($LastExitCode -Ne 0) { Throw "npm run build failed." }
Pop-Location
Write-Host "Deleting existing client files..."
[System.IO.Directory]::GetFiles("AK.Listor\Client", "*.*") | Where-Object {
	[System.IO.Path]::GetFileName($_) -Ne ".gitignore"
} | ForEach-Object {
	$File = [System.IO.Path]::GetFullPath($_)
	[System.IO.File]::Delete($File)
}
[System.IO.Directory]::GetDirectories("AK.Listor\Client") | ForEach-Object {
	$Directory = [System.IO.Path]::GetFullPath($_)
	[System.IO.Directory]::Delete($Directory, $True)
}
Write-Host "Copying new files..."
[System.IO.Directory]::GetFiles("AK.Listor.WebClient\build", "*.*", "AllDirectories") | Where-Object {
	-Not ($_.EndsWith("service-worker.js"))
} | ForEach-Object {
	$SourceFile = [System.IO.Path]::GetFullPath($_)
	$TargetFile = [System.IO.Path]::GetFullPath([System.IO.Path]::Combine("AK.Listor\Client", $_.Substring(26)))
	Write-Host "$SourceFile --> $TargetFile"
	$TargetDirectory = [System.IO.Path]::GetDirectoryName($TargetFile)
	If (-Not [System.IO.Directory]::Exists($TargetDirectory)) {
		[System.IO.Directory]::CreateDirectory($TargetDirectory) | Out-Null
	}
	[System.IO.File]::Copy($SourceFile, $TargetFile, $True) | Out-Null
}
Write-Host "Building application..."
dotnet build
If ($LastExitCode -Ne 0) { Throw "dotnet build failed." }

The build YAML, then is simple enough:

pool:
  vmImage: 'VS2017-Win2016'

variables:
  buildConfiguration: 'Release'

steps:
- script: |
    powershell -F build.ps1
    dotnet publish --configuration $(buildConfiguration) --output $(Build.BinariesDirectory)/publish	
  displayName: 'Run Build Script'

- task: ArchiveFiles@2
  inputs:
    rootFolderOrFile: '$(Build.BinariesDirectory)/publish' 
    includeRootFolder: false
    archiveType: 'zip'
    archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip' 
    replaceExistingArchive: true 
  displayName: 'Zip Build Output'

- task: PublishBuildArtifacts@1
displayName: 'Publish Build Artifacts'

The release pipeline, of course, is identical since both these applications are hosted as app services on Azure. Since deploying these applications to production, I have had to make a few changes and this CI/CD pipeline has proved very helpful.

On Service Fabric, Kubernetes and Docker

UPDATE (Nov 13, 2019) – My views on this have changed since I first wrote this. See this post for where I currently stand.

Let us get Docker out of the way first. Microservices and containers are quite the hype these days. With hype comes misinformation and hysteria. A lot of people conflate the two (fortunately there are wise people out there to set us all straight). If you have done your due diligence and decided to go with microservices, you don’t have to go with containers. In fact, one would argue that using containers for production might be a good crutch for applications that have too many tentacles and there is no appetite to port them or rewrite them to be “portable”. Containers do have other good use cases too. Docker being the leading container format (although starting to face some competition from rkt these days), all in all I am glad containers exist and I am glad that Docker exists. Just be aware of the fact that what you think you must use may not be what you need at all.

Service Fabric was written by Microsoft as a distributed process and state management system to run their own Azure cloud infrastructure. After running Azure on top of it for some years, Microsoft decided to release it as a product with the marketing pitch being geared towards microservices. Whether you are doing microservices or not, if you have any multitude of services or deployment units, Service Fabric is an excellent option. If you are running .NET workloads, then it is an even better option as you get to use its native features such as the Stateful model, among others. Since it is also a process orchestrator, and at the end of the day, a Docker container is just a Docker process, Service Fabric can also act as a decent container orchestrator. Kubernetes I am told has a better ingress configuration system, but Service Fabric has improvements coming related to that. The other big promise is around Service Fabric Mesh – which is a fully managed PaaS offering – so that may sway the decision somewhat.

If you have done your homework and find that containers are the way to go, then Kubernetes is a good choice for the orchestration system. Even Microsoft is all behind Kubernetes now with their Azure Kubernetes Service, which only makes sense given that it is in their best interest the more developer ecosystems adopt Azure. It also boasts having run years worth of production workloads at Google. If you are completely onboard with containers, then even if you are running .NET workloads (especially if you are running cross platform workloads such as .NET Core), perhaps you mitigate a lot of risk by sticking to Kubernetes. However, especially if you have a .NET workload, you have to decide whether you want to pick up what you have, put it in containers, and move into Kubernetes; or with some effort you can just make your application portable enough so you don’t need containers.

Where do I stand? For my purposes (which involves writing primarily .NET applications), I am partial to Service Fabric. However, take this fact within the context that I have been working with Service Fabric in production for almost a couple of years now, while I have only “played around” with Kubernetes. I do like the fact that Service Fabric can run on a set of homogenous nodes as opposed to Kubernetes that needs a master. However, as with stance on things, this one can change given new information and experience.

What should you do? Not listen to quick and easy decision making flowcharts. When you do listen to opinions, be aware of the fact that they are influenced by experience and also sometimes by prejudice. Finally, evaluate your own needs carefully rather than do something because “everybody is doing it”. I could be eating my words years from now, but based on their origins and use, I don’t think either Service Fabric or Kubernetes are at risk of being “abandoned” by their stewards (i.e. Microsoft and Google) at any point.

Reader/Writer Locking with Async/Await

Consider this another pitfall warning. If you are a frequent user of reader/writer locking (via the ReaderWriterLockSlim class) like I am, you will undoubtedly run into this situation. As more and more code we write these days are asynchronous with the use of async/await, it is easy to end up in the following situation (an oversimplification, but just imagine write locks in there as well):

async Task MyMethod()
{
	...
	myReaderWriterLockSlim.EnterReadLock();
	var thing = await ReadThingAsync();
	... 
	myReaderWriterLockSlim.ExitReadLock(); // This guy will choke.
}

This, of course, will not work. This is because reader/writer locks, at least the implementation in .NET, are thread-affine. This means the very same thread that acquired a lock must be the one to release it. As soon as you hit an await, you have dispatched the rest of the behavior to some other thread. So this cannot work.

This explains why other thread-synchronization classes such as SemaphoreSlim are not async/await savvy with methods like WaitAsync but not ReaderWriterLockSlim.

So, what are our options?

  1. Carefully write our code such that whatever happens between reader/writer locks is always synchronous.
  2. Relax some of the rules around reader/writer locking that requires it to be thread-affine and roll your own.
  3. Look for an already existing, widely adopted, mature library that handles this very scenario.

In the spirit of Option 3, Stephen Cleary has an AsyncEx library that includes this functionality and many others geared towards working efficiently with async/await. If that is too heavy-handed, may I suggest this post by Stephen Toub that lays out a basic implementation that you can build upon?

Listor- Showcasing React and .NET Core

For both React and for .NET Core (specifically ASP.NET Core and Entity Framework Core), I got sick of playing around with little prototypes and decided to build an application. Listor is my first proper application I have built using both these technologies. It is a simple list-maker application- nothing fancy. But I have been using it since I put it up and it has come in handy quite a bit.

I am quite impressed with .NET Core (or I should say “the new .NET” – to mean not just the .NET Core runtime, but .NET Standard, the new project system, ASP.NET Core, EF Core, and let’s say even the latest language improvements to C#). So much so, that it is going to suck a bit going back to writing traditional .NET Framework stuff for my day job.

My journey with React was much more convoluted. I started with it, took some time getting into the whole idea of JSX, but eventually was quite impressed by it. Then I got into the rabbit hole of setting up the environment, boilerplates, and then Flux and then Redux – and then I got jaded real fast. I had to then reset myself back into vanilla React to see if I could just get stuff done with it. As far as this little application is concerned, it did the job nicely. I used create-react-app to set it up.

EF Core worked out nicely for this application – I am not doing anything crazy anyway, so those basic use cases are already nicely covered. In terms of the build process, rather than treat the build output of the client application as content and simply use something like app.UseFileServer(), I opted to go with compiling them as embedded resources, and built an AssetServer middleware that would cache it in-memory and serve it out fast. Again, I didn’t have to, but it turned out okay.

I am using Let’s Encrypt for TLS, so the other middleware I put in place is a HttpsVerifier that I could use to easily set up an ACME verification endpoint for when I renew the certificate.

All in all, the application turned out to be useful, and I think it will serve as a nice playground for when I need to try out new technologies as they come out.

Listor the application is hosted at listor.aashishkoirala.com. The source code can be found on GitHub over here.

An Azure Service Fabric Restarter in F#

Trying to get beyond just writing quick scripts here and there in F#, I went through functional design patterns targeted at building mainstream applications. Railway-oriented programming specifically stuck with me. I decided to try it along with some of the other core functional concepts such as projecting to other domains with map and bind operations. My first foray into this was applying it to, surprise, surprise, yet another quick script I had in place. This one was something I had put together already using F# to recycle all code packages for a given application running on Azure Service Fabric.

Thus what was one end-to-end script became three distinct modules, as implemented in three different files:

The full source code is in this repository.

OAuth2 and OpenID Connect versus WS-* and SAML

I have mentioned how part of our replatforming project that saw us move to Azure was moving the security protocol from WS-Federation/WS-Trust to OAuth2 and OpenID Connect. I kept running into rumblings on the internet about how even though it was widely adopted, OAuth2/OpenID Connect were somehow less secure. Comparing a secure implementation of both side by side, I did not really see how this could be. Since our industry is not short on oversimplification and grand proclamations, I decided to pose this question to experts in the field.

I posted this question on the Information Security Stack Exchange site. The quality of the responses I got blew me away- carefully thought through and well articulated, to say the least.

I liked this answer by Karl McGuinness the best and thought it worthwhile to socialize it further through this blog post.

The key takeaway, though, is:

  • All these protocols are secure, but an implementation may be insecure if not properly done. In this spirit, the simpler the protocol, the better.
  • All these protocols use cryptographically signed tokens that support optional encryption.
  • There are some problems with OAuth2 by itself which are addressed by OpenID Connect.

I hope this can serve as a good resource to refute any other oversimplified statement to the contrary.

Moving to Azure PaaS and Service Fabric- Part 2

This is Part 2 of a two-part blog series:

  • Part 1 (Application- Services, Security and UI)
  • Part 2 (this one; Database, Configuration, Logging, Caching, Service Bus, Emails, Tooling, Rollout)

Database

We moved from our on-premises installation of SQL Server to the PaaS offering that is SQL on Azure. Other than the actual physical moving of the data, the additional challenge we had was that our system had a number of separate databases that were interconnected via synonyms. Since each SQL Database is an independent resource on Azure, this would not be possible without introducing external data sources which would still be performance prohibitive. We therefore had to remove the synonyms and rework some of our code to account for this. We opted to go with an Elastic Pool that was associated with all our databases. We also configured geo-replication for redundancy.

Additionally, part of our system includes a document storage mechanism that ends up storing the document BLOBs in SQL. We decided to take this opportunity to also move these BLOBs out of SQL and in to Azure BLOB Storage where they naturally belong. We wrote a tool to extract all BLOBs out of SQL and upload them into Azure BLOB Storage. We ran it a week before the actual rollout, with the plan being to upload the last week’s worth as part of the rollout. In order to move the SQL data, we provisioned empty databases on Azure and set up replication from the on-premises databases. This way, come rollout time, the on-premises databases and the ones on Azure would already be in sync, thus eliminating the riskiest step of massive data movement.

Configuration

We have a bespoke configuration system based around XML files. The major change here would be going from the files residing on some local share to them residing in BLOB storage. Code change needed for that was done, as well as the change to the settings deployment mechanism – which now went from simply copying files to the share to uploading the files to Azure BLOB storage.

Logging

We had two forms of logging: application level logging that uses log4net to write out to configured appenders. Since we went from file system to Azure Table Storage, this involved writing a new appender that would send the log entry off to Table Storage. The other piece was ETW-based tracing. We configured a VM extension as part of the Service Fabric cluster that would gather all ETW traces and upload them to Table Storage. The last part was redoing the log viewer tooling to read from Table Storage.

Caching

This was a significant change in going from AppFabric, which was being sunset, to Redis – which was the recommended distributed cache for Azure. We discovered a few leaks in our existing abstraction layer where some consuming code was dependent on AppFabric-specific features such as versioning. This needed some reworking of the code. Other than this, we quite extensively use the locking feature for distributed pessimistic locking. With Redis, no such feature natively exists. However, there is a published algorithm that states how it can be done. While our main Redis client library, StackExchange.Redis, did not have this implemented, we ended up using RedLock.net for this purpose.

Service Bus

I think this was the least work required, because we were already using Windows Service Bus – the almost out-of-support on-premises version of Azure Service Bus. So moving to Azure Service Bus was akin to a version upgrade without much rework needed on the code.

Emails

For all automated emails that the applications send out, we moved from an on-premises SMTP server to SendGrid. The initial setup was straightforward. Other considerations we had to work around was setting up SPF and DKIP with our DNS provider so that SendGrid could be designated as an authorized sender of emails from our domain. SendGrid has facilities to do all these so that step was easy. Additionally, we had to make sure DMARC setup was properly done on our side.

Tooling

Much of this has already been covered in other sections, but essentially boiling it down to a few bullet points, we had to work on the following aspects of tooling:

  • Deployment – we are now deploying to Service Fabric and not to IIS anymore, plus as part of deployment, we are also generating Fabric projects on the fly based on host model descriptors.
  • Logging – the tool we use to aggregate and view logs now talks to Table Storage.
  • Databases – tooling related to database backup/restore, export/import, etc. all have to work against Azure. Additionally, database maintenance scripts are now hosted using Azure Automation instead of SQL Server Agent.
  • Provisioning – we needed to write a brand new provisioning system using Azure Resource Manager APIs that would allow us to stand up and tear down platforms as needed on Azure.

Rollout

We stood up the entire environment a week prior to actual rollout, which gave us plenty of time to test and correct. Owing to the database migration strategy that involved replication, a lot of risk was mitigated. Thus, the actual rollout involved simply:

  • Start outage window.
  • Repoint DNS to new application.
  • Ensure replication has no transactions pending, and turn off replication.
  • Upload BLOBs that were created during the last week.
  • End outage window.

And with that, we were on Azure. Not a traditional lift-and-shift by any means. We were fully cloud-native on PaaS – wholesale- the entire system. Quite a feat if you ask me.

Moving to Azure PaaS and Service Fabric- Part 1

This is Part 1 of a two-part blog series:

  • Part 1 (this one; Application- Services, Security and UI)
  • Part 2 (Database, Configuration, Logging, Caching, Service Bus, Emails, Tooling, Rollout)

It has been an action-packed year at work. We moved our entire platform in one fell swoop from on-premises to Azure PaaS (Platform as a Service). Since this was a big re-platforming effort that would incur regression testing across the entire set of applications, we took this opportunity to include a few technology upgrades in the process. All in all, it was a daunting task and took quite a bit of research and preparation before the actual implementation could be done. I think it is worth it to highlight some of the key achievements. The move entailed the following key aspects:

  • Application- Services, Security and UI: The move from WCF on IIS to Web API over OWIN on Azure Service Fabric, the move from WS-Fed over WIF to OAuth2/OpenID Connect, and the move from server side MVC with jQuery based AJAX to full-on SPA with AngularJS
  • Database: The move from SQL Server on-premises to a combination of SQL Azure PaaS and Azure BLOB Storage
  • Configuration: The move from file system to Azure BLOB Storage
  • Logging: The move from file system to Azure Table Storage
  • Caching: The move from AppFabric to Redis
  • Service Bus: The move from on-premises Windows Service Bus to Azure Service Bus
  • Emails: The move from on-premises SMTP server to SendGrid.
  • Tooling: Building provisioning, deployment and other tools for the new platform
  • Rollout: The actual deployment to and switch to the new platform

I will attempt to do a quick overview of each of these aspects.

Application: Services, Security and UI

We have a microservices-like architecture- not all the way microservices though. We practice SOA and thus our system is decomposed into services that have dependencies on each other and GUIs that have dependencies on the services. This is decoupled from our hosting and deployment model. We group several services into hosts based on a number of factors. A host then is our unit of deployment and represents a runtime process. Additionally, the services don’t carry their configuration with them – the configuration is a single deployment artifact that is then centrally accessed by all services.

The existing on-premises hosting mechanism was using IIS to expose each host as an application that then exposed the underlying services over WCF. With the announcement of Service Fabric, we decided that was going to be our new hosting environment with our move to Azure. Having been dealing with deployment and management of a significant number of services and having seen the challenge of doing that on IIS without the help of any service orchestration system, we knew we needed something. Service Fabric proved to be the ideal solution:

  1. While it has container orchestration facilities like those provided by Kubernetes (maybe not all the way for containers like Kubernetes), that was not our use case. We had pure .NET workloads that had first-class support for application level orchestration through Service Fabric. This was the perfect scenario.
  2. Proven robustness – in that Service Fabric is what a lot of Azure services themselves run on top of.
  3. The opportunity to one day take advantage of the Stateful service model that is a capability you can take advantage of if you are running native .NET workloads on the fabric (as opposed to guest executables or containers).

We are enamored with it enough at this point that we would have adopted it even if we stayed on-premises. In fact, we still have an on-premises development environment (how that is maintained with abstraction layers so it can run both on-premises and with Azure PaaS is a whole another blog post that I won’t get into here) that runs on an on-premises installation of Service Fabric. With Service Fabric alleviating some of the complexity woes associated with making more granular microservice-style deployment, we considered moving to that scenario. However, looking at the memory footprint of each hosted service gave us pause for thought. We decided to stick with our existing model of grouping services into hosts that then would be the unit of deployment.

If I had to boil the steps involved down to a few bullet points:

  1. We created the concept of host model descriptors. Concretely, these are JSON files that describe how a host is constructed, what services are involved, how they call each other, and so on.
  2. We created a host generation mechanism that was part of the deployment tooling. This was able to read a host model descriptor along with pre-defined templates and on-the-fly generate Service Fabric service and application projects which could then be packaged up and published to Service Fabric.
  3. We created a script to read all our existing WCF host projects and convert them to host model descriptors.
  4. We needed an API gateway to expose the services running on the fabric. On top of this, we now had the added layer of mapping services to whatever hosts they were running on. This way, consumers of those services would not need to concern themselves with the host allocation and could simply address a service. In order to handle these needs and a bunch of others, we built a Router application that then became the public facing endpoint to all the services.

With all this in place, we could now take our existing services and without touching a line of business operation code, we could re-target them to run on Service Fabric. Pretty neat. In terms of security, while WS-Fed using WIF was serving us well, we had recognized that it was perhaps a bit too bloated and while it played well with WCF, our implementation of it, which included a homegrown STS based on it, required quite a bit of upkeep. Additionally, at the time there was not enough extension points on the WCF based listeners on Service Fabric.

All this and a creeping notion that WCF was on its way to becoming a dying technology soon) acted as catalysts for us to make the decision to ditch WCF, WIF and WS-Fed altogether in favor of a lightweight JSON-over-HTTPS based model running on OWIN/Katana with OAuth2 (specifically OpenID Connect) acting as the security protocol of choice. While this involved changes to application hosting, the big chunk of the effort was in rewriting the STS as an OWIN/Katana-based Angular app that used IdentityServer to handle OpenID Connect. While OpenID Connect would be the primary protocol, we still had parties that needed to get in via SAML – so we had to build adapters for that as well.

In terms of the UI, our frontend applications were ASP.NET MVC based with a hodgepodge of jQuery, Knockout here and there, etc. and were lacking in any common unifying pattern. The WIF integration also was a little too tight of a coupling for my liking. Since redoing our frontends on a core SPA driven pattern was always in our wish-list, this was the perfect time to do so since we would have to deal with removing WIF and integrating with our new OAuth2 based STS, and with hosting the frontends on the Service Fabric as well. We opted to go with AngularJS (1.x) as it was the best choice for us at the time (since then, for our newer efforts, we have evaluated the new Angular and React as well with the decision to be made soon). The makers of IdentityServer also provide a OIDC token manager that is built specifically for SPA applications to work with STS applications based on IdentityServer. We just had to build a few core modules and codify our patterns.

That’s quite enough for now. More to come in Part 2.

The New Way Forward for SPAs with Angular and React

Having worked with Angular 1.x for some time and having liked it quite a lot (I guess that one we’re supposed to call AngularJS, and the new one is just Angular – yes, that is not confusing at all, is it?), I must say I was quite spooked when I first saw the documentation for the new Angular. It indeed is a completely different framework. There is no easy migration path from AngularJS short of a rewrite, at which point you might as well evaluate all your options including React.

Having evaluated both the new Angular as well as React, I noticed a few things. First, looks like I will have to eat my words regarding TypeScript, as that is now the de-facto dialect of choice for Angular. With React, it seems to be ES6, which at this point, I still prefer over TypeScript (even though TS has more features, a gap that I hope will be bridged with ES7).

Second, the tooling around JS coding is crazy complex now! You can’t just throw together a bunch of ES5 scripts into your server-side project, whatever platform it may be in, and call it a day. JS has evolved so much with ES6/TypeScript, there is build and deployment tooling around it – all based on Node – there is package management with NPM and the like – it is a full blown programming ecosystem. What this means is that if you’ve been late to the game adopting JS tooling because you could get away with slapping ES5 scripts together, with these new frameworks or libraries, you will not have that choice anymore, so you’d better get on-board.

Third, and perhaps most interesting, is that as different as they appear, they are similar in spirit. Both the new Angular and React are component-based – and that, it turns out, is a good way to be as far as frontend development is concerned. Looking at both Angular and React from the AngularJS lens, and ignoring the internals and the performance and runtime implications, you could draw a comparison to directives. I remember thinking when working with AngularJS – why all this asymmetry? Why isn’t everything simply a directive? Well, looks like quite a few people agreed.

In any case, if you’re going to build a SPA today, you are going to have to weigh quite a few options and your decision-making process has become more difficult. You are probably going to see something better come along in a few months once you’ve adopted something – and while it can be frustrating, these are exciting times to be doing JS on the frontend.

Fiddling with F#

I have always had a keen interest in functional programming. While I still shy away from going completely functional for full-blown applications, I try to use the tenets of functional programming as much as I can even when writing C#. This is made much easier by the fact that C# has borrowed a lot of functional programming features as it has evolved. With each new version of the language, I find my code getting more concise and more expressive mostly owing to these features. That said, if you are looking for a functional-first experience, nothing beats a functional language. I like F# as it belongs to the .NET ecosystem but is derived from OCaml which itself is quite elegant.

There is somewhat of a barrier to entry with functional programming in terms of the learning curve. Sites like this amazing one by Scott Wlaschin make it much more appealing. Having said all that, since I am not quite at the point where I will build a full blown application with F# (plus I can’t at my day job anyway), but I have that itch to use it that needs to be scratched, I found a good use case for it, at my day job nonetheless.

Any time I need to write a quick script that does something, I default to F# these days. A good example of this is a bunch of project file cleanup scripts I needed to write that cleaned up CSPROJ files (references, Nuget packages and what not). I keep an F# program around (not even an actual compiled program – it’s a LINQPad script) that I run and add to as needed, and it works beautifully. Here are a few snippets from it that are generic enough that I can share here. Everything is in a module ProjectCleanup.

Some basic utility functions first:

let getFiles sourcePath pattern = 
    Directory.GetFiles(sourcePath, pattern, SearchOption.AllDirectories)

let excludeNones (xs: seq<'a option>) =
    xs
    |> Seq.filter (fun x -> match x with Some _ -> true | _ -> false)
    |> Seq.map (fun x -> match x with Some x' -> x' | _ -> failwith "Invalid")

let toElements (nodes:XmlNodeList) = seq {
    for node in nodes do
        yield node :?> XmlElement
}

let toMatches (ms:MatchCollection) = seq {
    for m in ms do
        yield m
}

let toMany (a:seq<seq<'a>>) = seq {
    for x in a do
        for y in x do
            yield y
}

A few helper functions to work with a bunch of files or a bunch of XML files (which CSPROJ files are).

let processFilesXml processor (files: seq<string>) =
    files
    |> Seq.iter (fun f ->
        let d = XmlDocument()
        d.Load f
        match processor d |> Seq.exists id with
        | true -> d.Save f
        | false -> () 
    )

let processFilesLines processor files =
    files
    |> Seq.iter (fun f ->
        let ls = File.ReadAllLines f
        let lcs = processor ls
        let changed = lcs |> Seq.exists (fun (c, _) -> c)
        let ls' = lcs |> Seq.map (fun (_, l) -> l) |> Seq.toArray
        match changed with true -> File.WriteAllLines(f, ls') | _ -> ()
    )

The following removes specific targets and imports that intrude into the CSPROJ when you install certain packages. They serve a purpose, but not in my case – so they have to go.

let removeBadTargetsAndImports sourcePath =
    printfn "Removing bad targets and imports from %s..." sourcePath
    getFiles sourcePath "*.csproj"
    |> processFilesXml (fun d ->        
        let importElementsToRemove =
            d.GetElementsByTagName "Import"
            |> toElements
            |> Seq.map (fun e ->
               match e.GetAttribute("Project").ToLower() with
               | p when p.Contains "nuget.targets" -> Some e
               | p when p.Contains "microsoft.bcl.build.targets" -> Some e
               | _ -> None 
            )
            |> Seq.toArray
        let targetElementsToRemove =
            d.GetElementsByTagName "Target"
            |> toElements
            |> Seq.map (fun e ->
                match e.GetAttribute("Name").ToLower() with
                | n when n.Contains "ensurebclbuildimported" -> Some e
                | n when n.Contains "ensurenugetpackagebuildimports" -> Some e
                | _ -> None
            )
            |> Seq.toArray
        Seq.concat ([|importElementsToRemove; targetElementsToRemove|])
        |> excludeNones
        |> Seq.map (fun x -> 
            x.ParentNode.RemoveChild x |> ignore
            true 
        )
    )

This one removes duplicate entries from packages.config files:

let dedupePackages sourcePath =
    printfn "De-duping packages in %s..." sourcePath
    getFiles sourcePath "packages.config"
    |> processFilesXml (fun d ->
        let elementHash = ref Map.empty            
        d.GetElementsByTagName "package"
        |> toElements
        |> Seq.map (fun e ->
            let id = e.GetAttribute "id"
            match elementHash.Value.ContainsKey id with
            | true -> Some e
            | false ->
                elementHash.Value <- elementHash.Value.Add(id, true)
                None
        )
        |> Seq.toArray
        |> excludeNones
        |> Seq.map (fun x ->
            x.ParentNode.RemoveChild x |> ignore
            true 
        )
    )

None of my projects want to target specific versions of assemblies. That is just tempting the binding redirects monster. So they have got to go.

let removeReferenceVersions sourcePath =
    printfn "Removing reference versions from %s..." sourcePath
    getFiles sourcePath "*.csproj"
    |> processFilesLines (fun ls ->
        ls
        |> Seq.map (fun l ->
            match l with
            | line when (line.Trim().StartsWith "<Reference Include=\"") && (line.Contains ",") && not (line.Contains "processorArchitecture=AMD64") ->
                let xml = line.Trim()
                let endTag = match xml.EndsWith ">" with true -> "</Reference>" | _ -> ""
                let doc = XmlDocument()
                doc.LoadXml (sprintf "%s%s" xml endTag)
                doc.DocumentElement.SetAttribute("Include", ((doc.DocumentElement.GetAttribute "Include").Split ',').[0])
                doc.DocumentElement.InnerXml <- ""
                let newXml = doc.DocumentElement.OuterXml.Replace("</Reference>", "")
                let newXml' = (match xml.EndsWith "/>" with true -> newXml.Replace(">", "/>") | _ -> newXml).Replace("//>", "/>")
                (true, line.Replace(line.Trim(), newXml'))
            | line -> (false, line)
        )
        |> Seq.toArray            
    )

Test setting files and test sections in solutions have got to go:

let deleteTestSettingFiles sourcePath =
    printfn "Deleting test setting files from %s..." sourcePath
    Seq.concat [|getFiles sourcePath "*.vsmdi"; getFiles sourcePath "*.testsettings"|]
    |> Seq.iter (File.Delete)

let removeSolutionTestSections sourcePath =
    printfn "Removing solution test sections from %s..." sourcePath
    getFiles sourcePath "*.sln"
    |> Seq.map (fun file -> (file, File.ReadAllText file))
    |> Seq.map (fun (file, content) ->
        (file, content, Regex.Match(content, "Project\\([\"\\w\\-\\{\\}]+\\) = \"Solution Items\", \"Solution Items\".+EndProjectSection\r\nEndProject", RegexOptions.Singleline).Value))
    |> Seq.filter (fun (_, _, x) -> not (String.IsNullOrWhiteSpace x))
    |> Seq.map (fun (file, content, solutionItems) -> (file, content.Replace(solutionItems, "")))
    |> Seq.iter (fun (file, content) -> File.WriteAllText(file, content))

Finally, I can call what I need to like so:

let main =
    let sp = @"C:\My_Source_Directory"
    removeBadTargetsAndImports sp
    dedupePackages sp
    removeReferenceVersions sp
    deleteTestSettingFiles sp
    removeSolutionTestSections sp
    0

For now, anyway, I find this is a nice balance and keeps me actively using both C# and F#, one for major application development, and the other one for these tools.