Mike Feingold's Adventures in programming

I was not sure it is possible, but it is. I managed to implement the approach I described here, found a workaround to (almost) every problem I ran into along the way and packaged all of it into one little Visual Studio extension package which allows you to organize your f# project like this.

You can download it here.
Using the package is really simple. Once you open an F# project after installing the package, the context menu on the project in the Solution Explorer will give you one new item:

Clicking on the item will enable the Project Extender for your project. From this moment on the behavior of the solution explorer will change in a number of ways.

First of all the order of items in the solution explorer will no longer reflect the compilation order. All files will be listed in the alphabetical order within their directories, right after the subdirectories listed also in the alphabetical order.

Context menus on the project node as well as subdirectories now also provide a way to create new (sub)directories. Renaming items changes the position of the item in the solution explorer if it is necessary to keep the alphabetical order of items.

All of the features listed above and a few more which are not listed are self - explanatory and very similar to the way the Solution Explorer works for C# project among others.

The file compilation order is not affected by any changes to the files performed through solution explorer. Compilation order is controlled by a new project property page:

Pressing Move Up and Move Down buttons changes the compilation order by moving the selected file up or down in the compilation order list.

As explained in the previous post ability to support compilation order across multiple directories involves some cheating. In practical terms it means that every time a project is loaded, the project file will be modified. When the project is closed it is modified back. It works ok with subversion, because the final state of the project file does not change, but with other source control systems it may force you to checkout the project file even though you have no intention of modifying it. This is one of the things I found no way around. I hope you can live with it as I plan to.

One more thing worthy mentioning is that once you enable the Project Extender you will get two more buttons on the solution explorer: Show All Files and Refresh:

Pressing the Show All Files button makes all files and/or subdirectories in the project directory visible in the solution explorer. Pressing this button again removes the elemens displaying such files from the solution explorer. Refresh button refreshes the list of files to bring it up to date. Again this functionality is similar to how it is done in other project systems, including ability to re-add previously excluded files back to the project.

And finally one last note: when you get really annoyed with the Project Extender you can easily disable it by using Disable F# project extender item on project context menu in the solution explorer.

I hope you will never use this command. Enjoy.

P.S. At this time the only platform supported is Visual Studio 2008/F# CTP 1.9.9.9. I am working on implementing it for Visual Studio 2010.

As of version 0.9.1.0 both VS2008 and VS 2010 are fully supported

Cheating is bad - right? But what if this is the only option available? As you may or may not know the project system shipped with F# compiler imposes limits on the  compilation order  for the files in non-flat directory structure. Look here for the explanation.

I was looking for a way to squeeze by this limitation and here is what I found out: all this strictness is only enforced when

• The project is being loaded.
• project elements are added or renamed.

In particular these rules are not validated or enforced when the project is compiled. So the solution I came up with is to give the F# project manager something it can live with and when it is happy with what's provided and looks the other way rearrange the elements in the project file in the order I want them to be. If I do that I can use my custom compile order dialog to arrange the compilation order any way I want - for as long as the F# project manager looks the other way.

When the project is saved it has to be brought back to the state compatible with the F# project manager, otherwise it will refuse to load it next time around. So I intercept the onAfterProject close event and reorder project elements to comply with the F# project manager demands. While doing so I decorate the elements I move with custom XML elements to preserve the information how should I move them back to restore the (non-compliant) order of the files.

I also have to be on the lookout for F# project manager turning its attention to the build file element order. When it is about to happen I will have to adjust the element order in the build file to F# project manager liking, and it turns its back on me again I will have to move the items back.

That's the plan at the moment, let's see how far it's gonna take us.

I love F#. Seriously. For many reasons. Working with F# code gives this fresh liberating feeling. It takes code conciseness on a whole new level.

My love to F# does not extend to the F# project system in Visual Studio though - not quiet. Do not get me wrong - it is solid and reliable and all of it. Not without  glitches but what real product is. It is just one particular thing about the F# project system that irks me - it just keeps getting in the way. I am talking about F# project main navigation tool - Solution Explorer tree.

The Problem

Any other project system I know of (C#, VB, etc.) would list the files in alphabetical order. Not F#. You see - with F# compilation order is important, so designers of the F# project system decided to use Solution Explorer to arrange the F# code files in the proper compilation order. So files in the Solution Explorer tree are listed in the order in which they will be compiled, and Solution Explorer provides tools to move files around to control such order. As reasonable as it sounds it creates a few problems. One of them is that if you have more than a few files in your project it becomes more difficult to look your file up. It becomes especially messy if your project includes other file types - html, js, what have you. One can argue that this is a minor point and I can concede that it is - to a degree.

But here is another one - F# project system does not support directories. There is no way to add a directory to your project through GUI. There is a back door way - you can go and edit the project file - manually add <compile include=""> tags with include pointing to a subdirectory. Than a directory entry will appear in the solution explorer tree. But once you do it, controlling the compilation order through this view becomes even more problematic.  Consider the following project directory structure:

There is no way to change compilation order so that Module2 is compiled after Module1 but before Module3. You can edit the build file to change the order of the corresponding <compile include=""> manually and if you run it with MSBuild it will compile but, the project system will not even load it. The reason is that showing this in the solution explorer would take two  Controllers directory nodes with the entry for Module2 right between them. I even found a way to break an F# project with subdirectories through GUI interface - just create several directories and then start adding files through right-click menu on the directories. Everything will seem to be fine, but after you add enough of them an attempt to save and re-open the project will fail with the same message - something to the effect of "cannot create several entries for the same directory".

The bottom line here is that the F# project system severely limits your ability to organize files in the project. It is especially painful if the project includes files of type other than *.fs - i.e. *.htm or *.js.

The Solution

The root of all this evil here is that the Solution Explorer tree is used for two distinct purposes - one is presenting the content of the project file so it can be easily reviewed and organized, and another one  to provide means to manipulate compilation order. If there would be a different screen to control compilation order, the Solution Explorer tree could be reverted to the model used by all other project systems.

This is exactly how I did it in my Bistro designer - I modified the behavior of the solution Explorer so that it follows the standard pattern - first virtual directories, then physical directories, than physical files, everything in alphabetical order - regardless of the order of the corresponding project items. To control compilation order I created a new project property page:

Changing compilation order in the project property does not change how the same files are presented in the Solution Explorer. Also having a separate screen for controlling the compilation order opens other interesting possibilities. One of them is to automatically order the files based on file dependencies provided by the user. This can be more convenient, because usually majority of the files depend on a small number of "common" files.

And if after all the compiler guys will find a way to build dependencies programatically, this property page can be dropped altogether

When you see somebody laying a claim like this, your first reaction should be - this is a hoax, there is no ultimate solution to double checked locking problem. And you will be correct.

Yet I do have an ultimate solution, although it only solves this problem for load/create on demand scenarios. It also involves some cheating. Let me explain myself.

Double Checked Locking - some background

Double checked locking is usually presented as a way to reduce contention around some global object used in a multi-threaded environment. It is pretty common in such systems to have global objects which are updated rarely but accessed very often. A solution to double checked locking problem would provide a way to first check whether the lock is necessary and only apply the lock if it is. When (and if) the lock is applied the usual pattern is used which also involves checking - hence the name 'double checked locking'

There are numerous, sometimes heated, discussions on the topic (as a starting point you can check here) but the bottom line is that all suggested generic solutions either have implicit locks or are not thread safe.

So, what about my claim?

Here is the scenario: Let us say you have a list of resources to be used by multiple threads. If resource creation is expensive you would want to ensure that every resource is created on demand the first time it is requested. So when a thread needs a resource it has to check first whether this resource was requested before and if it is, take the existing one rather than create another copy. The dictionary of the resources is that very object which would benefit from the double checked lock.

The thing is that in this particular case it is possible to safely implement the first check of the double checked lock schema outside of any locks. All you need to do is in addition to keeping a global dictionary make copies of the dictionary - one per processing thread and make every thread perform the check against its own dictionary. Any copy of the dictionary will be only accessed by the thread which owns it, so there is no locks necessary while checking this dictionary.

If the requested resource is located you can take it and use it. If it is not you have to check whether it was created by a different thread or you need to create it. To do that you have to  acquire a lock on the global dictionary, check if the resource is already there, create and add it to the global dictionary if it is not and, finally create a new copy of the global dictionary to replace the copy in your worker thread, the usual stuff - all of this while holding the lock.

This second part is no different from the straightforward implementation of the global resource dictionary. The difference is that if it were the only one, every request for the resource would have to go through a global lock which in the systems with a big number of simultaneous requests can create a severe bottleneck. Adding extra layer with per thread dictionaries remedies this problem - every thread which already has a resource in its local dictionary will access it without any locking.

Let me be explicit about some assumptions here.

• First of all it is implied that all threads share a single copy of the resource. It means that either it has to be immutable or you will have to build some synchronization over access to the resource itself - which takes you back to the square one.
• Secondly the dictionary itself has to be immutable as well so that the same copy can be shared by multiple threads.

If it sounds very F# than that's because it is - this is exactly how NDjango rendering engine is implemented and it is written in F#

There is one more thing I have to admit to. Speaking theoretically my approach does not solve anything - it just pushes the same problem to the process responsible for thread allocation. Think about it - you have a list of available threads and when a new request comes in, a thread has to be allocated to it and removed from the list. It is the same problem. Luckily for us people creating these parts of our software/hardware (like ASP.NET core) are really smart and they solved this problem for us in the most efficient manner, so for all practical purposes it does solve the problem.

Sounds strange - huh? Here is the deal: Imagine you have a service running. It is happily processing incoming requests returning whatever it is supposed to return. Now imagine that in doing so it has to take into account certain global settings. The usual way to implement the settings is to put every value into a dictionary keyed by the setting name. The dictionary is owned by the service and every time some part of the service needs a setting value it pulls it from the dictionary.

In my case the service is the NDjango rendering engine. And to give you an example of a setting: an error string, a string to be inserted into the rendering result every time a django variable fails to resolve.

Now, what if you need to change a setting while the engine is running? The first thing which comes to mind is just go and stick a new value into the dictionary. Well... there are a few complications which may or may not be a real issue depending on what is the nature of the setting and how it is implemented. What if the change hits right in the middle of a request processing? Depending on the setting the impact can be anywhere from inconsistent response all the way through to the service crash. There is a number of ways to prevent such problems. You can clone the settings for every request. You can suspend processing, wait till all active request complete processing and only then apply changes. Or you can write your code so that every setting is accessed only once. All these remedies are difficult to implement and easy to break. Some of them can also have negative performance impact.

So how immutability can help here? Or better yet how can you change settings if they are immutable?

Here is something I picked up from my F# coding: instead of trying to change the values owned by existing service, create a new instance of service with updated values. Then leave the old instance deal with the existing requests and make sure that the new ones are directed to the new instance. This is a design pattern which certainly can be implemented in any language. Implementing it in F# provides additional safety of ensuring on the language level that the values are read only.

This design pattern can be used for any number of things - not just settings. In NDjango this pattern gave me a way to implement system - wide template caching with no global locks for templates already in the cache. Global locks are only involved when templates are loaded and parsed.