NDjango: Getting cozy with ASP.MVC

August 2nd, 2010

Almost there. Just a few more touches, tests, spelling corrections... No, really, the new NDjango editor I was working on for some time now is almost ready. The major improvements over the existing one are in the overall usability as well as ASP.NET MVC compatibility. Here is a brief rundown on what's coming:

Code Completion for strongly typed models

Currently NDjango editor supports code completion for tag and filter names as well as for a few other syntax elements. With the new version two more elements are added to the list: block names and variables.

Code completion for variables works everywhere a variable reference can exist and shows a list of all variables available in the context. The code completion dropdown includes global (for the template) variables provided by the view engine as well as context based variables introduced by the django tags, i.e. forloop variable inside the for tag. A new tag <% model [Name:Type]... %> can be to used introduce additional global variables. The specified name will be shown on the code completion list.

Typing '.' after the name of a variable triggers another code completion dialog with a list of members/properties as defined by the variable type. This feature combined with the ability to provide the model type in the model tag provides great support for strongly typed models.

The list of predefined global variables depends on the project type. For Bistro projects the list will include all resources provided by the Bistro controllers. For ASP.NET MVC integration the context will only include the "Session" variable and the "Model" variable for strongly typed views (see below ASP.NET MVC integration)

Block names code completion dialog in the block tag shows a list of blocks available for overriding. The list includes names of the blocks defined in parent templates.

Add New Django View wizard

Add new view wizard simplifies creating new templates by providing dialogs to select base template, model type, and blocks to override

ASP.NET MVC integration

There are quiet a few changes in the ASP.NET MVC integration. The overriding goal is to make integration as seamless as possible. Specfically:

  • Django Views can be mixed and matched with WebForm views or any other types of views
  • Full support for Areas
  • HtmlHelpers can be used from inside of Django views
  • Views with strongly typed models provide full blown code completion support
  • New DjangoViewEngine constructor simplifies working with custom tag libraries
  • Session global variable provides direct access to the session object from the views
  • Extensive samples provide a good starting point for writing your own views/tags/filters

Sample Code

A new ASP.NET MVC sample project will be included in the shipment. The project is an extension of the stock ASP.NET MVC sample project created by the "Create a New ASP.NET MVC2 Web Application" wizard. In this project a new tab called "Django" is added to the menu. Also for the register new user view the aspx file is superceded by a django template.

This project can be used as guidance in respect to a few aspects of using NDjango view engine:

  • Using Django inheritance in lieu of master pages
  • Accessing context values with Django variables
  • Setting up NDjango to use custom tag libraries
  • Using HtmlHelpers from within Django templates
  • Using ascx controls from Django templates

Also included in the shipment is a Sample Library project. This project is a tag library with several tags providing access to various HtmlHelpers. These tags were written for the ASP.NET MVC sample project. They also can be used as starting point for developing your own custom tags.

I hope I did not miss anything. I hope I will be able to wrap it all up over the next week or so. I am also looking into the ASP.NET MVC3 and will include MVC3 integration as well if it will not get in the way too much.

See you soon.

F# Project System, meet Project Extender

April 26th, 2010
I was not sure it is possible, but it is. I managed to implement the approach I described here, found a workaround to (almost) every problem I ran into along the way and packaged all of it into one little Visual Studio extension package which allows you to organize your f# project like this. FSharpTree

You can download it here.
Using the package is really simple. Once you open an F# project after installing the package, the context menu on the project in the Solution Explorer will give you one new item: ProjExtenderMenu

Clicking on the item will enable the Project Extender for your project. From this moment on the behavior of the solution explorer will change in a number of ways.

First of all the order of items in the solution explorer will no longer reflect the compilation order. All files will be listed in the alphabetical order within their directories, right after the subdirectories listed also in the alphabetical order.

Context menus on the project node as well as subdirectories now also provide a way to create new (sub)directories. Renaming items changes the position of the item in the solution explorer if it is necessary to keep the alphabetical order of items.

All of the features listed above and a few more which are not listed are self - explanatory and very similar to the way the Solution Explorer works for C# project among others.

The file compilation order is not affected by any changes to the files performed through solution explorer. Compilation order is controlled by a new project property page:ProjExtenderProperties

Pressing Move Up and Move Down buttons changes the compilation order by moving the selected file up or down in the compilation order list.

As explained in the previous post ability to support compilation order across multiple directories involves some cheating. In practical terms it means that every time a project is loaded, the project file will be modified. When the project is closed it is modified back. It works ok with subversion, because the final state of the project file does not change, but with other source control systems it may force you to checkout the project file even though you have no intention of modifying it. This is one of the things I found no way around. I hope you can live with it as I plan to.

One more thing worthy mentioning is that once you enable the Project Extender you will get two more buttons on the solution explorer: Show All Files and Refresh:ProjExtenderShowAll

Pressing the Show All Files button makes all files and/or subdirectories in the project directory visible in the solution explorer. Pressing this button again removes the elemens displaying such files from the solution explorer. Refresh button refreshes the list of files to bring it up to date. Again this functionality is similar to how it is done in other project systems, including ability to re-add previously excluded files back to the project.

And finally one last note: when you get really annoyed with the Project Extender you can easily disable it by using Disable F# project extender item on project context menu in the solution explorer.

I hope you will never use this command. Enjoy.

P.S. At this time the only platform supported is Visual Studio 2008/F# CTP 1.9.9.9. I am working on implementing it for Visual Studio 2010.

As of version 0.9.1.0 both VS2008 and VS 2010 are fully supported

In regards to the virtue of cheating the F# project system

April 1st, 2010

Cheating is bad - right? But what if this is the only option available? As you may or may not know the project system shipped with F# compiler imposes limits on the  compilation order  for the files in non-flat directory structure. Look here for the explanation.

I was looking for a way to squeeze by this limitation and here is what I found out: all this strictness is only enforced when

  • The project is being loaded.
  • project elements are added or renamed.

In particular these rules are not validated or enforced when the project is compiled. So the solution I came up with is to give the F# project manager something it can live with and when it is happy with what's provided and looks the other way rearrange the elements in the project file in the order I want them to be. If I do that I can use my custom compile order dialog to arrange the compilation order any way I want - for as long as the F# project manager looks the other way.

When the project is saved it has to be brought back to the state compatible with the F# project manager, otherwise it will refuse to load it next time around. So I intercept the onAfterProject close event and reorder project elements to comply with the F# project manager demands. While doing so I decorate the elements I move with custom XML elements to preserve the information how should I move them back to restore the (non-compliant) order of the files.

I also have to be on the lookout for F# project manager turning its attention to the build file element order. When it is about to happen I will have to adjust the element order in the build file to F# project manager liking, and it turns its back on me again I will have to move the items back.

That's the plan at the moment, let's see how far it's gonna take us.

NDjango: Closing the gap on the Visual Studio ASPX editor

March 23rd, 2010

One of glaring weaknesses of a proposition to use a view engine different from .aspx is that Visual Studio provides a very nice .aspx editor full of features both for editing the UI as well as .aspx source code, while this 'other' engine usually offers nothing of the sort - just a notepad to type in anything you want.

Not that I would ever dream of singlehandedly build something comparable to the Visual Studio .aspx designer, but the NDjango editor makes at least some of the .aspx editor features available for the NDjango users.

Let us go through the list:

Feature aspx editor NDjango editor
Syntax highlighting - clearly marks view engine constructs by highlighting the brackets separating the constructs from the rest of the template aspx syntax highlight ndjangosyntaxhhighlight
Code Completion - Shows a context specific list of available options as you type aspxcodecompletion ndjangoCodeCompletion
As you type diagnostics - highlights invalid items in the source code and provides diagnostic messages in the Visual Studio ErrorList DiagASP DiagNdjango
Item Hierarchy display - visualizes the nested constructs hierarchy HierarchyASP HierarchyNDjango

As anyone can clearly see the NDjango editor still has a long way to go, but IMHO the features it already supports can be of considerable help for the NDjango users.

NDjango editor for VS2010 – 2 down 1 to go

March 15th, 2010

From VS2010 Beta 1 to Beta 2 and now to RC. I will still need to port it (the editor) once more - from RC to the RTM, but for now enjoy the RC compatible version of the editor

This post is about my experience with the port and it will be really short because the port was pretty uneventful.

First of all the Intellisense Source interfaces (IQuickInfoSource, ICompletionSource) no longer have Get...Information methods. They have Augment...Session instead. Which for me makes more sense because now you can not only add completion sets from the source but also examine and even change existing elements in the completion set. These interfaces now also sport a Dispose method to help with the resource cleanup.

The changes were easy  - move the logic from GetCompletionInformation/GetToolTipContent method to the appropriate Augment method and add an empty Dispose method. Another small change is that there SquiggleTag is no longer which only made my life easier because now instead of deriving my own ErrorTag from SquiglleTag I can use the predefined ErrorTag.

And this is pretty much it.

A better F# project system

February 8th, 2010

I love F#. Seriously. For many reasons. Working with F# code gives this fresh liberating feeling. It takes code conciseness on a whole new level.

My love to F# does not extend to the F# project system in Visual Studio though - not quiet. Do not get me wrong - it is solid and reliable and all of it. Not without  glitches but what real product is. It is just one particular thing about the F# project system that irks me - it just keeps getting in the way. I am talking about F# project main navigation tool - Solution Explorer tree.

The Problem

Any other project system I know of (C#, VB, etc.) would list the files in alphabetical order. Not F#. You see - with F# compilation order is important, so designers of the F# project system decided to use Solution Explorer to arrange the F# code files in the proper compilation order. So files in the Solution Explorer tree are listed in the order in which they will be compiled, and Solution Explorer provides tools to move files around to control such order. As reasonable as it sounds it creates a few problems. One of them is that if you have more than a few files in your project it becomes more difficult to look your file up. It becomes especially messy if your project includes other file types - html, js, what have you. One can argue that this is a minor point and I can concede that it is - to a degree.

But here is another one - F# project system does not support directories. There is no way to add a directory to your project through GUI. There is a back door way - you can go and edit the project file - manually add <compile include=""> tags with include pointing to a subdirectory. Than a directory entry will appear in the solution explorer tree. But once you do it, controlling the compilation order through this view becomes even more problematic.  Consider the following project directory structure:FSharpTree

There is no way to change compilation order so that Module2 is compiled after Module1 but before Module3. You can edit the build file to change the order of the corresponding <compile include=""> manually and if you run it with MSBuild it will compile but, the project system will not even load it. The reason is that showing this in the solution explorer would take two  Controllers directory nodes with the entry for Module2 right between them. I even found a way to break an F# project with subdirectories through GUI interface - just create several directories and then start adding files through right-click menu on the directories. Everything will seem to be fine, but after you add enough of them an attempt to save and re-open the project will fail with the same message - something to the effect of "cannot create several entries for the same directory".

The bottom line here is that the F# project system severely limits your ability to organize files in the project. It is especially painful if the project includes files of type other than *.fs - i.e. *.htm or *.js.

The Solution

The root of all this evil here is that the Solution Explorer tree is used for two distinct purposes - one is presenting the content of the project file so it can be easily reviewed and organized, and another one  to provide means to manipulate compilation order. If there would be a different screen to control compilation order, the Solution Explorer tree could be reverted to the model used by all other project systems.

This is exactly how I did it in my Bistro designer - I modified the behavior of the solution Explorer so that it follows the standard pattern - first virtual directories, then physical directories, than physical files, everything in alphabetical order - regardless of the order of the corresponding project items. To control compilation order I created a new project property page:propertyPage

Changing compilation order in the project property does not change how the same files are presented in the Solution Explorer. Also having a separate screen for controlling the compilation order opens other interesting possibilities. One of them is to automatically order the files based on file dependencies provided by the user. This can be more convenient, because usually majority of the files depend on a small number of "common" files.

And if after all the compiler guys will find a way to build dependencies programatically, this property page can be dropped altogether

HOW TO: Hurt your server really bad

January 15th, 2010

I am sure you can think of quite a few ways to do it. Here is mine: use the X509 certificate based authentication in a web app. Isn't it simple?

Here is what happened: after through testing we deployed our app to a farm of W2003 servers. It worked really well for a while. In a couple of months one of the servers started to slow down - the response time went up. We even started to see occasional timeouts in the logs. We looked at everything - CPU utilization, memory consumption, Windows handles - nothing to put your finger on. Meanwhile the server fell to its knees - 50% of the requests were timing out. Even to RDP into the server was taking up to 2 minutes. What's interesting - once you are in the server seemed to be pretty responsive, unless you try to start a program - launching notepad was taking up to 1 min. And again once it is up - it works fine. What scared us even more is that other servers in the farm started to deteriorate in the same way.

To solve the mystery we had to take a memory dump from production server. After looking at the dump we started to suspect that the problem has to do with the fact that some of our single sign on users are authenticated using X509 certificates. After some digging we stumbled upon this article . And that's where all the pieces finally fell into their places. Here is what's happening;

  • Apparently when validating an x509 certificate windows crypto module creates a temporary file in the Windows temp directory. I do not see any reason why an expense of creating a temp file would be necessary, but as stupid as it is, in itself this is not a problem.
  • What is a problem is a bug described in the article I mentioned. As explained in the article, as a result of this bug the temp file(s) created during x509 validation are not deleted. No big deal - just a few 0 length files nobody cares about - right?
  • Well, right - at least for some time. The problem is that the files keep piling up. When the number of files in the temp directory is low - below 65000, everything works fine, but when this number is exceeded, as stated in the article "the computer can experience significant delays". In our case it took our servers around 4 months to hit this number - and that is where we started to see these problems.

The solution was simple - just get rid of the files and everything goes back to normal. There is a hotfix available for the bug, but for some reason it is not included in the latest service pack. Because of this we decided against applying the hotfix. Instead we just configured a daemon job cleaning up temp directories overnight.

An Ultimate Solution to Double Checked Lock problem

November 27th, 2009

When you see somebody laying a claim like this, your first reaction should be - this is a hoax, there is no ultimate solution to double checked locking problem. And you will be correct.

Yet I do have an ultimate solution, although it only solves this problem for load/create on demand scenarios. It also involves some cheating. Let me explain myself.

Double Checked Locking - some background

Double checked locking is usually presented as a way to reduce contention around some global object used in a multi-threaded environment. It is pretty common in such systems to have global objects which are updated rarely but accessed very often. A solution to double checked locking problem would provide a way to first check whether the lock is necessary and only apply the lock if it is. When (and if) the lock is applied the usual pattern is used which also involves checking - hence the name 'double checked locking'

There are numerous, sometimes heated, discussions on the topic (as a starting point you can check here) but the bottom line is that all suggested generic solutions either have implicit locks or are not thread safe.

So, what about my claim?

Here is the scenario: Let us say you have a list of resources to be used by multiple threads. If resource creation is expensive you would want to ensure that every resource is created on demand the first time it is requested. So when a thread needs a resource it has to check first whether this resource was requested before and if it is, take the existing one rather than create another copy. The dictionary of the resources is that very object which would benefit from the double checked lock.

The thing is that in this particular case it is possible to safely implement the first check of the double checked lock schema outside of any locks. All you need to do is in addition to keeping a global dictionary make copies of the dictionary - one per processing thread and make every thread perform the check against its own dictionary. Any copy of the dictionary will be only accessed by the thread which owns it, so there is no locks necessary while checking this dictionary.

If the requested resource is located you can take it and use it. If it is not you have to check whether it was created by a different thread or you need to create it. To do that you have to  acquire a lock on the global dictionary, check if the resource is already there, create and add it to the global dictionary if it is not and, finally create a new copy of the global dictionary to replace the copy in your worker thread, the usual stuff - all of this while holding the lock.

This second part is no different from the straightforward implementation of the global resource dictionary. The difference is that if it were the only one, every request for the resource would have to go through a global lock which in the systems with a big number of simultaneous requests can create a severe bottleneck. Adding extra layer with per thread dictionaries remedies this problem - every thread which already has a resource in its local dictionary will access it without any locking.

Let me be explicit about some assumptions here.

  • First of all it is implied that all threads share a single copy of the resource. It means that either it has to be immutable or you will have to build some synchronization over access to the resource itself - which takes you back to the square one.
  • Secondly the dictionary itself has to be immutable as well so that the same copy can be shared by multiple threads.

If it sounds very F# than that's because it is - this is exactly how NDjango rendering engine is implemented and it is written in F#

There is one more thing I have to admit to. Speaking theoretically my approach does not solve anything - it just pushes the same problem to the process responsible for thread allocation. Think about it - you have a list of available threads and when a new request comes in, a thread has to be allocated to it and removed from the list. It is the same problem. Luckily for us people creating these parts of our software/hardware (like ASP.NET core) are really smart and they solved this problem for us in the most efficient manner, so for all practical purposes it does solve the problem.

Immutable settings make settings updates easy

November 25th, 2009

Sounds strange - huh? Here is the deal: Imagine you have a service running. It is happily processing incoming requests returning whatever it is supposed to return. Now imagine that in doing so it has to take into account certain global settings. The usual way to implement the settings is to put every value into a dictionary keyed by the setting name. The dictionary is owned by the service and every time some part of the service needs a setting value it pulls it from the dictionary.

In my case the service is the NDjango rendering engine. And to give you an example of a setting: an error string, a string to be inserted into the rendering result every time a django variable fails to resolve.

Now, what if you need to change a setting while the engine is running? The first thing which comes to mind is just go and stick a new value into the dictionary. Well... there are a few complications which may or may not be a real issue depending on what is the nature of the setting and how it is implemented. What if the change hits right in the middle of a request processing? Depending on the setting the impact can be anywhere from inconsistent response all the way through to the service crash. There is a number of ways to prevent such problems. You can clone the settings for every request. You can suspend processing, wait till all active request complete processing and only then apply changes. Or you can write your code so that every setting is accessed only once. All these remedies are difficult to implement and easy to break. Some of them can also have negative performance impact.

So how immutability can help here? Or better yet how can you change settings if they are immutable?

Here is something I picked up from my F# coding: instead of trying to change the values owned by existing service, create a new instance of service with updated values. Then leave the old instance deal with the existing requests and make sure that the new ones are directed to the new instance. This is a design pattern which certainly can be implemented in any language. Implementing it in F# provides additional safety of ensuring on the language level that the values are read only.

This design pattern can be used for any number of things - not just settings. In NDjango this pattern gave me a way to implement system - wide template caching with no global locks for templates already in the cache. Global locks are only involved when templates are loaded and parsed.

Moving to VS2010 Beta 2

October 21st, 2009

First let me tell you that as much as I liked VS2010 Beta 1 I like the VS2010 Beta 2 much better. I will leave it to other guys to write about the new UI, new features, much improved speed etc., so that in my post I can focus on what is closer to me, VS Integration.

What I want to share with you here is a journal of what I did to port my Django editor from Beta 1 to Beta 2 - I just got it running on Beta2. Let me start with telling you that I did not run into any major issues. Here is a list of what changes in the API affected Django editor and what I did to address the issues:

  • Interface Microsoft.VisualStudio.ApplicationModel.Environments.IEnvironment is gone. As a matter of fact the entire Microsoft.VisualStudio.ApplicationModel namespace is no longer. This interface was used in a number of methods. In Beta 2 all these methods (at least the ones I used) are one parameter short. So the fix was easy - I just changed the signatures.
  • Microsoft.VisualStudio.Language.Intellisense.Completion constructor signature has changed. I need the insert value to be different from the display value, and the only constructor which gives a way to do that now also wants some information about icons. This is nothing too - gave it a null and an empty string to keep it happy
  • The Microsoft.VisualStudio.Text.ContentType.TypeName which used to be "text" now is "plaintext". This is a little more subtle - primarily because the compiler does not catch it, just need to be thorough to change it everywhere. Also I relied on context to skip certain buffers, in particular the buffers created to hold the tooltip text. I had to find a different way to do that.
  • The Microsoft.VisualStudio.Language.Intellisense.IQuickInfoSource.GetToolTipContent now returns ReadOnlyCollection<object> rather than an object - fine, I do not care about returning multiple, just wrapped my string into a collection and returned it
  • A few changes in the F# - namely due to rename rethrow() to reraise() and deprecation of the OverloadID attribute.
  • And finally the biggest change. Both Code Completion and Quick Info used to relay on broker map services, respectively ICompletionBrokerMapService andIQuickInfoBrokerMapService. These services maintained maps of brokers to textViews. No longer. Now Visual Studio maintains a single instance of ICompletionBroker as well as IQuickInfoBroker both to be accessed by directly importing them when necessary. An extra positive from this change was that simplifying the code by throwing away the code related to mapping services showed me the path to streamline the way I work with projection buffers in HTML editor and XML editor.

And this is pretty much it. Of course I had to mess around with project settings, missing F# dlls, changed format for vsix manifest etc., but this is to be expected. Anyway now I have a version of the Django editor for Beta 2.  Right now I have it in the subversion. I will release it in binaries shortly.