Mike Feingold's Adventures in programming

I love F#. Seriously. For many reasons. Working with F# code gives this fresh liberating feeling. It takes code conciseness on a whole new level.

My love to F# does not extend to the F# project system in Visual Studio though - not quiet. Do not get me wrong - it is solid and reliable and all of it. Not without  glitches but what real product is. It is just one particular thing about the F# project system that irks me - it just keeps getting in the way. I am talking about F# project main navigation tool - Solution Explorer tree.

The Problem

Any other project system I know of (C#, VB, etc.) would list the files in alphabetical order. Not F#. You see - with F# compilation order is important, so designers of the F# project system decided to use Solution Explorer to arrange the F# code files in the proper compilation order. So files in the Solution Explorer tree are listed in the order in which they will be compiled, and Solution Explorer provides tools to move files around to control such order. As reasonable as it sounds it creates a few problems. One of them is that if you have more than a few files in your project it becomes more difficult to look your file up. It becomes especially messy if your project includes other file types - html, js, what have you. One can argue that this is a minor point and I can concede that it is - to a degree.

But here is another one - F# project system does not support directories. There is no way to add a directory to your project through GUI. There is a back door way - you can go and edit the project file - manually add <compile include=""> tags with include pointing to a subdirectory. Than a directory entry will appear in the solution explorer tree. But once you do it, controlling the compilation order through this view becomes even more problematic.  Consider the following project directory structure:

There is no way to change compilation order so that Module2 is compiled after Module1 but before Module3. You can edit the build file to change the order of the corresponding <compile include=""> manually and if you run it with MSBuild it will compile but, the project system will not even load it. The reason is that showing this in the solution explorer would take two  Controllers directory nodes with the entry for Module2 right between them. I even found a way to break an F# project with subdirectories through GUI interface - just create several directories and then start adding files through right-click menu on the directories. Everything will seem to be fine, but after you add enough of them an attempt to save and re-open the project will fail with the same message - something to the effect of "cannot create several entries for the same directory".

The bottom line here is that the F# project system severely limits your ability to organize files in the project. It is especially painful if the project includes files of type other than *.fs - i.e. *.htm or *.js.

The Solution

The root of all this evil here is that the Solution Explorer tree is used for two distinct purposes - one is presenting the content of the project file so it can be easily reviewed and organized, and another one  to provide means to manipulate compilation order. If there would be a different screen to control compilation order, the Solution Explorer tree could be reverted to the model used by all other project systems.

This is exactly how I did it in my Bistro designer - I modified the behavior of the solution Explorer so that it follows the standard pattern - first virtual directories, then physical directories, than physical files, everything in alphabetical order - regardless of the order of the corresponding project items. To control compilation order I created a new project property page:

Changing compilation order in the project property does not change how the same files are presented in the Solution Explorer. Also having a separate screen for controlling the compilation order opens other interesting possibilities. One of them is to automatically order the files based on file dependencies provided by the user. This can be more convenient, because usually majority of the files depend on a small number of "common" files.

And if after all the compiler guys will find a way to build dependencies programatically, this property page can be dropped altogether

I am sure you can think of quite a few ways to do it. Here is mine: use the X509 certificate based authentication in a web app. Isn't it simple?

Here is what happened: after through testing we deployed our app to a farm of W2003 servers. It worked really well for a while. In a couple of months one of the servers started to slow down - the response time went up. We even started to see occasional timeouts in the logs. We looked at everything - CPU utilization, memory consumption, Windows handles - nothing to put your finger on. Meanwhile the server fell to its knees - 50% of the requests were timing out. Even to RDP into the server was taking up to 2 minutes. What's interesting - once you are in the server seemed to be pretty responsive, unless you try to start a program - launching notepad was taking up to 1 min. And again once it is up - it works fine. What scared us even more is that other servers in the farm started to deteriorate in the same way.

To solve the mystery we had to take a memory dump from production server. After looking at the dump we started to suspect that the problem has to do with the fact that some of our single sign on users are authenticated using X509 certificates. After some digging we stumbled upon this article . And that's where all the pieces finally fell into their places. Here is what's happening;

• Apparently when validating an x509 certificate windows crypto module creates a temporary file in the Windows temp directory. I do not see any reason why an expense of creating a temp file would be necessary, but as stupid as it is, in itself this is not a problem.
• What is a problem is a bug described in the article I mentioned. As explained in the article, as a result of this bug the temp file(s) created during x509 validation are not deleted. No big deal - just a few 0 length files nobody cares about - right?
• Well, right - at least for some time. The problem is that the files keep piling up. When the number of files in the temp directory is low - below 65000, everything works fine, but when this number is exceeded, as stated in the article "the computer can experience significant delays". In our case it took our servers around 4 months to hit this number - and that is where we started to see these problems.

The solution was simple - just get rid of the files and everything goes back to normal. There is a hotfix available for the bug, but for some reason it is not included in the latest service pack. Because of this we decided against applying the hotfix. Instead we just configured a daemon job cleaning up temp directories overnight.

When you see somebody laying a claim like this, your first reaction should be - this is a hoax, there is no ultimate solution to double checked locking problem. And you will be correct.

Yet I do have an ultimate solution, although it only solves this problem for load/create on demand scenarios. It also involves some cheating. Let me explain myself.

Double Checked Locking - some background

Double checked locking is usually presented as a way to reduce contention around some global object used in a multi-threaded environment. It is pretty common in such systems to have global objects which are updated rarely but accessed very often. A solution to double checked locking problem would provide a way to first check whether the lock is necessary and only apply the lock if it is. When (and if) the lock is applied the usual pattern is used which also involves checking - hence the name 'double checked locking'

There are numerous, sometimes heated, discussions on the topic (as a starting point you can check here) but the bottom line is that all suggested generic solutions either have implicit locks or are not thread safe.

So, what about my claim?

Here is the scenario: Let us say you have a list of resources to be used by multiple threads. If resource creation is expensive you would want to ensure that every resource is created on demand the first time it is requested. So when a thread needs a resource it has to check first whether this resource was requested before and if it is, take the existing one rather than create another copy. The dictionary of the resources is that very object which would benefit from the double checked lock.

The thing is that in this particular case it is possible to safely implement the first check of the double checked lock schema outside of any locks. All you need to do is in addition to keeping a global dictionary make copies of the dictionary - one per processing thread and make every thread perform the check against its own dictionary. Any copy of the dictionary will be only accessed by the thread which owns it, so there is no locks necessary while checking this dictionary.

If the requested resource is located you can take it and use it. If it is not you have to check whether it was created by a different thread or you need to create it. To do that you have to  acquire a lock on the global dictionary, check if the resource is already there, create and add it to the global dictionary if it is not and, finally create a new copy of the global dictionary to replace the copy in your worker thread, the usual stuff - all of this while holding the lock.

This second part is no different from the straightforward implementation of the global resource dictionary. The difference is that if it were the only one, every request for the resource would have to go through a global lock which in the systems with a big number of simultaneous requests can create a severe bottleneck. Adding extra layer with per thread dictionaries remedies this problem - every thread which already has a resource in its local dictionary will access it without any locking.

Let me be explicit about some assumptions here.

• First of all it is implied that all threads share a single copy of the resource. It means that either it has to be immutable or you will have to build some synchronization over access to the resource itself - which takes you back to the square one.
• Secondly the dictionary itself has to be immutable as well so that the same copy can be shared by multiple threads.

If it sounds very F# than that's because it is - this is exactly how NDjango rendering engine is implemented and it is written in F#

There is one more thing I have to admit to. Speaking theoretically my approach does not solve anything - it just pushes the same problem to the process responsible for thread allocation. Think about it - you have a list of available threads and when a new request comes in, a thread has to be allocated to it and removed from the list. It is the same problem. Luckily for us people creating these parts of our software/hardware (like ASP.NET core) are really smart and they solved this problem for us in the most efficient manner, so for all practical purposes it does solve the problem.

Sounds strange - huh? Here is the deal: Imagine you have a service running. It is happily processing incoming requests returning whatever it is supposed to return. Now imagine that in doing so it has to take into account certain global settings. The usual way to implement the settings is to put every value into a dictionary keyed by the setting name. The dictionary is owned by the service and every time some part of the service needs a setting value it pulls it from the dictionary.

In my case the service is the NDjango rendering engine. And to give you an example of a setting: an error string, a string to be inserted into the rendering result every time a django variable fails to resolve.

Now, what if you need to change a setting while the engine is running? The first thing which comes to mind is just go and stick a new value into the dictionary. Well... there are a few complications which may or may not be a real issue depending on what is the nature of the setting and how it is implemented. What if the change hits right in the middle of a request processing? Depending on the setting the impact can be anywhere from inconsistent response all the way through to the service crash. There is a number of ways to prevent such problems. You can clone the settings for every request. You can suspend processing, wait till all active request complete processing and only then apply changes. Or you can write your code so that every setting is accessed only once. All these remedies are difficult to implement and easy to break. Some of them can also have negative performance impact.

So how immutability can help here? Or better yet how can you change settings if they are immutable?

Here is something I picked up from my F# coding: instead of trying to change the values owned by existing service, create a new instance of service with updated values. Then leave the old instance deal with the existing requests and make sure that the new ones are directed to the new instance. This is a design pattern which certainly can be implemented in any language. Implementing it in F# provides additional safety of ensuring on the language level that the values are read only.

This design pattern can be used for any number of things - not just settings. In NDjango this pattern gave me a way to implement system - wide template caching with no global locks for templates already in the cache. Global locks are only involved when templates are loaded and parsed.

First let me tell you that as much as I liked VS2010 Beta 1 I like the VS2010 Beta 2 much better. I will leave it to other guys to write about the new UI, new features, much improved speed etc., so that in my post I can focus on what is closer to me, VS Integration.

What I want to share with you here is a journal of what I did to port my Django editor from Beta 1 to Beta 2 - I just got it running on Beta2. Let me start with telling you that I did not run into any major issues. Here is a list of what changes in the API affected Django editor and what I did to address the issues:

• Interface Microsoft.VisualStudio.ApplicationModel.Environments.IEnvironment is gone. As a matter of fact the entire Microsoft.VisualStudio.ApplicationModel namespace is no longer. This interface was used in a number of methods. In Beta 2 all these methods (at least the ones I used) are one parameter short. So the fix was easy - I just changed the signatures.
• Microsoft.VisualStudio.Language.Intellisense.Completion constructor signature has changed. I need the insert value to be different from the display value, and the only constructor which gives a way to do that now also wants some information about icons. This is nothing too - gave it a null and an empty string to keep it happy
• The Microsoft.VisualStudio.Text.ContentType.TypeName which used to be "text" now is "plaintext". This is a little more subtle - primarily because the compiler does not catch it, just need to be thorough to change it everywhere. Also I relied on context to skip certain buffers, in particular the buffers created to hold the tooltip text. I had to find a different way to do that.
• The Microsoft.VisualStudio.Language.Intellisense.IQuickInfoSource.GetToolTipContent now returns ReadOnlyCollection<object> rather than an object - fine, I do not care about returning multiple, just wrapped my string into a collection and returned it
• A few changes in the F# - namely due to rename rethrow() to reraise() and deprecation of the OverloadID attribute.
• And finally the biggest change. Both Code Completion and Quick Info used to relay on broker map services, respectively ICompletionBrokerMapService andIQuickInfoBrokerMapService. These services maintained maps of brokers to textViews. No longer. Now Visual Studio maintains a single instance of ICompletionBroker as well as IQuickInfoBroker both to be accessed by directly importing them when necessary. An extra positive from this change was that simplifying the code by throwing away the code related to mapping services showed me the path to streamline the way I work with projection buffers in HTML editor and XML editor.

And this is pretty much it. Of course I had to mess around with project settings, missing F# dlls, changed format for vsix manifest etc., but this is to be expected. Anyway now I have a version of the Django editor for Beta 2.  Right now I have it in the subversion. I will release it in binaries shortly.

Yesterday was a big day for us - the day of the first NDjango editor roll out. Check this out:  NDjango Editor. This release is built for .NET 4.0 and Visual Studio 2010 - both in Beta 1 as of now. As they go to Beta 2, RTM etc. we will update the Editor. There are some problems in current release because of certain bugs in VS 2010 editor. I decided against building workarounds for them because according to Brittany Behrens of Microsoft they are fixed in the upcoming VS 2010 Beta 2.

I have some great ideas about where to take the editor and ndjango in general from here, but for now I will just sit and wait for some feedback from the users.

Last week I asked Alex to give the editor a 'real try' and try to break it. Guess what - he did it. All he did was he opened a template of a reasonable size and started typing. He types fast and as he typed the editor slowed to a crawl.

And understandably so. Here is what's happening. Every keystroke is a change to the buffer with the template source code. It triggers the Change event on the buffer, which in turn initiates re-parsing of the template using updated template source. Yes, parsing is happening in a separate thread, and a single parsing request is something the parser can easily handle in the background without noticeable effect on the UI, but for bigger templates it can take some time for parsing to complete and with the typing speed of 10-15 keystrokes per second the requests start to pile up.

Thankfully there is no need to do it anew every time a key is hit - all we need is to parse the last one. Essentially when I type fast I do not really care for the parsing results, but when I pause in my typing I would like to see how good is what I've done so far.

I already discussed the parsing speed issue in the Part 2 Background Parsing of the editor series. I listed there some things which can be done to address the speed issue. I implemented them all from the get go, except for the last one - the parsing requests queuing. And when Alex broke my beautiful editor, I realized that I am not getting away without implementing the queuing too.

Here is what I need: every time a parsing request comes in I schedule its execution for .5 sec in the future. If during the delay another request comes in, instead of creating a new request, I reschedule the active request so it is still set to run .5 sec after the last buffer modification. Once this delay is implemented parser will not run until there is a .5 sec pause in buffer modifications. In other words all requests with less then .5 sec time between them will be combined into one.

And here is the code to do this:

        public NodeProvider(IVsOutputWindowPane djangoDiagnostics, IParser parser, ITextBuffer buffer)
        {
            this.djangoDiagnostics = djangoDiagnostics;
            this.parser = parser;
            this.buffer = buffer;
            filePath = ((ITextDocument)buffer.Properties[typeof(ITextDocument)]).FilePath;
            buffer.Changed += new EventHandler(buffer_Changed);
            // we need to run rebuildNodes on a separate thread. Using timer
            // for this seems to be an overkill, but we need the timer anyway so - why not
            parserTimer =
                new Timer(rebuildNodes, buffer.CurrentSnapshot, 0, Timeout.Infinite);
        }
 
        /// Initiates the delayed parsing in response to the buffer changed event
        private void buffer_Changed(object sender, TextContentChangedEventArgs e)
        {
            // shut down the old one
            parserTimer.Dispose();
 
            // put the call to the rebuildNodes on timer
            parserTimer =  new Timer(rebuildNodes,  e.After,  PARSING_DELAY, Timeout.Infinite);
        }

If you are familiar with Visual Studio Integration SDK you already know that "...Creating a VSPackage is a powerful way to extend Visual Studio...". You probably also know that VSPackages use Services to interact with each other and the core of the Visual Studio. Many components of the Visual Studio core are packages and can be accessed as services. To access a service all you need is an appropriate service provider implementing the IServiceProvider interface. You also need to know the type of the service you want to access. Once you have both - call the GetService method on the provider and it will return the service you desire. All of this is good and handy when you are building your own VSPackage.

What about Visual Studio Editor extensions? What if all you are planning is an editor extension with just one small thing which goes beyond editor?

Guess what - all this wealth of the Visual Studio core services is still available to you without the formal frame of the VSPackage. It is available, that is, if you can get your hands on a service provider which is aware of the service you are after.

In my NDjango designer in addition to syntax colorization, code completion and other wonders of the Visual Studio Editor extensions I wanted to show the errors in both the Error List and a separate pane of the Output Window. There is a variety of ways to do it. 

In NDjango designer it is done in NodeProviderBroker object by accessing the IVsOutputWindow service. Once I have the service I use its CreatePane method to create the Django Output pane (an IVsOutuptWindowPane object). And then, in the NodeSnapshot object I generate the messages using the OutpuTaskItemString method. You can find examples of how to use these services on the web and in the Visual Studio SDK, as well as in the source code of the NDjango designer

What is somewhat less obvious is how do you get to the starting point - the IVsOutputWindow service. It is promised that in the future it will be as easy as importing the ServiceProvider, and then calling the GetService method on the service provider. We are not there yet, so here is what I did to get by for now:

        private object GetService(ITextBuffer textBuffer, Type type)
        {
            var vsBuffer = adaptersFactory.GetBufferAdapter(textBuffer);
            if (vsBuffer == null)
                return null;
 
            Guid guidServiceProvider = VSConstants.IID_IUnknown;
            IObjectWithSite objectWithSite = vsBuffer as IObjectWithSite;
            IntPtr ptrServiceProvider = IntPtr.Zero;
            objectWithSite.GetSite(ref guidServiceProvider, out ptrServiceProvider);
            Microsoft.VisualStudio.OLE.Interop.IServiceProvider serviceProvider =
                (Microsoft.VisualStudio.OLE.Interop.IServiceProvider)Marshal.GetObjectForIUnknown(ptrServiceProvider);
 
            Guid guidService = typeof(SVsOutputWindow).GUID;
            Guid guidInterface = typeof(IVsOutputWindow).GUID;
            IntPtr ptrObject = IntPtr.Zero;
 
            int hr = serviceProvider.QueryService(ref guidService, ref guidInterface, out ptrObject);
            if (ErrorHandler.Failed(hr) || ptrObject == IntPtr.Zero)
                return null;
 
            IVsOutputWindow taskList = (IVsOutputWindow)Marshal.GetObjectForIUnknown(ptrObject);
            Marshal.Release(ptrObject);
 
            return taskList;
        }

This code is based on the code posted by Noah Richards and it uses the text buffer (an object implementing IVsTextBuffer interface) as a gateway to access Visual Studio global services. Works for me.

Look here for complete source

Ok. This post is the last and, I hope, the shortest and the simplest one in the Django Editor series. All what's left to cover is the source for the code completion part of the NDjango editor.

To continue line of reasoning from the previous post, the code completion source works in very much the same way as the quick info (see posts 4 and 5 of the series). The source provider is registered as an MEF component and creates source objects for text buffers as necessary. Here is the code:

    [Export(typeof(ICompletionSourceProvider))]
    [Name("NDjango Completion Source")]
    [Order(Before = "default")]
    [ContentType(Constants.NDJANGO)]
    internal class SourceProvider : ICompletionSourceProvider
    {
        [Import]
        internal INodeProviderBroker nodeProviderBroker { get; set; }
 
        public ICompletionSource TryCreateCompletionSource(ITextBuffer textBuffer, IEnvironment environment)
        {
            if (nodeProviderBroker.IsNDjango(textBuffer, environment))
                return new Source();
            return null;
        }
    }

And the source itself is responsible for providing the code completion data:

    internal class Source : ICompletionSource
    {
        public ReadOnlyCollection GetCompletionInformation(ICompletionSession session)
        {
            List completionNodes = session.Properties[typeof(Source)] as List;
            if (completionNodes != null)
            {
                // Calculate the location of the textspan to be replaced with
                // the selection. We always want to replace the entire word
                ITextSnapshot snapshot = session.SubjectBuffer.CurrentSnapshot;
                int triggerPoint = session.TriggerPoint.GetPosition(snapshot);
                ITextSnapshotLine line = snapshot.GetLineFromPosition(triggerPoint);
                string lineString = line.GetText();
                // position of the first non-space character before the tag name
                int start = lineString.Substring(0, triggerPoint - line.Start.Position).
                    LastIndexOfAny(new char[] {' ', '\t', '%'})
                    + line.Start.Position + 1;
                // length of the word currently in the tag name position in the tag
                int length = lineString.Substring(triggerPoint - line.Start.Position).
                    IndexOfAny(new char[] {' ', '\t', '%'} )
                    + triggerPoint - start;
 
                CompletionSet completionSet = new CompletionSet(
                    "ndjango.completions",
                    session.SubjectBuffer.CurrentSnapshot.CreateTrackingSpan(
                    start, length, SpanTrackingMode.EdgeInclusive),
                    CompletionsForNodes(completionNodes),
                    null);
                return new ReadOnlyCollection(new List { completionSet });
            }
 
            return null;
        }
 
        private IEnumerable CompletionsForNodes(IEnumerable nodes)
        {
            foreach (INode node in nodes)
                foreach (string value in node.Values)
                    yield return new Completion(value, value, value);
        }
    }

Even though it is pretty easy to create a custom code completion presenter, I decided against creating my own presenter. And the only thing you need to do to make the standard one work is to feed it with a list of code completion sets in return value of the GetCompletionInformation method. One of the parameters you have to provide when creating your completion set is the tracking span. This tracking span determines what text will be replaced by the value selected by the user from the list of available values.

The rest of the voodoo in the code shown above has to do with calculation of this tracking span as well as building the list of values to show.

And this concludes our discussion of building django editor for Visual Studio 2010. As of the time of this writing I am getting ready to roll out a beta version of the django designer.

Looking back at as the project comes to completion, I can safely say that as planned the project consisted of two parts - extending the django parser for more detailed diagnostics and building the Visual Studio Editor capable of consuming the information. Of course, during the course of the project there were some design challenges. To my pleasant surprise almost all of them were related to the django parser rather than the actual editor code.

When I just started the project after the first look at the VSEditor I was saying that building custom editors in Visual Studio 2010 is a breeze. Now, when the project is at its completion, I still stand by my statement.

P.S. The project is open source, so feel free to browse the source code in the subversion repository. Currently the source code is in the development branch called VS2010Designer. The project is still in development, so there will be changes from the code published in the log. If you are want to look at the code matching exactly the code published in the posts look at the tag I created for this purpose. The tag name is Designer_blog_version. I hope you will find it helpful

Look here for complete source

Code completion is the final piece of the NDjango designer. With code completion out of the way, the NDjango template editor all of us care so much about will be complete. OK. Let us get it over with.

The overall structure of the code completion code is very similar to the QuickInfo code I covered in two previous postings. As was the case with the QuickInfo, there are two classes, Controller and ControllerProvider controlling the session - in this case the Completion session and two more classes Source and SourceProvider providing the data to be displayed. As with the QuickInfo the Controller class instances are created by the Controller provider for all relevant TextViews and once attached to the buffers, Controllers create and dismiss the sessions.

Here is the code of my code completion controller provider:

    [Export(typeof(IIntellisenseControllerProvider))]
    [Name("NDjango Completion Controller")]
    [Order(Before = "Default Completion Controller")]
    [ContentType(Constants.NDJANGO)]
    internal class ControllerProvider : IIntellisenseControllerProvider
    {
        [Import]
        internal ICompletionBrokerMapService CompletionBrokerMapService { get; set; }
 
        [Import]
        internal INodeProviderBroker nodeProviderBroker { get; set; }
 
        [Import]
        internal IVsEditorAdaptersFactoryService adaptersFactory { get; set; }
 
        public IIntellisenseController TryCreateIntellisenseController(ITextView textView, IList subjectBuffers, IEnvironment context)
        {
            bool brokerCreated = false;
            foreach (ITextBuffer subjectBuffer in subjectBuffers)
                if (nodeProviderBroker.IsNDjango(subjectBuffer, context))
                    brokerCreated |= (CompletionBrokerMapService.GetBrokerForTextView(textView, subjectBuffer) != null);
 
            if (brokerCreated)
                return new Controller(this, subjectBuffers, textView, context);
 
            return null;
        }
    }

Nothing really new or exciting here. The controller itself is somewhat longer. I still decided to include the complete source code for the class, but you are welcome to scroll all the way through the code to the notes down below:

    class Controller : IIntellisenseController, IOleCommandTarget
    {
        private IList subjectBuffers;
        private ITextView subjectTextView;
        private IWpfTextView WpfTextView;
        private ICompletionSession activeSession;
        private IEnvironment context;
        private ControllerProvider provider;
 
        public Controller(ControllerProvider provider, IList subjectBuffers, ITextView subjectTextView, IEnvironment context)
        {
            this.provider = provider;
            this.subjectBuffers = subjectBuffers;
            this.subjectTextView = subjectTextView;
            this.context = context;
 
            WpfTextView = subjectTextView as IWpfTextView;
            if (WpfTextView != null)
            {
                WpfTextView.VisualElement.KeyDown += new System.Windows.Input.KeyEventHandler(VisualElement_KeyDown);
                WpfTextView.VisualElement.KeyUp += new System.Windows.Input.KeyEventHandler(VisualElement_KeyUp);
            }
        }
 
        void VisualElement_KeyUp(object sender, System.Windows.Input.KeyEventArgs e)
        {
            if (activeSession != null)
            {
                if (e.Key == Key.Escape)
                {
                    activeSession.Dismiss();
                    e.Handled = true;
                }
 
                if (e.Key == Key.Enter)
                {
                    if (this.activeSession.SelectedCompletionSet.SelectionStatus != null )
                        activeSession.Commit();
                    else
                        activeSession.Dismiss();
                    e.Handled = true;
                }
            }
        }
 
        void VisualElement_KeyDown(object sender, System.Windows.Input.KeyEventArgs e)
        {
            ITextView textView = sender as ITextView;
            if (this.subjectTextView != textView)
                return;  // Make sure that this event happened on the same text view to which we're attached.
 
            if (!(e.Key >= Key.A && e.Key <= Key.Z))
               return;   // we only start the session when an alphanumeric key is pressed          
 
            if (activeSession != null)
                 return;  // if there is a session already leave it be
 
             // determine which subject buffer is affected by looking at the caret position
             SnapshotPoint? caretPoint = textView.Caret.Position.Point.GetPoint
                 (textBuffer =>
                    (
                        subjectBuffers.Contains(textBuffer)
                        && provider.nodeProviderBroker.IsNDjango(textBuffer, context)
                        && provider.CompletionBrokerMapService.GetBrokerForTextView(textView, textBuffer) != null
                    ),  PositionAffinity.Predecessor);
 
            if (!caretPoint.HasValue)
                return;   // return if no suitable buffer found
 
            List completionNodes =
                provider.nodeProviderBroker.GetNodeProvider(caretPoint.Value.Snapshot.TextBuffer).GetNodes(caretPoint.Value)
                    .FindAll(node => node.Values.Count() > 0);
 
            if (completionNodes.Count == 0)
                return;  // return if there is no information to show
 
            attachKeyboardFilter();   // attach filter to intercept the Enter key
 
            ICompletionBroker broker = provider.CompletionBrokerMapService.GetBrokerForTextView
                  (textView,caretPoint.Value.Snapshot.TextBuffer);
            ITrackingPoint triggerPoint = caretPoint.Value.Snapshot.CreateTrackingPoint(caretPoint.Value.Position, PointTrackingMode.Positive);
            activeSession = broker.CreateCompletionSession(triggerPoint, true);
            activeSession.Properties.AddProperty(typeof(Source), completionNodes);
            activeSession.Dismissed += new System.EventHandler(OnActiveSessionDismissed);
            activeSession.Committed += new System.EventHandler(OnActiveSessionCommitted);
            activeSession.Start();
        }
 
        void OnActiveSessionDismissed(object sender, System.EventArgs e)
        {
            detachKeyboardFilter();
            activeSession = null;
        }
 
        void OnActiveSessionCommitted(object sender, System.EventArgs e)
        {
            detachKeyboardFilter();
            activeSession = null;
        }
 
        public void ConnectSubjectBuffer(ITextBuffer subjectBuffer) { }
 
        public void Detach(Microsoft.VisualStudio.Text.Editor.ITextView textView)
        {
            detachKeyboardFilter();
        }
 
        public void DisconnectSubjectBuffer(ITextBuffer subjectBuffer)
        {
            WpfTextView = subjectTextView as IWpfTextView;
            if (WpfTextView != null)
            {
                WpfTextView.VisualElement.KeyDown -= new System.Windows.Input.KeyEventHandler(VisualElement_KeyDown);
                WpfTextView.VisualElement.KeyUp -= new System.Windows.Input.KeyEventHandler(VisualElement_KeyUp);
                detachKeyboardFilter();
            }
        }
 
        private void attachKeyboardFilter()
        {
            ErrorHandler.ThrowOnFailure(provider.adaptersFactory.GetViewAdapter(subjectTextView).AddCommandFilter(this, out oldFilter));
        }
 
        private void detachKeyboardFilter()
        {
            ErrorHandler.ThrowOnFailure(provider.adaptersFactory.GetViewAdapter(subjectTextView).RemoveCommandFilter(this));
        }
 
        // The code below intercepts the ECMD_RETURN command before it is sent to the editor window.
        private IOleCommandTarget oldFilter;
 
        private static readonly Guid CMDSETID_StandardCommandSet2k = new Guid("1496a755-94de-11d0-8c3f-00c04fc2aae2");
        private static readonly uint ECMD_RETURN = 3;
 
        public int Exec(ref Guid pguidCmdGroup, uint nCmdID, uint nCmdexecopt, IntPtr pvaIn, IntPtr pvaOut)
        {
            if (pguidCmdGroup == CMDSETID_StandardCommandSet2k && nCmdID == ECMD_RETURN)
                return VSConstants.S_OK;
            return oldFilter.Exec(pguidCmdGroup, nCmdID, nCmdexecopt, pvaIn, pvaOut);
        }
 
        public int QueryStatus(ref Guid pguidCmdGroup, uint cCmds, OLECMD[] prgCmds, IntPtr pCmdText)
        {
            return oldFilter.QueryStatus(pguidCmdGroup, cCmds, prgCmds, pCmdText);
        }
    }

With this one even though the overall approach is the same as with the Quick Info, there are a few things deserving some explanation.

First the similarities: As with the Quick Info when the controller is created it hooks itself up to the view. It does so by subscribing to appropriate events on the view and in the event handlers it supplies the Intellisense sessions are created, started and dismissed as necessary. Also the information to be used by the source is attached to the session using session.Properties.

Now the differences: First of all the events here are different. In QuickInfo we relied on the OnMouseHover event to fire up the session and the session dismissal was handled automatically based on the mouse moves. With the Code completion we have to relay on the keyboard events for this purpose. See the implementation of the KeyUp and KeyDown events for more details.

The second and a little more unsettling difference is the games we have to play with the standard editor window. The problem is that when the user presses Enter, this keypress in addition to being sent to our KeyUp event is also sent to the editor window itself. As a result, the window gets updated which causes the update events to fire which in turn causes the current selection in the CompletionSet to be reset according to the matching rules. In other words, if the user selects something from the completion dropdown using arrows and presses enter, before the selection is applied to the code it is reset to whatever is considered to be a match to the letters keyed in before that. From the user standpoint, it causes the code completion code to ignore the selection with the arrow keys.

The attachKeyBoardFilter method above shows a way to suppress passing KeyUp event to the standard window. Also take a note of the detachKeyBoardFilter method, implementing the cleanup to be performed when the code completion session is completed one way or another.

And this is all I have to say about code completion controller. Bear with me - the only thing left is the code completion source, which is coming up next.