The creativity and hard work that has gone into our journey during this time has been inspiring and fruitful.

Three things stand out as our biggest endeavors over the last few months:

• Advisors
• Alpha
• (The search for) Investors

So on the first topic of advisors, it is my pleasure to announce that we’ve recently completed the assembly of an international board of advisors consisting of top-tier consumer tech and game industry experts and luminaries.

They are:

• Wim Stocks – Former President/COO, Elephant Entertainment; Executive VP, Atari; Executive VP, GT Interactive
• Robert Stephens – Founder, The Geek Squad; and recently named CTO, Best Buy
• Maxime Julien – Former COO, Electronic Arts; VP Operations, Ubisoft
• Jim Rossignol – Author, This Gaming Life; founding blogger, Rock, Paper, Shotgun; writer for PC Gamer, Wired, and Edge
• [Name Not Currently Publicly Available] – Award-winning game company Co-Founder/Creative Director

On behalf of Echobit, I’d like to welcome Wim, Robert, Maxime, Jim, and You-Know-Who-You-Are to the team. Your experience and talents will bring many valuable perspectives on the gaming landscape in the weeks and months and years ahead. We are thrilled to have you working with us to make Evolve the best online gaming platform in the world.

Then yesterday, something changed. While I was out in the Spring sunshine on a 10-mile run, it happened:  I found myself  ”in the zone”.

Suddenly and unexpectedly, running started to feel good. No longer was I even thinking about it. No longer was I trying to convince my body to keep moving. It was easier. And not only was it easier, I was going faster.

So too, has been our experience at Echobit. The last few months have been challenging. And even though it hasn’t always felt like it, we’ve been quietly and steadily making important progress…

Since the beginning of this year:

• We’ve closed a small seed round of capital
• We’ve successfully applied for and been granted an H1B visa for our Co-Founder, Soren Dreijer
• Our engineering team has come onboard full-time
• We’ve updated our company and product websites; and
• We’ve put out three compelling releases of our VPN platform, LAN Bridger (in less than a month)

But perhaps what’s most exciting for us is how Echobit has found itself in a zone of its own. Our vision has never been clearer. Our updated plan is presenting us with an even more exciting path. And finally, our work on Evolve is starting to get easier—and gain momentum.

Q2 promises to be very satisfying, as we get into better and better shape. Goals over the next three months include more fundraising, another release of LAN Bridger, providing a public peek at Evolve, plus a whole lot more behind the scenes.

So here’s to a great Spring. And thanks for joining us on our journey.

If you’re like me, you still want to be able to run debug builds on your test machines. As always, you need the Debug CRT for that, which I’m sure you know isn’t part of the redistributable package. Microsoft has documented two ways of installing a debug CRT on test machines:

http://msdn.microsoft.com/en-us/library/aa985618%28VS.80%29.aspx

I personally use the second approach since I know exactly which files get copied to the target machine and it’s just less intrusive in general.

When you apply the security patch on your dev machine, the various libraries that you link against should’ve been updated as well. (You can always check the final manifest file in your build folder to see which version of the CRT you’re building against.) I usually just grab the latest debug DLLs from the Visual Studio folder at:

<Path to VS folder>\VC\redist\Debug_NonRedist\x86\Microsoft.VC90.DebugCRT
<Path to VS folder>\VC\redist\Debug_NonRedist\x64\Microsoft.VC90.DebugCRT

Just copy the DLLs and the accompanying .manifest file to the target folder.

You might have noticed that the security update has overwritten the existing DLLs in those folders. This means that if you want to debug an older build of your product, you’ll have to manually pull out the DLLs and the corresponding .manifest file from the WinSxs folder. That’s not terribly hard to do (albeit annoying), and I’ll describe that in a future blog post.

We entered the MN Cup 2009 earlier this year. It’s a competition for startup companies and their breakthrough ideas. The semifinalists were announced last week and we made it into the next round. We’re extremely excited to be one of the 49 companies that were selected from over a 1000 submissions.

More information to follow as we progress through the competition.

I always listen to music when working. At work, that’s not a problem since I’ve got all my music stored on my work computer. However, when on the road all my music is stored on my laptop. This becomes a problem since my laptop’s music player cannot be accessed without first minimizing the fullscreen remote desktop window.

Actually, there are two problems here:

1. There is no keyboard shortcut for minimizing a fullscreen remote desktop session. You have to use the mouse cursor and click on the minimize button on the “banner” at the top.
2. The media keys (play, pause, stop, etc.) that are present on so many laptops today are forwarded to the remote computer rather than to the music player on the local computer.

These problems become a real nuisance after a while when working primarily in a fullscreen remote desktop window, and so I decided to investigate if these problems could be addressed.

Terminal Services allow you to register add-ons that are loaded by mstsc.exe when you connect to a remote desktop server. Through a mechanism called virtual channels, the server- and client-side of the remote desktop connection can communicate with each other.

This mechanism is perfect for addressing the problems I identified above. For instance, when a hotkey is pressed in a fullscreen remote desktop session, the server requests that the client minimizes the remote desktop window, and when a media key is pressed it’s forwarded to the client, which injects the key press on the client computer such that it’s routed to e.g. the local music player without requiring the user to minimize the remote desktop window.

The add-on consists of three components described below. The client- and server-side components must be explicitly registered with Terminal Services at the client and server computers, respectively. You must be running with Administrator privileges when registering them.

Server-side component:

This component is responsible for intercepting the user input and forwarding it to the client computer. It does so by registering several hotkeys.

To register this component with Terminal Services, run the following command from Start->Run:

<path to component>\RdpServerControl.exe install

To unregister this component with Terminal Services, run the following command from Start->Run:

<path to component>\RdpServerControl.exe uninstall

The server-side component must be launched every time a new remote desktop session is initiated. You can do this manually by simply running the executable when you’ve connected to the server, or by having it start automatically when a user logs on. The component exits automatically when the user disconnects from the Terminal Services session.

Client-side component:

This component is responsible for executing the commands requested by the server-side component.

To register this component with Terminal Services, run the following command from Start->Run:

regsvr32 <path to component>\RdpClientControl.dll

To unregister this component with Terminal Services, run the following command from Start->Run:

regsvr32 /u <path to component>\RdpClientControl.dll

Configuration component:

This is just a small tool for configuring the server-side component. For instance, you can specify the keyboard shortcut for minimizing the fullscreen remote desktop window. It’s not required for using the add-on, which uses the default settings described below.

This component must be run on the server.

The default configuration is as follows:

• Alt+F1: Minimize fullscreen remote desktop window.
• Media keys are being forwarded by default.

Download

If you’d like to try out the add-on for yourself, you can get it here: x86 or x64. It requires the Visual C++ 2008 SP1 redistributable: x86 or x64.

Disclaimer:

Please note that it’s a very simple tool that was created for a very specific purpose, and the feature set is thereafter. If you experience any problems, drop me a line and I’d be more than happy to help you get it to work.

What cracks me up about this is that when she gets back to her own workstation, it’s not even locked! If you’re worried that people might be snooping around and you know for a fact that your account has unrestricted network access, why the hell do you leave your computer unlocked when you go somewhere.

Just a random note

Shortly after I had created my account, it turned out that I’d given them an incorrect mailing address because the agent at my apartment’s leasing office had accidentally written the wrong street number on the leasing papers (well done…). Even though only the last digit was wrong, and the street number one I’d been given didn’t even exist, it meant I had to get in touch with my bank, insurance company, etc., to make sure they all had my correct address.

Unfortunately, changing the mailing address with my bank turned out to be more difficult than expected.

They have a really nice home banking portal and one of the things they allow you to do is update your current address. So, I went ahead and filled out the form and it told me that I should allow 24 hours for the changes to take effect. Upon receiving an e-mail notification from the bank shortly after stating that I’d made changes to my account, I believed my address had been changed successfully (or at least was about to), and I happily forgot about the issue.

Later that same month, I received my monthly statement from the bank in the mail. The address on the letter hadn’t been corrected and was still pointing to my non-existing address, however. (I guess USPS is clever enough to figure out where I live because it’s such a slight error.) I tried calling the bank about the issue, but every time the automatic phone system attempted to forward me to a customer representative the call was dropped.

Being lazy, I refused to give in and go to the local bank office to get the issue fixed, so instead I looked around the home banking site some more. I was pleased to discover that it had a nice “Chat with a customer representative now” feature. Upon activating the feature, a chat window appeared and shortly after a representative connected to the session and asked me in the usual overly polite way what he might assist me with.

I explained the situation, which he was very happy to assist me with. He just needed to verify my identity before he went ahead and updated my mailing address. Fair enough. I patiently answered all of his questions about the number of accounts I had, where I lived (by providing him with my incorrect address), and so on. Eventually he went ahead and updated the address and told me that the change could take up to 24 hours to take effect.

I was happy. I’d found the online chat feature quite convenient as it had saved me a trip to the bank. Even better, though, this time the change had to be effective since it’d been done by a human.

Wrong!

A couple of weeks later I noticed by coincidence that the address listed on my home banking portal was still pointing to the incorrect one. Despite numerous appeals from my friends to just go to the bank to get it over with, I decided to give the customer representative chat feature a last chance.

This time around I made sure to let the representative know that his colleague had failed to do this correctly last time around and that I was running low on patience. He assured me that he would make the change successfully. I just had to answer a few security questions first.

It occurred to me that if I had successfully logged in to my home banking account through the multiple layers of security offered by the bank, why did I have to go through yet another process of verifying my identity by answering a bunch of security questions? A friend of mine suggested that maybe the customer representative didn’t have any information about me when I connected to the chat session. I doubted that, so I asked him:

You: I have a question too. When I joined the chat, does it say who I am or are you relying on whatever I tell you?

Nicholas: Yes, Soren. We have all the account information with us.

You: so why do you have to ask me all these questions? I mean, I’m logged into home banking so you already know who I am since I logged in

Nicholas: I understand that these questions are annoying; however, these are for the security of your accounts.

Nicholas: As your account security is of prime importance to us.

You: I know, I’m just failing to see what extra security it gives since I wouldn’t be chatting with you if I hadn’t already proven my identity when logging into home banking

Nicholas: I completely understand your concern in this matter.

Interesting. Apparently, the bank believes that if they verify my identity multiple times, their online site will be more secure. Honestly, I think this is most likely an issue with the computer system that the bank uses internally, which doesn’t distinguish between assisting customers phoning in or using an online feature. Either way, it just makes the whole customer experience worse since it means we’ll have to go through multiple hoops to do one thing. It also means customer representatives will have to spend more time assisting people.

Added bonus

As an added bonus, I’ve included the customer representative’s last few sentences before I left the chat. I think it really illustrates how overly polite these guys are (or are required to be). To me, it just seems superficial and unnecessary since I know he doesn’t mean it and it just sounds plain stupid:

Nicholas: I reassure you that your concern is very important to us, and will be resolved with utmost priority.

Nicholas: You are most welcome.

Nicholas: Thank you very much for giving us an opportunity to assist you.

Nicholas: It was a pleasure assisting an esteemed customer like you today.

This is a pretty simple and straightforward scheme that works quite well. With this post, however, I wanted to highlight how commonly used this approach has become that the stores (and their employees) seem to have forgotten why it was created in the first place.

My brother and I happened to end up in a store that used this exact approach. After having picked out the clothes we wanted to try on, we went to the fitting room area where we were met by a huge line of people waiting for their turn. When we finally got to the front of the line, it turned out that there were actually plenty of fitting rooms available but there was only one sales assistant around who could hand out badges. Everybody therefore had to wait for him to go through the line, one by one, and showing each customer to an available fitting room.

Unfortunately, he was so busy counting the number of items for the people entering the fitting rooms that he barely had time to look at the people coming out and instead they just dumped the badges on the nearest table and left. The sales assistant didn’t seem to care at all.

When my brother and I came out of the fitting room, the sales assistant was nowhere to be found. Other people came out too and looked similarly confused. Just like the people before us, we simply dumped our badges and the clothes we didn’t want to buy and left.

What annoyed me the most was that we’d spent more than 10 minutes in line for no reason at all. There was absolutely no point whatsoever in having the sales assistant hand out badges since he never checked them when people came back out.

In my opinion, this is what happens when security gets in the way of things. It’s understandable that the store wants to avoid shoplifting, but if they don’t follow through on their security measures then it just becomes a customer annoyance.

I hate wasting my time…

Most people probably wouldn’t have noticed, but as a security-conscious IT professional I immediately saw that the registration page wasn’t encrypted with SSL. This, in my opinion, is particularly bad practice for a government-controlled website that expects its users to enter confidential information — and we’re not “just” talking credit card information here.

Since I had to complete the form, I reluctantly filled out the remaining fields and hit Submit. I was redirected to a confirmation page, which told me that a confirmation e-mail had be been sent to me to verify the e-mail address I had entered.

Fair enough. That’s standard practice these days.

A couple of minutes later the confirmation e-mail arrived. I was horrified to learn, however, that all the information I had entered on the registration page had been reprinted in the e-mail — even my passport information.

That did it. I immediately fired off an e-mail to the Consulate trying to voice my concerns about the security of the site. I was fortunate enough to have a contact at the Consulate from a previous correspondence, and when I told her about my experiences she was kind enough to forward my e-mail to the person responsible.

I received a response within an hour (what a pleasant surprise) and it turns out the site was supposed to be SSL encrypted, but for some reason the main page was linking to the wrong version of the page. This just illustrates how easily things can go wrong, even if it was done with the best intentions.

The confirmation e-mail was deliberate, though, and the government official assured me that they’d address the (obvious) security issue in an upcoming large-scale redesign of the site in January.

I’m very pleased that the Consulate responded so quickly to my concerns. I think it happens way too often that sites remain broken and unsafe for long periods of time even though the security holes are known to the maintainers.

Kudos to the Consulate!

When a 16-bit application is launched on 32-bit Windows, NTVDM is used as a proxy application in order to launch the original application. NTVDM provides a complete virtual 8086 mode environment for the 16-bit application to run in. (In fact, all the proxied applications share a dedicated thread in NTVDM.) Since these applications are hosted internally by NTVDM, they only show up in Task Manager if the user has enabled the “Options->Show 16-bit tasks” menu option.

As can be seen in the screenshot below, two 16-bit applications (wowexec.exe and rdo001gl.exe) are hosted by ntvdm.exe on my computer. Wowexec.exe works together with ntvdm.exe to provide a 16-bit environment.

If you use Process Explorer from Sysinternals, these 16-bit processes won’t show up in the process list because they’re not considered “real” processes on a 32-bit operating system. Personally, though, I find it quite useful that I can view all the processes running on my system whether they’re 16-bit or 32-bit. It’s sort of weird if an application’s window is present in the taskbar but a corresponding process cannot be found in the process list.

So, how does Task Manager go about showing these 16-bit processes? It uses something called the Virtual DOS Manager Debug library (VDMDBG) (part of the Windows SDK), which lets you access 16-bit process information on a 32-bit operating system. For instance, VDMDBG lets you enumerate all VDMs currently running 16-bit processes (or tasks, as they’re referred to internally), or all the tasks running in a particular VDM.

Two functions are central in updating the process list view in taskmgr.exe: CProcPage::UpdateProcInfoArray and CProcPage::UpdateProcListview. The first function obtains a listing of all the processes currently running on the system by calling ntdll!ZwQuerySystemInformation and steps through each one and adds it to an internal array. The function also extracts various information about the process (image name, CPU time, etc.) and calls CProcInfo::SetData to set it internally. CProcPage::UpdateProcListview, on the other hand, is responsible for updating the GUI by tapping into the aforementioned internal process info array.

The CProcInfo::SetData function is particularly interesting because it checks to see if the current process is ntvdm.exe:

push    offset aNtvdm_exe ; "ntvdm.exe"
push    eax             ; wchar_t *
call    ds:__imp___wcsicmp
test    eax, eax
pop     ecx
pop     ecx
jnz     loc_100CA37

If it is, CProcInfo::SetData calls VDMDBG!VDMEnumTaskWOWEx to obtain information about the 16-bit processes currently being hosted by ntvdm.exe. The second parameter to the function is a pointer to a callback function, which is set to CProcPage::WowTaskCallback.

In the screenshot above of Task Manager, ntvdm.exe hosted two 16-bit applications, wowexec.exe and rdo001gl.exe. On my computer, we therefore expect CProcPage::WowTaskCallback to be called twice, once for each task. To verify, we can set a breakpoint in the function and take a look at the fourth and fifth parameters passed to it:

0:000> da poi(ebp+14)
001ae6f8  "RDO001GL"
0:000> da poi(ebp+18)
001ae701  "C:\PROGRA~1\BC31\BOOK\RDO001GL.E"
001ae721  "XE"

CProcPage::WowTaskCallback calls CProcPage::SetDataWowTask to obtain information about the process, and to add it to the internal process info array alongside the 32-bit processes. However, to disinguish the two types of processes (16-bit and 32-bit), Task Manager displays the 16-bit processes as sub-processes of the ntvdm.exe process by indenting them in the process list.

That’s all there is to it.

It had my name on it and it was even spelled with the Danish o-with-a-slash (ø) character (see picture below).

My first thought was that Microsoft certainly went all out to impress their conference attendees by individually customizing each bag, but it quickly turned out that both Ken and Steve had the same name on their bags. Little did I know that it was merely a coincidence (rather than a gesture) that the company, which made the bags, was simply named similarly to me.

It still made my day, though. What are the odds…

To start things off, consider the following code snippet:

void SomeFunction()
{
    for (int i = 0; i < 1; ++i);

    for (int i = 0; i < 1; ++i);
}

The function above contains two for-loops that do absolutely nothing useful (unless you consider looping useful). What makes these two for-statements interesting, though, is how they both use an iterator variable named i. According to the C standard (C98), i is local to the for-statement in which it is defined. That is, code outside the scope of the for-statement cannot access i.

Older Microsoft C++ compilers (pre-Visual Studio 2005) didn’t follow the standard and instead considered i to be valid outside the scope of the for-loop. Fortunately, Microsoft decided to make their newer versions of the compiler significantly more standards-compliant, and one of the things they fixed was the scope of for-loops. As a result, the two iterator variables (i) in the code snippet above are therefore considered to be separate entities.

However, it looks like they forgot to update the Visual Studio debugger to reflect this change. The screen shot of the Locals window below illustrates what I’m talking about:

For some reason, the variable i is shown twice in the list when execution hits the second loop, even though it has gone out of scope in the first loop and is no longer valid. The disassembly of the code reveals that i has been created as a local variable on the stack (a separate stack location exists for each loop variable). The compiler could easily have used the same local stack location for both of these loop variables, of course, since they’re never used at the same time, but for this post we’re only considering unoptimized debug builds.

The Visual Studio debugger and WinDbg differ somewhat in how they show these stale variables. The first simply shows all the variables in the Locals window as in the previous screenshot, which makes it really hard to distinguish them from each other. WinDbg, on the other hand, shows the variables that have gone out of scope as <Eclipsed>:

Executing “dv i” in WinDbg yields the following output:

0:000> dv i
    i = 1
    i = 1

This tells us that the debugger is very much aware of the presence of both of the iterator variables even though neither is valid anymore.

If we dig further, it turns out that variables declared in the body of a single-line for-loop also show up in the debugger once they have gone out of scope. The following piece of code illustrates that:

for (int i = 0; i < 1; ++i)
    int foo = 0;

for (int i = 0; i < 1; ++i)
    int foo = 0;

int i = 1;

// When we get here, we have three i's and two foo's in the Locals window

Contrast this to the behavior of e.g. while-loops where variables declared in the body of the loop are correctly removed from the Locals window once they go out of scope.

Interestingly, the issue seems to only manifest itself whenever scope brackets, { and }, aren’t used. Consider the following example:

for (int i = 0; i < 1; ++i)
{
    int foo = 0;
}

for (int i = 0; i < 1; ++i)
{
    int foo = 0;
}

{
    int bar = 1;
}

// When we get here, only the two i's are present in the Locals window

In this example, there is always only one foo or one bar shown in the debugger, and they disappear from the Locals window whenever the variable goes out of scope. As soon as the scope brackets are removed, though, we get the aforementioned behavior.

It’s worth mentioning that the assembly code of a for-loop with and without the scope brackets is, of course, completely identical.

This lingering local variable issue is most likely a bug in the debugger since the for-loops should essentially work just like while- and do-while-loops. It’s probably a relic from the days where the Visual C++ compiler didn’t conform as much to the C++ standard as it does now and variables declared in for-loops didn’t just have local scope.

Implications

At first, this behavior seems quite harmless. However, take a look at what happens in the following case:

int n = /* someArbitraryNumber */;

for (int i = 0; i < n; ++i);

int m = /* someOtherArbitraryNumber */;

for (int i = 0; i < m; ++i);

// .. A few more loops using i ..

int i = /* Complex calculation */;

...

// When looking at i in the Locals window at this point, it's difficult to determine exactly
// which one is used in the calculation below since multiple exist.
int result = i + /* some other variables */;

In the example above, the variable i is used multiple times. When you get to calculating the result (the last line), how do you know which i to look at in the debugger? If you’re lucky enough to step over the complex calculation in the debugger, the correct variable will most likely be highlighted in red by Visual Studio, but you could just as well have set a breakpoint further down the code path or have broken into the debugger due to an exception.

It’s fairly easy to get past this issue simply by looking at the actual instruction in disassembly mode to figure out the stack location that’s being referred to, or by hovering the mouse over the variable in the source file, but it seems to me that this is unnecessary work for something that the debugger should be able to tell you right away in the Locals window. After all, the previous instances of i have gone out of scope and are no longer valid, so why show them?

Test Script

I’ve uploaded a simple test script if you want to test out the behavior yourself: loop.cpp.

A couple of days ago, I hit a problem when testing the LAN Bridger central server, which is hosted on a Linux box. I do most of my development and testing on Windows, though, so as a result the LAN Bridger server runs on both Windows and Linux for ease of debugging. From time to time, and especially toward the end of a release cycle, I typically have to compile and test the server thoroughly on Linux to make sure everything works.

I usually do all of my testing with debug builds (contrary to Ken’s beliefs). Once I’m sure everything runs smoothly and I don’t get any assertions or erratic behavior, I turn to release builds. In this particular case, the server worked flawlessly for debug builds, but exhibited a rather strange behavior for release builds.

LAN Bridger uses a connection manager, which is responsible for timing out connections if they become idle. It basically checks to see when the last packet was received and if it’s more than a certain threshold, the connection is closed. When testing the debug build of the server, the connections were correctly timed out and removed. For release builds, not so much.

My first cause of action was to attach a debugger (gdb) to the server process. Unfortunately, since this was a release build, debugging became much harder. For instance, as shown in the stack trace below it was pretty difficult to identify which thread corresponded to the connection manager, even after showing the stack traces for each thread:

(gdb) info threads
  9 Thread 114696 (LWP 2979) 0x08129acf in nanosleep () at cryptlib.h:1144
  8 Thread 98311 (LWP 2978)  0x08129891 in accept () at cryptlib.h:1144
  7 Thread 81926 (LWP 2977)  0x081299b1 in recvfrom () at cryptlib.h:1144
* 6 Thread 65541 (LWP 2976)  0x08129acf in nanosleep () at cryptlib.h:1144
  5 Thread 49156 (LWP 2975)  __pthread_sigsuspend (set=0xb6f8dcdc) at ../linuxthreads/sysdeps/
                             unix/sysv/linux/pt-sigsuspend.c:56
  4 Thread 32771 (LWP 2974)  __pthread_sigsuspend (set=0xb778dcdc) at ../linuxthreads/sysdeps/
                             unix/sysv/linux/pt-sigsuspend.c:56
  3 Thread 16386 (LWP 2973)  __pthread_sigsuspend (set=0xb7f8dcdc) at ../linuxthreads/sysdeps/
                             unix/sysv/linux/pt-sigsuspend.c:56
  2 Thread 32769 (LWP 2972)  0x082936fa in poll ()
  1 Thread 16384 (LWP 2969)  __pthread_sigsuspend (set=0xbf97834c) at ../linuxthreads/sysdeps/
                             unix/sysv/linux/pt-sigsuspend.c:56

By manually setting a breakpoint in the connection manager’s main loop, I was able to determine that it was thread number 6. The stack trace for that particular thread was very uninformative:

(gdb) bt
#0  0x08129acf in nanosleep () at cryptlib.h:1144
#1  0x00000000 in ?? ()

The first thought that hit me was that I was trashing the stack, but setting a breakpoint in the connection manager thread clearly indicated that it was just a misleading stack trace and that the thread was running as intended.

What puzzled me the most was why connections were removed correctly for debug builds, but not for release builds. The server also ran flawlessly on Windows. The only real difference between the debug and release builds was the -O2 compiler flag. To rule out the possibility that the compiler was making a funky optimization error, I decided to upgrade gcc to the latest version (4.3).

This was my first real mistake – never assume that the compiler is doing something wrong until you’ve thoroughly examined all other possibilities. I guess I got blinded by certain optimization bugs I had read about in the past. Unfortunately, the bug also manifested itself even when the server was compiled with the updated version of gcc.

Having wasted valuable time upgrading the compiler and making both the server and all of its third-party dependencies compile and link, I went back to the root of the problem. By stepping through the connection manager, I noticed that the variable, which held the time the last packet was received, was zero. Interestingly, that particular variable was only being assigned values from a single function, GetTickCount(), which was supposed to return a monotonically increasing number, i.e. it should never be zero.

Here’s where I made the second mistake. Rather than just checking the return value of the GetTickCount() function, I decided to verify that the calculation in GetTickCount() was correct. (I’d recently updated the implementation of the function so I thought it quite possible that I’d made a mistake somewhere.) The result of the calculation, however, was correct.

Finally, I checked the actual return value of GetTickCount(). The value of the eax register was 0. This was impossible given that I’d just verified that the calculation was correct.. unless I’d forgotten to return the result!

My own fault

There were several reasons why this mistake had gone unnoticed. First of all, I had originally coded the function while on Windows and it never got compiled by the Visual Studio compiler since the function was only defined for Linux. Secondly, I had deliberately left out the ‘-Wall‘ compiler flag because I got a ton of warnings from third-party libraries. Unfortunately, this meant I didn’t get a warning when I forgot to return a value from the GetTickCount() function. (Compare that to the Visual Studio compiler which automatically promotes such [rather serious] warnings to errors.)

These are both mistakes on my end, and indeed some that I should’ve caught. However, the point of this story is to highlight just how easily small things can lead you astray. For instance, I didn’t notice that the problem was actually in the GetTickCount() function until much later in the process after taking a major detour.

I can’t help but wonder how quickly I would’ve caught this mistake if I was debugging on Windows. As you might have guessed, I’m much more comfortable with WinDbg and the Visual Studio debugger than I am with gdb, and it wouldn’t have taken me more than a few seconds to notice the incorrect value of the timestamp variable, or the incorrect return value of the GetTickCount() function. A bug like this really shows the importance of knowing your tools — and knowing them well. On the plus side, though, I got a chance to work much closer with gdb than I usually do.

Wrapping Up: Why it worked for debug builds

The GetTickCount() function is used to get a timestamp that can be used in timeout operations. The reason the connection manager worked correctly for debug builds is due to how the compiler repurposes the registers. On x86, return values are stored in the eax register, but since GetTickCount() didn’t return a value, eax was essentially left untouched and would instead contain the result of a previous calculation or the return value from another function.

For the debug builds, when receiving a packet and storing the current time in a timestamp variable, GetTickCount() might return zero due to the way the compiler had purposed the registers. When the connection manager later checked to see if the connection had timed out by subtracting the current value of GetTickCount(), which might be non-zero depending on the current value of the eax register, with the previously stored timestamp, the difference might be larger than the timeout threshold and the connection would be removed.

For the release builds, eax was always zero when GetTickCount() was called, and thus the timeout threshold was never exceeded and no connections were ever removed.

If you’d like to share your own stupid mistake stories, drop me a line at [dreijer at echobit dot net] and I’ll publish them here (with due credit, of course).

A few days ago I stumbled upon Herb Sutter’s latest remarks on his personal preference and some of the comments to his post sparked my interest. For instance, I find the following statement particularly bad:

“The compiler or IDE knows the type of the variable, so why do you need to prefix it?”

The answer is readability. I think John hit it spot on in his comment to Herb’s post:

“Variable names are for the code reader, not just the compiler. … Most people read code more often than they write it. The IDE is not always available (and it’s certainly not quick, if we’re talking about Visual Studio). The variable declaration is often far away from its usage. Having the type embedded in the name saves people from having to dig around for the declaration.”.

Personally, I don’t prefix variables with type information such as dw or ul since I don’t believe it makes the code any clearer. I’m more interested in how the variable is used. For instance, I use p to denote pointer types since the semantics are clearly different from using an integral type, as illustrated below:
“if (foo == 0)” and “if (pFoo == 0)”

One of my biggest pet peeves is member variables. Here at work, people have very different ideas on how member variables should be identified. Some just write them like they do local variables, others capitalize the first letter, and yet others prefix the variables with an underscore or m_. I prefer the latter simply because it improves readability. It enables me to quickly get an overview of a piece of code and the scope of its variables without having to use any features of the IDE.

To demonstrate to our coworkers the usefulness of showing the scope of their variables by augmenting the variable name, a friend and I compiled a list with three different scenarios:

• If intelligent syntax highlighting isn’t available, you cannot figure out whether you’re assigning a value to a local or a member variable just by looking at the code. Instead, you have to manually search through the function or rely on the IDE (e.g. by hovering the cursor over the variable name)
• Local and member variable name clashing:

  void SomeClass::SomeFunc()
  {
      // Define a local variable
      int foo = ...;
  
      // Assign the local variable to a member variable
      // We have to use the 'this' keyword since the two variables are named identically
      this->foo = foo;
  }

• Prefixing member variables makes auto-suggestion tools like IntelliSense and Visual Assist more accurate. For instance, typing ‘m_’ would cause the IDE to suggest only member variables, whereas typing the variable name would bring up local, global, and member variables in the suggestion box. (I know this point sort of contradicts the first scenario since it relies on the IDE, but I think it’s important to highlight how prefixing can also increase the usefulness of such tools.)

Before concluding this post, I think it’s also worth commenting on some peoples’ habits of embedding type information in variables. As I said before, I don’t personally do it, but not because I think it creates a huge maintainability issue. Changing the type of a variable (e.g. from int to float) changes the contract between the variable and the code, and the programmer is forced to go through every single location that uses it to make sure the code still behaves correctly. Not too bad, in my opinion…

These viewpoints are, of course, completely my own, and I’m sure a lot of people disagree with me. If you’re one of those, I’d love to hear your sentiments about why you think I’m wrong.

I just got a new laptop from work preloaded with the usual stuff such as an office suite and antivirus software. I tend to prefer manually installing only the software I need when I get my hands on a new computer, but since this was for work I was pretty limited in what I was allowed to do with it. So, I just shed a tear and then went along pretending I was happy.

Not surprisingly, it didn’t take long before I started noticing problems.

Here at work, we use Exchange and Outlook 2007 for e-mail. I’ve installed two profiles, one for my Exchange account and one for my personal account. I keep the Exchange profile active most of the day (after all, I’m at work..), but occasionally I exit Outlook and switch profiles so I can check my personal e-mail.

When trying to relaunch Outlook, however, nothing happens. A quick look in Task Manager reveals two instances of the Outlook.exe process, one of them consuming more memory than the other. Killing the first process (the one with the biggest memory footprint) causes the second to continue running and the new instance of Outlook to appear.

It would seem the old instance of Outlook isn’t shutting down correctly, and the lingering process is keeping all new instances of Outlook from loading properly. This issue is quite problematic for ordinary (read: non-technical) users who think the application has already exited because the Outlook window has disappeared. Instead, they probably try to launch the application five more times before giving up and finally calling tech support.

It’s not hard to guess who gets the blame: Outlook, yet again.

The prospect of falsely blaming Outlook, and because a coworker saw similar behavior with his copy of Outlook, caused me to start digging deeper to find the root cause of why the application was hanging at shutdown. Was it really a bug in Outlook, or was it caused by something else?

The first step was to figure out when the issue surfaced. It occurred to me that I’d only noticed the behavior when I was switching from the Exchange profile to my personal e-mail profile. This lead me to believe it might be an issue with Exchange since my personal e-mail accounts only used POP3 and IMAP. The company I work for uses an off-site e-mail provider, and as such we have to log on the very first time Outlook connects to the Exchange server, Digging deeper, it turned out Outlook didn’t hang if I dismissed the Log On dialog and instead quit the application right away. Furthermore, if I did log on and the first entry in my Inbox was anything but an e-mail, such as a calendar event, and I closed Outlook before opening any e-mails, the application wouldn’t hang upon shutdown either.

Unfortunately, the previous analysis didn’t shed any light as to why this issue only happened when accessing Exchange. The natural next step was therefore to attach a debugger to the lingering process. WinDbg spat out the following message right away:

Break-in sent, waiting 30 seconds...
WARNING: Break-in timed out, suspending.
This is usually caused by another thread holding the loader lock

Aha, sounds like we’re dealing with a deadlocked process. WinDbg revealed that only two threads were left in the process:

0:000> ~* k

.  0  Id: d34.5d4 Suspend: 1 Teb: 7ffdf000 Unfrozen
ChildEBP RetAddr
0013f7bc 7c90df3c ntdll!KiFastSystemCallRet
0013f7c0 7c8025db ntdll!NtWaitForSingleObject+0xc
0013f824 7c802542 kernel32!WaitForSingleObjectEx+0xa8
0013f838 77566f71 kernel32!WaitForSingleObject+0x12
0013f854 775146e7 ole32!CDllHost::ClientCleanupFinish+0x30
0013f880 77514657 ole32!DllHostProcessUninitialize+0x80
0013f89c 774ff231 ole32!ApartmentUninitialize+0xd6
0013f8b4 774fee98 ole32!wCoUninitialize+0x41
0013f8d0 05f9d912 ole32!CoUninitialize+0x5b
WARNING: Stack unwind information not available. Following frames may be wrong.
0013fd34 05fa6b01 saPlugin+0xd912
0013fd5c 7c923aba saPlugin!DllUnregisterServer+0x62e1 ; DllMain
0013fde0 7c81ca96 ntdll!LdrShutdownProcess+0x14f
0013fed4 7c81cb0e kernel32!_ExitProcess+0x42
0013fee8 78131720 kernel32!ExitProcess+0x14
0013fef0 78131a03 msvcr80!__crtExitProcess+0x14
0013ff2c 78131a4b msvcr80!_cinit+0x101
0013ff3c 300051f6 msvcr80!exit+0xd
0013ffc0 7c817067 OUTLOOK+0x51f6
0013fff0 00000000 kernel32!BaseProcessStart+0x23

1  Id: d34.17a8 Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr
00fffc0c 7c90df3c ntdll!KiFastSystemCallRet
00fffc10 7c91b22b ntdll!NtWaitForSingleObject+0xc
00fffc98 7c901046 ntdll!RtlpWaitForCriticalSection+0x132
00fffca0 7c91e395 ntdll!RtlEnterCriticalSection+0x46
00fffd18 7c90e437 ntdll!_LdrpInitialize+0xf0
00000000 00000000 ntdll!KiUserApcDispatcher+0x7

The stack trace for the main thread (thread 0) shows that the application is shutting down. In fact, it’s stuck trying to uninitialize COM in the saplugin module. Let’s have a look at who owns that module:

0:000> lmvm saplugin
start    end        module name
04e20000 04e57000   saPlugin  (export symbols)  C:\Program Files\SiteAdvisor\6261\saPlugin.dll
Loaded symbol image file: C:\Program Files\SiteAdvisor\6261\saPlugin.dll
Image path: C:\Program Files\SiteAdvisor\6261\saPlugin.dll
Image name: saPlugin.dll
Timestamp:        Fri May 16 18:40:18 2008 (482DB8F2)
CheckSum:         00040AA8
ImageSize:        00037000
File version:     2.6.0.6261
Product version:  2.6.0.0
File flags:       0 (Mask 3F)
File OS:          4 Unknown Win32
File type:        1.0 App
File date:        00000000.00000000
Translations:     0409.04b0

Interesting. The module, which is called SiteAdvisor, seems to be owned by McAfee and has been installed as part of McAfee VirusScan. According to their website, SiteAdvisor protects your computer against viruses, spam, and adware. I’d previously noticed it as a plugin in Firefox, but had decided against uninstalling it in order to figure out what it actually did (who knows, maybe it did something good).

What puzzled me, however, was why it suddenly appeared in Outlook. It hadn’t been installed as an add-in, so something had to cause saplugin.dll to be loaded by Outlook. I restarted Outlook and instructed it to break whenever saplugin.dll was loaded:

0:000> sxe ld:saplugin; g

ModLoad: 06000000 06037000   C:\Program Files\SiteAdvisor\6261\saPlugin.dll
eax=00000003 ebx=00000000 ecx=0602f050 edx=f6000000 esi=00266110 edi=00000000
eip=7c90e4f4 esp=0013dc54 ebp=0013dd48 iopl=0         nv up ei pl zr na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246
ntdll!KiFastSystemCallRet:
7c90e4f4 c3

0:000> kb
ChildEBP RetAddr  Args to Child
0013dc50 7c90d50c 7c91d956 00000990 ffffffff ntdll!KiFastSystemCallRet
0013dc54 7c91d956 00000990 ffffffff 0013dd2c ntdll!ZwMapViewOfSection+0xc
0013dd48 7c91624a 001d8d10 0013ddd4 0013e2fc ntdll!LdrpMapDll+0x759
0013e008 7c9164b3 00000000 001d8d10 0013e2fc ntdll!LdrpLoadDll+0x1e9
0013e2b0 7c801bbd 001d8d10 0013e2fc 0013e2dc ntdll!LdrLoadDll+0x230
0013e318 7c80aeec 0013e338 00000000 00000000 kernel32!LoadLibraryExW+0x18e
0013e32c 10001282 0013e338 003a0043 0050005c kernel32!LoadLibraryW+0x11
WARNING: Stack unwind information not available. Following frames may be wrong.
0013e780 7e44f8ee 00050003 000c1570 0013e7ac saHook!saHooker_Uninitialize+0xc2
0013e7b4 7c90e453 0013e7c4 00000080 00000080 USER32!__fnHkINLPCBTCREATESTRUCT+0x82
0013e840 7e43e1ad 7e43e18a 00000003 000c1570 ntdll!KiUserCallbackDispatcher+0x13
0013e868 74730f0a 000102df 00000003 000c1570 USER32!NtUserCallNextHookEx+0xc
0013e8b0 7e431923 00000003 000c1570 0013e910 MSCTF!SysCBTProc+0xd2
0013e8e4 7e44f8e7 00050003 000c1570 0013e910 USER32!DispatchHookA+0x101
0013e918 7c90e453 0013e928 0000007c 0000007c USER32!__fnHkINLPCBTCREATESTRUCT+0x7b
0013e9a0 7e42e389 7e42e34f 00000000 0013eec8 ntdll!KiUserCallbackDispatcher+0x13
0013ee44 7e42e442 00000000 0013eec8 00000000 USER32!NtUserCreateWindowEx+0xc
0013eef0 7e42d0d6 00000000 32a7d830 00000000 USER32!_CreateWindowEx+0x1ed
0013ef2c 32629a19 00000000 32a7d830 00000000 USER32!CreateWindowExW+0x33
0013ef94 326d3cc1 00000000 32a7d830 00000000 mso!Ordinal6700+0x301
0013efdc 326d3ac8 0008157e 0127ad00 7c80e4cd mso!Ordinal3679+0x6b

0:000> du 0013e338 
0013e338  "C:\Program Files\SiteAdvisor\626"
0013e378  "1\saPlugin.dll"

That explained it! SiteAdvisor was injected into all processes by the means of a global hook called sahook.

Now, the stack trace shown earlier, which identified saplugin as being part of the critical path that led to the deadlock, could just as well have been caused by an earlier event that eventually manifested itself when SiteAdvisor was being unloaded. As with classic memory corruption cases, it might be the case that some other module in the process had done something bad which later caused the process to deadlock. Consequently, more digging for proof was required before blaming SiteAdvisor.

Fortunately, I didn’t have to look far. It was clear that the main thread of execution was waiting for something — after all it was stuck in a call to WaitForSingleObject. The second thread in the process was attempting to acquire a lock as can be seen by the call to RtlEnterCriticalSection:

0:000> ~1 kb
ChildEBP RetAddr  Args to Child
00fffc0c 7c90df3c 7c91b22b 00000218 00000000 ntdll!KiFastSystemCallRet
00fffc10 7c91b22b 00000218 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
00fffc98 7c901046 0197b178 7c91e395 7c97b178 ntdll!RtlpWaitForCriticalSection+0x132
00fffca0 7c91e395 7c97b178 00fffd2c 00000004 ntdll!RtlEnterCriticalSection+0x46
00fffd18 7c90e437 00fffd2c 7c900000 00000000 ntdll!_LdrpInitialize+0xf0
00000000 00000000 00000000 00000000 00000000 ntdll!KiUserApcDispatcher+0x7

0:000> !critsec 7c97b178
CritSec ntdll!LdrpLoaderLock+0 at 7c97b178
LockCount          1
RecursionCount     1
OwningThread       5d4
EntryCount         64
ContentionCount    64
*** Locked

0:000> ~
.  0  Id: d34.5d4 Suspend: 1 Teb: 7ffdf000 Unfrozen
   1  Id: d34.17a8 Suspend: 1 Teb: 7ffde000 Unfrozen

The second thread, however, is trying to acquire the loader lock, which is already owned by the main thread. The latter is in turn waiting for COM to shut down (that second thread could very well be an APC call dispatched by the COM runtime), and voilá, we have a deadlock.

MSDN states in several places that “Because there is no way to control the order in which in-process servers are loaded or unloaded, do not call CoInitialize, CoInitializeEx, or CoUninitialize from the DllMain function.” In general, doing something non-trivial in DllMain is a big no-no, and I’d expect anyone developing DLLs to have read Microsoft’s excellent document titled Best Practices for Creating DLLs.

Unfortunately, the stack trace I printed in the beginning of this post clearly shows that the saplugin module is calling CoUninitialize in its DllMain function while the loader lock has been acquired. (The loader lock is acquired by the call to LdrShutdownProcess.) This is a clear violation of the restrictions highlighted above.

I’ve uninstalled SiteAdvisor and the issue seems to be gone. Unfortunately, this case illustrates how easy it is for third-party applications to make their host applications exhibit undefined behavior.

With the previous issues I’ve had with McAfee’s software, I’m really starting to dislike the invasiveness (and buginess) of their applications.