The case of the Windows Desktop Search CPU hog

My home computer is set up (using MBM 5) to sound off an alarm with the PC Speaker whenever CPU temperature exceeds 70 degrees Celsius. It is not that I overclock it, not at all, but rather because of the embarrassingly poor fan I got with the box (which I didn’t purchase personally) combined with the Pentium 4’s well-known poor thermal characteristics.

The alarm is quite audible and annoying. Fortunately, it is pretty unusual for the CPU to get that hot. I’ve noticed that it does this after more than a few minutes of running at near 100% CPU usage.

Earlier today, I could hear the annoying alarm from across the house. I sat down at the computer and quickly silenced it. Since this isn’t the first time I’ve heard the alarm, I already knew what I was looking for: a CPU hog. More often than not, the culprit is vmware-vmx.exe, VMware Workstation’s virtual machine process. If you run an MS-DOS VM without DOSIDLE or the like, the guest OS hogs the CPU and the blame is placed on vmware-vmx.exe. This can also happen with a modern guest OS like Windows in case the guest is misbehaving for some reason. However, this time, I didn’t remember leaving any VM running on the machine, and indeed, none was to be found.

I opened the Task Manager to examine the situation and after a few moments determined that cidaemon.exe was at 49% CPU usage. This actually means 100% on my system which has a hyper-threaded, single core Pentium 4.

My first reaction was that it’s probably just a random case of a CPU hog since I’ve never experienced it on my system before. Without asking too many questions, I killed the cidaemon.exe process and watched as CPU utilization plummeted to the healthy single-digits.

I got up from the computer and addressed other concerns (washed my car). It was about an hour later that I came back and heard the hectic CPU temperature alarm once again. I took another look at the process list and found out cidaemon.exe has been resurrected, and had returned to avenge its predecessor’s demise with another round of CPU hogging.

At this point I understood there is no recourse but to conduct a deeper investigation. I googled for cidaemon.exe being a CPU hog and pretty much only found the “solution” of disabling Windows Desktop Search. Yeah, well, not so fast. I fired up the debugger and attached to the CPU hogging cidaemon.exe process (there was another instance of cidaemon.exe which was being benign).

The first thing I did is get some basic information on where I found myself:

0:006> lmv m cidaemon
start end module name
01000000 01005000 cidaemon (pdb symbols) C:\WINDOWS\Symbols\exe\cidaemon.pdb
Loaded symbol image file: C:\WINDOWS\SYSTEM32\cidaemon.exe
Image path: C:\WINDOWS\SYSTEM32\cidaemon.exe
Image name: cidaemon.exe
Timestamp: Fri Aug 17 23:56:12 2001 (3B7D84EC)
CheckSum: 00007613
ImageSize: 00005000
File version: 5.1.2600.0
Product version: 5.1.2600.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: cidaemon.exe
OriginalFilename: cidaemon.exe
ProductVersion: 5.1.2600.0
FileVersion: 5.1.2600.0 (xpclient.010817-1148)
FileDescription: Indexing Service filter daemon
LegalCopyright: © Microsoft Corporation. All rights reserved.

OK, so cidaemon.exe is the indexing service’s filter daemon. An educated guess would be that the name is actually short for “content indexing daemon.” I knew that the indexing service is a part of the core Windows XP operating system (as is apparent from the file version of cidaemon.exe above) and not an additional service installed by the separate Windows Desktop Search package I have on my system.

The next thing I did was to see how many threads I am dealing with in this demonic daemon process:

0:003> ~
0 Id: afc.af4 Suspend: 1 Teb: 7ffdf000 Unfrozen
1 Id: afc.a5c Suspend: 1 Teb: 7ffde000 Unfrozen
2 Id: afc.858 Suspend: 1 Teb: 7ffdc000 Unfrozen
. 3 Id: afc.13b4 Suspend: 1 Teb: 7ffdb000 Unfrozen
4 Id: afc.1190 Suspend: 1 Teb: 7ffda000 Unfrozen
6 Id: afc.10d8 Suspend: 1 Teb: 7ffd8000 Unfrozen

OK, looks like I’ve got quite a few. Only the CPU hogging thread is of interest, so let’s check the CPU times, let the process resume for a while, and examine the times once again:

0:003> !runaway
User Mode Time
Thread Time
6:10d8 0 days 0:00:53.515
0:af4 0 days 0:00:00.140
4:1190 0 days 0:00:00.078
3:13b4 0 days 0:00:00.000
2:858 0 days 0:00:00.000
1:a5c 0 days 0:00:00.000
0:003> g
(afc.ca8): Break instruction exception - code 80000003 (first chance)
eax=7ffdd000 ebx=00000001 ecx=00000002 edx=00000003 esi=00000004 edi=00000005
eip=7c901230 esp=0078ffcc ebp=0078fff4 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246
ntdll!DbgBreakPoint:
7c901230 cc int 3
0:005> !runaway
User Mode Time
Thread Time
6:10d8 0 days 0:01:04.859
0:af4 0 days 0:00:00.140
4:1190 0 days 0:00:00.078
5:ca8 0 days 0:00:00.000
3:14b8 0 days 0:00:00.000
2:858 0 days 0:00:00.000
1:a5c 0 days 0:00:00.000

Well, no hard choice here. The other threads list the same CPU time and thread 6 is obviously the culprit. I examine the thread’s stack trace:

0:005> ~6 s
eax=00000103 ebx=00000aef ecx=00000103 edx=000043c3 esi=000043c0 edi=000000ef
eip=74da9a58 esp=00bdc2ac ebp=00bdc370 iopl=0 nv up ei ng nz ac po cy
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000293
USP10!ScriptTokenize+0x97:
74da9a58 663b4508 cmp ax,word ptr [ebp+8] ss:0023:00bdc378=0103
0:006> k
ChildEBP RetAddr
00bdc370 74da2d44 USP10!ScriptTokenize+0x97
00bdc398 74e85840 USP10!ScriptItemize+0x42
00bdc3d0 74e84a13 riched20!CUniscribe::ItemizeString+0x8a
00bdc644 74e5f269 riched20!CTxtBreaker::OnPostReplaceRange+0x200
00bdc664 74e6b436 riched20!CNotifyMgr::NotifyPostReplaceRange+0x25
00bdc6ec 74e65825 riched20!CRchTxtPtr::ReplaceRange+0x42c
00bdc750 74e65643 riched20!CTxtRange::ReplaceRange+0x127
00bdc7b8 74e66b3e riched20!CTxtRange::CheckLimitReplaceRange+0x179
00bdca04 74e5b32b riched20!CTxtRange::CleanseAndReplaceRange+0x833
00bdfa78 74e5b8b2 riched20!CLightDTEngine::ReadPlainText+0x2c7
00bdfd30 74e7ce5d riched20!CLightDTEngine::LoadFromEs+0x287
00bdfeb8 74e7e87b riched20!CTxtEdit::TxSendMessage+0x11c9
00bdff1c 00852b90 riched20!CTxtEdit::Open+0x244
WARNING: Stack unwind information not available. Following frames may be wrong.
00bdff44 00853e4e rtffilt!DllCanUnloadNow+0xaff
00bdff74 00853eb8 rtffilt!DllRegisterServer+0x1af
00bdffac 00853f11 rtffilt!DllRegisterServer+0x219
00bdffb4 7c80b683 rtffilt!DllRegisterServer+0x272
00bdffec 00000000 kernel32!BaseThreadStart+0x37

I was quite surprised and disappointed by this stack trace. As is apparent, no symbolic information is available for rtffilt.dll. Since my Windbg is configured to use the Microsoft web symbol store, the PDB should have been downloaded automatically. I pondered, perhaps a third party product has installed this filter into the indexing service? I examined who is to blame for this sorry state of affairs:

0:006> lmv m rtffilt
start end module name
00850000 0085a000 rtffilt (export symbols) C:\WINDOWS\system32\rtffilt.dll
Loaded symbol image file: C:\WINDOWS\system32\rtffilt.dll
Image path: C:\WINDOWS\system32\rtffilt.dll
Image name: rtffilt.dll
Timestamp: Tue Feb 06 01:36:06 2007 (45C7BF66)
CheckSum: 00014877
ImageSize: 0000A000
File version: 2006.0.6000.16431
Product version: 2006.0.6000.16431
File flags: 8 (Mask 3F) Private
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: rtffilt.lib
OriginalFilename: rtffilt.lib
ProductVersion: 2006.0.6000.16431
FileVersion: 2006.0.6000.16431 (vista_gdr(wmbla).070205-1422)
FileDescription: RTF Filter
LegalCopyright: © Microsoft Corporation. All rights reserved.

Oy vey. The RTF filter is actually a Microsoft DLL. Since this is an XP machine and the DLL version is “Vista-ish”, I quickly concluded that the DLL belongs to Windows Desktop Search 3.01, which aligns the desktop search engine on XP with Vista’s built-in search engine. The only other thing with “Vista-ish” versions on my system is Windows Script Host 5.7, which similarly, aligns WSH functionality with the Vista version. I figured the RTF filter is unrelated to that product.

It appears that whenever Microsoft releases an add-on for Windows like Windows Desktop Search, releasing debugging symbols to the symbol store is not a foregone conclusion, even if the original version of the component has its symbols already published by virtue of being a part of the Windows Vista operating system. It is obvious that the absence of the symbols is not a misguided attempt at protecting intellectual property, since they are available anyway for their Vista counterparts. For some reason, not all groups inside Microsoft have uploading public symbols to the symbol store as an integral part of their release process. When mishaps like this happen, their customers are left out in the cold to diagnose the issue on their own. Sure, in this case it was on my home computer, but it could have easily been an enterprise production debugging scenario. Missing symbols are unacceptable, Microsoft. Someone should get their ass fired.

Although taken aback, I did not give up. Since I did have symbols for the upper part of the stack trace, I could still have some impression of what was going on. I figured I’d let the filter run for a while and examine the stack again:

0:006> g
(afc.fbc): Break instruction exception - code 80000003 (first chance)
eax=7ffdd000 ebx=00000001 ecx=00000002 edx=00000003 esi=00000004 edi=00000005
eip=7c901230 esp=0078ffcc ebp=0078fff4 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246
ntdll!DbgBreakPoint:
7c901230 cc int 3
0:005> ~6 k
ChildEBP RetAddr
00bdc3e8 74e848d6 riched20!CBreakArray::GetBreak+0x2a
00bdc644 74e5f269 riched20!CTxtBreaker::OnPostReplaceRange+0xc3
00bdc664 74e6b436 riched20!CNotifyMgr::NotifyPostReplaceRange+0x25
00bdc6ec 74e65825 riched20!CRchTxtPtr::ReplaceRange+0x42c
00bdc750 74e65643 riched20!CTxtRange::ReplaceRange+0x127
00bdc7b8 74e66b3e riched20!CTxtRange::CheckLimitReplaceRange+0x179
00bdca04 74e5b32b riched20!CTxtRange::CleanseAndReplaceRange+0x833
00bdfa78 74e5b8b2 riched20!CLightDTEngine::ReadPlainText+0x2c7
00bdfd30 74e7ce5d riched20!CLightDTEngine::LoadFromEs+0x287
00bdfeb8 74e7e87b riched20!CTxtEdit::TxSendMessage+0x11c9
00bdff1c 00852b90 riched20!CTxtEdit::Open+0x244
WARNING: Stack unwind information not available. Following frames may be wrong.
00bdff44 00853e4e rtffilt!DllCanUnloadNow+0xaff
00bdff74 00853eb8 rtffilt!DllRegisterServer+0x1af
00bdffac 00853f11 rtffilt!DllRegisterServer+0x219
00bdffb4 7c80b683 rtffilt!DllRegisterServer+0x272
00bdffec 00000000 kernel32!BaseThreadStart+0x37

So it looks like we went from stack bottom doing some Uniscribe work up to actual work on the RTF by the implementation in the Rich Edit library. We do not appear to be stuck in an infinite loop in Uniscribe, at least. Letting the daemon run a few more times shows that some movement between Rich Edit 2.0 functions is occurring, but it is unclear what’s making it work so hard.

A corrupt RTF file is an obvious suspect as the cause of the problem. Let’s see what the filter is working on. I was hoping to use the “!handle” extension to find out about the file handle used by the cidaemon process, but disappointingly, I could not get the extension to show the file name for a given file handle. I opted for Sysinternals’ Handle tool, which I remembered had this functionality. I reviewed cidaemon.exe’s open handles:

C:\Program Files\SysInternals>handle -p cidaemon.exe
Handle v3.2
Copyright (C) 1997-2006 Mark Russinovich
Sysinternals - http://www.sysinternals.com
------------------------------------------------------------------------------
cidaemon.exe pid: 5016 NT AUTHORITY\SYSTEM
C: File (RW-) C:\WINDOWS\system32
40: File (RW-) C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_659
5b64144ccf1df_6.0.2600.2982_x-ww_ac3f9c03
68: Section \BaseNamedObjects\c::inetpub:catalog.wci__cisharedmem
F8: Section \BaseNamedObjects\__CiPerfMonMemory
------------------------------------------------------------------------------
cidaemon.exe pid: 2812 NT AUTHORITY\SYSTEM
C: File (RW-) C:\WINDOWS\system32
40: File (RW-) C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_659
5b64144ccf1df_6.0.2600.2982_x-ww_ac3f9c03
68: Section \BaseNamedObjects\c::system volume information:catalog.wci_
_cisharedmem
F8: Section \BaseNamedObjects\__CiPerfMonMemory
14C: File (---) C:\Program Files\Nero\Nero 7\Core\CDI\CDI_IMAG.RTF
1D4: Section \BaseNamedObjects\ShimSharedMemory
20C: File (RW-) C:\WINDOWS\WinSxS\x86_Microsoft.Windows.GdiPlus_6595b64144c
cf1df_1.0.2600.2180_x-ww_522f9f82
264: File (RW-) C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_659
5b64144ccf1df_6.0.2600.2982_x-ww_ac3f9c03
2A0: File (RW-) C:\Program Files\Nero\Nero 7\Core\CDI\CDI_IMAG.RTF

The offending cidaemon.exe has two open handles to an RTF file and in fact they are both for the same file. This file, CDI_IMAG.RTF, is a part of the Nero Burning ROM CD/DVD software installed on my system. This must be the file causing the RTF filter’s grief.

Attempting to confirm this is the case (although hardly anything would be able to persuade me otherwise at this point), I examine the thread’s stack yet again, this time also examining arguments:

0:006> kb
ChildEBP RetAddr Args to Child
00bdc3a8 74e795bb 00012414 00000100 00000000 riched20!CTxtPtr::FindOrSkipWhiteSpaces+0xb5
00bdc3dc 74e84889 00000001 00000101 00000100 riched20!CTxtPtr::FindWhiteSpaceBound+0xcc
00bdc644 74e5f269 00012414 00000000 00000001 riched20!CTxtBreaker::OnPostReplaceRange+0x76
00bdc664 74e6b436 00bdfe50 00012414 00000000 riched20!CNotifyMgr::NotifyPostReplaceRange+0x25
00bdc6ec 74e65825 00000001 00000001 00bdd2e8 riched20!CRchTxtPtr::ReplaceRange+0x42c
00bdc750 74e65643 00000001 00bdd2e8 00000000 riched20!CTxtRange::ReplaceRange+0x127
00bdc7b8 74e66b3e 00000001 00bdd2e8 000000ee riched20!CTxtRange::CheckLimitReplaceRange+0x179
00bdca04 74e5b32b 000003c6 00bdca78 00000000 riched20!CTxtRange::CleanseAndReplaceRange+0x833
00bdfa78 74e5b8b2 00bdfe4c 00bdfee4 00000000 riched20!CLightDTEngine::ReadPlainText+0x2c7
00bdfd30 74e7ce5d 00bdfe4c 00001011 00bdfee4 riched20!CLightDTEngine::LoadFromEs+0x287
00bdfeb8 74e7e87b 00000449 00001011 00bdfee4 riched20!CTxtEdit::TxSendMessage+0x11c9
00bdff1c 00852b90 000ba5b0 00bdff34 00000130 riched20!CTxtEdit::Open+0x244
WARNING: Stack unwind information not available. Following frames may be wrong.
00bdff44 00853e4e 00bdff64 b7ee486a 7ffdd000 rtffilt!DllCanUnloadNow+0xaff
00bdff74 00853eb8 b7ee48b2 7ffdd000 7ffdf000 rtffilt!DllRegisterServer+0x1af
00bdffac 00853f11 00bdffec 7c80b683 0007e384 rtffilt!DllRegisterServer+0x219
00bdffb4 7c80b683 0007e384 7ffdd000 7ffdf000 rtffilt!DllRegisterServer+0x272
00bdffec 00000000 00853f04 0007e384 00000000 kernel32!BaseThreadStart+0x37

Neither file handle 0x14c nor file handle 0x2a0 is immediately visible in this stack trace. Presumably the Rich Edit 2.0 library is dealing with the RTF stream in memory. Once again I am bitten by the absence of the RTF filter’s symbol and FPO information. If the stack trace had not been black-holed in the middle, perhaps associating the ongoing processing with an handle would have been easier. I realize in this case only one RTF file is open in the offending process, but I was still interested in how one could associate this operation with a specific file if the filter had been written to process many RTF files in bulk, simultaneously.

While we can’t readily see the handle’s association with the current Rich Edit processing, we can at least find old, nested function calls on the stack that used it. We can search the stack for the 0x2a0 handle, for example (little endian byte order):

0:006> s @esp L1000 a0 02
00bdc994 a0 02 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................

Let’s have a look at the stack in the vicinity of the 0x2a0 value we found to determine whether this is a false positive or actual use of the handle:
0:006> dps 0x00bdc994-20
00bdc974 74e64eab riched20!CTxtRange::Update_iFormat+0x15c
00bdc978 00000001
00bdc97c 00000000
00bdc980 00000000
00bdc984 00bdfe4c
00bdc988 00000000
00bdc98c 7c90e288 ntdll!NtReadFile+0xc
00bdc990 7c801875 kernel32!ReadFile+0x16c
00bdc994 000002a0
00bdc998 00000000
00bdc99c 00000000
00bdc9a0 00000000
00bdc9a4 00bdc9cc
00bdc9a8 00bdca78
00bdc9ac 00000ffe
00bdc9b0 00000000
00bdc9b4 7c801898 kernel32!ReadFile+0x1ca
00bdc9b8 00bdfe4c
00bdc9bc 00bdfee4
00bdc9c0 00000000
00bdc9c4 00000001
00bdc9c8 00bdfee4
00bdc9cc 00000000
00bdc9d0 00000ffe
00bdc9d4 00000000
00bdc9d8 00000000
00bdc9dc 00bdc9b8
00bdc9e0 00000000
00bdc9e4 00bdff68
00bdc9e8 7c839aa8 kernel32!_except_handler3
00bdc9ec 7c809ba8 kernel32!`string'+0xc
00bdc9f0 ffffffff

We can see the remains of a call to the ReadFile API with a file handle to the suspected RTF file on the offending thread’s stack. Had this process had more threads processing RTF files, we’d be able to make the connection between the CPU hogging thread and the problematic RTF file this way. If the process was using a thread pool and each thread was being recycled to process an RTF file after another, results could have still been inconclusive, however.

For those without Nero 7 on their systems who are wondering what’s the deal with that RTF file, it is, in fact, not a rich text format file at all. Ahead made the mistake of calling a non-RTF file with the RTF file extension, persuading the Indexing Service’s RTF filter to attempt to index it. Given all the talk from Microsoft these days on their Security Development Lifecycle, Threat Modeling and all the fuzzing they claim to be doing, it is curious that relatively modern code like the search engine’s RTF filter would happily go into a CPU hogging frenzy on an RTF file that doesn’t even bother with an RTF opening tag. Some basic sanity validation is amiss here.

I looked it up out of curiosity and apparently CDI_IMAG.RTF has something to do with the now very defunct CD-i technology, which White Book classic VCDs are required to be compatible with. I was surprised there’s actually something older than those Video CDs…

For the time being, I renamed the file so its extension won’t be RTF, to prevent the RTF filter from embarrassing itself and killing my box while doing so. Hopefully Nero won’t mind as long as I stick to 21st century technology.

Another happy ending.

A JScript interactive interpreter shell for the Windows Script Host

A few weeks ago a friend of mine who was starting to code WSH scripts asked me if there was an interactive shell for it. I told him I didn’t know of one, but it got me thinking. Python users are well aware of its interactive shell, and indeed, take it for granted. The ability to execute statements immediately, one line at a time, is pretty fundamental. Yet, Windows Script Host offers no such built-in functionality.

As a new WSH script writer, I would often find myself in a cycle of Edit, Save, Run, quite similar to the Edit, Save, Compile cycle of native code, with small snippets or even one-liners. As my scripts exceeded a certain threshold of complexity, I would find myself using the script debugger. Unfortunately, since it provides a read-only view of the debugged script, when I wanted to make my changes and test them, I would have to switch to the editor window, make my changes and start the session all over again.

I recall one of my first programming experiences, around the age of 9. I was toying with my brother’s old Atari 800XL, initially in BASIC. The machine had a BASIC interpreter built-in to its ROM and had a measly 64K of memory. When it was turned on, a friendly “READY” banner written in white over a blue background greeted you to the BASIC interpreter. The ability to interpret statements for rapid modeling was considered so fundamental it was this, not a disk operating system, that was the core of the machine.

Fast forward back to the present. My friend’s question had me searching for a solution. I did not find an interpreter targeting WSH, but I did find a variety of JavaScript shells for the web browser, like this one. These are good candidates for brushing up on the HTML DOM, but are less useful to those using WSH. For instance, attempting to model automation controllers quickly brings you into the realm of warnings and denials from the browser’s security apparatus. In the case of the specific shell in question, its approach of having the user use Shift-Enter for multi-line entry was inconvenient, since you had to keep doing so until your code block was complete.

JScript lends itself well for implementing a self-hosted shell through the “eval” keyword. As I was examining the input mechanisms available to a command-line WSH script, I saw that the WScript.StdIn object was a TextStream, only supporting newline-terminated input. This means I could not implement the same Shift-Enter based approach for multi-line input used by the browser hosted shell mentioned above.

During my search, I also found two JavaScript shells that are not browser-based but do not target the Windows Scripting Host. One was a part of Spidermonkey, which is Mozilla’s classic JavaScript implementation (which is set to be retired and replaced by the JIT-based Tamarin, the open source version of Adobe Flash’s ActionScript VM, in future versions of Firefox). The other was a part of Rhino, an implementation of JavaScript in Java.

I examined their source to determine what was their approach to multi-line input. It appeared that both the Spidermonkey and the Rhino shells used the underlying script language implementation’s functionality for determining whether a given string is a “compilable entity.” They would keep on reading lines until that condition was met.

Unfortunately, it did not seem like I could adopt a similar approach. Calling “eval” repeatedly until successful is problematic. Even if I were to implement the shell in native code using the Active Scripting hosting interfaces instead, it did not appear as though IActiveScript or the related interfaces provided a similar “compile testing” method.

Defeated, I opted for a simple approach where a blank line initiates multi-line input and two consecutive blank lines terminate it.

Pardon me for the coarse, unpolished illustration code:

function hex(n) {
    if (n >= 0) {
        return n.toString(16);
    } else {
        n += 0x100000000;
        return n.toString(16);
    }
}
var scriptText;
var previousLine;
var line;
var result;
while(true) {
    WScript.StdOut.Write("jscript> ");
    if (WScript.StdIn.AtEndOfStream) {
        WScript.Echo("Bye.");
        break;
    }
    line = WScript.StdIn.ReadLine();
    scriptText = line + "\n";
    if (line === "") {
        WScript.Echo(
            "Enter two consecutive blank lines to terminate multi-line input.");
        do {
            if (WScript.StdIn.AtEndOfStream) {
                break;
            }
            previousLine = line;
            line = WScript.StdIn.ReadLine();
            line += "\n";
            scriptText += line;
        } while(previousLine != "\n" || line != "\n");
    }
    try {
        result = eval(scriptText);
    } catch (error) {
        WScript.Echo("0x" + hex(error.number) + " " + error.name + ": " +
            error.message);
    }
    if (result) {
        try {
            WScript.Echo(result);
        } catch (error) {
            WScript.Echo("<<<unprintable>>>");
        }
    }
    result = null;
}

This is simple enough and is quite useful for the majority of cases. It does have its disadvantages, however. Notably, the surrounding code of the shell is leaked into the namespace accessible by the interpreted snippets. For example, typing “hex” exposes the error code conversion function. However, for my needs, I found this quite satisfactory.

If anyone can offer an improved implementation, I’d be happy to see it in the comments.

Save this code to a file, like shell.js, and use “cscript shell.js” to start it. Multi-line input is performed as described above. Ctrl-Z can be used to quit.

A nice stunt you can pull with this is wrap the shell in a .WSF referencing your favorite type libraries. For example, consider this shell.wsf:

<job>
    <reference object="Scripting.FileSystemObject" />
    <script language="JScript" src="shell.js" />
</job>

If you start a shell with “cscript shell.wsf”, the shell instance will have access to type library constants like “ForReading”, “ForAppending” and so forth.

Although I’m not much of a VBScript fan, I considered doing something similar for it, since it could be quite handy for testing those pesky automation objects that take SAFEARRAYs and are thus not that JScript friendly. However, VBScript’s distinction between expressions and statements (and its Eval function vs. the Execute & ExecuteGlobal keywords) make such a thing a bit more complicated. It is also not clear whether the interpreter should opt for executing statements using Execute or ExecuteGlobal, and in what cases. If anyone is up for implementing this, I’d love to see it.

Have fun.

A lightweight approach for exposing C++ objects to a hosted Active Scripting engine

Microsoft’s Active Scripting architecture allows application developers to host the same implementations of the JScript and VBScript scripting languages used by Internet Explorer for scripts in HTML pages, Active Server Pages (the old, pre-.NET implementation) for server-side dynamic content or the Windows Scripting Host for independent scripts. Additionally, third party scripting engines can and have been developed, for Python, Perl and other interpreted languages.

Hosting a scripting engine involves implementing the IActiveScriptSite interface, providing a method to pass script code to the IActiveScript and IActiveScriptParse interfaces and is extensively documented in the literature. Therefore, I shall not discuss the mechanics of hosting itself and will elaborate only on the topic of exposing objects from the host to the engine.

Enabling scripting in your application only adds value over the external Windows Script Host if you expose unique, internal application functionality to the hosted scripts. If your application already exposes its functionality as COM automation objects to automation controllers that can be used out-of-process, there isn’t much point in hosting. However, if your application is document-oriented, for example, providing scripts with access to the document context can be very useful to your users.

A scripting host can make its object model available to hosted scripts by providing the engine with an IDispatch interface for each object it wants to make available. This interface is the foundation of OLE automation and is used by the scripting languages for late binding.

Since the IDispatch interface is basically a rather raw reflection mechanism, implementing it from scratch for a moderately complex object is tedious and error-prone.

If your application already implements COM objects regardless of scripting, it probably already makes use of a framework for doing so, be it ATL, MFC or the CLR. In that case, you have already paid the framework tax and implementing another interface is no challenge. Specifically, ATL offers the convenient IDispatchImpl class for implementing dual interfaces while the CLR makes it ridiculously simple to implement dispatch interfaces (by default, a .NET class is also a COM dispatch object).

However, a dependency on the CLR might not be a welcome requirement. Similarly, complicating a substantial existing code base with the tedium of COM class registration is an adventure that may not be suitable for the faint of heart. If you do not wish to expose automation objects to external clients like WSH, you have no need or desire to modify the registry and maintain that information across installations, uninstallations, upgrades and the like.

However, both ATL and MFC do not go to any reasonable lengths to facilitate the implementation of internal, unregistered COM objects. The IDispatchImpl class requires that you provide it with type information for your dual interface, but ATL’s only ITypeInfo wrapper, CComTypeInfoHolder, is oriented towards retrieving that from a type library residing in a file, either an independent .TLB or an embedded resource in your .EXE or .DLL file. This means that for exposing an object, you need to describe it in IDL, have your build process generate a .TLB for it with MIDL and possibly embed it as a resource using RC. At run time, you need to take care of the logistics of interface and type library registration. All of this for what you only want as internal functionality.

Apart from being tedious, that approach is also characterized by being rigid and static. Manipulating your exposed objects by making runtime decisions that could change the type information does not go well with them being static embedded resource entities.

I considered what would it to take to come up with binary type information from a source that isn’t a file or a resource. At first glance, the LoadTypeLib API is definitely file-oriented. However, a light bulb turned on in my head when I noticed that if the file name given does not exist, the string is treated as a moniker. I was hoping I could generate binary type information in .TLB format from IDL, store it in a flexible manner and provide LoadTypeLib with a moniker to the type information. I then paused as I realized there was an unanswered question – “a moniker to what?”. As is not uncommon in Microsoft’s documentation, elaboration on this point was scarce. I later found this newsgroup post on the matter. The original poster had the same question as mine and the reply pointed me in the right direction.

Although the responder was incorrect in assuming the pointer moniker implementation actually implemented IMoniker::GetDisplayName, a deficiency for which I can find no excuse, the OBJREF moniker provides a suitable alternative. The OBJREF moniker is a superset of the pointer moniker that supports out-of-process references, although no such functionality is required by me for this purpose, just getting a display name to feed LoadTypeLib.

I promptly implemented a skeleton IUnknown that would simply print what interface was requested on every call to QueryInterface and then return E_NOINTERFACE. I created an OBJREF moniker for this IUnknown implementation and supplied LoadTypeLib with the moniker’s display name. I figured this way, I would figure out what LoadTypeLib is expecting the supplied object to implement as an alternative to being given a file name.

I was disappointed when I saw what happened next – LoadTypeLib was asking my object for an ITypeLib implementation, and nothing else. This basically means that LoadTypeLib’s moniker support is completely useless – it returns an ITypeLib for an ITypeLib you already have.

My next attempt to tap into the existing binary type information parser involved writing a test program that called LoadTypeLib on a .TLB file for the purpose of finding if it loaded the information to memory and then promptly used intermediate functionality on the in-memory data that was also accessible to me. I examined the type library loader’s high level flow using Windbg:
0:000> bp oleaut32!LoadTypeLib
0:000> g
Breakpoint 2 hit
eax=0012ff00 ebx=7ffda000 ecx=81818d85 edx=10313d00 esi=0012fdc8 edi=0012ff5c
eip=771279e5 esp=0012fdbc ebp=0012ff68 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
OLEAUT32!LoadTypeLib:
771279e5 8bff mov edi,edi
0:000> wt -m oleaut32 -l 2 -ns
Tracing OLEAUT32!LoadTypeLib to return address 004117d0
7 0 [ 0] OLEAUT32!LoadTypeLib
43 0 [ 1] OLEAUT32!LoadTypeLibEx
16 0 [ 2] OLEAUT32!InitLoadInfo
44 16 [ 1] OLEAUT32!LoadTypeLibEx
65 0 [ 2] OLEAUT32!InitAppData
51 81 [ 1] OLEAUT32!LoadTypeLibEx
9 0 [ 2] OLEAUT32!LHashValOfNameSys
58 90 [ 1] OLEAUT32!LoadTypeLibEx
25 0 [ 2] OLEAUT32!OLE_TYPEMGR::LookupTypeLib
66 115 [ 1] OLEAUT32!LoadTypeLibEx
46 0 [ 2] OLEAUT32!FindTypeLib
72 161 [ 1] OLEAUT32!LoadTypeLibEx
25 0 [ 2] OLEAUT32!OLE_TYPEMGR::LookupTypeLib
89 186 [ 1] OLEAUT32!LoadTypeLibEx
185 0 [ 2] OLEAUT32!GetOffsetOfResource
101 371 [ 1] OLEAUT32!LoadTypeLibEx
79 0 [ 2] OLEAUT32!CreateFileLockBytesOnHFILE
117 450 [ 1] OLEAUT32!LoadTypeLibEx
23 0 [ 2] OLEAUT32!LoadTypeLib2LockBytes
126 473 [ 1] OLEAUT32!LoadTypeLibEx
17 0 [ 2] OLEAUT32!FileLockBytesMemory::Release
133 490 [ 1] OLEAUT32!LoadTypeLibEx
156 0 [ 2] OLEAUT32!OLE_TYPEMGR::TypeLibLoaded
145 646 [ 1] OLEAUT32!LoadTypeLibEx
15 0 [ 2] OLEAUT32!UninitLoadInfo
155 661 [ 1] OLEAUT32!LoadTypeLibEx
5 0 [ 2] OLEAUT32!__security_check_cookie
157 666 [ 1] OLEAUT32!LoadTypeLibEx
9 823 [ 0] OLEAUT32!LoadTypeLib

It was clear from the trace that LoadTypeLib created an ILockBytes over the .TLB file and promptly provided it to LoadTypeLib2LockBytes. Unfortunately, neither this internal function nor any other leading to its functionality is exported from the OLE automation library. The binary type information parser is not accessible externally for in-memory data. What was missing is that LoadTypeLib did not attempt to QueryInterface for ILockBytes when given a moniker, if ITypeLib is not implemented by the object directly. This approach, therefore, had to be scrapped.

I was hoping I could use MIDL to generate binary type information for me and the notion of implementing ITypeLib completely on my own for in-memory representation seemed like a daunting task. If this is the trade-off, surely reverting to ATL and dealing with the evils of registration would be the better approach?

Not so fast. It turns out there is another approach for getting the type information you need for exposing your C++ object, without generating a full-fledged type library or implementing your own type information provider. The marvelous CreateDispTypeInfo API. You provide it with a INTERFACEDATA structure describing your object and get the type information you need. Combined with CreateStdDispatch, it becomes easy to expose simple objects to automation.

Reviewing the sample included in the MSDN documentation of CreateDispTypeInfo is indicative of the sorry state of affairs in Microsoft’s documentation group, seeing as it is quite incomplete and makes use of macros like METHOD0, METHOD1 and PROPERTY, which are nowhere to be found and must have existed in whatever project the sample code has been copy-pasted from. Detailed discussion of the function’s usage is scarce, but existent, on the Web, primarily in newsgroups. Allow me to illustrate with an example. Consider the following hypothetical C++ class one wishes to expose to scripting:

class MyObject
{
public:
virtual void __stdcall f(int i);
virtual BOOL __stdcall g(float f);
};

As is evident, this class is pretty plain and certainly has nothing to do with COM. Just the sort of class your existing application with no use of COM might have. To expose it, we need to fill some descriptor structures so type information can be generated for it. We add a few static members:
class MyObject
{
public:
virtual void __stdcall f(int i);
virtual BOOL __stdcall g(float f);
static PARAMDATA f_paramData;
static PARAMDATA g_paramData;
static METHODDATA methodData[];
static INTERFACEDATA interfaceData;
};

Let’s fill those babies up:

PARAMDATA MyObject::f_paramData = {
OLESTR("i"), VT_I4
};
PARAMDATA MyObject::g_paramData = {
OLESTR("f"), VT_R4
};
METHODDATA MyObject::methodData[] = {
{ OLESTR("f"), &MyObject::f_paramData, 1, 0, CC_STDCALL, 1, DISPATCH_METHOD, VT_EMPTY },
{ OLESTR("g"), &MyObject::g_paramData, 2, 1, CC_STDCALL, 1, DISPATCH_METHOD, VT_BOOL }
};
INTERFACEDATA MyObject::interfaceData = {
MyObject::methodData,
sizeof(MyObject::methodData) / sizeof(METHODDATA)
};

For each method of our object, we describe the method’s parameters, giving them name and type in a PARAMDATA structure. We then fill a method table for the object with complete information, including the parameter data, return value type, calling convention and such. The INTERFACEDATA wraps the whole thing in a nice little package to feed CreateDispTypeInfo with.

We now proceed to create an automation wrapper for our pure object:

CComPtr<ITypeInfo> pMyobjTypeInfo;
hr = CreateDispTypeInfo(
&MyObject::interfaceData,
LOCALE_SYSTEM_DEFAULT,
&pMyobjTypeInfo);
CComPtr<IUnknown> pMyobj;
hr = CreateStdDispatch(NULL, &myobj, pMyobjTypeInfo, &pMyobj);

At this point, pMyobj is a full fledged COM object implementing IDispatch and wrapping the MyObject class instance myobj, which had no knowledge of COM originally and now bundles tables describing its methods.

The scripting site’s implementation of IActiveScriptSite::GetItemInfo should now return pMyObj, the object’s IUnknown and potential IDispatch, and pMyobjTypeInfo, its ITypeInfo, when requested to do so by the hosted scripting engine. We register the object we wish to expose with the engine:

hr = pActiveScriptEngine->AddNamedItem(
L"myobject",
SCRIPTITEM_ISSOURCE | SCRIPTITEM_ISVISIBLE | SCRIPTITEM_ISPERSISTENT);

If our GetItemInfo does its job when asked for “myobject”, assuming we host the JScript engine, we can now do things like
myobject.f();
var b = myobject.g(0.4);

in script code running in our host.

I find this approach to automation object exposition attractive because it is non-intrusive. If desired, the tables describing the exposed class need not be members of the actual class, but can be stored separately. Notice that you do not even have to generate a CLSID for the exposed class. It is also possible to expose only a certain subset of class methods to the scripting environment.

However, maintaining the type information tables can become a clear scalability issue with more complicated classes. For these cases, rolling an automatic code generation solution may be desired, since MIDL’s functionality in this department cannot be reused. The class and its methods could be described in an XML file, and a tool iterating over its DOM or even an XSLT transformation could generate a C++ header file from the description, complete with the INTERFACEDATA information. This would ensure the method tables and the actual method signatures remain synchronized over the extended life-time of the class.

Finally, a pointer to some tips and a few words of caution to those interested in this solution, this newsgroup post. Let me add to it that CreateDispTypeInfo only seems to work correctly with the __stdcall calling convention, even if you specify CC_CDECL in your type information. Using CC_STDCALL and making sure your classes use __stdcall made everything work. Before that, symptoms included method arguments receiving seemingly random values when called by the scripting engine, due to stack imbalance.

Hey, I said the approach is lightweight, not the post ;-)