ParanoidMike

Stalk Google's stalking of you

2012-01-27T15:09:00.000-08:00

What with all the attention on Google right now because of the sweeping and selfish privacy policy changes they just publicized, it's smart, fun and sometimes a little disturbing to see just what Google's advertising arm thinks it knows about you. Play along at home - browse to this page from whatever browser(s) you like to use, and compare with what I found.

http://www.google.com/ads/preferences

(1) Here's what Google thinks based on the cookie in my iPhone's Twitter cache:

Your categories
Below you can review the interests and inferred demographics that Google has associated with your cookie. You can remove or edit these at any time.
Arts & Entertainment - Comics & Animation - Comics
Arts & Entertainment - Humor - Live Comedy

Your demographics
We infer your age and gender based on the websites you've visited. You can remove or edit these at any time.
Age: 35-44
Gender: Male

(2) Here's what Google says about me (the same user) according to my
iPhone's browser cookie (i.e. in Safari):

Your categories
Below, you can review the interests and inferred demographics that Google has associated with your cookie. You can remove or edit these at any time.
Beauty & Fitness
Jobs & Education - Jobs
Reference - General Reference
World Localities - North America - USA - Pacific Northwest - Oregon - Portland (OR)

Your demographics
No demographic categories are associated with your ad preferences so far. You can add or edit demographics at any time.

(3) Here's what Google thinks of me according to Chrome on my laptop:

Your categories and demographics

No interest or demographic categories are associated with your ads preferences so far. You can add or edit interests and demographics at any time.

(4) Here's who they think I am according to IE on my laptop:

Your categories

Below you can review the interests and inferred demographics that Google has associated with your cookie. You can remove or edit these at any time.

Computers & Electronics - Computer Security - Antivirus & Malware

Computers & Electronics - Software - Freeware & Shareware

Computers & Electronics - Software - Software Utilities

Online Communities - File Sharing & Hosting

Reference - General Reference - Time & Calendars

Your demographics

We infer your age and gender based on the websites you've visited. You can remove or edit these at any time.

Age: 25-34

Gender: Male

Frankly, I'm glad that Google has multiple fragmented profiles on me (at least for this usage models of this ad service they sell). If I was to see the exact same data across all apps and devices) I'd be chilled to the bone.

But I'm not naive enough to think they're not deliberately and inevitably taking us (along for the ride) there. The closer try get to a single, unified, aggregated view of my behaviour and interests, the more they'll be able to charge their advertisers for every impression and click-through (or whatever new metrics they're currently stewing up).

Share that revenue stream with me? Then I'm all for the kind of surreptitious gathering that you're doing. Steal that from me and pretend that "more relevant advertising" is fair compensation (legal: "consideration")? You know where you can stuff it.

Got the "2012 security suite" infection? This should fix it

2012-01-20T10:55:00.000-08:00

I heard from colleagues who've been infected with the "2012 security suite" - yet another in a long line of malware that pretends to be an antivirus program, able to shut down legitimate antimalware like McAfee, and gets itself embedded deeply in your system.

Another colleague posted the following instructions on how they got rid of the "2012 security suite" infection. I'm posting as-is with no firsthand knowledge of how well this would work for others, but in the spirit of sharing whatever smarter people have tried in hopes it helps a few others.

"I had to fix this one for a friend recently. I had to use 3 different tools and it took several passes to get in entirely cleaned out. Here’s what worked in the end (I make no guarantees of course):

Download, install, and update Malwarebytes and SpyBot Search & Destroy. They are free. Do this on two computers, the one that’s infected and another one.
Remove the drive from the system and connect it to the other system as a secondary drive. Boot that system into Safe Mode with Networking. Scan with Malwarebytes and allow it to remove anything it finds. Reboot into Safe Mode with Networking and run SpyBot and allow it to remove anything it finds.
DO NOT REBOOT YET (if you reboot at this point you will be re-infected). First delete the following:

%AllUsersProfile%\Application Data\ LocalAppData\kdn.exe
%LocalAppData%\Temp\%UserProfile%\Templates
All files in c:\windows\temp (Yes you will lose your browsing history and cookies, sorry)
Your Temporary Internet Files (on the infected drive, not the system you’re running on)

Now remove the drive from the second computer and put it back in the original computer.
Boot into Safe Mode with Networking
Turn off System Restore.
7. Delete the following registry keys if they exist:

HKEY_USERS\.DEFAULT\Software\Microsoft\Internet Explorer\BrowserEmulation "TLDUpdates" = '1'
HKEY_CURRENT_USER\Software\Classes\.exe\shell\open\command "(Default)" = '"%LocalAppData%\kdn.exe" -a "%1" %*'
HKEY_CURRENT_USER\Software\Classes\exefile\shell\open\command "(Default)" = '"%LocalAppData%\kdn.exe" -a "%1" %*'
HKEY_CLASSES_ROOT\.exe\shell\open\command "(Default)" = '"%LocalAppData%\kdn.exe" -a "%1" %*'
HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\FIREFOX.EXE\shell\open\command "(Default)" = '"%LocalAppData%\kdn.exe" -a "C:\Program Files\Mozilla Firefox\firefox.exe"'
HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\FIREFOX.EXE\shell\safemode\command "(Default)" = '"%LocalAppData%\kdn.exe" -a "C:\Program Files\Mozilla Firefox\firefox.exe" -safe-mode'
HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\IEXPLORE.EXE\shell\open\command "(Default)" = '"%LocalAppData%\kdn.exe" -a "C:\Program Files\Internet Explorer\iexplore.exe"'
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Security Center "AntiVirusOverride" = '1'
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Security Center "FirewallOverride" = '1'

Run Malwarebytes and allow it to clean anything it finds.
Reboot into Safe Mode with Networking
Run Spybot and allow it to clean anything it finds.
Assuming you are able to run your antivirus software, run a full scan of your system.
Reboot normally.
Scan with both tools and your anti-virus software.
Re-enable system restore.

NOTE: If at any time during this process you find you can no longer start your applications, follow the instructions here: http://www.techerator.com/2010/03/virus-blocks-exe-files-from-opening-2/ You’ll find a fix that you can download and run."

2012 Manifesto

2012-01-06T21:43:00.001-08:00

Here is my manifesto for 2012: to move in a more creative career direction, I will think more visually by working more visually.

I will become my own style of visual designer, user experience engineer, crackpot scientist of ideas. This will happen by spending more of my working day more visually - drawing, whiteboarding, illustrating the noise in my head. I will create sketches, wireframes, storyboards, flow charts, ideographs, gonzo business apps, unintelligible charts and results that scare the shit out of me.

Status and failure of CacheMyWork development

2011-10-26T09:04:00.001-07:00

Every month or so I get an email like the following:

Hi, Love your app CacheMyWork. Left some suggestions for improvement on your website a while ago. Any chance you'll release an updated version with more functionality any time soon? Seems to my it has huge potential once it's updated.

I get this kind of email - wondering when I'll finish the app, asking about status, wondering why it seems incomplete or why I haven't integrated [easy fix "X"] - for only one of the open source projects I ever released: http://CacheMyWork.codeplex.com.

Every time I get one of these, I feel like a jerk for not finishing (or continuing) what I started, and I realize that if I was just a better coder, I'd have a lot of happy people out there.

I could never get my head around the databinding that is necessary to connect the WPF front end i built for the "version 2" of my app to the well-formed, totally functional app-finding algorithm I built literally years ago.

I feel like I'm disappointing a lot of people by not getting this back underway, but I really don't have a clue how to fix the damned thing. I think my failure to maintain/improve this project is one reason why I'm giving up on my dream to ever be a professional coder.

I swear something must be wrong with my brain - every time I try to re-learn databinding concepts for .NET they look like they're pretty simple for the author, but when I try to apply the ideas to my code, it never seems to work. I've coded three different data classes, I've tried every combination of parameters in the binding (both the XAML and the code-behind) I could find, but at best I get code parameters in the UI - never anything that hints that the bound data is leaking through (even though I can clearly see the data in the data class when I set breakpoints in the debugger). It's like I'm not "getting" something about how this is supposed to work - it reminds me of how I was a week or two behind in introductory calculus class, when my brain couldn't visualize what it was we were manipulating with those damned equations. (I finally got the calculus, though I think by now I'd have to start all over again.)

What would you do in this situation? I'd really like to get this going again - at least make good on the unfinished "new" release, and give myself some closure on that chapter of my geek life.

Review: Jinx

2011-10-23T12:49:00.003-07:00

Jinx by Brian Michael Bendis
My rating: 4 of 5 stars

I picked this up at a book signing by Bendis last week. I haven't read it in years and years, and I've read so much of Bendis since that time, I wasn't sure how this would hold up. As indie comics go, this is a winner. It's amazing how much of Bendis' talent for dialogue was already there in the beginning, and how well he tuned into that talent and focused on it. I'm also impressed by the art renderings - I totally forgot that Bendis used to do the art for his books, and I would've encouraged him to keep up this side of it (there's much worse from non-writing artists out there, believe me), had his career not taken off so much as a writer. The story is pretty thin - just a few set pieces in which to let his characters breathe - but there's three more books' worth of plot after this too, so it's hard to judge the full merits of this fraction of the story arc (and I don't remember the rest of the storyline so I won't embarrass myself by guessing at it). Most of all, this reminds me why I like Bendis' work so much - he tunes in closely to how people act around each other, and what happens when unusually unlike personalities synthesize into something unlike either of them.

View all my reviews

Excommunicating myself from the Church of Alan Moore

2011-10-22T15:40:00.000-07:00

I'm coming out of the closet: I actively dislike Alan Moore's "masterpieces".

As a kid I drank up works like Watchmen (fresh! Post-modern capes!), but even then I secretly struggled to enjoy V for Vendetta - simply getting through it without losing any grip on a sense of a cohesive story was a challenge, and I worried there was something wrong with me.

As I returned to comics as an adult, I quickly revisited Moore (elder respected statesmen are supposed to be admired, revered and read, so I did my duty to god and the queen) and slogged through Watchmen (again), then Promethea (directionless softcore), Top Ten (interesting and enjoyable but probably due more to Gene Ha's art) and then League of Extraordinary Gentlemen (curious, teasing with its layers of detail, but ultimately thinly veneered story - more like a series of staged scenes in which to scatter all his historical research).

I told myself I was enjoying it (I really needed to believe I saw what everyone else was talking about), but eventually I got to a point where I saw how many times I would put V on the pile of books but dig underneath it to something I was enjoying - and berating myself for not doing my "required reading". Then I realized that if I didn't enjoy something *that* much, there's actually no good reason why I *should* force myself to finish it. So I didn't. And felt liberated for the epiphany.

My review on Goodreads of V for Vendetta is simply this: "Awful, stilted writing. Reads like an assigned book from a first-year Literature class - not something to be enjoyed but to be excruciatingly endured. Should be required reading for the Guantanamo set."

My opinions on Alan Moore's masterpieces started to turn when I started to read some dissent from fellow readers/reviewers on Goodreads and really crystallized when I started commenting in the recent "book club discussion" on Goodreads.

The more I think about Watchmen the book vs. Watchmen the movie, the more I realise that Moore's greatest achievements are writing the wordy equivalent of a Where's Waldo puzzle. His dialogue's not bad and at least the plots aren't telegraphed from page 1, but his characterisations are absurdly inhuman and the work is needlessly overwritten with irrelevant trivia. It's like an artist who has a bad case of OCD - he can't put the damned brush down, even after he's layered on so much paint that the original picture is completely inscrutable underneath the layers of random strokes and irrelevant tangents, that all look pretty but have destroyed anything worth rendering to the world.

Hiding your pretty storyline behind (or burying it under) details that don't advance the story is a way of teasing the smarties in the audience and making making them feel smarter than "other readers", but the conceit falls down once you realise that "other readers" don't care to crack an encyclopaedia every page or two - we lose the rhythm of the plot (if there is one) in trying to figure out what that next metal trap is all about.

As a kid I loved how I could feel smarter than my peers while reading this kind of stuff (even though truthfully I only understood about 1/4 of the references and just imagined that there was weighty importance to the rest). Now, I find things like Watchmen, LoXG or that turgid thesis V just tiresome and laborious to slog through, especially when trying to ignore Moore's desperate pleas to "look at how smart I am" and find the underlying story that's still worth enjoying. Somehow the movies were able to emulate Moore's staging and still find a way to *connect* with the moviegoer - but just barely, considering how slavish they were to nearly every page of layout and dialogue.

I feel at once brave and stupid for putting this out to the world - like I'm going to lose a number of people who would've otherwise tried to like me, but are put off because I just called their favourite baby ugly. I can only offer that my standards have changed since I first started reading comics, and this no longer holds the line for most exciting works in comics for me. "At the time his work was amazing" is an historical conceit, but if that's the best reason I can come up with to convince myself to like it, I'll move onto something I just *naturally* enjoy today. There's a wide world of books and tastes out there these days, and for me I'm glad to have more than Moore to choose from.

I still like Moore's cleverness (despite my self-consciousness at why I like it), and what I don't like about his work, many others do. We don't have to agree on what's best, nor would I enjoy the medium as much if there was no variety to choose from or be surprised by. I appreciate that others don't agree with me, and in many cases I'm ready to understand something I've missed that others see. I've thought a lot about Moore's work over the years and my opinions have evolved - but hasn't entirely stagnated. Who knows? Maybe in another ten years I'll see something there that isn't apparent to me now, and I might be horribly embarrassed by these current opinions.

Considering learning Python - idle thought (until something catalyses it)

2011-10-22T13:11:00.000-07:00

A friend-of-a-friend asked me this week:

Hi Mike, [mutual friend] referred me to you as a good person to ask: what's the best way to learn Python for someone like me, whose programming skills are essentially 1990-era (I know Perl and C, but haven't made the leep to object oriented stuff)? I'd like to leapfrog into the present era, and make web 2.0-ish-looking sites and experiments. Is there a particular web hosting service I should use? Thanks for any advice you might have.

Funny you should ask - I was just wondering this week whether I shouldn't dive into Python as a quick-and-dirty prototyping language. I'd fancied myself for years as someone who might be able to reinvent myself as a programmer, and I've muddled around with C# and VB for a few years now - but only in short spurts. Every time I come back to it, I feel like I'm climbing a steep hill all over again.

For some reason, I get the impression that for folks that are just whipping something together quickly, interpreted scripting languages like Python, Perl or Javascript are easier to deal with - less overhead, less setup, just diving in and getting to the business of making something happen. I've always felt like if I wanted to call myself a coder, that I'd be "cheating" by taking this route, so I never allowed myself the freedom to try this out. But at the same time, I was never brave/patient enough to mess around with low-level code like C or C++ (who wants to write hundreds of lines of memory-handling routines that managed code gives you for 'free'?), so I guess I've left myself between a rock and hard place - not quite as easy as "just do it" but not really forcing myself to learn the really "worthy" stuff either.

How to learn Python? An almost-colleague of mine took the leap and blogged his process for going deep - start at the bottom:

http://www.curphey.com/category/python/

I've never done it myself, but I trust that Mark is a smart guy who doesn't muck around for the sake of making himself "look smart" (feel miserable).

For me, forcing myself to learn to code was an exercise in frustrating false starts - until I found a problem I couldn't solve any way but coding it myself, and a problem that pissed me off enough to keep slogging through failures and dead ends until I got something working.

Web hosting? No idea. I know a few big names (AWS, Rackspace, Google Apps) but I have no clue where to get the pre-built infrastructure to just upload .py and let fly.

Is this helpful? Do you have something specific in mind? If you're working on something specific and looking to work loosely with one or a few others, I'd be interested in hearing what it is and whether it fires my "that sucks!" instinct enough to want to contribute/walk alongside.

Got a new iPhone 4S? Worried about how to "maximize the battery"? Me too

2011-10-14T23:53:00.000-07:00

Just got the iPhone 4S today, and before I even *powered it up*, I made a point of digging around the Interwebs to find out whether I should (a) drain the battery & fully charge it a few times (to 'condition' it), or (b) charge it as much and as frequently as I can. I've literally heard both urban legends over the years, and even though I keep buying Li-ion battery'd devices, I can simply never keep it straight (or feel confident that the story is true and consistent).

Today's research yielded a couple of great links, and while I'm sure this isn't 100% definitive (I didn't even go through the 2nd page of Google results), it's compelling enough for me:

http://forums.macrumors.com/showthread.php?t=943276

"Preparing new lithium-ion for use

Unlike nickel and lead-based batteries, a new lithium-ion pack does not need cycling through charging and discharging. Priming will make little difference because the maximum capacity of lithium-ion is available right from the beginning. Neither does a full discharge improve the capacity of a faded pack. However, a full discharge/charge will reset the digital circuit of a 'smart' battery to improve the state-of-charge estimation"

http://batteryuniversity.com/learn/article/how_to_prolong_lithium_based_batteries

"Similar to a mechanical device that wears out faster with heavy use, so also does the depth of discharge (DoD) determine the cycle count. The smaller the depth of discharge, the longer the battery will last. If at all possible, avoid frequent full discharges and charge more often between uses. If full discharges cannot be avoided, try utilizing a larger battery. Partial discharge on Li-ion is fine; there is no memory and the battery does not need periodic full discharge cycles other than to calibrate the fuel gauge on a smart battery.

A partial discharge reduces stress and prolongs battery life. Elevated temperature and high currents also affect cycle life."

Somehow the battery lasts *longer* and retains a *greater* maximum charge if you fully discharge it as *little* as possible. "Topping it off" as much as possible is the best way to keep it from running down.

Further, these articles made it clear to me that there's no such thing as "conditioning" a Li-ion battery - unlike a car engine, which urban legends make clear should be run at lower speeds for the first XXX miles, the Li-ion battery seems to be as good as it gets when it leaves the factory, and needs no special handling.

Suh-weet - just act like I always have with my previous phone and I'll be laughing.

Usability and UX - are you a practitioner? How to hone my instincts?

2011-10-12T16:43:00.000-07:00

My career has been veering for a couple of years now towards the squishy-but-satisfying creative arenas of Usability & UX (User Experience), and I'd like to make this an explicit career direction. That means (a) meeting other folks already practicing these disciplines, (b) learning how to behave more like them, and (c) sharing my experiences with any who are looking to learn from newer practitioners. I expect the full transition to take a number of years, but where I have creative freedom in my job to flex in these directions I'll be taking advantage of every chance I get.
As for immediate changes:

I'm joining some local user groups here in Portland (PDX UX already, Chifoo and IxDA next)
I've been re-reading "Don't Make Me Think" from my iPhone while commuting
I've bookmarked this discussion on Slashdot (! yes, ironic to find Linux geeks who can spell UX)

What's next? Who should I be talking to? Where should I seek out like-minded folks? What should I be reading/doing/learning next?

Review: S.H.I.E.L.D.: Architects of Forever

2011-09-08T19:07:00.001-07:00

S.H.I.E.L.D.: Architects of Forever by Jonathan Hickman
My rating: 5 of 5 stars

Fantastic ideas incorporated into this series. No one is investing more into re-imagining the Marvel Universe's already well-established history, and boy is Hickman pouring real effort into this and the Secret Warriors. What with all the multiple-titles-per-month writing going on in Marvel these days, it's a rare treat to get someone who sits down and thinks up more than appears on the page.

Weaver's gorgeous artwork lavishly renders each panel, and he does some Big Scenery well. This book for me demanded a careful read and a number of flips back and forth to make sense of all the details. Some may say this isn't a good story or that they didn't like it; for me this is exactly the kind of work I've been looking for, and I enjoyed the challenge, the surprises and the sheer force of imagination that this book brought to me. I borrowed this book from the library but I'm going out to buy it now, and I'll put the next book on my wish list too. This deserves a place of honor among my most prized graphic novels, and I'll happily re-read it just for the "better than petty men" inspiration.

View all my reviews

Review: The Other Side

2011-08-27T14:48:00.001-07:00

The Other Side by Jason Aaron
My rating: 5 of 5 stars

Fascinating insights into two very different training & motivational approaches to mustering an army, and how it affects not only the individual actions of soldiers but also the outcome of the whole war. Then the real slog hits - holy god is this hard to get through, unrelenting, painful, numbing stuff. I had to put it down about 2/3 the way through, take a break, read some funny stuff on the Internet, grab a snack. It was *that* hard for me to experience just how unrelentingly depressing, inhumane and soul-crushing the actual *experience* of fighting this war was for the boys on the ground. Which makes this an all-star, kudos-earning, compelling storytelling effort on the part of the creators. When I'm that repulsed by the work, that means they did an incredible job reaching out from the pages and hitting me square in the emotive muscle.

View all my reviews

Review: Atomic Robo Vol. 3: Atomic Robo and the Shadow from Beyond Time

2011-08-14T16:33:00.001-07:00

Atomic Robo Vol. 3: Atomic Robo and the Shadow from Beyond Time by Brian Clevinger
My rating: 5 of 5 stars

Funny. Inventive. And the word "fresh" just keeps coming to me, even though my subconscious critic knows somehow all these stories are regurgitated. What Clevinger does with Lovecraft is just slightly insane and wonderful.

What he does with the later chapters are mind-blowing - combining insane scenes with an incredible fluency with science and science fiction. My brain actually *thinks* while reading this one, and wonders how real & bleeding edge some of these ideas really are. That happens so rarely and my intellectual standards in comics are so low, that this is a real gem.

View all my reviews

How can I automatically crosspost/publish Google Reader "shares" to Google+, Facebook or Twitter?

2011-07-30T14:52:00.001-07:00

Google Reader is such a great, simple way as a desktop user to keep up with dozens or hundreds of RSS feeds - but even moreso, to be able to selectively share and comment on the blog articles in those feeds with your friends and with various "external" social services (Facebook, Twitter, Delicious, Stumbleupon, and lots of even-more-ancient-sounding services).

What *isn't* Google Reader great at?

sharing your favourite articles (with commentary) via a purely mobile experience
sharing those same faves with your burgeoning Google+ circles

Mobile: the Google Reader team have added a tiny number of widgets for sharing to the mobile web experience. Other third-party iPhone apps have done a better job of integrating to the foreign services' APIs, but it's not like they voraciously keep up or try to stay ahead of the next wave. [Reeder's been good to me but hardly stellar - the FB integration works, but I guess I'd rather it just added this as a background operation for all sharing rather than me having to take the time and extra steps to bother sharing outside of the native Google Reader/Buzz feed.]

G+: And what the heck is with Google leaving no clues as to the status of Google Buzz, and whether/when they'll move operations (like automatic posting of your Google Reader "sharing" activity to the Buzz feed) over to a native G+ implementation? After a year on Buzz, I accumulated no further "organic" growth of friends & followers than the folks that I originally scraped together on day one. I'm seeing a little better "natural uptick" of followers on G+, but still nothing like I see on Twitter - though to be fair, I think if I got new bot followers every time I posted to the Public feed on G+, I'd probably lose my mind.

So what's up with Google Reader? Has it effectively been zombied, or starved of any reasonable squad of hungry developers and Product Owners? Or is this still a feature with a future? Hell, if I have to find yet *another* place to host my RSS feeds I'm probably gonna lose my mind. And that's likely the reason why Google has starved the Reader team of any serious developer resources: there's no competition left. Google is the last place to offer free, voluminous RSS scraping and it shows in their complacency. Up to the rest of us to kludge together some pretty Rube Goldbergian workflows to make it worth reading/sharing on Reader in the first place.

So what does *your* Goldberg machine look like?

So far, the furthest I've gotten is to ensure that Sharing & Notes are still working from the mobile Reader and the Reeder iPhone app. Nothing in the Reader "Send To" list even seems to hint at a G+ interaction - and I haven't gotten creative enough yet to figure out how to make a 'custom link' post to one place, that will eventually (and richly - i.e. without cryptically condensing and stripping good content off the original shares) end up on FB, G+ & Twitter without me having to take a half-dozen manual steps at each post. Last I looked, Friendfeed does a cryptic job; does Seesmic have anything to offer here? Any new services from Silicon Valley that I should be looking into?

Review: 52: Vol. 4

2011-07-30T14:47:00.001-07:00

52: Vol. 4 by Geoff Johns
My rating: 4 of 5 stars

Summary: too bad about the never-say-die multiverse addiction at DC. Dibny's story? Awesome. Week 43 Day 1? Like a five-year-old wrote it.

Black Adam's finale was not written by that five-year-old, and works much better.

And the build-up to the final climax? Cool. I felt like I participated in something actually pretty wondrous.

Too bad about bringing back the multiverse tho. Seems like DC is just addicted to its easy outs - the many, many variations on the same themes that make it so easy to explain away any discontinuities. Is DC just fundamentally lazy? Some describe it as "enabling creativity", but sometimes endless possibilities leave you without having to think hard, make choices or come up with elegant solutions.

DC is the brute force method of writing superhero stories. Just keep bashing on the same characters until one of the iterations finally comes out with something good.

I still liked the ending to 52, but it sure reminds me of how tiring these "restarts" get every time DC lets its writers paint them into a corner.

View all my reviews

How can I publish GoodReads reviews on Google+, Facebook or Twitter?

2011-07-30T11:54:00.001-07:00

Hey all my geeky friends: I'm trying to figure out the most automated way (least number of extra steps) to enable all the reviews I write on GoodReads to show up on my Google+ and/or Facebook streams. I realize that with the amount of effort I've put into these reviews over the last year or two, I might as well be getting more value out of the postings than from the dozen or so 'friends' I've accumulated on GoodReads itself. Has anyone come up with a good scheme for pulling or pushing GoodReads reviews to their social walls? I'm currently experimenting with ways to pull or push reviews using this blog as a staging ground - see if I can find a way to pull or push from Blogger to G+ or Facebook.

Right now, I do all my reviewing from the GoodReads iPhone app, and based on my previous experiments, it looks like all of the automation is still just "semi-automated", and only really available from a desktop browser. That is, GoodReads will certainly cache the locations and credentials necessary to cross-post from GR to the 'foreign' services; however, it still requires me to (a) fire up my laptop, (b) browse to GR, (c) find the review I just published via GR on iPhone, (d) click the checkbox for the 'foreign service' to which I want to post, and (e) push the review to that service.

My ideal behaviour: I'd LOVE GoodReads if they could enable the following scenario:

I write a review on GoodReads via the iPhone app
I switch the "shelf" from "currently-reading" to "read"
GoodReads publishes the review automatically (with no further actions on my part) to all previously-configured social platforms: Blogger, Facebook, Twitter (and in the near future, Google+)

Who do I have to get drunk to make this happen?

Review: Final Crisis

2011-07-30T11:42:00.001-07:00

Final Crisis by Grant Morrison
My rating: 2 of 5 stars

Found the first few pages waaaaay too abstract to bother reading the rest. However it keeps coming back at me, and after realizing it was the linchpin between Batman RIP and Batman Reborn, I finally gave in.

This book feels like it's all over the place - intentionally. That's good for the DC geeks who know every minor character or sub-plot that gets a cameo here, but holy hell is it uninteresting to a DC amateur like me.

And frankly the hardest part of reading this book was trying to find and follow the narrative. I mean, there's good mind-bending storytelling that Morrison is famous for, and then there's just putting the plot points, characters and dialogue in a blender and simply churning until it's an unrecognizable mass.

I'm sure if I read it again I'd get much more out of it, and maybe I will someday after I've spent a couple of years catching up on the DC universe (after five years I've focused so much on Marvel).

Until then I'm going to lament Morrison's dubious decision to narrate most dialogue in that most insipid and stilted style that was popular back when comic books were still printed using droplet ink. I totally felt duped into reading this when I saw how little airtime Batman and his "death" actually got. I didn't know anything more about the continuity jump between Batman RIP and Batman Reborn than before I'd read this unholy mess.

View all my reviews

Building an Outlook add-in: troubleshooting an add-in that won’t start

2010-10-31T14:55:00.001-07:00

I made some changes to my code a couple of weeks ago and didn’t document them for myself, nor did I try to debug the code for a week or so. So by the time I did try it out, I didn’t remember what I’ve done, and of course there’s some reason the add-in isn’t loading anymore. It’s still listed among the 14 add-ins listed as “Loading Add-ins” as Outlook starts up, but the context menu isn’t available where it should be anymore.

Digging deep into Outlook 2010 to see what goes on with the add-in (Outlook File menu > Options > Add-Ins > COM Add-ins Go… button > click on the add-in in the list), the Load Behavior text says the following:

Not loaded. A runtime error occurred during the loading of the COM Add-in.

Google that error message, a few articles come back, and I bookmarked a couple to check on later. I’ve reviewed them and tried the following:

http://blogs.msdn.com/b/vsod/archive/2008/04/22/troubleshooting-com-add-in-load-failures.aspx

Examined Registry entry (HKCU\…\Outlook\Addins\) to confirm two things:

LoadBehaviour setting is still = 3 (it is)
file path listed in Manifest value is still valid (it is)

Ensured add-in isn’t on the Disabled list (it isn’t)
Ran Dbgview.exe while loading Outlook to see if there are any obvious errors being thrown that aren’t otherwise being noticed (there aren’t any entries remotely related to my add-in)

http://social.msdn.microsoft.com/forums/en-US/vsto/thread/f2f2a8f6-3ed5-4c1f-ab6b-17120e64f13d/

Set the VSTO_SUPPRESSDISPLAYALERTS=0
Observed the following error:

Microsoft Office Application Add-In
An add-in could not be found or could not be loaded.
Details button:

Exception from HRESULT: 0x8004063E

************** Exception Text **************
System.Runtime.InteropServices.COMException (0x8004063E): Exception from HRESULT: 0x8004063E
   at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
   at System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32 errorCode, IntPtr errorInfo)
   at Microsoft.VisualStudio.Tools.Office.Runtime.DomainCreator.CreateCustomizationDomainInternal(String solutionLocation, String manifestName, String documentName, Boolean showUIDuringDeployment, IntPtr hostServiceProvider, IntPtr& executor)
   at Microsoft.VisualStudio.Tools.Office.Runtime.DomainCreator.Microsoft.VisualStudio.Tools.Office.Runtime.Interop.IDomainCreator.CreateCustomizationDomain(String solutionLocation, String manifestName, String documentName, Boolean showUIDuringDeployment, IntPtr hostServiceProvider, IntPtr& executor)

Found a bunch of discussions referencing this error, and started with the first result: http://social.msdn.microsoft.com/Forums/en-US/vsto/thread/d0ffb791-43bb-41f3-a2ca-c22bd959f4c3
Per instructions, I shut down Outlook, started VS2010, chose “Clean [project]” from the Build menu – this seems to have removed the HKCU registry entry for the add-in

Worked like a charm – next time I debugged the add-in, it came up exactly as I expected it to.

Additional Reference

Debugging in Application-Level Projects http://msdn.microsoft.com/en-us/library/ms269003.aspx

Building an Outlook add-in: binding data to XAML (questions)

2010-10-19T22:26:00.001-07:00

ToDo’s (from last post)

Need to figure out how to bind the NameValueCollection to the XAML for my WPF-based add-in. Then I need to find my a** with two hands and a map. And after that, perhaps I’ll look into the complexity of referral chasing to get the non-GC-replicated user attributes.

Bind AD Data to XAML

The more I look at this snarl of code I’ve pieced together from different samples, the more worried I get that I’ve created an intractable problem. I have three major classes developed so far (plus the XAML):

ContextMenuItem.cs – inherits Office.IRibbonExtensibility, and implements the IRibbon extensibility member and the Ribbon callbacks (i.e. the context menu click handler). The click handler instantiates the Pop-Up window, converts the Outlook contact to an email address and runs the AD query.
UserDetailsPopup.cs – implements the ElementHost WinForms object that wraps the WPF control.
ActiveDirectoryUser.cs – implements the query against the Global Catalog, and returns a collection of requested data.
UserDetailsLayout.xaml – implements the XAML UI in which we’d like to drop the requested data.

How should I store the data, and how can I get the XAML to consume it (and ensure it gets displayed in the UI)?

Where to Store the Data?

Let’s assume for the moment that the ContextMenuItem class gets instantiated when the user right-clicks on a contact, and doesn’t die until the context menu and/or the pop-up window it generates are all dead. That means any Property of the ContextMenuItem will exist during that timeframe, and can be read by other members of the add-in.

So exposing the data the add-in needs as a read-only Property of the ContextMenuItem class seems the only smart way to go here.

How to “Push” Data to UI?

Next, we need some way to cause something in the pop-up window (the XAML, the C# code-behind for the XAML, and/or the WinForms ElementHost that contains the WPF control) to consume the Property’s data, and then refresh the UI of the pop-up window if necessary.

I’m used to the fairly simple .NET app development model, where the first class instantiated is the UI, and it “pulls” in all code calls until the UI receives a callback (user click or some other event) that performs an atomic function. The best approach I can think of at the moment (given the classes I’ve constructed) is to create some function in either the UserDetailsPopup or the UserDetailsLayout (whichever’s code can cause the intended effect) that reads the ContextMenuItem’s property and then… what, “paints” the data into the XAML labels? Writes it to a class-local collection to which the XAML is already bound at design time? Or is there some other elegant feature of VSTO, WPF or .NET 4 that I am totally missing?

Here’s where my inexperience with (1) sophisticated design patterns or (2) WPF is coming to bite me on the butt. However, it occurs to me that the fundamental problem isn’t WPF but rather a VSTO-and-Outlook-centric design pattern that ferries remote data to local content-driven UI. Getting the data to the WinForms equivalent in this model must be the majority of the problem; pushing the data the last mile through the WinForms-to-WPF wrapper can’t be all that hard, can it?

Next Steps

So I guess my next quest is to find an Outlook VSTO code sample that creates a new WinForms object in response to user input (rather than all the examples that just graft VSTO code into an existing Outlook “Form” such as an email window, Appointment item and the like).

The final quest in making a functional object is to re-examine the WPF-in-ElementHost examples for Outlook VSTO add-ins, to see how external data gets bound or otherwise propagated into the XAML.

Found: Minimum permissions to edit Site Columns in SharePoint 2010

2010-10-18T20:51:00.001-07:00

Our organization has a requirement to delegate the maintenance of a number of Site Columns (re-used throughout our business application) to colleagues who do not have Site-wide responsibilities or authority. We didn’t want to simply grant one of the all-powerful permissions/groups (such as Site Collection Administrators, Site Collection Owner, [Site] Owners, or just granting Full Control) as that would leave them open to accusatory questions every time the site or application started behaving badly. We’d like them to be able to act fairly autonomously, without having to worry that they could cause unintentional damage to the site or application due to excessive permissions.

Knowing that most every Microsoft product has a rich, granular set of DACLs applied at every object level, and having exercised some of the rich permissions available in SharePoint 2010, I was pretty sure there should be a way to combine some permissions together to enable Site Column edits, without giving away the farm. However I was unable to find any documentation or research that definitively asserted they knew what those permissions were.

Finally I asked my colleague Dale Cox to pair up with me and, together with four hours and an experimental sub-site, we were able to work out a tight sub-set of permissions that (a) definitely allow Site Columns edits to propagate to inheriting Lists and (b) were ratcheted down stepwise until we were comfortable there was no place lower to go.

Permissions: Summary

There are two sets of permissions we tested for: site-level permissions needed to edit an existing Site Column (that is being used as a column in one or more Lists in the SharePoint site), and List-level permissions needed to propagate Site Column changes down to inheriting columns in a List.

The former permissions *only* exist at the site level (even though some permissions in the permissions set are labelled “site permissions” and others “list permissions”) – we’ve found there are *no* permissions that can be managed directly related to Site Columns, but only indirectly via the site-level permissions.

That we also require the latter permissions was surprising at first (once we configured the inheritance, why should we need ongoing permissions on every inheriting List?), but that’s just a design decision that Microsoft’s SharePoint team made. It means we have to manually *track* all the Lists that inherit changes from Site Columns, and make sure our site column maintenance people have enough permissions to the Lists.

Site-level Permissions

Manage Lists (labelled “List Permissions”)
View Items (labelled “List Permissions”)
Add and Customize Pages (labelled “Site Permissions”)
Browse Directories (labelled “Site Permissions”)
View Pages (labelled “Site Permissions”)
Open (labelled “Site Permissions”)

List-level Permissions

Manage Lists (individual permission – which includes View Items, View Pages and Open)
Contribute (permission set)

There is one case we did *not* explicitly test for, but found was part of the resultant permissions we arrived at: creating a new Site Column. While we believe this probably requires only a subset of the site-level (site and/or List) permissions, we didn’t spend the extra cycles to isolate the minimum permissions needed there. Perhaps it’s implied in the results, and if we need to know we’ll pursue it, but it’s not something we needed to know right now.

Steps We Followed To Distill These Permissions

We had to work as a team to test each permissions combination, as I’d found early on that as soon as I removed myself from the all-powerful groups, I was unable to restore myself or change permissions later on. I manipulated the permissions, and Dale ran the test cases. I pre-created the following groups in our experimental site and assigned them specific site-level permissions, to make it easy to switch from one permission set to another:

Group name	Permission Levels assigned to group (custom permissions included)
Approve	Approve
Design	Design
Manage lists	Manage Lists (Manage Lists, View Items, View Pages, Open, Manage Personal Views)
Manage web site	Manage Web Site (View Items, Manage Web Site, Add and Customize Pages, Browse Directories, View Pages, Enumerate Permissions, Browse User Information, Open)
Site viewers	Contribute, Read, View Only
Edit Site Columns	(Add and Customize Pages, Browse Directories, View Pages, Open)

Dale’s starting position:
- Permissions: member of Site Viewers group
- Result: can view site Libraries and Lists but cannot even load the Site Columns page (/_layouts/mngfield.aspx)
Added Dale to “Manage Lists” group
- Result: can load Site Settings page but still cannot access Site Columns page
Removed Dale from “Manage Lists” group, added to “Manage web site” group
- Result: Can load Site Columns page but cannot edit site columns
Granted Dale the Contribute permission on one of the Lists that inherits a Column definition from the Site Column being tested.
- Result: can still load Site Columns page but still cannot edit the site column being tested
- Dale even tried just updating the site column definition without propagating the change to the inheriting Lists, but that still didn’t work
- NOTE: from here onwards we started making a point of trying to separate out the ability to edit the site column definition from the ability for that site column definition to propagate to inheriting lists
Removed Dale from “Manage web site” group, added to “Approve” group
- Result: Dale cannot load Site Columns page
Added “Manage Web Site” permission to a new permission group called “Approve + MWS”
- That is, copied the “Approve” permission group, then checked the “Manage Web Site” permission (and left the dependent permissions that came with it)
- Result: Dale can load the Site Columns page, but cannot commit changes to any Site Columns
Removed Dale from “Approve + MWS”, added Dale to “Manage Hierarchy” group
- Note: while the difference in included permissions between “Manage Web Site” and “Manage Hierarchy” is large, still the “Manage Hierarchy” permission set is a superset of “Manage Web Site”
- Result: Dale can access the Site Columns page, but cannot commit changes to Site Columns
- However, we noticed that Dale can create a new Site Column, and can edit that new Site Column
- Dale attempted just a non-propagating edit with one of the existing Site Columns, and was successful.
- Theory: current permissions on the Lists that inherit from the existing Site Columns are what’s blocking the previous Site Column edits
Granted to Dale “Contribute” permission on all of the three Lists which inherit from the existing Site Columns
- Result: Dale cannot commit a propagating edit on an existing, inherited Site Column
- Theory: the individual permission “Manage Lists” is what’s needed on each inheriting List for the Site Column propagation to succeed
Granted to Dale “Design” permission (which includes the individual “Manage Lists” permission) to one of the three inheriting Lists
- Result: Dale could not commit a propagating edit on an existing, inherited Site Column. Further, the change did not even inherit to the inheriting List (where the permissions needed for propagation could have allowed the change).
- Theory: propagating edits to a Site Column are a transactional request – if the request doesn’t succeed on every inheriting List, then the edit “transaction” is rolled back, rather than partially succeeding.
Changed the following permissions to all three inheriting Lists: changed from “Design” + “Contribute” to “Manage Lists” + “Contribute”
- Result: Dale’s propagating edit finally succeeded!
Decided to go back over the site-level permissions to determine whether there was an unknown interaction between the site-level and list-level permissions that we didn’t account for.
Kept the List-level permissions as-is, but changed Dale’s site-level permissions – removed him from “Manage Hierarchy” and added him to “Approve + MWS” group.
- Result: propagating edit failed, and non-propagating edit failed.
Removed Dale from “Approve + MWS” group, added Dale to Design group (which has Design permissions set).
- Result: propagating edit succeeded.
Compared the permissions between the “Design” and the “Manage Hierarchy” permissions groups, and found the following eight permissions in common:
- Add and Customize Pages
- Browse Directories
- View Pages
- Browse User Information
- Use Remote Interfaces
- Use Client Integration Features
- Open
- Edit Personal User Information
Analysis of these permissions, looking for the “secret sauce” that they share:
- The “Manage Web Site” site-level individual permission isn’t the one that Dale needs (as it isn’t part of the Design permission set, and yet Dale was still able to propagate the edit when he had the Design permission)
- Based on the descriptions of each individual permission listed above, the most likely candidate is “Add and Customize Pages”
Created a new Group called “Edit Site Columns”. Created a new permissions set called “Edit Site Columns”. To that permissions set, we added only “Add and Customize Pages”. Removed Dale from Design group, added him to Edit Site Columns group.
- Note: when “Add and Customize Pages” is checked, the following permissions were automatically checked: “Browse Directories”, “View Pages”, “Open”, “View Items”.
- Result: propagating and non-propagating edits failed. Adding a new Site Column also failed. Dale could access the Site Columns page.
Added back all eight site-level permissions that were in common between the Design and Manage Hierarchy permissions sets.
- Note: the “Edit Site Columns” permission set at this point doesn’t include any of the List-level permissions that I’ve so far ignored. Continued failure means that to edit Site Columns, a user needs some of these “list permissions” assigned at the site level. [Confused yet?]
- Result: propagating and non-propagating edits failed. Dale is still able to access the Site Columns page.
Re-examined the “list permissions” that were included in both of the site-level permissions sets (Design and Manage Hierarchy) that were successful for Dale:
- Manage Lists - Create and delete lists, add or remove columns in a list, and add or remove public views of a list.
- Override Check Out - Discard or check in a document which is checked out to another user.
- Add Items - Add items to lists and add documents to document libraries.
- Edit Items - Edit items in lists, edit documents in document libraries, and customize Web Part Pages in document libraries.
- Delete Items - Delete items from a list and documents from a document library.
- View Items - View items in lists and documents in document libraries.
- Open Items - View the source of documents with server-side file handlers.
- View Versions - View past versions of a list item or document.
- Delete Versions - Delete past versions of a list item or document.
- Create Alerts - Create alerts.
- View Application Pages - View forms, views, and application pages. Enumerate lists.
Added all of the above permissions to the “Edit Site Columns” permission set except View Items.
- Result: propagating edits were successful.
Stripped down the site permissions from the “Edit Site Columns” permission set to leave only the following: ACP, Browse Directories, View Pages, Open.
- Result: propagating edits were successful.
Stripped out a number of List-level permissions from the “Edit Site Columns” permission set to leave only the following: ML, VI. Left the site permissions as in the last step.
- Result: propagating edits were successful.
Wanted to ensure this result is reproducible, so we created a new permissions set “Edit Site Columns – Confirmation”, added the same permissions to this new set (i.e. ML, VI, ACP, BD, VP, Open), created new group “Edit Site Columns – Confirmation”, granted the new permission set to the new group, added Dale to the new group and removed him from the older “Edit Site Columns” group.
- Result: all operations were successful – view Site Columns page, commit propagating and non-propagating edit of inherited Site Columns, add new Site Column.

Building an Outlook Add-in: completing work on the Active Directory query

2010-10-17T22:57:00.001-07:00

ToDo’s (from last post):

Implement the GC (domain-less) query, rather than orient to the current implicit binding to the user's current domain

Big question to answer: are all the fields of interest being replicated in the AD GC?

Implement a reasonable collection that will be useful to the calling code, and return the user info in the return value
- After 15 minutes in MSDN Library and some wide-swath google searches, it sounds like the NameValueCollection class is fairly suitable: hashtable-based, strings-only, and allows multiple values for a single key (which may come in handy later for multi-value directory data, of which AD allows a few)
Start testing this beast against the company AD

…

Y’see, this is one of the problems with doing code development in your spare time – I’m facing a major uphill climb, trying to remember (1) where I was going next, (2) where the next layer of function calls was supposed to get plugged into, and (3) which article I was reading that headed me down this design path. Arrgh.

So I’m starting over with the first “article” (actually MSDN video demo) that caught my fancy, and downloaded the sample code. I have an idea of (1) – to plug the utility class that scrapes AD for user data into the code that calls the Winforms-wrapping-WPF control, and display the user data in that blank context menu window I’ve already sketched out. By wandering around the sample code, I’m hoping to figure out (2) without having to review every line of code and compare to my own design “pattern”. Presumably by retracing the steps outlined in my blogs, I’ll be able to quickly stumble across (3) – again, without having to review every word I’ve written.

God I hope this works.

Acquiring the user’s Email Address

Fortunately, by looking in the sample code’s ThisAddIn.vb class, I’ve already discovered a helpful Function that I knew I needed somewhere. A quick trip to a VB-to-C# converter and I’ll drop this into my own ThisAddIn.cs class.

However, there’s no direct property of an IMsoContactCard object that corresponds to the contact’s email address – for that I had to find this sample code that does a conversion.

D’oh! Nope, even that just ends up returning the X.500 version of the Exchange mail address. Instead, I ended up back at this article that I’d read before, which provides a great GetSmtpAddress() function in VB. Oh, *right* – which I’d already implemented in my project’s code a few weeks ago. See what I mean about this “once a week coding” nonsense?

Here’s my completed OnClick() method for the Contact context menu item, that calls into my ActiveDirectoryUser class:

public void onGetUserDetailsClick(Office.IRibbonControl control)
{
    try
    {
        Office.IMsoContactCard card = control.Context as Office.IMsoContactCard;

        if (card != null)
        {
            // Here's where we need to instantiate the UserDetailsPopup() object
            UserDetailsPopup UserDetailsInfoForm = new UserDetailsPopup();
            UserDetailsInfoForm.Show();

            string emailAddress = GetSmtpAddress(card);
            ActiveDirectoryUser user = new ActiveDirectoryUser();
            
            NameValueCollection coll = user.getGcUserData(emailAddress);
        }
        else
        {
            // Here's where we handle this edge case
        }
    }
    catch (Exception ex)
    {
        // Handle the exceptions
        Console.WriteLine("Error spat out" + ex.Message);
    }
}

…

Much later, after a couple of hours of debugging this slapped-together code, I’ve finally arrived at a method that’s finally deriving the output of live queries against AD. And I have an answer to the big question: no, not all of the really useful attributes are being replicated to the GC. The most obvious one that’s missing would be the job description-oriented attribute. Of all the User object custom attributes *not* to replicate, why did they pick on this one? Is it really so hard to believe that someone would want to query this value in a global directory? Looks like I’m going to have to implement some referral chasing and see if that doesn’t tank the performance of the add-in.

Completed GC-querying Method

Here’s my GC-querying method call to derive the GC-replicated attributes (using “department” as substitute for the the custom attributes internal to the organization that I’m going after):

public NameValueCollection getGcUserData(string emailAddress)
{
    NameValueCollection returnValue = new NameValueCollection();
    
    DirectoryEntry gc = new DirectoryEntry("GC: ");
    DirectoryEntry _root = null;
    using (gc)
    {
        //there is only 1 child under "GC: "
        foreach (DirectoryEntry root in gc.Children)
        {
            _root = root;
            break;
        }
    }

    //Note: the filter must be searching for a GC replicated attribute!
    string filter = String.Format(
        "(mail={0}) ",
        emailAddress
        );

    DirectorySearcher ds = new DirectorySearcher(
        _root,
        filter,
        null,
        SearchScope.Subtree
        );

    using (SearchResultCollection src = ds.FindAll())
    {
        foreach (SearchResult sr in src)
        {
            returnValue.Add("department", sr.Properties["department"].ToString());
        }
    }
    return returnValue;
}

Next Steps

Need to figure out how to bind the NameValueCollection to the XAML for my WPF-based add-in. Then I need to find my ass with two hands and a map. And after that, perhaps I’ll look into the complexity of referral chasing to get the non-GC-replicated user attributes.

Acknowledged: the two simple articles that saved my ass twice now

Extending the User Interface in Outlook 2010

Customizing the Context Menu of a Contact Card in Outlook 2010

Building an Outlook add-in: Context Menu of Contact Card in Outlook 2010

2010-09-27T22:35:00.001-07:00

Before going *any* further with VSTO code, I am taking some very good advice I gave myself a couple of years ago and installing some of the VSTO Power Tools (just the VSTO_PT.exe package) – specifically to obtain the VSTO Developer Cleaner and the VSTO Troubleshooter. I recall both of these being very helpful, the last time I went down this path.

Note also: I am *not* going to pursue the Office Interop API Extensions. I am aware they’re targeted specifically at easing C# development of VSTO apps in Word, Excel and Outlook. However, after reading PhillipHoff’s blog that “documented” them, the near-total lack of a deployment story is enough to encourage me to tough it out without ‘em:

“You are correct in that the User Guide does not explicitly state that you cannot use VSTO_PTExtLibs.exe to redistribute the extension assemblies, but as you found, the installer has a dependency check which prevents it from being used for that purpose.

“You have two basic options for distributing the extension assembly: either a custom installer that places it in the GAC or another common location, or side-by-side the referencing assembly.”

Heh, yeah right. A year or two ago I feel down the rabbit hole of developing a custom setup package for a Word VSTO app add-in. That sucked me dry for months, and I never got any satisfaction out of it (tho it’ll probably save me a couple of months this time around). *Voluntarily* adding to my pain by trying to install a helper assembly of dubious provenance and poor support? Not bloody likely my friends.

Creating a WPF Control for use within Outlook 2010

Diving back into this video tutorial. Already stuck on one of the first steps – adding the WPF “User Control” to the ElementHost winforms container. Even though I’m able to successfully build the project, there’s no WPF user control to add to the ElementHost (at least that’s what the ElementHost “Select Hosted Content” taskpane is indicating. Google it is… searching on elementhost "select hosted content" none:

…actually, I just remembered how many articles I’ve been skimming are focused so strongly on .NET Framework 4, so first I decided to re-target the VSTO project to .NET 4. For some reason that seems to have wiped out the piece of code that contained the ElementHost-including form, so perhaps that’ll clear up the problem I’ve been having.

I ended up Building the project and failing, needing to add a Reference to System.Xaml (from the .NET 4 namepsace) to one of the .CS files. Once I did that, magically I was able to add either of the XAML user controls I’d tried to create as children of the ElementHost object. Amazing.

I’m re-trying this approach with an Office Ribbon (XML) item as is outlined in Customizing the Context Menu of a Contact Card in Outlook 2010. I change the button element’s id, label, and onAction XML parameters, then run this article’s VB-only code through a VB-to-C# converter to import into my code. Each time I have to fill in a few blanks along the way – figuring out which method to add to my code, to which CS library, and matching some undeclared variable names and using statements up with orphaned code in the sample. What moron would expect a “Visual How To” from Microsoft to ever run without a ton of band-aid work to actually fill in boring-but-essential components?

Once I figured out how this sample code would instantiate the Ribbon (XML) item, I only have two more tasks before I start binding the XAML to its data source:

instantiating the XAML-encapsulated-in-WinForms control, and
creating the Callback Methods that are hinted at (but not spelled out) in the Ribbon (XML) //TODO section (automatically generated by Visual Studio when I created the Ribbon (XML) object.

Yeah, that should be easy. Callbacks – is that like in the theatre? Hell, for all *I* know, (2) is where I *implement* (1).

Aha! Apparently I’m not far off from the truth – the Walkthrough: Creating a Custom Tab by Using Ribbon XML says:

You must add onAction callback methods for the Insert Text and Insert Table buttons to perform actions when the user clicks them.

Right right right – so the meaning of “callbacks” is coming slowly back to me – at least in the context of button pushing. :)

Problem: UserControl doesn’t Show()

I spent some time looking for a suitable approach to rendering the WPF Element Host-ing user control, and tried a couple of approaches, but none work so far. The approach I’ve landed on instantiates a new instance of the Class that hosts the ElementHost (here called “DataDetails”), then calls the .Show() method for the object instance, like so:

DataDetails dataDisplay = new DataDetails();
dataDisplay.Show();

However nothing appears visible when I click the new context menu item in Outlook 2010. Here’s what I’ve tried, none of which tells me why this won’t display:

follow the .Show() call with calling the BringToFront() method, in case somehow the user control was being hidden behind Outlook
added a straight System.Windows.Forms class to the project (called it my Debug window), then instantiate it as for the DataDetails class instance and call the ShowDialog() or Show() method [which works]
did a line-by-line debug of this area of my code [which proves that every line of code is being called – nothing skipped – but the ElementHost is still not showing up]

Re-examining the Debug vs. DataDetails objects, I’m pretty convinced that using the Form class is easy, but there’s some step I’m still missing when using the UserControl class. This is borne out when I merely substitute “Form” in place of the original “UserControl” for the partial inheritance, like so:

public partial class DataDetails : Form
{
public DataDetails() { InitializeComponent(); }
}

Further Research References

Aside: the specific Context Menu I’m manipulating here is sometimes called the Persona Context Menu (at least by the article Extending the User Interface in Outlook 2010). I have a feeling that will be a very useful phrases in my current research.

Other articles I’ll probably need to almost immediately:

Here are some other articles I expect I’ll need in the future:

Finally, here are some possible sources of working code that I’ve found in my travels:

Building an Outlook Add-in: customizing the Contact Card, other options?

2010-09-25T17:53:00.001-07:00

[I’m working on an app for work – enable me to pull job title, team name, etc. from a proprietary internal database, since our Exchange Address Book doesn’t contain that info. Just to set the context for what the h*** I’m up to.]

Did a broad search for “Outlook 2010 VSTO” to find some beginners’ guide to writing something that will run in Outlook 2010. Lots of possible stuff to read, but this 20-minute video seemed like my best bet:

How Do I: Create a WPF Control for Use Within Outlook 2010

Yes the question should be asked: are you better off writing an Outlook 2010 add-in using WPF or a Windows Forms? I haven’t yet found a definitive answer, and they may both be supported to some degree, but the majority of articles I stumbled across were using WPF not Windows Forms. That *may* just be a tyrannical bias coming from Microsoft, where XAML is all the rage and you’re ‘not allowed’ to talk about legacy development there anymore. Regardless, if I see WPF and Outlook 2010 mentioned more than a half-dozen times, that *has* to be supported pretty well, so I’ll go with that for now.

Contact Card

Next I need to know what kind of Outlook “object” I’m trying to attach to. I am aiming for that new intermediate Contact panel that shows up when you double-click the email address of an email in your Inbox (this example borrowed from Microsoft Support):

It already has a tabbed interface, so adding another tab makes intuitive sense as a user.
It’s the first interface you get to detailed info on a user, so it’s the fewest clicks to get to an extended UI I could add.
It’s new in Outlook 2010, which (I hope) means that the Outlook Interop object model has implemented first-class .NET support for interacting with that object. [As opposed to some of the legacy objects in Outlook for which there is pretty crappy support in the .NET Interop model.]

Searches for “Outlook 2010 new features” take me to pages like this, which refer to this new interface as “a new easy-to-access contact card”. Digging into the official Outlook blog there’s a detailed article “Using the contact card to learn who someone is”, so I’m definitely on the right track. Now if only this was the same nomenclature used in the Outlook object model (developer side of the world) – it’s rare that the underlying code and the marketing names ever get aligned, so I’m not surprised much anymore if I don’t get that fantasy fulfilled. [Heh, and just to show how easy it could’ve been, I realize now that the Contact Card is available as a context menu item when you right-click the email address.]

So my next search – Outlook 2010 “contact card” “object model” – turned up an article called “Customizing the Context Menu of a Contact Card in Outlook 2010”. Among other things, this article states that “The IMsoContactCard object is defined in the primary interop assembly, Office.dll, of the Microsoft Office object library, not in Outlook 2010.” So this is apparently an Office-wide object, available to other Office apps – not just Outlook (though it’s entirely possible Outlook is the only app that bothered).

Which also happens to lead to a Code Gallery sample called “Outlook 2010: Visual How Tos: Code Samples” that includes the sample code. Unfortuately the sample is adding a new entry to the context menu, not to the Contact Card itself (which is a mite confusing, as the object model doesn’t make a really clear distinction between the two). HOwever, this gives me a great lead on what area of the object model to focus my attention on. And worse come to worst, I can always start with a really crude hack of adding a context menu selection that just pulls up the internal directory data for the selected user (or as I’m seeing from the code sample, the “Recipient” – gotta get your nomenclature aligned when you dive into an Office object model exploration).

Next let’s see if anyone out there has been monkeying with this object or this part of the object model – searches on Stack Overflow turn up nothing, but MSDN Social Forums hits some info on:

And a search of MSDN Library for IMsoContactCard led to one non-reference article: Extending the User Interface in Outlook 2010. According to this article, adding a new item to the context menu when you right-click an email sender or recipient is done using the ContextMenuContactCardRecipient context menu.

Given that these articles all seem to say that it’s impossible to extend the contact card itself, I find myself with two alternatives:

Add a right-click menu option that pulls the internal directory info for the selected recipient.
Add another tab to the ‘legacy’ Outlook Properties window (which was the default in Outlook 2007 & 2003 when you double-click on a user).

Context Menu vs. Properties window

Comparing Option (1) and (2), I come to these benefits & drawbacks:

The programming for the Contact Card context menu was just added in 2010, which probably makes it behave more consistently and robustly than the COM-based crap that comes with legacy Outlook features.
Adding a tab to the Properties window (form) would assist me more easily if I wanted to “crawl up the address book” (i.e. look for the same information on the managers of the recipient I’m exploring). I find I *can* get to a context menu for the recipient’s manager, but it’s hellishly buried and I’d probably be one of three people who’d find it (or who’d bother taking this path).
From my recollection, Office (and Outlook in particular) can be really picky about exactly how and when to dispose of objects – generally resulting (from <100% perfect code) in deadlocks, memory leaks or difficulties in shutting down the hosting app. I would imagine the Interop Assemblies have a harder time communicating reliably with the legacy COM object model (e.g. Properties window) than with objects only recently introduced (e.g. Fluent UI).
While the Office Interop Assemblies have been incredibly forgiving about providing backwards-compatibility to all the COM objects that have been accumulated over the decades, I have to believe that Fluent UI customizations have a better future in coming versions of Office than COM-based customizations. This should be especially true of Outlook, since that team took a “wait and see” approach to the Fluent UI in the Outlook 2007 generation. If they’re still on board, they’ve benefited from the delay *and* it is likely they’re more committed than if they’d gotten burned by jumping in early.
If I’m reading this right, Office Communicator (probably 2007 R2 and later) implements support for the IMsoContactCard – so a Fluent UI approach might actually give us coverage in Outlook *and* Communicator. I don’t know how useful that really would be, but it *sounds* cool.

Thinking as an end user, I’d find another tab on the Outlook Properties window more intuitive, but I’d also be extremely unforgiving if my Outlook user experience slowed down or destabilized. I don’t like the Context Menu approach that I seem to be left with in customizing the Fluent UI, but I can be optimistic that a more integrated approach will become apparent as my research continues – and in the meantime I’ll have a Fluent UI-compatible set of code to build on.

Details: Fluent UI

All these articles I’m finding talk about these Contact Card customizations in terms of customizing the “Fluent UI”. I’m not sure, but I had believed that this Fluent UI was primarily introduced as a wrapping layer of menu/ribbon ‘cleanup’ of the Office UI that was long overdue by Office 2007. These references make it sound as if the Fluent UI is where all new UI improvements are “homed” in Outlook 2010.

As I dig a little further, there are some pretty clear indications this is true:

Customize the Office UI

In Office 2010, the Office Fluent UI is fully customizable. This includes the ribbon, the Quick Access Toolbar, and the built-in context menus. By using the flexible XML-based markup and callbacks, you can create context menus by updating Open XML Format files or by using add-ins that are created in Microsoft Visual Studio.

Customizing Context Menus in Office 2010

In Office 2010, you can customize built-in context menus just as you can the other components of the Ribbon UI. This XML-based context menu extensibility model is based on the familiar Ribbon extensibility model. This means that you can use the same XML markup and callbacks that you currently use to customize the Ribbon UI.

Encouraging, but not specifically helpful other than a lot of hand-waving and empty promises. Having clear documentation on what the object model does is the critical piece, and all I’ve got here so far is a Context menu (which is hardly an intuitive UI approach). However, if that’s what I’ve got then it’ll have to do. Off to implement code based on Customizing the Context Menu of a Contact Card in Outlook 2010 and see how well that treats me.

Silverlight training, note to self: adding non-default Assemblies for additional controls

2010-03-09T10:55:00.000-08:00

Problem: In my Silverlight training class, we were instructed to try using a TabControl in a newspaper Silverlight app. I followed part of the instructions to add the Assembly as an xmlns reference in the page code, but I couldn't get the sucker to compile. Here's the line of code I added to the page:

xmlns:my="clr-namespace:System.Windows.Controls;assembly=System.Windows.Controls"

It was giving me this error:

The type or namespace name 'TabControl' does not exist in the namespace 'System.Windows.Controls' (are you missing an assembly reference?)

Solution: (Big forehead slap) Every single time I try to get back into .NET code, I keep having to learn this ridiculously, embarrassingly simple lesson: for whatever reason, when I add a reference to a .NET assembly in my code, I also have to add a reference to the References tree in the Solution Explorer:

In the Solution Explorer, browse to the Project where this code is failing
Right-click the References folder, choose Add Reference, and add System.Windows.Controls from the .NET tab

Silverlight training, note to self: blank page

2010-03-09T10:34:00.000-08:00

Problem: Every time I try to debug a new project that's derived from training materials, I end up with a blank page. When I "View Source" on the debugging page, all I see is this:

<form id="form1" method="post" name="form1" action="default.aspx"> <div> <input id="__VIEWSTATE" value="/wEPDwULLTE2MTY2ODcyMjlkZGlFdbf1fP59jHcHgqEoRgovlZBY" type="hidden" name="__VIEWSTATE"> </div> <div> </div> </form>

Solution: need to set the default page in the Web project:

In Solution Explorer, browse the Project suffixed with ".Web"
Right-click on the .aspx page you wish to launch by default, and choose "Set As Start Page"

Lenovo T61? High CPU usage? Look for bad drivers (…spewing hardware interrupts)

2010-02-15T18:43:00.001-08:00

For a couple of months, I noticed some pretty unusual CPU utilization on my Lenovo T61 Thinkpad (running Windows XP SP3). I run Process Explorer (ProcExp) in the background all day, to keep an eye out for unusual system performance behaviour, and lately I’ve noticed that almost an entire “core” on my Intel dual-core system is consumed by something running in Kernel space.

I have ProcExp configured to display the CPU meter in the “systray” (aka notification area), and it’s really easy to know when there’s a kernel-consumer on your system – that’s when the CPU utilization graph will show a sizeable proportion of the line graph in red, rather than its more typical green (for User space consumers).

Pop open ProcExp under this condition, and if you’ve spent as much time watching these behaviour profiles as I have, and you’ll be as surprised as I was to find the “Interrupts” process (really, just an abstract aggregate) reporting CPU utilization in the 30’s or 40’s.

Summary of the solution: bad drivers, needed to be updated. In my case it was a network driver that had just been updated by the vendor; for others the culprits seem to be video drivers. I’m sure other failures exist, but these are the two most common.

Step 1: Device Manager

When hardware and/or drivers are acting up like this, check to make sure your drivers are loading properly in Windows:

Open the Device Manager control panel applet (e.g. run “devmgmt.msc”)
Look at the list to see if any categories are expanded
In every expanded category, examine the details of any device that has a yellow exclamation mark (indicating there is some problem with the driver). If at all possible, resolve the issue that’s been identified.

Until all driver issues have been resolved, any other suggestions you follow below could be moot. If you really can’t fix the problem(s) that Device Manager is presenting, but want to try other approaches, then there are a couple of tactical workarounds you may wish to try:

Install a newer (or older) version of the driver, to see if it’s just the current driver version that’s having problems.
Disable the device temporarily, and make sure you are still seeing the excessive hardware interrupts being generated for a sustained period. (If not, then perhaps that was the offending driver, and perhaps you’ll be able to run with the device disabled while the vendor addresses the issue in a later version of the driver. Lucky you.)

Step 2: KernRate

Microsoft has an interesting little package, developed for use on Windows XP, to dig into a little detail on the Kernel-related activity on the system. It’s not exactly a wizard I’d send to my mom, but it’s helped at least narrow the potential issues to track down for me.

You can download KernRate from here.

2.1 Setup & Config for a valid KernRate result

Install Windows Debugging Tools
Install KernRate
Configure valid symbols PATH variable – see this thread for details
- Create a new directory C:\Symbols
- Create a new environment variable by clicking on: Start menu > Control Panel > System > Advanced > Environment Variables > click New under "user variables for …", then type:
  - Variable name: _NT_SYMBOL_PATH
  - Variable value:
Copy DBGHELP.DLL and SYMSRV.DLL from the install location for Windows Debugging Tools (usually C:\Program Files\Debugging Tools for Windows (x86)) to the install location for KernRate (usually C:\Program Files\KrView\Kernrates)

2.2 Use KernRate to profile kernel activity

Once it’s installed and configured correctly, you can just fire up Kernrate_i386_xp.exe, let it run for a few minutes (during the period when the Kernel activity is not what you’d expect), and then hit [Ctrl]-C to get the results of the Kernrate run.

Here’s a typical output from my system during normal operating conditions (NOTE: for every Kernrate output, I’ve left out the entries from the final table that have no significant “hits” i.e. no significant volume of kernel events):

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe
/==============================\
< KERNRATE LOG >
\==============================/
Date: 2009/06/09 Time: 13:50:38
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
Starting to collect profile data

***> Press ctrl-c to finish collecting profile data
===> Finished Collecting Data, Starting to Process Results

------------Overall Summary:--------------

P0 K 0:00:11.296 (17.7%) U 0:00:06.171 ( 9.7%) I 0:00:46.328 (72.6%) DPC
0:00:00.078 ( 0.1%) Interrupt 0:00:00.031 ( 0.0%)
Interrupts= 76929, Interrupt Rate= 1206/sec.

P1 K 0:00:12.328 (19.3%) U 0:00:07.718 (12.1%) I 0:00:43.750 (68.6%) DPC
0:00:00.109 ( 0.2%) Interrupt 0:00:00.531 ( 0.8%)
Interrupts= 15687, Interrupt Rate= 246/sec.

TOTAL K 0:00:23.625 (18.5%) U 0:00:13.890 (10.9%) I 0:01:30.078 (70.6%) DPC
0:00:00.187 ( 0.1%) Interrupt 0:00:00.562 ( 0.4%)
Total Interrupts= 92616, Total Interrupt Rate= 1452/sec.

Total Profile Time = 63796 msec

                                       BytesStart          BytesStop         Byt
esDiff.
    Available Physical Memory   ,       184655872,       180600832,        -4055
040
    Available Pagefile(s)       ,      1075990528,      1069088768,        -6901
760
    Available Virtual           ,      2132107264,      2131058688,        -1048
576
    Available Extended Virtual ,               0,               0,
0

                                  Total      Avg. Rate
    Context Switches     ,       949736,         14887/sec.
    System Calls         ,      2754324,         43173/sec.
    Page Faults          ,       799583,         12533/sec.
    I/O Read Operations ,         5382,         84/sec.
    I/O Write Operations ,         4657,         73/sec.
    I/O Other Operations ,        92064,         1443/sec.
    I/O Read Bytes       ,     15853022,         2946/ I/O
    I/O Write Bytes      ,      1968451,         423/ I/O
    I/O Other Bytes      ,      9179346,         100/ I/O

-----------------------------

Results for Kernel Mode:
-----------------------------

OutputResults: KernelModuleCount = 229
Percentage in the following table is based on the Total Hits for the Kernel

Time   11604 hits, 25000 events per hit --------
Module                                Hits   msec %Total Events/Sec
ntkrnlpa                               9041      63796    77 %     3542933
hal                                    1155      63796     9 %      452614
win32k                                  604      63796     5 %      236691
mfehidk                                 179      63796     1 %       70145
intelppm                                163      63796     1 %       63875
igxpmp32                                112      63796     0 %       43889
HIPK                                     98      63796     0 %       38403
……

================================= END OF RUN ==================================
============================== NORMAL END OF RUN ==============================

And here’s the output from my system during one of these CPU over-consumption cases:

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe
/==============================\
< KERNRATE LOG >
\==============================/
Date: 2009/06/10 Time: 8:41:26
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
Starting to collect profile data

***> Press ctrl-c to finish collecting profile data
===> Finished Collecting Data, Starting to Process Results

------------Overall Summary:--------------

P0 K 0:02:26.906 (80.8%) U 0:00:21.640 (11.9%) I 0:00:13.328 ( 7.3%) DPC
0:00:01.078 ( 0.6%) Interrupt 0:01:18.406 (43.1%)
Interrupts= 3919032, Interrupt Rate= 21548/sec.

P1 K 0:02:14.000 (73.7%) U 0:00:32.703 (18.0%) I 0:00:15.171 ( 8.3%) DPC
0:00:00.640 ( 0.4%) Interrupt 0:01:02.890 (34.6%)
Interrupts= 3837550, Interrupt Rate= 21100/sec.

TOTAL K 0:04:40.906 (77.2%) U 0:00:54.343 (14.9%) I 0:00:28.500 ( 7.8%) DPC
0:00:01.718 ( 0.5%) Interrupt 0:02:21.296 (38.8%)
Total Interrupts= 7756582, Total Interrupt Rate= 42648/sec.

Total Profile Time = 181875 msec

                                       BytesStart          BytesStop         By
esDiff.
    Available Physical Memory   ,       564551680,       571240448,         668
768
    Available Pagefile(s)       ,      2194202624,      2233167872,        3896
248
    Available Virtual           ,      2132402176,      2131353600,        -104
576
    Available Extended Virtual ,               0,               0,
0

                                  Total      Avg. Rate
    Context Switches     ,      1764882,         9704/sec.
    System Calls         ,     14821228,         81491/sec.
    Page Faults          ,       188084,         1034/sec.
    I/O Read Operations ,        31768,         175/sec.
    I/O Write Operations ,        27985,         154/sec.
    I/O Other Operations ,       321733,         1769/sec.
    I/O Read Bytes       ,     24755975,         779/ I/O
    I/O Write Bytes      ,     14122756,         505/ I/O
    I/O Other Bytes      ,     31150904,         97/ I/O

-----------------------------
Results for Kernel Mode:
-----------------------------

OutputResults: KernelModuleCount = 227
Percentage in the following table is based on the Total Hits for the Kernel

Time   62240 hits, 25000 events per hit --------
Module                                Hits   msec %Total Events/Sec
ntkrnlpa                              40159     181875    64 %     5520137
hal                                   10990     181875    17 %     1510652
igxpmp32                               2381     181875     3 %      327285
HECI                                   2170     181875     3 %      298281
win32k                                 1495     181875     2 %      205498
intelppm                               1335     181875     2 %      183505
mfehidk                                 931     181875     1 %      127972
HIPK                                    660     181875     1 %       90721
Ntfs                                    361     181875     0 %       49621
usbuhci                                 285     181875     0 %       39175
PGPwded                                 223     181875     0 %       30652

……

================================= END OF RUN ==================================
============================== NORMAL END OF RUN ==============================

2.3 KernRate detailed profiling on individual modules

So what I’m seeing is (like many other folks on the ‘Net) that the NTKRNLPA module is consuming the lions’ share of the CPU (being consumed by Kernel-space threads). Following some of the threads on the Sysinternals message boards, I dug into the details of what images were being loaded by NTKRNLPA during these kernel events:

2.3.1 NTKRNLPA

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe -s 5 -z ntkrnlpa
/==============================\
< KERNRATE LOG >
\==============================/
Date: 2009/06/10 Time: 8:49:42
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe -s 5 -z ntkrnlpa

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
CallBack: Finished Attempt to Load symbols for 804d7000 \WINDOWS\system32\ntkrnlpa.exe

Starting to collect profile data

Will collect profile data for 5 seconds
===> Finished Collecting Data, Starting to Process Results

------------Overall Summary:--------------

P0 K 0:00:01.984 (39.7%) U 0:00:00.406 ( 8.1%) I 0:00:02.609 (52.2%) DPC
0:00:00.031 ( 0.6%) Interrupt 0:00:01.421 (28.4%)
Interrupts= 97650, Interrupt Rate= 19530/sec.

P1 K 0:00:02.125 (42.5%) U 0:00:00.453 ( 9.1%) I 0:00:02.421 (48.4%) DPC
0:00:00.062 ( 1.3%) Interrupt 0:00:01.406 (28.1%)
Interrupts= 95410, Interrupt Rate= 19082/sec.

TOTAL K 0:00:04.109 (41.1%) U 0:00:00.859 ( 8.6%) I 0:00:05.031 (50.3%) DPC
0:00:00.093 ( 0.9%) Interrupt 0:00:02.828 (28.3%)
Total Interrupts= 193060, Total Interrupt Rate= 38612/sec.

Total Profile Time = 5000 msec

                                       BytesStart          BytesStop         Byt
esDiff.
    Available Physical Memory   ,       563130368,       560410624,        -2719
744
    Available Pagefile(s)       ,      2230300672,      2226749440,        -3551
232
    Available Virtual           ,      2132402176,      2104180736,       -28221
440
    Available Extended Virtual ,               0,               0,
0

                                  Total      Avg. Rate
    Context Switches     ,        73005,         14601/sec.
    System Calls         ,       214633,         42927/sec.
    Page Faults          ,         3814,         763/sec.
    I/O Read Operations ,         1175,         235/sec.
    I/O Write Operations ,          433,         87/sec.
    I/O Other Operations ,         4569,         914/sec.
    I/O Read Bytes       ,       776604,         661/ I/O
    I/O Write Bytes      ,       129278,         299/ I/O
    I/O Other Bytes      ,       575190,         126/ I/O

-----------------------------
Results for Kernel Mode:
-----------------------------

OutputResults: KernelModuleCount = 227
Percentage in the following table is based on the Total Hits for the Kernel

Time   1762 hits, 25000 events per hit --------
Module                                Hits   msec %Total Events/Sec
ntkrnlpa                               1307       4984    74 %     6555979
hal                                     246       4984    13 %     1233948
HECI                                     44       4984     2 %      220706
igxpmp32                                 38       4984     2 %      190609
intelppm                                 36       4984     2 %      180577
win32k                                   33       5000     1 %      165000
mfehidk                                  15       5000     0 %       75000
HIPK                                     12       5000     0 %       60000
……

===> Processing Zoomed Module ntkrnlpa.exe...

----- Zoomed module ntkrnlpa.exe (Bucket size = 16 bytes, Rounding Down) -------
-
Percentage in the following table is based on the Total Hits for this Zoom Module

Time   1307 hits, 25000 events per hit --------
Module                                Hits   msec %Total Events/Sec
KiDispatchInterrupt                     787       4984    46 %     3947632
READ_REGISTER_USHORT                    383       4984    22 %     1921147
READ_REGISTER_ULONG                     342       4984    20 %     1715489
READ_REGISTER_UCHAR                      41       4984     2 %      205658
KeReleaseInStackQueuedSpinLockFromDpcLevel         13       4984     0 %       65208
ZwYieldExecution                         10       4984     0 %       50160

……

================================= END OF RUN ==================================
============================== NORMAL END OF RUN ==============================

2.3.2 HAL

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe -v -s 5 -z hal
/==============================\
< KERNRATE LOG >
\==============================/
Date: 2009/06/10 Time: 18:25:00
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe -v -s 5 -z hal

KERNRATE: current IMAGEHLP SymOptions: UNDNAME DEBUG

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
KERNRATE: IMAGEHLP symbol search path is: .;symsrv*symsrv.dll*c:\symbols*http://
msdl.microsoft.com/download/symbols;C:\WINDOWS\System32\Drivers;C:\WINDOWS\System32;C:\WINDOWS
CallBack: Loading symbols for hal.dll...
DBGHELP: No header for hal.dll. Searching for image on disk
DBGHELP: C:\Program Files\KrView\Kernrates\hal.dll - file not found
DBGHELP: hal.dll not found in .
DBGHELP: hal.dll not found in C:\WINDOWS\System32\Drivers
DBGHELP: C:\WINDOWS\System32\hal.dll - OK
DBGHELP: C:\WINDOWS\System32\hal.dll found
DBGHELP: .\halmacpi.pdb - file not found
DBGHELP: .\dll\halmacpi.pdb - file not found
DBGHELP: .\symbols\dll\halmacpi.pdb - file not found
CallBack: Finished Attempt to Load symbols for 806e4000 \WINDOWS\system32\hal.dll

DBGHELP: hal - public symbols
c:\symbols\halmacpi.pdb\9875FD697ECA4BBB8A475825F6BF885E1\halmacpi.pdb
Starting to collect profile data

Will collect profile data for 5 seconds
===> Finished Collecting Data, Starting to Process Results

------------Overall Summary:--------------

P0 K 0:00:02.531 (50.6%) U 0:00:00.453 ( 9.1%) I 0:00:02.015 (40.3%) DPC
0:00:00.000 ( 0.0%) Interrupt 0:00:01.890 (37.8%)
Interrupts= 119072, Interrupt Rate= 23814/sec.

P1 K 0:00:02.609 (52.2%) U 0:00:00.656 (13.1%) I 0:00:01.734 (34.7%) DPC
0:00:00.031 ( 0.6%) Interrupt 0:00:01.437 (28.7%)
Interrupts= 114272, Interrupt Rate= 22854/sec.

TOTAL K 0:00:05.140 (51.4%) U 0:00:01.109 (11.1%) I 0:00:03.750 (37.5%) DPC
0:00:00.031 ( 0.3%) Interrupt 0:00:03.328 (33.3%)
Total Interrupts= 233344, Total Interrupt Rate= 46669/sec.

Total Profile Time = 5000 msec

                                       BytesStart          BytesStop         Byt
esDiff.
    Available Physical Memory   ,       111005696,       117772288,         6766
592
    Available Pagefile(s)       ,      1635250176,      1639911424,         4661
248
    Available Virtual           ,      2132402176,      2131013632,        -1388
544
    Available Extended Virtual ,               0,               0,
0

                                  Total      Avg. Rate
    Context Switches     ,        82262,         16452/sec.
    System Calls         ,       292287,         58457/sec.
    Page Faults          ,         6431,         1286/sec.
    I/O Read Operations ,         1592,         318/sec.
    I/O Write Operations ,          506,         101/sec.
    I/O Other Operations ,         7687,         1537/sec.
    I/O Read Bytes       ,       344208,         216/ I/O
    I/O Write Bytes      ,       457023,         903/ I/O
    I/O Other Bytes      ,       951393,         124/ I/O

-----------------------------

Results for Kernel Mode:
-----------------------------

OutputResults: KernelModuleCount = 227
Percentage in the following table is based on the Total Hits for the Kernel

Time   1761 hits, 25000 events per hit --------
Module                                Hits   msec %Total Events/Sec
ntkrnlpa                               1223       5000    69 %     6115000
hal                                     258       5000    14 %     1290000
intelppm                                 74       5000     4 %      370000
HECI                                     55       5000     3 %      275000
win32k                                   32       5000     1 %      160000
igxpmp32                                 29       5000     1 %      145000
HIPK                                     22       5000     1 %      110000
mfehidk                                  15       5000     0 %       75000
……

===> Processing Zoomed Module hal.dll...

----- Zoomed module hal.dll (Bucket size = 16 bytes, Rounding Down) --------
Percentage in the following table is based on the Total Hits for this Zoom Module

Time   258 hits, 25000 events per hit --------
Module                                Hits   msec %Total Events/Sec
READ_PORT_UCHAR                         143       5000    25 %      715000
READ_PORT_ULONG                         139       5000    24 %      695000
READ_PORT_USHORT                        139       5000    24 %      695000
HalpQueryPerformanceCounter              24       5000     4 %      120000
KeReleaseQueuedSpinLock                  21       5000     3 %      105000
KeAcquireInStackQueuedSpinLock           21       5000     3 %      105000
KfLowerIrql                               8       5000     1 %       40000
ExAcquireFastMutex                        7       5000     1 %       35000
KeGetCurrentIrql                          7       5000     1 %       35000
HalpGetCmosCenturyByte                    7       5000     1 %       35000
HalpReleaseSystemHardwareSpinLock          7       5000     1 %       35000
HalBeginSystemInterrupt                   6       5000     1 %       30000
KeAcquireQueuedSpinLock                   5       5000     0 %       25000
……

================================= END OF RUN ==================================
============================== NORMAL END OF RUN ==============================

2.3.3 IGXPMP32

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe -v -s 5 -z igxpmp32
/==============================\
< KERNRATE LOG >
\==============================/
Date: 2009/06/10 Time: 18:25:11
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe -v -s 5 -z igxpmp32

KERNRATE: current IMAGEHLP SymOptions: UNDNAME DEBUG

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
KERNRATE: IMAGEHLP symbol search path is: .;symsrv*symsrv.dll*c:\symbols*">">">">">">http://
msdl.microsoft.com/download/symbols;C:\WINDOWS\System32\Drivers;C:\WINDOWS\System32;C:\WINDOWS
CallBack: Loading symbols for igxpmp32.sys...
DBGHELP: No header for igxpmp32.sys. Searching for image on disk
DBGHELP: C:\Program Files\KrView\Kernrates\igxpmp32.sys - file not found
DBGHELP: igxpmp32.sys not found in .
DBGHELP: C:\WINDOWS\System32\Drivers\igxpmp32.sys - OK
DBGHELP: C:\WINDOWS\System32\Drivers\igxpmp32.sys found
DBGHELP: .\igxpmp32.pdb - file not found
DBGHELP: .\sys\igxpmp32.pdb - file not found
DBGHELP: .\symbols\sys\igxpmp32.pdb - file not found
SYMSRV: The server name or address could not be resolved
SYMSRV: The server name or address could not be resolved
SYMSRV: c:\symbols\igxpmp32.pdb\01C98FF4DD054B6AB11509CF9CF04F171\igxpmp32.pdb
not found
SYMSRV: http://msdl.microsoft.com/download/symbols/igxpmp32.pdb/01C98FF4DD054B6
AB11509CF9CF04F171/igxpmp32.pdb not found
DBGHELP: C:\WINDOWS\System32\Drivers\igxpmp32.pdb - file not found
DBGHELP: C:\WINDOWS\System32\Drivers\sys\igxpmp32.pdb - file not found
DBGHELP: C:\WINDOWS\System32\Drivers\symbols\sys\igxpmp32.pdb - file not found
DBGHELP: C:\WINDOWS\System32\igxpmp32.pdb - file not found
DBGHELP: C:\WINDOWS\System32\sys\igxpmp32.pdb - file not found
DBGHELP: C:\WINDOWS\System32\symbols\sys\igxpmp32.pdb - file not found
DBGHELP: C:\WINDOWS\igxpmp32.pdb - file not found
DBGHELP: C:\WINDOWS\sys\igxpmp32.pdb - file not found
DBGHELP: C:\WINDOWS\symbols\sys\igxpmp32.pdb - file not found
DBGHELP: C:\WINDOWS\System32\Drivers\igxpmp32.pdb - file not found
DBGHELP: D:\ccViews\autobuild1_BR-0810-1787_14.36_Snapshot\gfx_Development\dump3
2\igfx\xp\gtune\AIM3Lib\igxpmp32.pdb - file not found
CallBack: Finished Attempt to Load symbols for b942f000 \SystemRoot\system32\DRI
VERS\igxpmp32.sys

DBGHELP: igxpmp32 - export symbols
Starting to collect profile data

Will collect profile data for 5 seconds
===> Finished Collecting Data, Starting to Process Results

------------Overall Summary:--------------

P0 K 0:00:02.812 (56.1%) U 0:00:00.265 ( 5.3%) I 0:00:01.937 (38.6%) DPC
0:00:00.000 ( 0.0%) Interrupt 0:00:01.812 (36.1%)
Interrupts= 114753, Interrupt Rate= 22879/sec.

P1 K 0:00:02.937 (58.8%) U 0:00:00.562 (11.3%) I 0:00:01.500 (30.0%) DPC
0:00:00.062 ( 1.3%) Interrupt 0:00:01.578 (31.6%)
Interrupts= 109952, Interrupt Rate= 21990/sec.

TOTAL K 0:00:05.750 (57.4%) U 0:00:00.828 ( 8.3%) I 0:00:03.437 (34.3%) DPC
0:00:00.062 ( 0.6%) Interrupt 0:00:03.390 (33.9%)
Total Interrupts= 224705, Total Interrupt Rate= 44941/sec.

Total Profile Time = 5000 msec

                                       BytesStart          BytesStop         Byt
esDiff.
    Available Physical Memory   ,       119107584,       107962368,       -11145
216
    Available Pagefile(s)       ,      1640800256,      1629646848,       -11153
408
    Available Virtual           ,      2132402176,      2102800384,       -29601
792
    Available Extended Virtual ,               0,               0,
0

                                  Total      Avg. Rate
    Context Switches     ,        78959,         15792/sec.
    System Calls         ,       301466,         60293/sec.
    Page Faults          ,         8704,         1741/sec.
    I/O Read Operations ,          711,         142/sec.
    I/O Write Operations ,          197,         39/sec.
    I/O Other Operations ,         5430,         1086/sec.
    I/O Read Bytes       ,      1212541,         1705/ I/O
    I/O Write Bytes      ,        14701,         75/ I/O
    I/O Other Bytes      ,      1486190,         274/ I/O

-----------------------------

Results for Kernel Mode:
-----------------------------

OutputResults: KernelModuleCount = 227
Percentage in the following table is based on the Total Hits for the Kernel

Time   1764 hits, 25000 events per hit --------
Module                                Hits   msec %Total Events/Sec
ntkrnlpa                               1232       5000    69 %     6160000
hal                                     235       5000    13 %     1175000
intelppm                                 93       5015     5 %      463609
HECI                                     42       5015     2 %      209371
win32k                                   34       5000     1 %      170000
igxpmp32                                 32       5015     1 %      159521
mfehidk                                  23       5000     1 %      115000
HIPK                                     21       5000     1 %      105000
kmixer                                   11       5000     0 %       55000
KSecDD                                   10       5015     0 %       49850
……

===> Processing Zoomed Module igxpmp32.sys...

----- Zoomed module igxpmp32.sys (Bucket size = 16 bytes, Rounding Down) -------
-
Percentage in the following table is based on the Total Hits for this Zoom Module

Time   32 hits, 25000 events per hit --------
Module                                Hits   msec %Total Events/Sec
xc_copp_agent                            30       5015    100 %      149551

================================= END OF RUN ==================================
============================== NORMAL END OF RUN ==============================

2.3.4 HECI

C:\Program Files\KrView\Kernrates>Kernrate_i386_XP.exe -v -s 5 -z heci
/==============================\
< KERNRATE LOG >
\==============================/
Date: 2009/06/10 Time: 18:48:17
Machine Name: MIKESL-MOBL1
Number of Processors: 2
PROCESSOR_ARCHITECTURE: x86
PROCESSOR_LEVEL: 6
PROCESSOR_REVISION: 0f0b
Physical Memory: 2007 MB
Pagefile Total: 3856 MB
Virtual Total: 2047 MB
PageFile1: \??\C:\pagefile.sys, 2006MB
OS Version: 5.1 Build 2600 Service-Pack: 3.0
WinDir: C:\WINDOWS

Kernrate User-Specified Command Line:
Kernrate_i386_XP.exe -v -s 5 -z heci

KERNRATE: current IMAGEHLP SymOptions: UNDNAME DEBUG

Kernel Profile (PID = 0): Source= Time,
Using Kernrate Default Rate of 25000 events/hit
KERNRATE: IMAGEHLP symbol search path is: .;symsrv*symsrv.dll*c:\symbols*">">">">">">http://
msdl.microsoft.com/download/symbols;C:\WINDOWS\System32\Drivers;C:\WINDOWS\System32;C:\WINDOWS
CallBack: Loading symbols for HECI.sys...
DBGHELP: No header for HECI.sys. Searching for image on disk
DBGHELP: C:\Program Files\KrView\Kernrates\HECI.sys - file not found
DBGHELP: HECI.sys not found in .
DBGHELP: C:\WINDOWS\System32\Drivers\HECI.sys - OK
DBGHELP: C:\WINDOWS\System32\Drivers\HECI.sys found
DBGHELP: .\HECI.pdb - file not found
DBGHELP: .\sys\HECI.pdb - file not found
DBGHELP: .\symbols\sys\HECI.pdb - file not found
SYMSRV: The server name or address could not be resolved
SYMSRV: The server name or address could not be resolved
SYMSRV: c:\symbols\HECI.pdb\0A57FC20FA904FB8BAEC77C53EE39C821\HECI.pdb not found
SYMSRV: http://msdl.microsoft.com/download/symbols/HECI.pdb/0A57FC20FA904FB8BAE
C77C53EE39C821/HECI.pdb not found
DBGHELP: C:\WINDOWS\System32\Drivers\HECI.pdb - file not found
DBGHELP: C:\WINDOWS\System32\Drivers\sys\HECI.pdb - file not found
DBGHELP: C:\WINDOWS\System32\Drivers\symbols\sys\HECI.pdb - file not found
DBGHELP: C:\WINDOWS\System32\HECI.pdb - file not found
DBGHELP: C:\WINDOWS\System32\sys\HECI.pdb - file not found
DBGHELP: C:\WINDOWS\System32\symbols\sys\HECI.pdb - file not found
DBGHELP: C:\WINDOWS\HECI.pdb - file not found
DBGHELP: C:\WINDOWS\sys\HECI.pdb - file not found
DBGHELP: C:\WINDOWS\symbols\sys\HECI.pdb - file not found
DBGHELP: C:\WINDOWS\System32\Drivers\HECI.pdb - file not found
DBGHELP: d:\ccviews\autobuild1_br-0707-1049_2.6_snapshot\amt_development\sw\src\
drivers\heci\driver\objfre_wxp_x86\i386\HECI.pdb - file not found
CallBack: Finished Attempt to Load symbols for ba308000 \SystemRoot\system32\DRI
VERS\HECI.sys

DBGHELP: HECI - no symbols loaded
Starting to collect profile data

Will collect profile data for 5 seconds
===> Finished Collecting Data, Starting to Process Results

------------Overall Summary:--------------

P0 K 0:00:02.296 (45.9%) U 0:00:00.343 ( 6.9%) I 0:00:02.359 (47.2%) DPC
0:00:00.000 ( 0.0%) Interrupt 0:00:01.359 (27.2%)
Interrupts= 115350, Interrupt Rate= 23070/sec.

P1 K 0:00:02.203 (44.1%) U 0:00:00.531 (10.6%) I 0:00:02.265 (45.3%) DPC
0:00:00.015 ( 0.3%) Interrupt 0:00:01.093 (21.9%)
Interrupts= 110550, Interrupt Rate= 22110/sec.

TOTAL K 0:00:04.500 (45.0%) U 0:00:00.875 ( 8.8%) I 0:00:04.625 (46.3%) DPC
0:00:00.015 ( 0.2%) Interrupt 0:00:02.453 (24.5%)
Total Interrupts= 225900, Total Interrupt Rate= 45180/sec.

Total Profile Time = 5000 msec

                                       BytesStart          BytesStop         Byt
esDiff.
    Available Physical Memory   ,       116039680,       114077696,        -1961
984
    Available Pagefile(s)       ,      1642790912,      1640251392,        -2539
520
    Available Virtual           ,      2132402176,      2105364480,       -27037
696
    Available Extended Virtual ,               0,               0,
0

                                  Total      Avg. Rate
    Context Switches     ,        91187,         18237/sec.
    System Calls         ,       270278,         54056/sec.
    Page Faults          ,         6482,         1296/sec.
    I/O Read Operations ,          269,         54/sec.
    I/O Write Operations ,           41,         8/sec.
    I/O Other Operations ,         5361,         1072/sec.
    I/O Read Bytes       ,      1551742,         5769/ I/O
    I/O Write Bytes      ,         2836,         69/ I/O
    I/O Other Bytes      ,      2381876,         444/ I/O

-----------------------------
Results for Kernel Mode:
-----------------------------

OutputResults: KernelModuleCount = 227
Percentage in the following table is based on the Total Hits for the Kernel

Time   1754 hits, 25000 events per hit --------
Module                                Hits   msec %Total Events/Sec
ntkrnlpa                               1273       5000    72 %     6365000
hal                                     222       5000    12 %     1110000
intelppm                                 70       5000     3 %      350000
HECI                                     47       5000     2 %      235000
igxpmp32                                 34       5000     1 %      170000
win32k                                   30       5000     1 %      150000
mfehidk                                  14       5000     0 %       70000
HIPK                                     12       5000     0 %       60000
KSecDD                                   12       5000     0 %       60000
……

===> Processing Zoomed Module HECI.sys...

Symbol Enumeration failed (or no symbols found) on module HECI in CreateZoomedModuleList, Error Code= 0x000001e7
No Hits or No symbols found for module HECI.sys
================================= END OF RUN ==================================
============================== NORMAL END OF RUN ==============================

Step 3: Analysis & Research

In my case, it appears that there’s some process (or processes) that’s calling the RtlIpv6StringToAddressA kernel API an excessive number of times. From what I’ve seen in my research, not only is this a common complaint, but it’s been reported by at least two other T61 users in the last year or so – without any definitive solution.

And what’s this RtlIpv6StringToAddressA API do exactly? Well, it translates a string form of an IPv6 address to a more strongly-typed form of the address. Here’s what I know (and what I don’t) about IPv6 in my T61 and Windows XP SP3:

The “teredo” driver hasn’t been installed in Windows XP SP3 (“teredo” is the code name used by the Microsoft Windows team that developed the IPv6 stack)
There are a number of third-party packages on my system, but the newest to have been introduced are:
- the AMT 2.6-supporting software and driver (v “2.6.30.1014 built by: WinDDK”, driver date 7/12/2007)
- the PGP whole disk encryption package
- the Intel 4965AGN wireless driver (v12.4.0.21), installed earlier
- the Intel PROSet/Wireless Wifi Software (v12.1.1.0), installed earlier
Knowing that AMT is meant to provide a remote networking interface to Intel chipset functionality, and knowing that this problem didn’t occur before the AMT & PGP installs took place, my primary suspect (as the source of these excessive IPv6-related API calls) becomes the AMT software.
However, I consulted with colleagues on my team that are intimately familiar with the internals of HECI.sys, and they have confirmed that it doesn’t call the RtlIpv6StringToAddressA() Win32 API
That prompted me to search the strings embedded in all the drivers and libraries I could find in C:\Windows\System32. Here were the files where FINDSTR.EXE could find that API call:
- dnsapi.dll
- mswsock.dll
- ntdll.dll
- ntkrnlpa.exe
- ntoskrnl.exe
- Drivers\http.sys

Disabled the following Devices (which didn’t help):

Ricoh Memory Stick Controller
Ricoh SD/MMC Host Controller
Ricoh xD-Picture Card Controller
OHCI Compliant IEEE 1394 Host Controller
ThinkPad Modem
Ricoh R/RL/5C476(II) or Compatible CardBus Controller
SDA Standard Compliant SD Host Controller

Downloaded and installed the following driver updates:

bluetooth v5.5.09.6400
Ultranav v11.1.21.2
Power Manager v1.60
ACPI Power Management v1.53
Integrated camera v5.8.49004.0
Hotkey driver v2.20.0001

Step 4: Make sure you try later and earlier driver versions

Bottom line: I replaced the network driver on my system with one that appeared *months* after I started down this road, and I finally stopped seeing these CPU spews occur.

I was lucky – the vendor dropped a new driver version that happened to resolve this error. Until that point, I was stuck – nothing I could do to my OS to resolve it.

And if your vendor hasn’t released a *later* driver that eliminates the issue, I’ll recommend something radical – try an *earlier* version of the driver (e.g. like these folks here). Sometimes programmers introduce errors that didn’t exist in previous versions of their code, even if they didn’t intend to affect such a broad issue. (Especially when they didn’t intend to.) You might find a 6-, 12- or 24-month-old driver that still works well enough (for the time being) and doesn’t exhibit the performance drag you’re seeing. What’s worse – an older driver that works but doesn’t have the latest features, or a driver that has the features but also craters your system performance?

Last resort: you may find that disabling the device in the OS will finally shut up a misbehaving driver (i.e. that the driver won’t load, so it won’t spawn all these excessive interrupts). This isn’t a great long-term solution, and it’s only viable for hardware you don’t need to use (e.g. a biometric device such as fingerprint reader), but it’s one last thing to consider before junking your whole system.

Notes: related articles

Sysinternals Forums

Lenovo Forums

My Mac Mini HTPC saga: TV tuners

2009-10-17T21:54:00.001-07:00

Do I really need a tuner?

One of the major hang-ups I had – in making the switch from Windows Media Center to a Mac Mini – was worrying whether I’d be able to plug in enough TV tuners into the Mac Mini to be able to record all the TV I usually watch.

However, after having been *without* an HTPC for almost a year, it’s amazing how little I feel like I *must* get record whatever current-run TV shows are being played. What with the availability of almost every TV season on Netflix instant watch, or a full season on DVD, or (theoretically) a quick download via bittorrent, there’s almost no sense of urgency left to the act of watching TV, and if we happen to miss it – oh well, I’m sure I’ll be able to get it real soon. I’ve caught up on a year’s worth of The Big Bang Theory just recently, and I’m about to re-orient myself to two years’ worth of Supernatural (so I have the possibility of watching current-run episodes within the next few weeks).

All that said, I can’t quite bring myself to give up my ability to accumulate a “taped” copy of the shows that were broadcast to me, for later viewing when I have more time. Breaking the cycle of dependence between me and the timeslot the broadcaster decides to air the show is a powerful mental leap to make, and makes me feel less like a prisoner of the networks (and their increasingly hostile attitude towards their audience). If only to make myself feel like I have a modicum of control over my TV habits, I’ve decided to drop another $200 on an Elgato EyeTV 250 Plus tuner.

Why that brand and model? Couple of reasons:

My research into Mac-based HTPC’s and Mac-compatible tuners seems to mention the Elgato products more than any other brand. It feels like what I saw with Hauppauge for Windows Media Center, and if my satisfaction with the Hauppauge products is any indication, the Elgato products should be good.
It’s not just the hardware – apparently the EyeTV software is the best available for recording and managing playback of recordings on the Mac.

Finding all the digital channels available to be recorded with your tuner

If you’re going the OTA route, then this seems like a great way to see what all is available without a prescription (ha – I mean, subscription):

http://www.equinux.com/us/products/tubestick/tuningmap.html

If you’re stuck (like me) with wanting to get the few remaining channels that haven’t gone the full-streaming/Hulu/TV.com type of approach, and you *need* that subscription to bastards like Comcast, then your tuner has one of two uses:

Analog tuner (receiving the analog output from a DTA box), in which case be prepared to buy an “IR blaster” (device that allows your HTPC to direct “change channel” commands to the IR receiver of the DTA box). This way, you get all the channels to which you’ve subscribed – whether the cable company is encrypting those QAM signals or not.
Digital tuner (receiving whichever unencrypted (aka “clear QAM”) channels your cable company hasn’t yet obfuscated with the 56-bit DES encryption that (so far) is only supportably-decryptable by the DTA and/or full cable box solutions they hook up on your behalf). This way, you get all the channels your cable company hasn’t yet encrypted – usually including the “extended basic” channels, and probably a few others that might not be high on their list of “must-have” channels.

The EyeTV software that comes with Elgato products includes a feature called “Exhaustive Scan”, which picks up the digital channels that aren’t sent out on the typical frequencies that are publicized by Comcast. I was able to find about 80% of the non-local channels that I normally watch (and a bunch that I’ll *never* watch, no matter *how* high-def they are). However it took me most of the afternoon to map those channels out.

Be careful though: I don’t know why, but somehow all the configuration that I did to map those channels went up in smoke the next time I rebooted the Mini. Now maybe the system crashed before shutting down EyeTV – but is this software *really* so fragile that it doesn’t cache these settings as soon as they’re configured? And make sure that the file that stores these settings doesn’t get wiped out by an errant half-open write operation or something?

Future question:

Is there a similar Plex plug-in for EyeTV as is available for Boxee?

What do you like? Your ratings in multiple places – do you want to sync them?

2009-10-08T23:54:00.001-07:00

The modern “what do you like?” systems are driving me nuts lately. Every time I turn around, there’s a website offering to help me socialize, and what better way to find new friends (and new recommendations for media to consume) than by accumulating a bunch of ratings for the media I’ve already consumed?

Friends (and hopefully, to some degree) passers-by can then peruse my virtual “media shelves”, see what I’ve rated and how, and either (a) get or (b) give recommendations for stuff that’s related to what they/I have already seen/read/heard and liked.

Web 1.0: Amazon Recommendations

I’ve got > 1000 ratings for graphic novels accumulated on Amazon.com – which was a great way for years for me to get recommendations on other books by the same authors, but even more importantly and gratifyingly – to get recommendations on other books by authors I hadn’t yet read. (I’ve always assumed some sort of Bayesian analysis that results in “people who liked your 4- and 5-star books also bought/owned/rated these items”.)

[I’d further tried to accumulate lists of ratings for music (for CDs I’ve purchased), but Amazon’s interface for this wasn’t nearly as sophisticated or predictive for music as it seems to be for books, so I’ve never gotten quite the same gratifying experience for music on Amazon.com.]

Web 1.0: Netflix Recommendations

I’ve got > 1000 ratings for movies accumulated on Netflix.com – which was a great way for years for me to get recommendations on other movies I’d like that have many of the same ephemeral qualities I enjoyed in the movies I rated most highly. (Netflix has had a highly-publicized contest – which recently wrapped up – to come up with new ways of improving user recommendations, which to me meant that they’d exhausted all the available research into Bayesian and other mathematical analysis of the huge aggregations of data on what people liked, didn’t like, watched and marked “not interested”.)

Long before I was a Netflix subscriber though, I was (and am) a diehard advocate for IMDB.com. [Hell, I was one of the lunatic adopters in the early days back when you had to submit queries to the IMDB via email. Yeah, imagine browsing your favourite actor’s movies [and forget about TV – that didn’t count] by submitting a cryptically-formed email message and waiting the minutes it took for their servers to generate a response. Fred Flintstone-style browsing.] I’ve occasionally submitted a rating for movie through that site too, though I haven’t gotten any real benefit from it (except the knowledge that I’m helping to build the geek-slanted ratings that are the killer data set available from IMDB).

Web 2.0: social + recommendations

In the past year or so, I’ve fallen deeply in love with Facebook, Twitter and all the most interesting integrations with these “social platforms”. With these platforms have come brand-new applications that allow you to rate movies/TV/books/music/whatever and not only get some kind of recommendations back from “the system”, but also to get much more specific and immediate feedback from those of your friends (or even “friends”) who’ve also signed up to use the application. They see what you’ve rated, then respond with comments/replies/their own ratings, and can make much more specific (and personal, though statistically less predictive) suggestions of other stuff they want you to see/read/hear.

I love these – and while I’ve experimented with a bunch of these apps, I’ve gravitated to those apps that appear to have the greatest critical mass. Not so much because I want a horde of strangers to help me find stuff, but because I’d like to reconnect with as many friends as possible and I hope they’re also at the apps I’ve picked.

So I’ve got ratings slowly accumulating at Goodreads.com, Flixster and a couple of others. Goodreads has a great mobile site that makes it dead-easy to post a rating “on the go” with very few excess clicks, and the Flixster iPhone app is awesome *and* easy. And there are dozens of other great sites where lots and lots of people are accumulating lots and lots of ratings data.

Problems: stale data, incomplete data, spread-too-thin efforts

After a while, I’ve noticed I’m spread thin across multiple places where these ratings are being accumulated. It’s an unfortunate consequence of the abundance of such great sites and platforms, that I’m finding it hard to keep my ratings “in sync” between multiple places at once. I have good intentions – and occasionally I’ll even follow through on those good intentions. :)

For example, I’ve got a ton of movie ratings in Netflix, but my primary interest for “sharing” movie ratings is moving to the Flixster app – mostly because it gives me a chance to get immediate feedback from a larger group of friends who catalogue their ratings and mini-reviews there via either Facebook or the iPhone app. However, while I’m getting immediate gratification for my posts to Flixster, it’s not doing me any good in terms of system-generated (Bayesian) recommendations for other movies I want to watch. And when I go to my Netflix queue to add movies, I sometimes forget whether I’ve seen something (since I haven’t always rated those movies I’ve seen recently).

Similarly, the primary place I currently capture my ratings for graphic novels is in GoodReads (usually via the mobile-optimized web site that I access from my iPhone). I’m not even getting any instant feedback from friends there, nor have I found any way to use the “crowd” of GoodReads users as a source for new recommendations. However, there’s no alternative in my Web 1.0 world: Amazon doesn’t seem to have any way to add ratings on the go. If you’re not going through their full browser, then you’re SOL. (The Amazon iPhone app doesn’t do squat here, and neither does the mobile browser version – it’s almost as if they don’t care whether their customers like what they bought.)

I’m now split between worlds, and I suspect the world of Facebook/social media and other Web 3.0 apps will only make this worse – there’ll be more and more sites that all want you to provide some “sticky” information, that lures in more users ‘cause there’s a “crowd” there, and yet those ratings won’t be re-usable elsewhere.

This Ratings Data Ain’t Portable, My Friends

Yeah. Twitter might have finally gotten religion that you “own” your Tweets, and Facebook *looks* like they’re convinced that you can and should have the ability to control your personal information/updates, but good luck trying to convince the thousands of little start-up apps out there, all hoping to lure you into their little walled garden and *keep* you there.

I expect that in 5-10 years, all these systems will be able to freely consume and re-use this data – the business world will have finally gotten over thinking this is their only “value add” (and will have found some even more sexy way to separate you from your money).

However, for the forseeable future, these multiple ratings systems will continue to live as non-interoperable data islands. That means heavy “data generators” like me will have to make some pretty dopey (and unavoidable) decisions:

When you find a new, even-more-attractive place to catalogue your consumption and how much you enjoyed it, do you abandon all the invested effort you put into the last one?
If you don’t want to abandon all that “legacy data”, how will you migrate it to the new system? Just devote a freakin’ weekend to the prospect of clicking like a spastic lab rat, replicating each rating from one system to the other? Or do you go even further down the rabbit hole and learn how to export the data (if that’s supported) from one and import to another – or go completely over the cliff edge into writing yourself some web-scraping scripts that pull the data by force out of systems that don’t have a supported import/export interface (API)?
And if you actually *want* to maintain a presence in more than one system – e.g. if you find some ongoing benefit in having current presence in both Netflix and Flixster? Well gods help you then – you’re screwed into a life of regular repeated self-inflicted punishment.

I’ve really lost my mind – thinking there’s got to be some way to actually pull off (3) without feeling like a character in a Kafka novel (and no, I’ve never read Kafka, so don’t crucify me for a misspoken cultural reference).

How’s About a Ratings Sync App?

Yeah, why the hell not? Why not just burn the next years’ worth of weekends writing an extensible framework for us to be able to download, manipulate, upload and synchronize (i.e. manage and resolve the inevitable conflicts) the ratings data? Isn’t is just like me, to think of doing some thankless job like this, in the hopes that some morsel of thanks comes through from some other hapless geek like me?

Sure, what the hell.

In fact, I’ve invested a whole bunch of time into this harebrained notion already. Yes, I’ve written myself a bunch of code that attempts to provide an extensible, pluggable framework in which multiple “ratings” providers could be wired in, and between which synchronization could occur.

I actually dream that one day, users like me could:

Fire up this app
Connect to one of their ratings aggregating web sites
Download all ratings for whatever “things” are rated on that site
Select another compatible ratings web site (e.g. another books-rating web site if you just download book ratings)
Configure a translation between the two web sites (e.g. one site rates 1-5, the other site rates 1-10, so map 1 = 2, 2 = 4, 3 = 6, 4 = 8, 5 = 10; or if you’d prefer, 1 = 1, 2 = 3, 3 = 5, 4 = 7, 5 = 9)
Upload the translated ratings to the second site, thus synchronizing your ratings from one site to the next

This’ll require mapping out the APIs for each ratings-aggregating site, implementing an incredible flexible and robust local schema for the data, and figuring out all the different ways that different sites identify what to the human mind is an easy-to-identify product.

[And to think, this whole idea came about as a way to figure out how to migrate the ratings from my old Netflix account (to which I’ve still got access, but only just barely – due to the grace of an old housemate) to a new one that *I* own, and from which I could actually do Netflix Instant Watch in my own personality (and with my own IW queue). I’m seriously considering just paying them for their subscription for the rest of my life, so I never have to lose those 2956 movie ratings.]

Anyone out there got a better idea?

Anyone else crazy enough to want to help out with this?

Anyone out there want to see this app see the light of day?

My Mac Mini HTPC saga: software & configurations

2009-10-04T14:02:00.001-07:00

Now that I’ve done the whole hardware upgrade and full, clean install of Mac OS X 10.6, I have to start all over again (this time with a little practice under my belt) on getting all the stuff assembled for a well-oiled home theatre machine.

Plex, Boxee, EyeTV, VLC, Transmission, Silverlight, MacTheRipper

Lots of software, so little attention span. :)

Plex

As my chosen media front-end, I intend to do as much as possible from here, and only veer into the other apps I’m using when necessary
I choose to install the following apps: Hulu, Apple Movie Trailers, South Park, Netflix, The Daily Show, Pandora, PBS, Picasa Web, Trailer Addict, YouTube

Boxee

I hear this is the only way to get full streaming of the CBS TV shows in a 10’ UI experience (rather than fire up Safari/Firefox and click away directly)
So I logged into Boxee.tv, downloaded the Mac OS X alpha of Boxee (which is simply yet another XBMC fork/port) and started looking at the “Applications” – Videos > CBS > Full Episodes shows listings for How I Met Your Mother (which I watch) but not for The Big Bang Theory (which I *slavishly* watch)

CBS: Fail.

EyeTV

Once I decided to definitely get a tuner (more on that in another blog article), the overwhelming number of times I’ve heard Elgato’s products recommended (not just their tuner, but their pretty great EyeTV software app to manage the tuner) was the clincher
The trickiest part was upgrading EyeTV from 3.0.x to 3.2 (which just came out a few days ago). It turns out that on systems running Mac OS X 10.6, EyeTV 3.0.x won’t even launch – Apple in the infinite wisdom intentionally put a “block” in place so that it won’t start.
It turns out there’s an “under the covers” way to bypass such Mac OS X “blocks”, by launching the application’s actual binary file, rather than using the “user friendly shortcut” that is presented in the Applications folder (and which I, like most Mac users, seem to happily use until something prevents us from getting in the easy way).
I found an article at Elgato’s site that outlined the process: http://support.elgato.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=3628
Once I got EyeTV 3.0.x (I think it was 3.0.3 that shipped with my EyeTV 250 Plus tuner) running, I was able to use the EyeTV “Check for Updates” menu option to get the 3.2 download and be back working like a charm.

VLC Media Player

I remembered that VLC was the easiest way to get access to all the significant codecs, and have a nice media player in the background in case any of the rest of ‘em weren’t working out for me.
Downloaded, installed, and already confirmed that VLC can play VOB files that were copied directly off a DVD (for backup purposes, naturally – who wants to scratch their only copy of Robot Chicken?)

Transmission BitTorrent client

Ever since hearing about PeerGuardian for Windows (a piece of software that prevents your computer from connecting to “blacklisted” IP addresses – e.g. those servers setup by the RIAA, government agencies and others to track what you’re doing with your downloads and other torrent-like activity), I’ve been dreading the conversion over to my Mac Mini
Then somehow today, my searches through Lifehacker’s archives for “p2p file sharing ‘mac os x’” turned up an article from 2008 on Transmission – a BitTorrent client for Mac OS X and Linux.
It turns out that the friendly and nice folks at Transmission have integrated the PeerGuardian functionality into their app, including automatic downloads of the Bluetack blocklists
That’s enough for me – I’ve already got Transmission installed and slurping down some great media for later watching.

Silverlight 3.0 (for Netflix)

Silverlight is necessary for playing the Netflix Instant Watch streaming movies, and it doesn’t seem to get installed “in the background” by Plex, Boxee or any of the other media-front-end apps that provide a Netflix wrapper
Makes sense now that I think about it, but I can’t say I wasn’t disappointed by the fact that the Mac software community hasn’t made all this stuff entirely hidden from my relatively novice eyes.

MacTheRipper

Great little app for making backup copies of your DVDs onto a local hard drive. Essential in this day and age.

Issue: Mac OS X 10.6 intercepts all Apple Remote commands

As of OS X 10.6, the operating system itself intercepts all Apple Remote commands, and cannot be overridden by application-specific configurations (as was apparently the case with Plex and probably others in the past). Instead, hitting the Menu button on the Apple Remote will always bring up Front Row; hitting the volume buttons will always change the system-wide volume, and other irritating effects (for an HTPC user who’s using third-party HTPC-oriented software) are seen as well.

While there’s no supported, by-design way in OS X to disable this “feature”, there are some known workarounds – which I’ve implemented:

http://forums.plexapp.com/index.php?showtopic=8658

edited the /System/Library/LaunchAgents/com.apple.RemoteUI.plist file (after making a copy of the file in the same directory, in case I later want to revert back to the original settings)
deleted the /System/Library/LoginPlugins/BezelServices.loginPlugin (after making a copy of the file – ibid)

http://www.tech-recipes.com/rx/2754/os_x_edit_file_using_textedit_as_root_superuser/

I had to use this article’s cluefulness to allow me to interactively edit the RemoteUI.plist file using TextEdit.

http://support.apple.com/kb/TA25121

I also got bit by the “blank admin password” problem – wasn’t able to fire up sudo when my current account had a blank password. So I changed it, did the sudo-enabled edit, and changed the password back.

Next Steps…

Wondering – do I really *need* a TV tuner, to enjoy the world’s best offerings of TV and other media?

My Mac Mini HTPC saga: upgrading hardware, optimizations

2009-09-30T16:55:00.001-07:00

Replace RAM, Hard drive with beefier options I bought myself

I wanted to have the maximum available ‘headroom’ in this box before I committed a lot of time installing and configuring lots of software (e.g. if I wanted to run software-driven encoding, have multiple big apps running at the same time, or even to run a virtualized instance of Windows whenever I felt the ‘itch’).

So after reading about what some of the braver souls have done under the hood, I decided that I’d purchase the lowest-end Mac Mini (with the exception of getting the fastest processor, which aren’t upgradeable AFAIK) and then purchase 4 GB of RAM & a 320 GB 7200 RPM SATA drive.

4 GB of RAM (or 8?)

While deciding on a brand of RAM is usually a decision with way too many options, on what should be but sometimes isn’t a commodity purchase, I cheated. When browsing around Amazon.com for what kinds of prices they had available for Mac Mini’s, I noticed their “what other people purchase with this item” was consistently coming up with one package of Corsair 2 x 2GB DDR3 (PC3-8500) RAM – this one, for (at the time I purchased)

Note: I’ve done as much reading as I could about the new 64-bit capabilities unlocked by Mac OX 10.6 (Snow Leopard), and while there’s promising speculation that the Mac Mini could take 2 x 4GB RAM (i.e. 8GB total), the reality is the price of the 4GB PC3-8500 modules is astronomical compared to 2GB modules. I’ll keep that in mind for the future, but for now the 4GB is plenty of extra capacity for now, and I can keep the extra couple hundred dollars for something else.

Which Hard Drive?

Over the decades, different hard drive manufacturers have produced drives of higher or lower quality, and I can never keep up with “which manufacturer is the king for each drive size and capacity”. For this I trundled down to my local PC supply shop with two specifications in hand: it has to be a 2.5” drive (the bigger ‘desktop’ drives won’t fit) and it has to be a 7200 RPM drive (my reading suggests the faster drive speeds make a big difference in a Mac Mini HTPC).

The dude behind the counter was very definitive – according to him, the most reliable and best-performing 2.5” 7200 rpm drive on the market today is from Western Digital (aka the “Scorpio Black” line of drives).

If you’re looking at other brands, and you’d like to know if other Mac Mini owners have been successful upgrading with them, I found a really useful resource that could help: “Mac Drive Upgrades/Compatibility Database” at http://forums.xlr8yourmac.com/drivedb/search.drivedb.lasso

Procedure to Open the Mac and Upgrade the Components

Search Google for “Mac Mini RAM upgrade” and/or “Mac Mini hard drive upgrade” – there are many step-by-step guides available (e.g. this one), and a number of great YouTube videos that take you visually through the process. I just followed the first couple that I found and everything seemed to work fine (at least, for this “old hat” at constructing my own PCs for so many years).

For me, the only tools I needed were:

a putty knife or scraper blade (anything thin, metal and long enough to wedge in a few inches into the Mac Mini case)
a Philips (the “x” kind) screwdriver that is thin enough to get into a very narrow plastic tube (eyeglasses screwdrivers are usually good for this), and has a small/thin enough head that it can unscrew (and not strip) the screws with the very narrow “x” channel.
- As some pointed out, magnetizing the screwdriver head is very handy for extracting those tiny little, fidgety screws – otherwise they tend to drop into the Mac Mini guts, and then you’re shaking it hard enough to get it loose (but not too hard so that you risk “damaging” some microscopic parts – not that I think there is any real danger of this with what I saw, but even an “old hat” gets worried there’s some new level of miniaturization that makes some part extremely sensitive to impact from even a 1/4-ounce screw.

Remember: Format your new Drive

Oh, and don’t forget to format the drive using Disk Utility (accessible from the top menu bar after the Mac OS X installer has asked which language you want to install). I went through the install twice, both times wondering why the new 320GB disk wasn’t available as a selection for “where do you want to install Mac OS X". I figured that the installer was so smart that it recognized an unformatted disk, and would automatically format it for me. (Soon, perhaps, but not at present.) Embarrassing, but easily corrected – once I realized that I *had* checked all connections twice, and there was really very little that *could* prevent the drive from being recognized at the physical level.

Reinstalling Mac OS X and the rest of the bits from the two DVDs

This part was a freakin dream – so few questions, everything possible automated – it’s like Apple realizes that most people don’t care to fiddle, they just want the sucker to work.

The only extra steps I had to take were installing the optional bits (mostly utilities, but also Safari, Mail, iCal and some other useful things) from the OS X install DVD, and then installing whatever comes on that DVD labelled “”. (Heck, it only consumed 4GB of my new 320GB drive, and until I know more about what I *don’t* need, I’m pretty comfortable taking 4GB “just in case”.)

Misc optimizations:

enable SMB sharing in Mac OS X, so that Plex (or Boxee) can find and scour any locally-attached external drive for photos, music and video. (Apparently the XBMC codebase always expected to find all media on a network-shared device, since the original XBox hardware itself couldn’t retrieve or store that much media locally on the puny 20GB XBox drive.)
Add an entry point in the Front Row menu for Boxee – does this work for Plex as well?
Added link to 10-foot-UI-friendly version of my Google Reader account (via http://mymediaexperience.com/feeds-as-newspaper-in-your-living-room/)
Something about reconfiguring audio for Plex: “As for setting up Plex with a DTS or DD receiver. Go Configure The System -> System -> Audio Hardware from the menu. Then change the audio output from analog to digital. ” (http://www.123macmini.com/forums/viewtopic.php?t=20440)

My Mac Mini HTPC saga: filesystem selection for external media drive

2009-09-17T22:57:00.001-07:00

Quest: best filesystem for my external hard drive

I have a 750 GB Seagate FreeAgent drive, where I’ve kept most of my media (photos, music, video, backups) for the past couple of years. Until now, I was operating in an all-Windows world of my own design, and NTFS was the best (most reliable, performant, flexible) filesystem for that drive.

Now that I’ve introduced a Mac into my life, and because I’m planning to plug it into the Mac Mini for the vast majority of the time (to host media I record from or wish to display via the Mini) I’m faced with what feels like an imperfect choice:

FAT32 doesn’t support > 4 GB filesize, which in this day and age of recorded/downloaded TV (not to mention the potential for virtualization in my future, in case I want to keep experimenting/developing) is damned easy to exceed.
NTFS isn’t natively supported in Mac OS X (at least, not for writes, though a read-only NTFS driver is available).
Mac-only filesystems (e.g. HFS+) feel like a bit *too* much commitment to a platform that isn’t the dominant in my lifestyle yet – e.g. what if I want to unplug that drive from the Mac mini and hook it to my Windows box to do some USB 2.0-speed backups or other file transfers? I know that most of my file transfers will probably work fine with an SMB share over the Wifi network (it’s all 802.11 g or better, in a pretty confined space), but sometimes I just want to get something done quickly. Not to mention I’m not yet familiar with what steps I would have to take if I were to hook up my external drives (the 750GB – cleaned off now - and my backup one - FAT32), format the 750 gigger and try to copy over all my FAT32 data into reasonable facsimiles of the data I replicated off the 750 (before I repartitioned it).

There’s some fairly predictable chatter about lower performance of 3rd-party (R/W) NTFS drivers when used within Mac OS X (http://www.123macmini.com/forums/viewtopic.php?t=23243&highlight=ntfs), and I’d expect similar concerns about 3rd-party HFS+ drivers (R/W) running in Windows. I’m not worried about booting from the external drive, so that limitation of NTFS-3g isn’t a concern for me (http://www.123macmini.com/forums/viewtopic.php?t=23057&highlight=ntfs).

However, after thinking about what my *majority* usage will be, it’s clear to me that I’ll end up leaving the drive mounted to the Mac Mini 95% of the time, so I should optimize it for that scenario and not the “just in case” fear-based scenario. If it turns out that I’m using the drive directly attached to my Windows box *that* often, I can always use a one-time read-only HFS+ driver in Windows to get the data off the drive, then reformat with NTFS. Getting the data off the drive isn’t a worry of mine (these drivers all seem good enough for at least the “disaster recovery – get my data off the drive *eventually*” scenarios); so I’d be an idiot not to optimize for the day-to-day performance issues I’d otherwise be facing with a non-native filesystem for the host where it’ll be plugged in 95% of the time.

Next time

Quest: a tuner I can “set and forget” (i.e. reliable, stable, robust performance)
Quest: improve the screen drawing response time in “Screen Sharing” (aka VNC)

My Mac Mini HTPC saga: Plex or Boxee as the front-end?

2009-09-14T09:39:00.001-07:00

After opening my mind to the possibility that I could find in the Mac Mini a worthy replacement for the aging (and rotting) promise of Windows Media Center as an HTPC, it didn’t take long to find out just how far the Mac HTPC community has come. Between the streaming media front ends like Plex & Boxee, and the first-class support from digital-TV-tuner manufacturers like Hauppauge, and the number of people who’ve blazed the trail ahead of me, it seemed like a no-brainer.

However, unlike an Apple TV (a low-powered CPU ‘appliance’ that was intended merely to download movies and TV from the iTunes store), the Mac Mini isn’t specifically designed as a home theater PC (HTPC), and doesn’t present itself to the uninitiated user as a direct complement to their TV.

That said, with not a whole lot of effort and a few add-on bits of hardware, this Mac Mini is easily the equal of the functionality of my old Windows Media Center system, and has the advantage of being a lot smaller and ridiculously quieter than a clunky monster PC. [Outside of an extra $grand or two for a custom-designed HTPC, if you’ve got more money than brains.]

Which streaming media front end?

There are plenty of “Plex vs. Boxee” articles/forum chatter for anyone who’s looking (these are just the ones I found in my first couple of pages of Google results: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12). There’s a lot of active development in both projects, and there’s a lot of enthusiasm for both, but there seem to be some common themes to the discussions I’ve read:

Plex is more “fiddly” to setup your library of existing, local content – finding the folders and assigning the metadata. Boxee seems to have mastered the “point it at folders and it’ll organize it into a library” approach.
Boxee is more socially oriented, as in it lets other members of Boxee find out what you’ve watched (at least among the online sources – presumably it doesn’t advertise what you’re watching of your locally-stored media).
One covers the BBC online media; the other covers integration of the Netflix instant watch streaming. (Or at least that’s what I recall reading.)
Since they’re both based on the XBMC codebase, they’ll both benefit tremendously from future enhancements to the
Boxee seems to have already skewed towards/created a culture of “cut the cable” (i.e. “help me get rid of Comcast/Time Warner stat!”), while Plex (perhaps as a Mac-only product) has a little less of the prickly attitude.
They’re both generally considered inferior to Front Row from a UI/usability perspective, but neither one is ever accused of being “unusuable”.
They both integrate with the Apple Remote (or if you’re a Logitech weenie, with most Harmony remotes – personally, after trying and failing for over a year to get a Harmony to operate as a set-it-and-forget-it, “universal one-touch” remote, I’ll avoid those hack-jobs like the plague).

While my girlfriend is quite adept at her Macbook, our discussions so far have led me to believe that the less fiddly/quirky this setup is on a day-to-day basis, the more comfortable she’ll be at using this (and the less I’ll have to coach her down off the ledge and into the world of computer-based home media.

Thus I’m convinced by what I’ve read to veer towards Plex (and not bother her with it until I’ve got the existing content adequately catalogued in the Plex library).

Next Steps: setting up Plex

Apparently, as one poster indicated, “The key seems to be putting Plex in Library mode. Once done, its scraper will dl the video info.”

There’s also some question whether Plex yet provides integrations with Netflix and Flickr. I’ll have to dig through the PlexApp plug-in announcement archives.

Then I’ll be interested to dig further into the question of whether, by installing VLC media player, the codecs that it uses will be available to all media players, or just to my Mac.

Finally, I’d like to revisit the question of which desktop resolution is best suited to my 1080p HDTV – I’d tried 1920x1080, but wasn’t able to see the Mac menu bar (so e.g. I couldn’t click on the little Apple menu to reboot), so now I’m at something in the 1300-pixel-width range.

Namaste

2009-02-13T11:29:00.001-08:00

namaste - bumper sticker

Dude Abides
The Dude Abides - bumper sticker

Lots of geese
200+ Canada Geese just gathered in Hillsboro OR - what are they plotting?

Posted via Pixelpipe.

Untitled

2008-12-29T09:30:00.001-08:00

Frito all-natural oil

what were these chips made with BEFORE?

Posted via Pixelpipe.

Untitled

2008-12-19T20:26:00.001-08:00

if you're not living on the edge, you're taking up too much room

scrawled on a building in SE Portland

incredible renovation

beautiful monster home in Ladd's Addition

Posted via Pixelpipe.

Untitled

2008-12-19T20:18:00.001-08:00

awesome brew

one of the best Canadian beers, for half the price of it's branded counterpart

Posted via Pixelpipe.

InfoPath forms error – one of the most obscure I’ve ever seen…

2008-11-24T22:01:00.001-08:00

I had an InfoPath web form configured, working well, and ready to deploy. I let my colleague know about it and when he tried it out a day or two later, he was getting an error message I hadn’t myself seen:

Warning

There has been an error while processing the form.

Click Continue to resume filling out the form. You may want to check your form data for errors.

Click Start Over to load a new copy of the form.

Show error details

When I browsed the Details of the error message, InfoPath Forms Services showed me this:

An error occurred accessing a data source.

An entry has been added to the Windows event log of the server.
Log ID: 5566

Not much help there, but I kept trying to figure this out. I tried from different browsers (same error) and from different PCs (still same).

I got the bright idea to check the settings in the Data Connection in the InfoPath template I’d designed, but everything looked the same – even the URL that I was using to retrieve an XML representation of the List data (according to this helpful blog entry on “Accessing SharePoint List Data as XML” and “Populating form data from SharePoint List Views”). However, when I tried completing the Data Connection wizard for this existing connection, I was getting blasted with this error:

The file is not a valid XML file.
The file is not a valid XML document.
A document must contain exactly one root element.
Line 1, Position 0

I wondered if I’d messed up the permissions on the List, so I verified that I and my colleague had Read & Contribute permissions (so the Form would load), and I even added back the “System Account” with its Limited Access (using the “trick” documented in this blog article). Still nothing.

When I tried to load that URL in a browser, the page it returned was blank, and even when I looked at the View Source, there wasn’t anything helpful there:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=windows-1252" http-equiv=Content-Type></HEAD>
<BODY></BODY></HTML>

I retried this over and over for a few days, re-examining the URL in the Data Connection, triple-checking the GUIDs (thinking they must’ve gotten changed somehow during a recent tweak to the List), and wondering what I’d done wrong (or what had gotten horked up on the servers).

And the Solution Is…

Eventually I tried some googling on “A document must contain exactly one root element”, and finally returned to my old friends, the InfoPathDev.com and SharePointBlogs.com sites. When I arrived back at the “Accessing SharePoint List Data as XML” article, one of the comments jumped out at me:

I can't get this to work on a List that contains a Lookup field. I just get a blank page in the browser. Have you encountered this too? Any ideas?

Bingo! One of the recent tweaks I’d made was to convert a field in the list from using the “Choice” data type to the “Lookup” data type. (Note: using a Lookup to a separate List enables me to delegate maintenance of that list of Choices to my team members, without asking them to muddle around in the bowels of my List’s Column settings.)

All I had to do was to remove the one Lookup field from the View I was retrieving in this Data Connection, and the web Form started working again (without any changes or re-Publishing necessary, even!). I even confirmed that the URL, when used in a browser, would return a readable XML document (though you have to View Source on the returned page, which looks just as blank in IE8 as the failing one).

Lesson: if you’re going to leverage SharePoint Lists as XML data sources in your InfoPath Forms, make sure that the View you select does NOT include a Lookup field. I told you this was obscure…

Obama White House - starting to feel a lot like Christmas

2008-11-22T09:32:00.001-08:00

I'm getting awfully excited about the team that Obama is building. Biden's track record & poise, the behind-the-scenes bulldog who's going to be chief of staff, Clinton's abilities/intelligence, Richardson's sheer ability to articulate... It's like the Bizarro world version of the Bush White House. Especially after the past two years' painfully-drawn out campaigning, the forced antagonism, the hyper scrutiny of every little difference between presidential candidates. Now, to see some of the strongest, smartest Democratic candidates all coming together to form such a solid team - it's like this is the reward for sufering the last couple of decades of rampant Republican deconstructivism. Actually, the first thought when I realized that we had these five (and others) I'm the same cabinet was, "this is starting to make me think of The West Wing [the fantasy White House that most of us had wished was really running the country]" - above all, with more collected intelligence and good intentions than I could've hoped would come together to lead this country. I've been saying for the last couple if years that I'd be happy with a president that accomplished exactly *nothing* during their term, after the retarded clearcutting & international embarrassment of the tenure of the Connecticut Chimp. Now I'm actually looking forward to not just a period of healing from the road rash W has left behind, but real improvement in the lives of significant numbers of people (both inside and outside the US) that really *need* the help [instead of just lining the pockets of those who've already enriched themselves at the expense of the hard workers whose backs bore the burden of the conquistadors]. I'm not quite retardedly naive to think this will be a scandal- or failure-free era, but it's awesome to not have to grit my teeth every time the president is about to open his mouth and prove how "special" he really is.

Can non-Microsoft ERM (electronic rights management) be integrated into MOSS 2007?

2008-10-31T00:01:00.001-07:00

Fascinating question: can an organization that has deployed MOSS 2007 plug in another ERM/IRM (Electronic Rights Management) technology into the MOSS back-end, so that documents downloaded from MOSS would be automatically protected with that non-Microsoft ERM technology?

MOSS 2007 (aka SharePoint 2007) provides integration with the Microsoft Information Rights Management (IRM) technology – any documents that are uploaded to an “IRM-enabled” Document Library will automatically be (encrypted and) protected with a specific IRM policy whenever that document is downloaded again. This depends both on the Microsoft implementation of IRM (RMS) policies (known as “Information Management Policy” in the MOSS SDK) as well as the inclusion of the Microsoft IRM “lockbox” (security processor) library on the MOSS server farm. As I understand it, the procedure is basically:

MOSS receives the download request from a remote client
MOSS looks up the information management policy that is associated with the document’s List or Content Type (depending where the policy is applied)
MOSS calls an instance of the IRM security processor (installed with the RMS Client on the front-end servers) to (a) encrypt the document, (b) generate the IRM license based on the associated policy, and (c) encrypt the content encryption key with appropriate RM Server’s public key.
MOSS delivers the protected document to the remote client – otherwise the same way that it would deliver an unprotected document.

Guessing How Third-Party ERM Could Integrate Into MOSS

So theoretically, for a third-party ERM solution to properly intercept the steps in this sequence:

the MOSS server would have to request a method/API that is “pluggable”
the MOSS server would have to support the ability to “plug” alternative ERM policy services in place of the native Microsoft IRM policy services
the MOSS server would have to support the ability to “plug” an alternative security processor in place of the native Microsoft RM security processor
the ERM solution would have to implement the pluggable responder for the “policy lookup” service, as well as a replacement UI and business logic framework for the server-side ERM policy “creation/assignment” capability that MOSS provides for IRM
the ERM solution would have to support a thread-safe, multi-threaded, rock-solid-stable security processor that could run in a potentially high-volume server environment

Given how much effort Microsoft has gone to in the past couple of years (not without external incentives, of course) to make available and document the means for ISV’s to interoperate with Microsoft client and server technologies, I’d figured there must be some “open protocol” documentation that documents how an ISV would create compatible ERM components to plug into the appropriate locations in a MOSS environment.

I scoured the SharePoint protocols specifications, but there were no specific protocols documents, nor any mention of “information management” in any of the overview documents.

There are some occasional references in the Microsoft Forums and elsewhere that hint at details that might be relevant to a third-party ERM plugin for MOSS, but I can’t tell if this is actually related or if I’m jus chasing spectres:

Aha! It Appears the Answer is “Yes”

(I thought about erasing and rewriting the above, but there’s probably someone somewhere who thinks the same was I do about this, so I’ll leave it and just share my new insight below).

As always, I really should’ve started with the WSS 3.0 SDK and then branched out into the MOSS SDK and other far-off lands.

It turns out that the WSS SDK had the “secret” locked up in a page entitled “Custom IRM Protectors” (not to be confused with the forum post linked above). My theory above didn’t nearly guess correctly, but it most closely resembled the “Autonomous Protector”:

Create an autonomous protector if you want the protector to have total control over how the protected files are rights-managed. The autonomous protector has full control over the rights-management process, and can employ any rights-management platform. Unlike the process with an integrated protector, when Windows SharePoint Services invokes an autonomous protector, it passes the specific rights that the user has to the document. Based upon these rights, an autonomous protector is responsible for generating keys for the document and creating rights-managed metadata in the correct format.

The autonomous protector and the client application must use the same rights-management platform.

So for a third-party ERM vendor to support an integrated experience in MOSS, while still using its non-Microsoft ERM client (i.e. not the Microsoft RMS Client), it would have to:

provide a COM component on each MOSS web server that implements the I_IrmProtector interface and an I_IrmPolicyInfo_Class object (analogous to my theorized “alternative ERM policy service”).
provide a rights management platform that protects (at the server) in a way that’s compatible with protections enforced by their rights management client (e.g. an alternative security processor available either locally or remotely from each MOSS web server)
override the default “integrated protectors” for Microsoft Office document types, and (presumably) support the ability to protect the Microsoft Office document types with the autonomous protector(s)

If I’m reading this right, then with a server-accessible rights management platform and one or more autonomous protectors, MOSS would be able to handle the rest of the required functionality: policy storage, UI, management interfaces (business logic), etc.

Now I wonder if anyone has actually implemented this support in their ERM solution…

100 Mile Constitution Free Zone

2008-10-29T12:02:00.000-07:00

http://www.emergentchaos.com/archives/2008/10/100_mile_constitution_fre.html

This is pretty ridiculous abuse of power, and of opportunistic broadening of the interpretation of what must've been more narrowly defined laws. Seems pretty typical under the Stalinist (er, Bush) regime. How's about a little regime change on the domestic front eh?

"you most closely resemble survey respondents within the Omnivores typology group"

2008-09-28T13:15:00.000-07:00

Apparently I too am a technology/information Omnivore, according to the Pew Internet & American (?) Life project.

Basic Description Members of this group use their extensive suite of technology tools to do an enormous range of things online, on the go, and with their cell phones. Omnivores are highly engaged with video online and digital content. Between blogging, maintaining their Web pages, remixing digital content, or posting their creations to their websites, they are creative participants in cyberspace.

Defining Characteristics You might see them watching video on an iPod. They might talk about their video games or their participation in virtual worlds the way their parents talked about their favorite TV episode a generation ago. Much of this chatter will take place via instant messages, texting on a cell phone, or on personal blogs. Omnivores are particularly active in dealing with video content. Most have video or digital cameras, and most have tried watching TV on a non-television device, such as a laptop or a cell phone.

Omnivores embrace all this connectivity, feeling confident in how they manage information and their many devices. This puts information technology at the center of how they express themselves, do their jobs, and connect to their friends.

When I read this description, it reminded me of something my wife Robin has repeatedly pointed out about me: I am one of the least social people she knows. I rarely make social plans with anyone (including her), I don't naturally or voluntarily engage with people outside of work, and I am quite comfortable (or at least not dissatisfied) staying home with my TV, laptop and dogs.

So how to reconcile these two states of being - the "meatspace" Mike, who doesn't engage in any social contact, and the "cyberspace" Mike, who engages with strangers, colleagues and friends with nary a second thought, on a frequent, bleeding edge basis? I have to assume that the cyberspace activities of an Omnivore are not just a pleasant and easy means of interacting with friends/family/colleagues, but are in part a way of shielding ourselves from the demands of the meatspace environment where the interactions are somehow (cf. Introvert) more draining, demanding and threatening.

I am the same person, but in these differing situations I am able to engage in much different ways - and it appears that I'm on a trajectory that will increase the divergence between my social and cyber personalities.

I don't know what to make of that - whether it's a natural progression of the aging Introvert, or if there's some progressive neurochemical change that's making it harder to engage in person (and thus I'm biased more towards the distant/electronic/asynchronous interaction), or maybe I just don't have the strength, adrenaline and childish curiosity that I did in my 20's.

Apparently I too am a technology/information Omnivore, according to the Pew Internet & American (?) Life project.

DLL Injection in Windows: what security countermeasures can you use?

2008-09-09T16:21:00.001-07:00

Manage the Administrators group

Examine any default install of Windows since NT4 SP6. You’ll notice that the SeDebugPrivilege is assigned by default only to the .\Administrators local group of the Windows host. While this isn’t exactly unusual for users to be members of Administrators on their own PC, don’t think that every user or process automatically gets this capability in Windows.

Countermeasure: If you want to assert an explicit distinction between those who do and do not have the SeDebugPrivilege on a Windows system, explicitly manage the membership of the Administrators local group. This is especially useful (and applicable) to Windows Servers, where most of your users won’t have (or have need for) this membership.

How to implement:

run the net localgroup command locally e.g. with these parameters: “net localgroup Administrators NAME_OF_USER_OR_GROUP_TO_REMOVE /Delete” (or run it remotely via a remote-shell tool such as psexec.exe)
configure a Group Policy (e.g. using Active Directory group policy) that sets the membership of the Administrators group using the Restricted Groups security setting (either overwriting the existing membership or incrementally adding/deleting specified security principals)

Manage the SeDebugPrivilege

The obvious flipside of the default SeDebugPrivilege assignment is that you can change the security principals to whom the privilege is assigned. In fact, if you review (or have implemented) the Microsoft Security Security Accelerators for Windows (Windows XP, Windows Server 2003, Windows Vista, Windows Server 2008), you’ll find they recommend

Countermeasure: remove the Administrators group from the SeDebugPrivilege.

How to implement:

run the ntrights.exe Resource Kit command-line tool locally e.g. with these parameters: “ntrights.exe –u Administrators –r SeDebugPrivilege” (or run it remotely via a remote-shell tool such as psexec.exe)
configure a Group Policy (e.g. using Active Directory group policy) that removes all security principals that are assigned the SeDebugPrivilege privilege

Run Apps, Services as lesser-privileged user

So the first two BKMs are great and all, but there are still lots of situations where you can’t make these blanket changes to the entire OS (though thankfully virtualization is reducing these “shared system” problems). You may have to find ways to launch one or more processes with different security context or privileges than the rest of the system – sometimes having to run something with more privilege than the rest of the system (e.g. try Sudo for Windows), but usually wanting to strip privilege and permissions away from specific processes.

Countermeasure: use Windows’ Software Restriction Policy (aka SRP or “SAFER”) to strip the Token of as many groups and privileges as the application can tolerate. You don’t have to set a restrictive policy for the whole system – you can set this on an application-by-application basis (which can be practical in server environments, where you may only have a few critical applications to have to protect from each other).

How to implement:

Download and use Michael Howard’s SetSAFER application, which will strip varying levels of privileges and groups from the security token assigned to the process (thus making it more difficult for the process to access privileged objects in Windows). If you want to dig into the code for this, and if the source code isn’t available, you can take a look at the code included in the original article (on which SetSAFER was based), or fire up DotNet Reflector and inspect the MSIL for the SetSAFER “executable”.
You could also try “psexec –l” (which implements one of the approaches taken by SetSAFER – one of the “stripped-down profiles”).

Something about this feels like I’ve missed another approach that should be mentioned in this context, but I’m sure there’s smarter folks than I reading this who can add any missing details to the picture. Thanks, and have fun with this!

Well that's a sucky answer...

2008-07-23T20:57:00.000-07:00

I was really hoping for someone a little less useless - but I'm sure my wife would agree with how useless this figured out I am:

Installshield – great for developers, sucks ass for victims (aka everyone else)

2008-06-10T00:12:00.001-07:00

Holy crap, what a safari this is turning into. I’m trying to uninstall a piece of software (the Intel Debugger v9.1.x) that was apparently packaged with Installshield. However, every time I tell Windows to uninstall it, it returns to me the following error:

Idiot move #1: look for help from InstallShield Corp

So I Google for this error and come up with a half-dozen links to various “support” articles from InstallShield and related on how to resolve this error. It tells me to download various versions of their runtimes and engine installers, none of which make any improvement in the situation.

I went away for a while, came back again today and tried a whole different attack:

Forget the vendor, just debug it yourself (using Process Monitor)

Launch Process Monitor, filtering out all running processes except msiexec.exe
Near the end we finally see some activity that’s related to the problem:

MsiExec.exe RegQueryKey HKCR\CLSID\{8B1670C8-DC4A-4ED4-974B-81737A23826B}\LocalServer32

MsiExec.exe RegQueryValue HKCR\CLSID\{8B1670C8-DC4A-4ED4-974B-81737A23826B}\LocalServer32\(Default) Data: C:\PROGRA~1\COMMON~1\INSTAL~1\Driver\8\INTEL3~1\IDriver.exe

Then there are four attempts to launch the IDriver.exe, all of which immediately halt
Lastly, there’s an update to the MSI log file which says this:

1: The InstallScript engine on this machine is older than the version required to run this setup. If available, please install the latest version of ISScript.msi, or contact your support personnel for further assistance.

=== Logging stopped: 09/06/2008 14:02:26 ===

At least I know which file is “older than the version required”.

However, the next problem is figuring out how to get the ‘right one’ executed in its place:

Where Your Hero* Learns Just How Screwed Up InstallShield’s Model Really Is

* aka “just some dick on the Internet”

From what I can tell, Installshield only cares about one person: the dork who blindly builds the Installer package for their one little application. Apparently, if you need to call on the Installshield components, don’t ever even try to discover whether they’re already installed on the target. Instead, assume that they must *not* be installed (presumably because every developer on the planet has the privilege of being the first to get software installed on each PC where it’s being used), and always install a copy of some Installshield dependency on the end-user’s PC. And then for good measure, make sure that there’s a hard-coded dependency on the version of the InstallShield bits that went with the installer.

They sure as hell don’t seem to care about the lowly end-user or IT administrator, who might have to actually *deal* with the nightmare of conflicting/overwriting/installed-to-every-conceivable-corner-of-the-filesystem versions of these hard-coded InstallShield dependencies.

Just for s**ts and giggles, try this at home:

fire up REGEDIT.EXE
Press [Ctrl]-F to bring up the Search dialog
Type Installshield and [Enter]
Click the [F3] button a few dozen (or hundred) times

A bit more digging on my own system:

Current version of IDriver.exe in the logged directory = 8.0.0.123

One article in the InstallShield/Macrovision/Acresso library confirms the noted location is where the IDriver.exe version 7 or 8 should be found.

Once more, I downloaded and installed the latest InstallScript 8 package (which turns out to be the 8.0.0.123 I already had installed), so I then decided to try downloading all the later versions and install them one by one as well. I was hoping that the Registry setting that resolves to this particular IDriver.exe would be overridden (at least in the “Version Independent ProgID” or something similar) by a later install. Here’s one set of settings that I figured were related:

CLSID: {8B1670C8-DC4A-4ED4-974B-81737A23826B}
(Default) value: InstallShield InstallDriver
AppID: {1BB3D82F-9803-4d29-B232-1F2F14E52A2E}
LocalServer32: C:\PROGRA~1\COMMON~1\INSTAL~1\Driver\8\INTEL3~1\IDriver.exe
ProgID: ISInstallDriver.InstallDriver.1
VersionIndependentProgID: ISInstallDriver.InstallDriver

Yep, after installing IScript9.msi, the CLSID under the entry HKCR\ISInstallDriver.InstallDriver changed to {B3EDE298-AE75-4A1C-AB7E-1B9229B77BBE}. However, the uninstall “Fatal error” continued to crop up. Apparently the fatal application’s uninstaller doesn’t chase the ProgIDs but some other reference instead.

Then by some wild fortune, I happened to stumble on a very obscure directory in which the “later version” of the InstallScript MSI installer (isn’t there some irony embedded in that?) was actually still cached. WHY this wasn’t available from the vendor’s own web site, I’ll never know. However, installing this version of IScript.msi did overwrite the ProgID once again, and the version of IDriver.exe installed in the target location was 8.1.0.293.

Somehow, finally, that did the trick. Finally got that Installshield-driven crap off my system. Trying to resist the impulse to wipe all traces of Installshield product off my system as well, and reminding myself that I could create this same hell for myself ten times over in so doing.

So, apparently all it takes is for some ancient application to overwrite the better version of an “Installation Engine”, and all hell breaks loose. I’m beginning to see why the Installshield product line has been bought and sold more times than… well, I’m drawing a blank on a family-friendly comparison, so let’s just say this was not one of the more profitable software businesses out there. And no freakin’ wonder.

ComponentChk.cpp - I resolved my first C++ compiler "fatal error"!

2008-06-01T16:06:00.001-07:00

Yes, for those of you with even more grey in your beard/hair than I, this is probably a "meh" event. However, for me, who's always looked at C++ code and wondered how the H*** anyone could ever make sense of it, this is a big event. [Patting myself on the back.]

Situation

I'm working on finishing up the Setup bootstrapper for my VSTO add-in Word2MediaWiki.NET. I'm targeting Word 2003 and Word 2007 (despite Microsoft's best efforts/worst neglect to the contrary), and I'm trying to achieve what many have done before, but which seemed impossible for me: allow the Setup.exe to detect which version of Word is installed, and then conditionally install the prerequisite software (i.e. the so-called Office PIAs) for whichever version of Word is discovered.

Problem 1: most folks in the VSTO space seem to think that, despite the fact that a version of the .NET framework is required to support VSTO, the Setup.exe that goes with your VSTO add-in install package should use UNmanaged code to detect whether the pre-requisites are installed.

Problem 2: I know squat about writing unmanaged code.

Problem 3: Microsoft's VSTO & Office teams left it up as an exercise to the VSTO app developer to figure out how to assemble the amazing number of parts necessary to make a VSTO install work transparently to the end user.

Problem 4: There've been many articles written posthumously (I mean, long after VSTO v2 was released) on various aspects of automating VSTO-add-in installation, but none of them addressed the very inevitable scenario where the app developer needs to support both Word 2003 and Word 2007. [Don't even *think* about pre-2003 versions of Office — you'd have to be clinically insane.]

Manna from heaven

One ridiculously brave soul, Mark Hogan, actually took the time to not only figure out how to build such a conditional pre-requisite detection app in C++, but he was so overcome with glee that he beat Microsoft at something even they were too scared to do, that he published the full set of source code and XML configuration files for the entire world to use.

Now, masochist that I am, I took it upon myself to try to integrate the code that Mark Hogan published into the existing code that I'd already slotted into my Office PIA bootstrapper files. However, I didn't anticipate the foreseeable result that, because I'm coding this stuff in my spare time, and usually late at night, I would (a) not finish the integration work in one sitting, (b) forget what I'd originally set out to do and (c) forget to integrate any of the critical code that actually performed the conditional logic.

Mike chases his tail

Miraculously, I was left with a .CPP file that would compile and that appeared to be significantly different from the original file I started from, and that threw off an error code that created a spectacular wild goose-chase:

"Unable to satisfy all prerequisites for Word2MediaWikiDotNET. Setup cannot continue until all system components have been successfully installed."
Details:
"Prerequisite check for system component Microsoft Office 2003/2007 Primary Interop Assemblies (Word Only) failed.
See the setup log file located at 'C:\DOCUME~1\msmithlo\LOCALS~1\Temp\VSD33.tmp\install.log' for more information."

And in the INSTALL.LOG file was the illusory answer:

Running external check with command 'C:\DOCUME~1\msmithlo\LOCALS~1\Temp\VSD33.tmp\Office2003PIAor2007_WordOnly\ComponentChk.exe' and parameters '/delete /save /word {1EBDE4BC-9A51-4630-B541-2561FA45CCC5}'
"Process exited with code 1607"
Setting value '1607 {int}' for property 'SupportedWordVersionInstalled'
…
Setting value '1607 {int}' for property 'PIAsRegistered'

This led me down the path of chasing InstallShield errors, since the only Google search results that looked at all related were things like these. After a few days of trying to figure out how the InstallShield scripting engine could be related to a Visual Studio SETUP.EXE or custom C++ application, I finally got my brain back and tried to isolate where in the code or configuration files the problem existed. That led me to a line of C++ code that threw ERROR_UNKNOWN_COMPONENT, which as it turns out also shares the 1607 error/exit value.

[The whole gruesome story is illustrated in the Bug I filed to myself here.]

Back to the original goal, with a new mission

Once I realized what I'd left out of the C++ code, I quickly got it to the point where I could try compiling it. Now a new mission emerged - how to debug a C++ compiler error?

C:\>cl.exe /Oxs /MT /GS ComponentChk.cpp advapi32.lib
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.21022.08 for 80x86
Copyright (C) Microsoft Corporation. All rights reserved.
ComponentChk.cpp
Microsoft (R) Incremental Linker Version 9.00.21022.08
Copyright (C) Microsoft Corporation. All rights reserved.
/out:ComponentChk.exe
ComponentChk.obj
advapi32.lib
ComponentChk.obj : error LNK2019: unresolved external symbol __imp__WaitForInputIdle@8 referenced in function "int __cdecl MyStart(char const *,char const *)" (?MyStart@@YAHPBD0@Z)
ComponentChk.exe : fatal error LNK1120: 1 unresolved externals

That command line (cl.exe /Oxs /MT /GS ComponentChk.cpp advapi32.lib) was derived from the original article from which we were all working (or stumbling around half-blindly, it felt to me). I just ran this without a second thought, assuming that since Mark Hogan didn't seem to mention any modifications to it, any errors it showed must've been my fault.

Where is "__imp__WaitForInputIdle@8" I wondered? It didn't show up in the source code when I searched for it, and nor did "int __cdecl MyStart".

After staring at this for a while longer, I figured some substring searches would work, which after a few attempts finally showed me that I was actually looking for the code

WaitForInputIdle(pi.hProcess, INFINITE);

which is called in the function MyStart(). I tried some silly things first, of course, but I eventually realized that if WaitForInputIdle() didn't exist as a named function in the source code, perhaps it existed in another library that wasn't yet connected to the code? A quick MSDN Library search told me this function was actually part of USER32.DLL, and it wasn't too painful a leap of logic to try adding USER32.LIB to the compilation parameters, like so:

cl.exe /Oxs /MT /GS ComponentChk.cpp advapi32.lib user32.lib

And when I loaded up the now-successfully-compiled COMPONENTCHK.EXE into DEPENDS.EXE, it confirmed exactly what I expected to see:

Mark, many thanks to you, and my apologies for calling into question your mad C++ skillz.

Do you Wiki? Word2MediaWiki.NET first release is live!

2008-03-30T21:09:00.001-07:00

I just wanted to let my faithful readers know that I have finally completed a beta version of the Word2MediaWiki.NET add-in for Microsoft Word 2003.

This'll be useful to anyone that regularly contributes to Wikipedia, or those that have a bunch of Word documents that they'd like to share on Wikipedia.

It's just as useful for any of the other hundreds of wikis that use the same server software as Wikipedia (known as MediaWiki), and it'll be of interest to those folks who've used Word2MediaWikiPlus (from whose VBA source code I converted much of the basis of Word2MediaWiki.NET).

Grab a copy here:

http://www.codeplex.com/Word2MediaWikiDotNET

And please, let me know about any issues or experiences you have with it by posting a Discussion item here:

http://www.codeplex.com/Word2MediaWikiDotNET/Thread/List.aspx

Random distraction of the day: play the Red cards

2008-03-07T11:30:00.001-08:00

Take the Magic: The Gathering 'What Color Are You?' Quiz.

ATOM, RSS & feeds - have YOU ever known which was the "right" one to choose??

2008-03-04T10:43:00.001-08:00

What a friggin relief:

http://dev.live.com/blogs/devlive/archive/2008/02/27/213.aspx

"Microsoft is making a large investment in unifying our developer platform protocols for services on the open, standards-based Atom format (RFC 4287) and the Atom Publishing Protocol (RFC 5023)."

Finally I can try to stop worrying about which of the jillion "feed types" I should select to make sure that the feed subscriptions I've amassed are as "future-proof" as possible. Good gravy, the number of times I've gone to subscribe to a certain feed, only to be faced with the choice among 2-6 different feeds (all apparently for the same set of articles) is just paralyzing:

Do I want the one with RSS in the final suffix?
Should I try to figure out which protocol is most popular/widest supported?
Do I need to try to figure out which one provides the most metadata with each downloaded article?
And which one might be sending the least data back to the author? [This is ParanoidMike after all… wouldn't want to disappoint my fans with a rare moment of rational thinking now]

I don't normally like any one behemoth [aside: wasn't that one of Godzilla's opponent?] dictating to me a single format for anything, and I'm especially wary of any such edicts from Microsoft (having been privy to watching the sausage get made there for six years), but any time they make such an unequivocal commitment to an RFC standard and away from their "not built here" crap, I'm all in favour.

MindManager 7 ToDoList AddIn development Part 6: the Mysteries of COM interop

2008-03-02T16:45:00.001-08:00

Sometimes there are inevitable mysteries uncovered when writing code. Well, I've bumped into quite a rich source of mysteries in trying to use an aspect of the MindManager object model that thunks through a brittle COM interop module, known as CmjDocumentCollectionComObject. Here's just a couple of examples that have come up recently:

"unable to create document"

Here is the 'offending' code:

public static MMInterop.Document GetMap(string filename)
{
    MMInterop.Document toDoListMap;
    MMInterop.Documents maps;
    string toDoListMapFullPath = Connect.applicationObject.get_Path(MMInterop.MmDirectory.mmDirectoryMyMaps) + filename;
    maps = Connect.applicationObject.get_Documents(true);
    try // open a ToDoList map
    {
        toDoListMap = maps.Open(toDoListMapFullPath, String.Empty, true); // here's where the COMException is raised
    }
}

Here is the exception raised:

System.Runtime.InteropServices.COMException occurred
Message="Object 'CmjDocumentCollectionComObject' reports an error: 'unable to create document'"
Source="MindManager.Application.7"
ErrorCode=-2147220992
StackTrace:
at Mindjet.MindManager.Interop.DocumentsClass.Open(String pFileName, String pPassword, Boolean Visible)
at ParanoidMike.MindManager.ToDoList.ToDoListMap.GetMap(String filename) in C:\personal\VS Projects\MM7TODOList\ToDoList.cs:line 187
InnerException:

And do you want to know what that exception really means? "The specified file does not exist" would be my interpretation. If I understand the MindManager function maps.Open() correctly, this is meant to open an existing document. I've just asked it to open a non-existent document, so I'd expect "can't find it", but I'm puzzled by "unable to create document". It's like whoever wrote the error strings for the CmjDocumentCollectionComObject is interpreting the name of the Win32 CreateFile() API literally.

I am attempting to use a Try...Catch approach to testing whether the requested file exists. If the file exists, then maps.Open() would succeed; if it didn't exist, then I'd use the Catch block to instead create the named file. I didn't expect to have to catch a System.Runtime.InteropServices.COMException, nor in figuring out how to catch only the exception with a specific ErrorCode/HRESULT returned by the COM object.

But then again, "Nobody expects the Spanish Inquisition"…

{"Retrieving the COM class factory for component with CLSID {5B9EA9CE-76A3-4878-9A6B-22D0A3042774} failed due to the following error: 80040154."}

Here's the code:

MMInterop.Document toDoListMap;
try
{
    toDoListMap = new MMInterop.Document();
}
catch (System.Runtime.InteropServices.COMException e)
{
    throw;
}

Here's the .NET exception that gets thrown:

{"Retrieving the COM class factory for component with CLSID {5B9EA9CE-76A3-4878-9A6B-22D0A3042774} failed due to the following error: 80040154."}

And here's the HRESULT that is being thrown by the COM object: -2147220992

In various posts, the common theme seems to be that the "component with the noted CLSID" needs to be re-registered. Searching for this CLSID on my system (predictably) leads to Mindjet.MindManager.Interop, Version=7.0.323.0, Culture=neutral, PublicKeyToken=19247b5ea06b230f and Mindjet.MindManager.Interop, Version=7.1.388.0, Culture=neutral, PublicKeyToken=19247b5ea06b230f. [However, there's no ProgID registered for Mindjet.MindManager.Interop. Is that bad — I know it's not good for typical apps, but is a COM interop assembly a "typical" COM app?]

I remember reading somewhere that the MindManager 7 Primary Interop Assemblies were installed by default when installing MM7, so I thought perhaps and Add/Remove Programs "Repair" operation would suffice.

Unfortunately no — damned MindManager, I ran the full Repair, and it even reinstalled all the 41 default templates (so I know it did completely successfully), but my code is still throwing this same error. So I went looking for the file path, which meant examining the Properties of the "MindManager" Reference, and it reports that this "ActiveX" file is stored here: C:\WINDOWS\assembly\GAC_MSIL\Mindjet.MindManager.Interop\7.1.388.0__19247b5ea06b230f\Mindjet.MindManager.Interop.dll. Fire up a Command Prompt in that directory, run "REGSVR32.EXE Mindjet.MindManager.Interop.dll", and I end up with this error:

REGASM.EXE

Here's a riddle: how does one initialize a variable that cannot be initialized?

When I try to create a new mindmap document using a .NET AddIn, I'm finding that the variable I declare to hold a reference to the new mindmap is not usable: (a) it won't work until it's initialized, but (b) when it's initialized it throws the error documented here:
http://tech.groups.yahoo.com/group/MindManagerDev/message/447

I've tried to dig up any hints that would help me figure out what to do to resolve this error, but so far nothing has worked (including trying to Register the Interop DLL, and Repairing the installation of MindManager).

There are plenty of people asking about this kind of issue — some related to straight COM objects, others talking about COM Interop assemblies:
http://www.thescripts.com/forum/thread731361.html
http://forums.cnet.com/5208-6141_102-0.html?forumID=8&threadID=216925&messageID=2313201
http://channel9.msdn.com/ShowPost.aspx?PostID=333247
http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=765439&SiteID=1
http://groups.google.com/group/microsoft.public.dotnet.framework.interop/browse_thread/thread/e8d8ccb58221f843/2263fa2b3db5895b

(1) I tried loading up the Interop DLL in DEPENDS.EXE, but there appear to be no missing dependencies.
(2) I tried registering the assembly using REGASM.EXE, according to the article here: http://www.simple-talk.com/dotnet/visual-studio/build-and-deploy-a-.net-com-assembly/
using the following command:

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727>regasm C:\WINDOWS\assembly\GAC_MSIL\Mindjet.MindManager.Interop\7.1.388.0__19247b5ea06b230f\mindjet.mindmanager.interop.dll
Microsoft (R) .NET Framework Assembly Registration Utility 2.0.50727.1433
Copyright (C) Microsoft Corporation 1998-2004. All rights reserved.
Types registered successfully
C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727>

Now I'm getting the same .NET exception, with a different HRESULT: -2147221164. That HRESULT appears to be associated with needing to re-register Atl.dll, so I gave that a shot. Unfortunately, it gave me the same error and the same HRESULT. I also tried re-running the above REGASM.EXE command, with no change — still throwing the same exception.

Cheap Hack: avoid init issues with dummy calls

I'm almost embarrassed to even discuss how I'm getting around this intractable problem…but not embarrassed enough not to do it, so I might as well own up to it. Who knows? Maybe one of you will know what I'm overlooking. Maybe I'll find this article in six months' time and remember why I put this hack in place. Maybe this will just be my own personal legacy to the world. :)

Let's recap: my design for this AddIn is to use a separate map to list copies of all Tasks found in the searched maps. If the named map already existed, just open it; if not, a new map should be created. However, I kept getting blocked by not "initializing" the local variable to hold the new/opened Document handle before returning it to the calling function. However, initializing the local variable with a "new MMInterop.Document()" call just caused other problems.

I was thinking that because Visual Studio was throwing the CS0165 error "Use of unassigned local variable 'toDoListMap" on the final line of code "return toDoListMap", and not on any of the preceding where the toDoListMap variable was being assigned, what Visual Studio might be reacting to is that toDoListMap wouldn't have a non-null value on all code paths through the function. I'd ignored this thought because (a) the code paths where it wouldn't get a value were intentional, and (b) the documentation on CS0165 kept indicating that I needed to initialize the variable.

However, after exhausting all the "right way" options, I finally just dropped some initializations (assignments?) into the code paths where toDoListMap wasn't previously being touched:

public static MMInterop.Document GetMap(string filename)
{
    MMInterop.Document toDoListMap;
    MMInterop.Documents maps;
    string toDoListMapFullPath = Connect.applicationObject.get_Path(MMInterop.MmDirectory.mmDirectoryMyMaps) + filename; // TODO: remove this hard-coded pathing to the ToDoList map
    maps = Connect.applicationObject.get_Documents(true); // assign/initialize the maps variable so the array can be searched by maps.Open().  "Connect" is the Class that implements a static variable "applicationObject" which is assigned to the "application" object that's captured during AddIn initialization.

    try // open the existing ToDoList map
    {
        toDoListMap = maps.Open(toDoListMapFullPath, String.Empty, false);
    }
    catch (System.Runtime.InteropServices.COMException e) // if a ToDoList map by that name does not exist, create one
    {
        if (e.ErrorCode == -2147220992) // "Object 'CmjDocumentCollectionComObject' reports an error: 'unable to create document'"
        {
            System.Windows.Forms.MessageBox.Show("Exception opening map: " + e.Message); // TODO: remove once initial debugging is complete
            string templateFullPath = Connect.applicationObject.get_Path(MMInterop.MmDirectory.mmDirectoryTemplates) + templateFileName; //construct reference to the template that's required when creating a new Document

            toDoListMap = maps.AddFromTemplate(templateFullPath, String.Empty, false);
        }
        else // this is never intended to be run - it's just here to convince the compiler that the toDoListMap variable has been initialized on all code paths, as it otherwise throws an exception "Use of unassigned local variable 'toDoListMap'.
        {
            System.Windows.Forms.MessageBox.Show("If you see this message, record the error and report it to the developer - this should never be seen: error code = " + e.ErrorCode.ToString());
            toDoListMap = new MMInterop.Document();
        }
    }
    catch (Exception e) // this is never intended to be run - it's just here to convince the compiler that the toDoListMap variable has been initialized on all code paths, as it otherwise throws an exception "Use of unassigned local variable 'toDoListMap'.
    {
        System.Windows.Forms.MessageBox.Show("If you see this message, record the error and report it to the developer - this should never be seen: error code = " + e.ErrorCode.ToString());
        toDoListMap = new MMInterop.Document();
    }

    return toDoListMap;
}

And now, miraculously this damned code compiles and runs correctly. I really wish I had some better ideas than these "new" calls, 'cause I've got a feeling they'll come back to bite me far down the line. I've added some MessageBox "please let me know if you see this" dialogs, but that won't really solve the problem — just make me look a little more like an amateur code-jockey.

[I'm still not sure why in this case, I was able to assign the toDoListMap variable without first initializing it — it's not one of the simple datatypes, so I don't think it's being implicitly initialized by the compiler, and I thought I'd just re-learned that a complex Object always needs to be declared, initialized and then assigned. Once again there's something I don't get about this, but I'll leave that to dig up in the future, as yet another great surprise. :)]

MindManager 7 ToDoList AddIn development Part 5: minimizing Object overhead between classes

2008-03-02T11:25:00.001-08:00

I've created quite a puzzle for myself. In writing an AddIn that calls functions from custom Classes, I am forcing the AddIn to pass in all the data that would be needed in the custom code. I've added a few helper functions to the ToDoListBuilder class including GetAllTasks(), IsTaskCompleted() and IsTopicATask(). These are easy because I'm passing simple variables into each function.

However, I haven't yet wired up the primary function that will be called from the AddIn's mmCommand_Click() event, and I haven't decided what I need to pass down from the button_Click() to the cascade of encapsulated functions, nor how far down to pass any object like a baton.

It seems like it'd be correct to pass the MMInterop.Application object into the ToDoListBuilder code, but in passing the Application object to a GenerateToDoListItems() method, I'll have to pass the Application object to at least one further layer of called functions such as GetAllToDoListMaps(). Or will I?

Now that I'm looking around, the Connect class has a private applicationObject variable. While it seems unlikely I'd be able to make reference to that Class' private variable from a called Class, there should be no reason why I couldn't create another private variable in the ToDoListBuilder class as soon as the GenerateToDoListItems() method is called, and then call on that Class-wide variable from then on.

Reducing Memory Footprint of Application Object(s) without Passing References Everywhere

Here's how the code around the Application object comes out from the Visual Studio AddIn Template wizard by default:

    public class Connect : Object, Extensibility.IDTExtensibility2
    {
        private Mindjet.MindManager.Interop.Application applicationObject;
        private Mindjet.MindManager.Interop.Command mmCommand;
        ...
    }

Changing the first declaration to public from private makes that applicationObject object available outside of the assembly as well as within it, so that pointers (references) to the object don't have to be explicitly passed around the code.

Assuming that nothing can affect the state of the applicationObject, there should be no reason not to declare it as "public". I'm not entirely naive though — I assume there's reasons why you'd want to do one and not the other, but I have to believe that any code running in the MindManager application should necessarily want/try to use a single application object anyway.

Solution: Static/Shared modifier

After bashing my skull into the wall of my ineptitude for a few days, I finally dislodged a bone fragment of useful info: the "static" modifier. Somehow this does what I had expected the "public" keyword to accomplish — makes it possible to access the variable from outside the class, without having to instantiate another copy of the object.

Open Questions

Now I'm left wondering two things:

What's the lowest level of accessibility that the object needs? Does it still need to be public, or would protected or less be acceptable?
Under what circumstances would it be inadvisable to use the static modifier? I can imagine that in theory, anything that is supposed to represent multiple objects should not be marked "static", but that still leaves a ton of room for interpretation (and for subtle mistakes in code that will bite me only later).

Aside: Registry entries in Setup project Aren't Automatically Installed

Idiot assumption of the day: just because I entered the correct Registry settings in the Setup project, doesn't mean that when I Debug the Solution those settings will be automatically added to the computer's Registry. (Sigh, sometimes I surprise myself with how dense I can be.)

So which "hack" would be better — should I hand-enter the Registry settings I need (which seems pretty lame) or should I build the Setup project and actually install this AddIn (which might end up leaving behind stuff that I'll need to rip out later after I've rev'd the AddIn a few times)?

I guess I'm going down the path of hand-entering the Registry settings. I don't like doing this, and I really wish debugging MindManager AddIns didn't require this lame step, but it looks like I've got no better option.

NullReferenceException: When will I ever learn?

I can't believe the number of times I get caught by this seemingly predictable error:

        private System.Collections.ArrayList toDoListItems; // building list of all Tasks to be emitted as the items for the ToDoList
        private Mindjet.MindManager.Interop.Application applicationObject; // local instance of the Application object

        public System.Collections.ArrayList GenerateToDoListItems(MMInterop.Application application)
        {
            System.Collections.ArrayList toDoListMaps; // collection of all maps to be searched for Task topics
            applicationObject = application; // sets the local variable equal to the value of the passed-in parameter
            toDoListMaps = GetAllToDoListMaps(); // generate the collection of maps to be enumerated and searched
            toDoListItems = new System.Collections.ArrayList(); // declare this to avoid a NullReferenceException when it's assigned below
            
            foreach (MMInterop.Document map in toDoListMaps)
            {
                    toDoListItems.AddRange(GetAllTasks(map));
            }
            return toDoListItems;
        }

I really don't get why I don't have to declare the "new toDoListMaps" object, but I have to declare the "new toDoListItems". Is there something about creating the object inside the method that implicitly initializes it, but this implicit initialization doesn't occur for objects that are created outside of the method?

And why does creating an int object not require initialization, but creating an ArrayList object does? This might make sense to veteran coders, but my god it's confusing for those of us just trying to get their first few apps out the door…

Gotta love it when Microsoft catches Microsoft doing something insecure...

2008-02-26T18:23:00.001-08:00

The Office document conversion utility (that converts Office 2007 documents for use in Office 2003) seems to have gotten caught trying to execute (stack? heap? I still don't know which) memory devoted to data. Naughty naughty, wonder how they missed that? :)

MindManager 7 ToDoList AddIn development Part 4: Fixing the Registry Settings in the Setup Project

2008-02-18T13:10:00.001-08:00

The MM VS2005 AddIn wizard created a set of Registry settings for my project, and the friendly folks at Mindjet's Dev Support team helped me fix a problem with my project (and in the process had to remove most of those Registry settings). Well, now we're taking it back [anyone else remember Bono's intro to Helter Skelter?].

It took me a bit of time to track down the hidden place where Visual Studio "hides" the Registry settings for a Setup project: right-click the Setup Project ("AddInSetup_en" in this case), choose View, Registry. There's a stub for each hive, but not all of them contain actual Registry Keys or Values.

For my project, the only Key with any entries was HKLM\Software\Mindjet\MindManager\7\AddIns, with the Values (though different settings) that I'd noted earlier. The ProgID that it's setting is ParanoidMike.ToDoListBuilder.AddIn which isn't really representative of what I'm intending to release - ToDoListBuilder is just one portion of the functionality I want to provide (i.e. I've also spec'd out a ToDoListManager class that'll be a peer to ToDoListBuilder).

Also, if I'm going to define a namespace rooted in ParanoidMike, I figure I should use the application name as the next-level-down, and then describe the actual code/app/add-in as the third-level-down. That gives me the namespace ParanoidMike.MindManager.ToDoListAddIn, so I've replaced the existing ProgID with this as the new ProgID (and .NET namespace for my source code). I deleted the old Registry entries that reference this ProgID, added the new ProgID and its entries, and also added the HKCR settings that were still missing when I published this article.

That results in the following Registry entries populated in the AddInSetup_en project:

Complete Set of Registry Entries for MM7 Add-in

HKLM\Software\Mindjet\MindManager\7\AddIns

Value	Data type	Setting
Description	REG_SZ	MindManager ToDoList AddIn for MM7 - Built by ParanoidMike
FriendlyName	REG_SZ	MM7ToDoList
LoadBehavior	REG_DWORD	2

HKCR\ParanoidMike.MindManager.ToDoListAddIn

Value	Data type	Setting
(Default)	REG_SZ	ParanoidMike.MindManager.ToDoListAddIn

HKCR\ParanoidMike.MindManager.ToDoListAddIn\CLSID

Value	Data type	Setting
(Default)	REG_SZ	{33B60353-E8F3-4F38-98B6-41C7E5C6D32B}

Note: this little tip (HowTo: Create a Default Registry Value in Registry Editor) saved me a lot of trouble futzing around with these infrequently-created values.

Note: if you've used the MM7 AddIn template for Visual Studio, the CLSID above can be found in the Connect.cs file just under the /// <summary> comment, like so:
GuidAttribute("33B60353-E8F3-4F38-98B6-41C7E5C6D32B")

Open Issue: ProgID versioning

There's conflicting evidence on whether it's best to create the ProgID using the ".1" suffix notation, so that the ProgID can be rev'd, or whether it's just as good to leave it without the ".1" suffix and replace it upon every install (upgrade?). For the moment, until I stumble across anything better, I'm just going to leave it without.

Open Issue: ProgId in AddIn's source code

After cleaning out the old entries from my computer's Registry and testing this new set of code, I noticed that there was still the old ProgId referenced in the Connect.cs code:

    /// <summary>
    ///   The object for implementing an Add-in.
    /// </summary>
    /// <see also class='IDTExtensibility2' />   
    [
     CLSCompliant(false),
     ComVisible(true),
     GuidAttribute("33B60353-E8F3-4F38-98B6-41C7E5C6D32B"), ProgId("ParanoidMike.ToDoListBuilder.AddIn")
   ]

Since the add-in loaded and executed successfully, I'm now left to wonder what effect this ProgId entry is really supposed to have. Clearly it doesn't affect the operations of debugging the code (i.e. hitting F5 from within Visual Studio 2008 and waiting for MindManager to launch), but I can't imagine it's just there for decoration either. It's possible that, because I haven't rebooted since I cleaned out the Registry and updated the AddInSetup_en project settings, there's still enough of this information cached to allow the Add-in (with the same CLSID) to still work.

I'm taking no chances of a vaguely-specified error down the line, so I've updated this ProgId entry to the current value, but if anyone knows how these entries are related, I'd sure love to hear it.

MindManager 7 "ToDo List" Add-in development: Notes to Self

2008-02-18T10:26:00.001-08:00

Notes to Self: Object Model stuff I Might Use in my Add-in

Application.ActiveDocument: the currently-displayed Map
Application.Visible: enables/disables the MM UI
DocumentObject.Guid, DocumentObject.ResetDirty: this seems like a generic object type within the MM object model — might be useful for manipulating a whole collection of different objects at once
Document.Attributes: collection of document's custom attributes
Document.CentralTopic
Document.Guid
Document.IsModified
Document.Properties
Document.AssignMapMarkers(): assigns a map markers template to the document
Document.GetAttributes()
Document.Range(): returns a new Range collection of all the specified objects (topics, boundaries, relationships) in the document
Topic.AllSubTopics: subtopics connected to the topic
Topic.Attributes
Topic.Document: the document in which the Topic is found
Topic.Guid
Topic.Icons, Topic.UserIcons
Topic.IsFirstSibling: TRUE if the topic is the first in a set of siblings
Topic.IsMainTopic: TRUE if the topic is a child of the Central topic
Topic.IsSubTopic: TRUE if the topic is a child of another topic
Topic.Level
Topic.NextSibling
Topic.ParentRelationship
Topic.ParentTopic
Topic.SubTopics, Topic.UnfilteredSubTopics
Topic.Synchronization: ????
Topic.Task
Topic.Text
Topic.TopicLabel
Topic.TopicPrefix: sets the prefix for the topic — something to do with "numbering"
Topic.Type
Topic.AddControlStripType()
Topic.AddSubTopic()
Topic.CreateHyperlinkToTopicByGuid(): Cool! This could be used to hyperlink topics in ToDoList back to their original Topics. Might be useful to sync properties, attributes automatically.
Topic.GetAttributes()
Topic.NewTopicFinder(): helpful for traversing sub-topics from the current topic.
Topic.ResetDirty(): resets dirty bit of this and all subsequent objects. [Could be used to determine if a Topic has been changed since last "sync".]
Topics.Add()
Topics.AddWithGuid(): creates new Topic with a given Guid
CustomProperty.Value
CustomAttributes.GetAttributeValue(), CustomAttributes.SetAttributeValue()
Task.IsDone
Task.IsValid
Task.Priority
Task.Topic: the parent Topic for this object
TextLabels.AddTextLabel()
Icons.AddStockIcon(), Icons.AddCustomIcon()
Icon.Delete()
Utilities.Execute

Notes to Self: Questions to Research

What's the difference between a CustomProperty and a CustomAttribute?
- Attributes are persistent
- Nick Duffill has more info here
Is a TextLabel the same thing as a Text Marker?
What's the difference between a CategoryMarker, a CustomIconMarker and a CustomIcon?
What Events are available to which to respond (such as Topic.Moved)?

Nick Duffill alludes to "topic-delete" and "topic-add" here

Is there a way to find the current position among siblings for a Topic? Could there be a Property that indicates the Topic's position?

There is the Topics.Item property…

Can you get a handle to a Topic when you know its Guid? Do you need to be attached to its Document, or can you search all open Documents (e.g. searching through the AllDocuments or Documents collections)?
Is there some way (e.g. derive a class from the Topic class) that would allow me to define a Property whose Value was equal to its 1-index position among its siblings?
Does the Dirty flag get toggled automatically as soon as any change is made to the object? Or is this a flag that must be explicitly set?
Does the Dirty flag inherit up the tree — i.e. if any Object in a Document has the Dirty flag set, does that implicitly "set" the Dirty flag on the Document as well?
Can you reset the Dirty flag on a Document, and would that reset the Dirty flag on every Object in the Document all at once?
Could the InsertCustomProperties event be used to automatically propagate a Property to another attribute on the Topic object (e.g. taking the Index of a Topic next to its ToDoList siblings and convert that to a custom ToDoListPriority property, or to use the Priority icon's value as a ToDoListPriority value)?
Is the SetCustomPropertyValue event the one that'll let me trigger when the user changes the Topic's ToDoListPriority value (either automatically, by moving the Topic to a different position among its List siblings, or manually, by typing the value in or selecting it from a pre-populated list) to propagate that value back to the source Topic?
- If so, could I reset the Dirty flag on the Topic in the ToDoList, but leave open the possibility that the ToDoList overall is still "Dirty" and prompt the user to save unsaved changes?

Notes to Self: MM-specific Objects to Manipulate

There's all these new Object types defined in the MM object model:

BusinessTypeRegistry/Business Topics (see Nick Duffill's expert explanation of these Object classes here)
External Topics
Map Parts
MapShortcutCollections
Baselines
DocumentBars
Filter
Control Strips

Notes to Self: Code Tidbits

"use the Document.Range() method to iterate over all Objects" (Create and Use Custom Attributes)

Add Topics

Iterate over Topics (includes sample code for using the TopicFinder, an iterator)

Work with the Object ID (includes reference to FindByGuid() method that finds DocumentObjects by GUID in any Range, such as Document.Range)

Get an Object's Type (which refers to an enumeration of DocumentObject that includes mmDocumentObjectTypeTopic)

MindManager 7 "ToDo List" Add-in development Part 3: Initial Working Code

2008-02-17T20:45:00.001-08:00

Successful running of the add-in! Once I added the required Registry settings, the add-in finally made itself known (through the MessageBox'es). I learned that the HKCR settings are also necessary for other calls to succeed, but the essential behaviour of being recognized and loaded by MindManager was entirely dependent on having those Registry settings populated in the \Addins key.

At first I just used a .REG file to populate the Registry settings, following the model of the MM7 Sample Addin. I had to do this as a side-effect of the work the helpful folks on the Mindjet "Dev Support" team did to rip out a bunch of "custom actions" stuff from my MM7 add-in solution, and this appears to have included all the Registry settings as well. [Note: the Mindjet dev support folks are really helpful, and I'm not blaming them for this — the MM7 VS project template was (a) developed under Visual Studio 2005 not 2008 (which I'm using), and (b) was last updated last summer (and is no currently under active development I'd guess because the person who originally created it has since left the company — at least, that's the usual way these things occur).

However, it's not quite perfectly operating yet — just before the OnStartupComplete dialog box comes up, the following error is thrown by MM7:

According to this thread, this means that Addin ID must match the first parameter to applicationObject.Commands.Add(). In my code that parameter is "ParanoidMike.ToDoListBuilder.AddIn", and my first guess is that the key under HKLM\Software\Mindjet\MindManager\7\AddIns needs to have the same name (which it currently doesn't). Upon changing it to match, I finally saw the Add-Ins Ribbon tab that I hadn't been able to expose until now.

On to a little code…

First stage: Ribbon Button That Counts Tasks in Current Map

I wanted to use as much of the code from the MindManager7 Sample Add-in as possible… however somehow I ended up not using much or any of that. This was my original plan:

Create variable for MM application
- Sample\MmxUtility\Helper.cs: MmxUtility.Helper.GetMmxMap()
Beg/borrow/steal ribbon button code from existing add-in
- Sample\MmxUtility\RibbonTabBase.cs (base class for creating a new Ribbon tab)
- Sample\MmxUtility\ControlStripBase.cs
Attach function to Ribbon button
Create variable for Current Map
- Application.ActiveDocument
Find the Base Topic and enumerate
- Sample\MmxUtility\Helper.cs: Helper.BaseTopicFinder()
Create variable for Count
Create For Each loop of Task objects, and count 'em
- Sample\MmxUtility\Helper.cs (Topic attributes)

Instead, though, I ended up trying to separate out the UI from the data manipulation code, and leveraging the code I'd written from the MindManager DevZone article "How to Create a MindManager 7 Add-in Using C#".

This is what resulted:

        private void mmCommand_Click()
        {
            // Process the MindManager command and say hello
            MessageBox.Show("Hello world! (C#)", addInDisplayName);

            // Get the current Map
            Mindjet.MindManager.Interop.Document currentMap;
            currentMap = applicationObject.ActiveDocument;

            // Count the current Map's Tasks and display for the user:
            int count;
            ToDoListBuilder taskCounter = new ToDoListBuilder();
            count = taskCounter.countTasks(currentMap);
            MessageBox.Show("There are " + count.ToString() + " Tasks in this Map");
        }

    class ToDoListBuilder
    {
        public int countTasks(MMInterop.Document mindMap)
        {
            int tasksCount; // count of the number of Topics that are Tasks
            MMInterop.Range range; // just a collection of whatever we can find in the Document :)
            range = mindMap.Range(MMInterop.MmRange.mmRangeAllTopics, true);
            tasksCount = 0; // initialize to avoid error CS1065
            foreach (MMInterop.Topic topic in range)
            {
                if (topic.Task.Complete >= 0)  // this is intended to test whether the "topic" has Task attributes attached to it, and is how ResultsManager characterizes a Task
                {
                    tasksCount++; // increment this counter variable
                }
            }
            return tasksCount;
        }
    }

The key magic in this code was the test for whether a Topic is a Task or not:

if (topic.Task.Complete >= 0)

I'd spotted this approach in a macro that was published on ActivityOwner, but it seemed more complicated and indirect than should be necessary. I wondered whether something like topic.Task.IsValid() would identify whether a topic had "valid" Task characteristics. I looked into what documentation for this method was available, but the available info is pretty sparse. If not for the generous help from the MindManager development community like Nick Duffill, I would've been forced to work through this by trial & error.

Which Security Event Log audit categories are most useful on a Windows client?

2008-02-12T17:17:00.001-08:00

Let's say you're looking to maximize the value of the data logged to the Security Event Log on a huge number of Windows clients (say, Windows XP SP2).

Further, let's assume that you're not inspecting such logs on a regular basis, but instead you just want to keep the most critical events in case you have to track down some "suspicious activity" later. [Suspicious activity would probably include such things as successful intrusions into the PC (whether by attackers or malware), which is going to be a losing battle but worth trying.]

You have two different sets of knobs to twiddle: which categories of security events will be logged, and how the security Event Log will be configured. The categories are the more involved thinking, so let's start with the Event Log configuration first, shall we?

Security Event Log configuration

The default Security Event Log size on Windows XP is a paltry 512 KB. [It got boosted on Windows Vista, so don't go yelling at Microsoft — they heard ya already.] The question isn't if you should increase its size, but by how much?

When it comes down to a "best practice", I've always found it to be an arbitrary choice. This choice should be informed by the level of activity you expect (or tend) to see — many customers who turn on all logging options can fill up a 10 MB log in the space of a week, but those who make more judicious choices can survive on 2048 KB for sometimes a month.

The upper limit is somewhere in the neighbourhood of 300 MB, but that limit includes all Event Logs (even custom event logs created by other applications, I believe) — this is documented in Chapter 6 of Threats and Countermeasures. So for example, if you've already set the System and Application logs to 50 MB apiece, I would strongly advise a Maximum log size of somewhere around 150-200 MB for the Security event log. [Note: there is a bug that causes problems with Security event logs over 50 MB, which hopefully has not only been fixed in Windows Server 2003 but also Windows XP SP2.]

Aside: I'm sure there are others you might find, but on my own Windows XP box I've got four additional custom Event Logs:

Microsoft Office Diagnostics
Microsoft Office Sessions
Virtual Server
Windows PowerShell

The next setting to consider is how the logs will respond when they (inevitably) fill up. There's a setting innocuously labelled When maximum log size is reached, and there's no perfect selection for everyone. I've generally advised people to choose Overwrite events as needed, since most times, my customers would be interested in having a record of the most recent activity on the PC (e.g. tracking down details of a recent virus outbreak or suspected break-in attempt).

Finally, if you're really anal about your Security Event logs (and what security geek doesn't ideally want to keep them around forever?), you can enable one or two other specialized settings created just for you — but should you?

WarningLevel: recent versions of Windows can warn the Administrator when the Security Event log is nearly full (the usual recommendation is 80 or 90% threshold). Windows will record a single System event with EventID = 523. However, this is really only useful in cases where the Administrator wants to archive all Security Event Log records for later analysis or compliance checking, and they don't already have an infrastructure for collecting and centralizing this logging info. Warning someone of imminent failure, when they have no way to avert disaster, is really just a tease. Thus, the more useful setting is…
AutoBackupLogFiles: Rather than let the log files overwrite themselves, some would prefer to archive all log entries. This registry setting enables Windows to automatically backup and empty the specified Event Log, so that all the entries are stored in a local file on disk. This isn't perfect (a malicious attacker could wipe them out, for instance) but in cases where you just can't imagine copying the security Event log between the time the 90% alarm goes off and you get the time to deal with it, this can be an effective alternative. The most significant consequence of this is, over time, you may end up filling the OS volume with these archived files. However, shunting such saved data to a separate, non-OS volume — or monitoring for disk space — are the kinds of problems that aren't difficult to solve.

Security Event Log Category choices

Now the tough part: deciding which Success & Failure event categories to enable. Leaning on Eric Fitzgerald and Randy Franklin Smith, here's the current thinking I'm advising my customer for keeping the noise down (and which you're welcome to leverage, if our thinking seems to fit):

Account Logon

This'll identify the local (i.e. SAM-based) usernames that users have attempted to logon at this PC
If you're interested in tracking actual user activity and successful break-ins, then enable Success auditing.
If you're interested in (and plan to actually investigate) attempted but failed break-ins, and if your users don't use local accounts (and thus won't be the overwhelming cause of failed account logon attempts due to fat-fingering their password), then enable Failure auditing. Under such circumstances, this shouldn't be a significant contributor to the security logs.
Recommendation: enable Success and Failure auditing.

Account Management

This'll identify such things as account creation, password reset and group membership changes.
Under normal circumstances these should be highly useful records (both the successful changes and the attempts) — especially if you don't often manipulate local accounts on your XP clients.
Recommendation: enable Success and Failure auditing.

Directory Service Access

pointless — this only applies to Domain Controllers
Recommendation: No Auditing

Logon events

In a non-domain context, this doesn't add much value over and above Account Logon auditing
Recommendation: No Auditing

Object Access auditing

This is a tricky one. It logs little or nothing by default, even when Success and Failure auditing are enabled for this.
Used correctly, you can collect information with a fairly high signal-to-noise ratio.
Used incorrectly, however (and I was as guilty of this as anyone in my early career, and am still guilty today), and you'll wipe out any useful information that the security log might've otherwise kept for you.
For example, I'm currently recording "Handle Closed" and "Object Access Attempted" events dozens or hundreds of times an hour. What is being accessed? LSASS. Why? Because of a single "Everyone: Full Control" auditing entry I added to the EFS\Current Keys registry key, to try to track down some odd behaviour a few months ago. I'd forgotten about this ever since, and now I'm filling my 10 MB security log every 36 hours.
If you follow a VERY specific set of SACLs as in the EricFitz article linked above, then you will get some real value out of this category.
Recommendation: only enable Success and Failure auditing if you have specific activity you're looking for, but be VERY careful when setting any SACLs on the system.

Policy Change

I've never seen anything in this category that helps really track down malicious behaviour
While it may be interesting to highlight attempted (or successful) changes to Audit policy or assigned user rights, I'm extremely skeptical that any of this information would be conclusive.
However, with Windows XP SP2 and the use of Windows Firewall, there are a number of very specific audit records (e.g. Event IDs 851, 852, 860) that track changes in the Windows Firewall configuration. [It's unfortunate that there's not better info on the source of those changes.]
If you're using the Windows Firewall in XP SP2, these records could well be useful in isolating the source, cause, or spread of a malware outbreak.
Recommendation: enable Success and Failure auditing when using Windows Firewall.

Privilege Use auditing

One of the greatest sources of log pollution, with little practical application.
This looks very useful to a security geek on paper, but in practice 99% of the recorded events will be (a) legitimate behaviour and (b) completely harmless.
Recommendation: No Auditing

Process Tracking

Aka "Detailed Tracking" (which is how these events are labelled in the security Event Log)
A great way to swell the size of your security logs, unless your PCs run a very small number of applications for very long periods of time.
However, when you're using Windows Firewall, Failure auditing will record (in Event ID 861) a number of potentially useful pieces of information about any application that attempts to open an exception in the Firewall rules.
This logging can be very frequent (I show over 2000 events in the last 36 hours on my PC), but will give very detailed information on the the Port opened, the process that bound it, and whether the process is a service or RPC application.
(One good non-security use for this auditing capability is to troubleshoot unknown application behaviours.)
Recommendation: enable Failure auditing when using Windows Firewall.

System events

The only semi-useful information I've ever found from this auditing are the startup and shutdown events, and they're much more useful in determining uptime statistics (and otherwise unseen BSOD events) than they are for security.
Unfortunately, these events get buried under the amazing number of 514, 515 and 518 events that accumulate in the space of a few days.
Recommendation: No Auditing

Summary: Windows XP Security Event Log auditing category recommendations

Security Event Log Category	Recommended Audit Level
Account Logon	Success, Failure
Account Management	Success, Failure
Directory Services access	No auditing
Logon events	No auditing
Object Access auditing	No auditing*
Policy Change	No auditing*
Privilege Use auditing	No auditing
Process Tracking	No auditing*
System events	No auditing

* except in unusual circumstances, see above.

Advanced Oddities

Per-user Auditing

As of Windows XP SP2, auditing can be enabled or disabled for any or all users
Each category can be separately configured as well
On a PC with many user accounts, this would be useful to help remove the less interesting entries
However, where few accounts exist, and for PCs not joined to a domain, per-user auditing is not advised

Windows Firewall auditing

As I hinted above, there are some aspects of Windows Firewall's operations that can be logged to the Security Event Log, and which don't get logged to the pFirewall.log.
For organizations using Windows Firewall, and especially those that don't have a perfect idea of all the exceptions they need to open up on their user's systems, this auditing can be extremely useful.
Recommendation: To capture this data, you should enable Policy Change (success and failure) and Process Tracking (failure) auditing on the target systems

File/Registry access auditing

If you're interested in detecting attacks that tamper with system files, then EricFitz has some fascinating work you should examine
His work became the input for the Security Configuration Wizard in Windows Server 2003 SP1
Having had a quick look at it, there's nothing that looks dangerous or unsuitable for an XP client
Recommendation: if you'd like a quick & dirty way to detect changes to system files, cut and paste those "file access auditing" settings from the SCW templates, and make sure that you've also enabled Object Access auditing (success and/or failure, depending on whether you're after actual changes or just attempted changes)

Full Privilege Auditing

You can toggle a Registry setting known as (duh) FullPrivilegeAuditing, but be warned: these are default disabled for good reason
Recommendation: do NOT enable this setting

Audit the access of global system objects

Ever since this got added late in the NT4 service pack cycle, I've never quite figured out what this really tells me. Eric doesn't seem to interested in this either for most of us.
Recommendation: turn this setting Off

Audit the use of Backup and Restore privilege

This setting blows me away — it'll fill up the most generous security event log, 'cause it creates an entry for each file that is backed up or restored
Recommendation: do NOT enable this setting

CrashOnAuditFail aka "Shut down system immediately if unable to log security audits"

Are you nuts? Have you ever met a sysadmin that voluntarily puts in place a predictable Denial of Service attack?
If you're that one-in-a-million organization that can actually implement this setting, I want to hear from you. Yours is a tale I just gotta hear…
Recommendation: duh, do NOT enable this setting

For More Information...

Eric Fitzgerald is an old colleague of mine from my days at Microsoft, and I have an incredible amount of respect for the depth and persistence with which he pursued issues in the Auditing subsystem of Windows over the years. He's like the Rain Main of Windows security eventing, except I don't think he's much of a fan of Wapner. ;) Eric's "Windows Security Logging and Other Esoterica" blog is chock full of Windows security auditing goodness.

Windows Security Log Encyclopedia — Randy Franklin Smith's take on Security Event Logs

Technet Events & Errors Message Center — detailed information backing up each security Event ID and what it means.

Deciphering Account Logon Events — in case you wonder what "Logon Type 5" really means…

Account Management — disabling the noise — and we're done!

[Apologies to anyone monitoring my external blog, as this is a straight repost. However, I'm assuming very few of you know about both, so I'm going to start reposting anything that's applicable to both audiences.]

As David Hsing says: Best. Troll. Ever.

2008-02-05T11:01:00.001-08:00

Holy crap that's funny:

http://talkback.zdnet.com/5208-12355-0.html?forumID=1&threadID=31199&messageID=579806&start=43

Reproduced here for those (like me) too lazy to click through:

You are kidding aren't you? Are you saying that this linux can run on a computer without windows underneath it, at all? As in, without a boot disk, without any drivers, and without any services?

That sounds preposterous to me.

If it were true (and I doubt it), then companies would be selling computers without a windows. This clearly is not happening, so there must be some error in your calculations. I hope you realise that windows is more than just Office? It's a whole system that runs the computer from start to finish, and that is a very difficult thing to achieve. A lot of people don't realise this.

Microsoft just spent $9 billion and many years to create Vista, so it does not sound reasonable that some new alternative could just snap into existence overnight like that. It would take billions of dollars and a massive effort to achieve. IBM tried, and spent a huge amount of money developing OS/2 but could never keep up with Windows. Apple tried to create their own system for years, but finally gave up recently and moved to Intel and Microsoft.

It's just not possible that a freeware like the Linux could be extended to the point where it runs the entire computer from start to finish, without using some of the more critical parts of windows. Not possible.

I think you need to re-examine your assumptions.

YES (for the sarcasm-impaired), this is a joke, and it's NOT my writing. Don't bitch at me if you are rabidly anti-Windows — click on the link above and rant away to your heart's content.

Debugging a Word 2003 runaway thread...but not successfully

2008-02-02T18:58:00.001-08:00

I just experienced one of the usual "hangs" in Microsoft Word 2003 that happen pretty regularly when working on multiple, large documents for any significant length of time. The WINWORD.EXE process is taking up 50% of my CPU (which as a dual-core processor, means that there's a thread that's somehow taking up 100% of the logical CPU for which it's scheduled), and has been doing this for at least ten minutes now with no letup.

In my experience, these "runaway consumers of CPU cycles" just never quiesce — eventually I have to decide to kill WINWORD.EXE from Task Manager or Process Explorer, or else the offending process will consume that "CPU" from now until the end of time.

Maybe I was just bored today, 'cause rather than just kill the runaway process, I decided to see if I could dig a little deeper. [I think Mark Russinovich has infected me with the idea that these are surmountable problems — though I wouldn't dream of trying to make a favourable comparison between my haphazard hacking and Mark's mad skillz.]

Process Explorer

Let's have a look at a few screenshots, shall we?

(Performance stats, in case that's useful to anyone — though it doesn't provide me any telling evidence)

(Listing of the threads currently instantiated in WINWORD.EXE including the main thread, which is the one causing all the problems)

(Stack contents for the WINWORD.EXE thread)

(Stack contents for GdiPlus.DLL thread, which was the only other thread with any activity under the "CSwitch Delta" heading)

Process Monitor

Once I decided to investigate, I fired up Process Monitor and limited it to WINWORD.EXE. The activity logged is almost entirely like this:

Don't strain your eyes too badly on this — I've included this just to note the incessant nature of the major activity here: a rapidly-repeating WriteFile operation on a single Temporary file (~WRS1954.tmp), interrupted once in a while by a smaller (Length of anywhere between 512 and 3072) ReadFile operation on the same file:

Interestingly, these ReadFile operations occur in an irregular but repeating pattern:

Also of note is the fact that this temporary file is constantly growing in size, and not just temporarily swelling the data stored within a pre-allocated file — I confirmed that by right-clicking on the Path in Process Monitor, chose "Jump to Location…" and simply Refreshed on the folder to observe the reported file Size was incrementing every time (over a span of 50 minutes, it grew by approx. 222 Kb, or 233657856 bytes).

If I look closer at the Handles for WINWORD.EXE, I notice that this is one of many Temporary files open by Word, which implies that the problem we're experiencing is very specific to one type of unexpected activity (and not just affecting Word's handling of Temporary files):

(Note: I intentionally "hashed" out the original filename, which is the last entry in the list pictured.)

One other piece of information: I tried to resize the Window in which the active document was being displayed. Word appended "(Not Responding)" to its Title Bar, and that seems to have changed the behaviour profile of the WINWORD.EXE thread. Since that point in time, Process Monitor did not record any further increase in the size of the ~WRS1954.tmp file, but recorded one additional ReadFile operation on the WINWORD.EXE file itself (Offset: 3998720, Length: 4096). [WINWORD.EXE File version = 11.0.8169.0, Digital signature timestamp = May 31, 2007 12:38:03 PM]

Finally, I grabbed a full memory dump of the WINWORD.EXE process, using windbg.exe and the .dump /ma command. I can't say I know much about debugging a dump file, but I've got it on the off-chance that I ever find a good guide to debugging.

What Caused This?

Three circumstances I think contributed to this, though in my opinion none of them should lead to hung process (since I've done this more often without incident):

I had opened a Word 2003 document directly from Outlook (it was attached to an email).
The document had Track Changes enabled, and I'd already added Comments throughout the document.
In the Comment I was just editing, it had scrolled off screen…

…and I had just attempted to apply formatting (I'd typed [Ctrl]-B and [Ctrl]-I rapidly,to bold and italicize) to a single word in the Track Changes panel below the document (the one that opens automatically when you keep typing in Comments that have already "scrolled off screen").

(Note: I intentionally redacted the confidential text — but it sure ain't artistic)

Caveat: While my experience with Word over the years has taught me that heavy use and abuse of the Comments feature leads to instability, I'm still miffed that I'd lose the recent batch of edits just because I'd foolishly tried to emphasize my point using basic formatting in a Comment.

So What Can We Conclude So Far?

I don't know much about reading a stack trace, so this is all guesstimation on my part (plus a little intelligence gathered from a good Russinovich article). The WINWORD stack indicates that Word has called ntkrnlpa.exe aka the Windows kernel. It looks like it's basically stalled (KiDispatchInterrupt) while creating a new thread (KiThreadStartup). Looking lower in the stack, the first caller in WINWORD is labelled only "0x1a772b" — whatever that is, it's beyond my skills to unearth the identity of that API.

The next one down in the stack, however, is wdGetApplicationObject(). There's no information in MSDN that references this function, though a few pages on the 'net do allude to it (mostly in the same kinds of searches I made). The best info I could find was here, which I'm guessing is Word's way of getting a handle to the overall Word "application object". However, without any further context, it's very hard to imagine what is really going on here.

Turning to the GdiPlus stack, it looks like another kernel call that's stalled (many similar references to "WaitForMultipleObjects" functions), all boiling down to a call to the GdipCreateSolidFill() API. From what MSDN documents, this seems like a pretty innocuous function, having nothing to do with temporary files, only to do with UI. I can understand this — by the time I'd looked at the GdiPlus stack, I believe the UI had "hung" (aka it was non-responsive). So while this thread was also active, it's almost impossible for it to be involved in this issue.

Then the only thing I know for sure is the temp file was growing due to some runaway operation, and the runaway operation (which was probably related to an attempt to format Comment text) at some point obtained a handle to the Word application object.

I'm guessing that the only way to get any closer to the root cause would be to dig into the memory dump. And…bugger me, the dump I grabbed ended up with this as its STACK_TEXT (from !analyze -v):

0adaffc8 7c9507a8 00000005 00000004 00000001 ntdll!DbgBreakPoint
0adafff4 00000000 00000000 00000000 00000000 ntdll!DbgUiRemoteBreakin+0x2d

Guess that's "the wall" for me.

ParanoidMike

Stalk Google's stalking of you

Got the "2012 security suite" infection? This should fix it

2012 Manifesto

Status and failure of CacheMyWork development

Review: Jinx

Excommunicating myself from the Church of Alan Moore

Considering learning Python - idle thought (until something catalyses it)

Got a new iPhone 4S? Worried about how to "maximize the battery"? Me too

Usability and UX - are you a practitioner? How to hone my instincts?

Review: S.H.I.E.L.D.: Architects of Forever

Review: The Other Side

Review: Atomic Robo Vol. 3: Atomic Robo and the Shadow from Beyond Time

How can I automatically crosspost/publish Google Reader "shares" to Google+, Facebook or Twitter?

Review: 52: Vol. 4

How can I publish GoodReads reviews on Google+, Facebook or Twitter?

Review: Final Crisis

Building an Outlook add-in: troubleshooting an add-in that won’t start

Building an Outlook add-in: binding data to XAML (questions)

ToDo’s (from last post)

Bind AD Data to XAML

Where to Store the Data?

How to “Push” Data to UI?

Next Steps

Found: Minimum permissions to edit Site Columns in SharePoint 2010

Permissions: Summary

Site-level Permissions

List-level Permissions

Steps We Followed To Distill These Permissions

Building an Outlook Add-in: completing work on the Active Directory query

ToDo’s (from last post):

Acquiring the user’s Email Address

Completed GC-querying Method

Next Steps

Building an Outlook add-in: Context Menu of Contact Card in Outlook 2010

Creating a WPF Control for use within Outlook 2010

Problem: UserControl doesn’t Show()

Further Research References

Building an Outlook Add-in: customizing the Contact Card, other options?

Contact Card

Context Menu vs. Properties window

Details: Fluent UI

Silverlight training, note to self: adding non-default Assemblies for additional controls

Silverlight training, note to self: blank page

Lenovo T61? High CPU usage? Look for bad drivers (…spewing hardware interrupts)

Step 1: Device Manager

Step 2: KernRate

2.1 Setup & Config for a valid KernRate result

2.2 Use KernRate to profile kernel activity

2.3 KernRate detailed profiling on individual modules

2.3.1 NTKRNLPA

2.3.2 HAL

2.3.3 IGXPMP32

2.3.4 HECI

Step 3: Analysis & Research

Step 4: Make sure you try later and earlier driver versions

Notes: related articles

Sysinternals Forums

Lenovo Forums

My Mac Mini HTPC saga: TV tuners

Do I really *need* a tuner?

Finding all the digital channels available to be recorded with your tuner

Future question:

What do you like? Your ratings in multiple places – do you want to sync them?

Web 1.0: Amazon Recommendations

Web 1.0: Netflix Recommendations

Web 2.0: social + recommendations

Problems: stale data, incomplete data, spread-too-thin efforts

This Ratings Data Ain’t Portable, My Friends

How’s About a Ratings Sync App?

My Mac Mini HTPC saga: software & configurations

Plex, Boxee, EyeTV, VLC, Transmission, Silverlight, MacTheRipper

Plex

Boxee

EyeTV

VLC Media Player

Transmission BitTorrent client

Silverlight 3.0 (for Netflix)

MacTheRipper

Issue: Mac OS X 10.6 intercepts all Apple Remote commands

Do I really need a tuner?

Dude Abides
The Dude Abides - bumper sticker

Lots of geese
200+ Canada Geese just gathered in Hillsboro OR - what are they plotting?