Since it’s going to be rather technical, the details are hidden behind the cut…

Except this is OSX (Tiger in my case). So there is no Gems. Awesome. Before you head off to install it, be aware that the Darwin Ports version of it seems broken. The best way is going directly for the source.

curl -O http://rubyforge-files.ruby-forum.com/rubygems/rubygems-0.9.0.tgz
tar xzvf rubygems-0.9.0.tgz
cd rubygems-0.9.0
sudo /usr/bin/ruby setup.rb

OK, gems is installed. Next stop: simple-rss, a prerequisite for rubyrss. Conveniently enough it’s available as a gem. Except the rdocs are broken and cause gem to spit out a flood of error messages. Seems the developers know about it, too – but why rush a new gem just for a couple of warnings…. Anyways: disabling doc installation does that trick.

sudo gem install --no-ri --no-rdoc simple-rss-1.1.gem

That leaves the main package. And because it’s so much fun, it’s source-only, not packaged as a gem. Installation pains are to be expected, so I created a gem for you (and me) to use: rubyrss-1.1.gem

sudo install rubyrss-1.1.gem

And after all this, I’m finally ready to work on the main task. More on watching directory changes in a later post…

Tags: ruby osx rss

So how does it all work? The most important piece in the puzzle is the fact that almost all Apple Applications are using ObjectiveC – a message based programming language. That means no function calls, just message dispatches. It’s not that important to plugin development, but crucial to reverse engineering. Every message is dispatched through one central function, by name.

The question then is, how do you get to read the messages? We need to somehow hook into a running application. When I started this project, I thought about a rather convoluted solution using mach_inject – it allows you to inject arbitrary code into a running application. I thought I’d use that combined with mach_override to override the central dispatch function objc_sendmsg.

A quick trip to the extendamac mailing list on SourceForge, and the resident gurus had set me straight, though. Objective C already has message logging functionality built in. All I needed was to call instrumentObjcMessageSends(YES); inside iCal’s application space.

There are, as far as I know, two choices to do that. Either you use mach_*, which is a tricky business, or you write an input manager – all OSX applications load input managers – which is a slightly less tricky business.

This is where a second major piece comes in – SIMBL, the Smart InputManager Bundle Loader. It’s a piece of software that uses an input manager to load user-written plugins. All the heavy lifting is done in there, and you just need to create a bundle (sort of a DLL, for you Windows guys) with one entry point.

The code to activate message logging then boiled down to:

+ (void) install
{
    instrumentObjcMessageSends(YES);
}

Compile that as a bundle, and SIMBL will make sure it gets loaded at startup. All messages the hooked up sends from then on are logged. As you’ll soon find out, Cocoa Apps send a lot of messages, so prepare for some digging through multi-megabyte text files.

After a little while, you will have figured out which message does what. Once you are there, you need to somehow take over or intercept those messages to do your business. Since ObjectiveC is a relatively dynamic language, it’s easy to replace a method on a class with your own method – just what we want to do. The CocoaDev website has a great explanation of MethodSwizzling.

Now we’re almost there – we hooked the critical messages, but we actually need to modify data on objects to achieve any visible effect. This is where our third helper comes into play – class-dump, which can be downloaded here.

class-dump reads an executable and presents you with an API declaration that you can simply include in your source. Once you do that, you can call all the object modification functions you want. In short, you become a part of the executable you hooked.

All this allowed me to create iCalFix with less than 100 lines of code. It’s rapidly growing, since I’m busy adding new features, but the core hack is extremely simple – thanks to a lot of hard work done by others.

If you’d like even more detail, Mike Solomon, the author of SIMBL, has a great page on reverse engineering Cocoa programs.

Tags: osx, cocoa, SIMBL, objective-c, plugin, reverse engineering

While VS.NET does offer you conditional breakpoints, it does so by evaluating an expression every single time that breakpoint gets hit. Clearly, that’s not acceptable – especially if the breakpoint is hit often and the condition is rarely true.

You can speed this up quite a bit by moving the condition into actual (compiled) code. For that, you need a line of code that is only reached when the condition is true. I.e.

if(condition) {
   DoSomethingWithoutConsequences();
}

A breakpoint there will only be triggered if the condition is true, and the condition is evaluated in compiled code – just what we want.

The only drawback is that doing something without consequences gets ripped out by the optimizer ruthlessly. Which causes the if() statement to be removed too, and our whole scheme collapses. The most widely known way to work around this is incrementing a volatile global – but that adds even more code to type, and we can’t do it as a simple paste operation in a single place.

Thankfully, there’s an easy solution to this – the optimizer never touches inline assembly. So turn the above into:

if(condition) {
   __asm{ nop };
}

Then, in the debugger, put a real breakpoint on the assembly instruction, and you’re good to go.

Tags: debugging,Visual Studio,conditional breakpoint

All the freely available site counters give you a piece of JavaScript. I personally used one from SiteMeter, but all the other sites work pretty similarly.

With WordPress, you can’t simply put it into index.html – it’s template based. You need to embed this code in a template that is always displayed – for most WordPress themes, that’s the SideBar.

So how do you edit it? Log in as site adminstrator. In the Administration page, go to the Presentation Page and select the Theme Editor. Select the Sidebar template. It will show up as a PHP file – but don’t worry. It really is HTML with a bit of PHP thrown in for good measure.

The whole sidebar is usually embedded in a single <div> tag – find the end of that tag. Right before the end of that tag, paste your javascript that you got for the side counter. That’s all there is too it.

On some WP installations, you might not be able to edit your themes through the web interface – in this case, below the textbox, you will see the string “If this file was writable you could edit it.“

That means you need to actually log into the shell for your account and modify the file there. Thankfully, the web page gives you the path to the file relative to the WP installation directory as a title above the text box – so just cd to there, and edit the file. Et voila, your site counter is active!

Know the landscape

Since we’re talking strategy, let’s stick with a “war” analogy for a moment. As in any battle, knowing the landscape is key. If you don’t understand the environment you are in, you might miss clues, misinterpret others and put way more effort into finding the problem than necessary. This means

• Know the general ideas and data paths of the debugee
• Know your tools
• Know the environment. OS, hardware, etc.

Make it reproducible

Once you have a general understanding, you are prepared to actually tackle the bug. And the first step to tackling it is making it reproducible. Make it at least semi-consistently reproducible. If you cannot reproduce it easily, it is very hard to track down any problems. And before you fix it, always create a test that reproduces it 100% – otherwise, you’ll never be certain you fixed it.

Keep a log

Once you can reproduce it, write it down. You will forget it otherwise. Believe me, I’ve been there. In general, write down everything relevant so you can recall it easily. Some people keep it in their development log. For me, the best approach has always been putting all known facts on a whiteboard. This way, team mates can often offer insights.

Collect data

Now that you can make the problem happen, it is tempting to speculate what the problem is and just fix the code that is “obviously wrong”. Don’t give in to that temptation – most likely, it’s something completely different.

Any good theory is based on observations, and at this point, the observing has been slim – you only saw a manifestation of the bug. You don’t know if it’s the only manifestation. You don’t know if there’s more than one cause. You don’t know jack.

That’s fine, though. Now is the time to collect data. It’s nothing else than the scientific process – you collect data, form a theory, create testable hypotheses. Lather, rinse, repeat until your theory matches reality.

The collection of data can take many forms. Step through the code, add instrumentation to the code, use any kind of analysis tools available to you. Keep in mind that instrumentation quite possibly might change behavior – so be sure to test your hypotheses with the instrumentation removed, too. Heisenberg is alive and kicking.

No fear

Don’t let emotions override the process. Yes, sometimes the pressure is insane – you’ve got three hours before burning a disc, and you just found a crash bug. That’s OK – as long as you keep your head, you will find it. Just calmly collect data, and examine every hypothesis against the known data. Not only will it help you get there faster – it will also calm down your teammates, help them keeping their head.

Reduce the noise

“Hold on”, you say. “My code base is way too big to step through it!”. Yes, it is. It always is. That’s where you do like the Romans did: Divide & Conquer. Split your code into smaller and smaller pieces, excluding the parts that are not contributing to the problem.

One of the best ways to do that is find a piece of code where the problem shows itself, and then follow the bad data up the call chain. That usually gives you a good idea of what systems are involved. (Here’s where knowing the road map is important!). As much as possible, shunt things that are not involved.

Another way to reduce the clutter is to reduce the amount of data – fix inputs not contributing to the actual problem. Force them to always be the same value.

If you have a “known good” version, it becomes even easier – look at the changes since then. If your build pipeline allows it, do a binary search through all the changes until you’ve narrowed it down to a single problem. (Know your tools!)

So now you’ve narrowed it down enough to build a theory that matches your observations. You’re ready to make the fix. You have a test case that reproduces the problem every time.

How do hedgehogs make love? – Very carefully!

Step back for a second, take a deep breath, and repeat after me: “One change at a time!”. Go step by step. See if the change yields results that fit your theory. If you change multiple things, tracking interactions between them is incredibly hard.

If you still can’t find it

Of course, sometimes things aren’t that easy. There are two pieces of advice for the “I can’t find it!” bugs.

1. Double-check your assumptions. If the monitor is dark, and you know it’s plugged in, do yourself a favor – see if it is really plugged in.
2. Find a debugging buddy. Explain the symptoms – often, that’s enough to smack your forehead. Either way, don’t share your theories, since they come with built-in assumptions and might mislead.

Happy hunting

That’s it. That is all there is to successfully debugging even the toughest problems. Yes, special problems require special techniques. But this is the general strategy. I’ve been following it for quite some time, and it seems to work for me. If you try it, let me know how it works for you.

Safari, InfoWindows, and formatting

Google Maps’ InfoWindows (the bubbles that appear when you click on a marker) are broken for Safari. It will add linebreaks after pretty much every word. After a bit of googling around, Google Groups (notice a pattern here?) reveals a helpful thread. Basically, you need to wrap all your text into a <div> tag with a width style element. Like this:

  <div style="width:204px;">some text</div>

Better XML

It had to happen – somehow, invalid elements snuck into my XML. I don’t know where they came from, since my source data is HTML – at least a close enough relative. They still happen. And if they sneak into the XML data file for the markers, the markers will just not appear.

And, just to compound matters, the offending entities are different for Safari and IE. Safari complains about entities like &oelig; “&oelig” – which is silly, since the file should be treated as XML, and those entities are meaningless unless defined in a DTD. IE chokes on “&”, which is less silly, since it’s part of, for example &lt;, which is used to escape “<” inside tags. (Of course, Safari can stand that. And Firefox has yet another set of issues…)

The point remains, though – it breaks my application, and there’s no way to predict how and why. So the solution is, “replace all unknown entities”, and “Check the data file in each browser”. Thankfully, they even offer the line number where they choked, so it’s fairly easy to fix if I run into a new issue. Technically, I probably should be wrapping the text into a CDATA section. It’s on the to-do list

Previous Articles

If you just stumbled upon this page and want to know more about challenge maps, here are the previous ones:

• Challenge Map II – Google Maps
  Covers the Google Maps part of the application, and pitfalls encountered therein.
• Challenge Map I – Python Web Scraping
  Covers extraction of data from web sites using Python – which ends up being surprisingly easy.

So let’s look at the same example in Python:

registryKey   = \
  'SYSTEM\CurrentControlSet\Control\Session Manager\Environment'
variableName    = 'SOMEVAR'
value         = 'Another String bites the dust!'

registryHandle = OpenKey( HKEY_LOCAL_MACHINE, registryKey ) SetValue( registryHandle, variableName, REG_SZ, value ) CloseKey( registryHandle )

SendMessageTimeout( HWND_BROADCAST, \ WM_SETTINGCHANGE, \ 0, 'Environment', \ SMTO_ABORTIFHUNG, 5000 ) So what are the advantages? At first glance, you’ll see the Python program is short – half as short as the C++ sample (13 vs. 27 lines ). “How come?”, you ask. One main reason for this: Exceptions

Error cases in Python are consistently handled as exceptions. There’s no need for lines and lines of error checking that do little else besides propagate the error to the next layer – which is what most C++ error handling does.

“But… aren’t exceptions only for truly exceptional cases?”

Short answer: Yes. Long answer – if a function doesn’t do what I expect it to do, that is an exceptional case. And at least with exceptions, I’ll know the moment it happens. Most C++ code silently ignores return codes, which hardly qualifies as a better solution.

Plus, I happen to think that error codes really are out-of-band communications, which is much better expressed by exceptions than by return values or extraneous parameters.

“But you could do that in C++, too!”

Yes, I could. If I wanted to write even more code. Exceptions are not part of the language philosophy, which means many of the APIs don’t support them. Python internalized exceptions, and the API reflects it. My goal is to write short, expressive code. If there are standardized building blocks to do that, I’m happy to use them – but C++ really doesn’t have much in terms of a support library. Which happens to be a point in favor of any post-Java language – they come with massive APIs that solve most of the basic problems for you, so you can concentrate on the problem at hand.

There are a couple more pluses to Python, so let me mention them really quick:

• Dynamic Typing
  I don’t have to spell out that a string is really a string, and a number is really a number. Variables in Python take the type of whatever they contain. Types in C++ are for the most part busy work – you’re just spelling out something again that you’ve already said a few times.
• Indentation
  Yes, that’s right. I love the mandated indentation format for flow control – because it forces even the sloppiest of programmers to indent their code so I can read it. C++ programmers do the same thing – they just call it a coding standard and fight to the death over it. Me, I’d rather focus on the problem instead of layout opinions.

There’s plenty more, but that’s for another day.

“If Python is so great, why didn’t you use it right away?”

Fair question. I actually did use it to prototype the solution, but two things conspired to make me convert this. First, I wanted to use SendNotifyMessage() – and there’s no Python binding for it yet. (Yes, that is a drawback of Python – I didn’t say it’s perfect!). Second, the article was mostly a summary of older notes for a colleague at work – so I picked a language he’s more comfortable with.

“So what other reasons are there NOT to use Python”

Oh, plenty. The main one I’m hearing is plain old fear of the new. I hope that’s not what’s stopping you from trying it out. However, there are some real limitations. If you need to go “direct to the metal” – i.e. interface with the OS or some hardware – you need bindings. Which usually require a dose of C/C++.

Even more importantly, if you need raw performance, stay away. If your application is slinging around hundreds of megabytes per second, Python might not be your tool of choice. You do pay a price for all those exceptions and dynamic types. The usual advice in that case is to “write the performance critical part in C/C++, the rest in Python”. That might work – I’ll defer you to others who’ve actually tried it.

Or maybe you already did try and can tell me if it worked?

In a nutshell, you set the registry key corresponding with your environment variable and send a WM_SETTINGCHANGE message system-wide. And since it’s a system-wide send, you need to send with SendMessageTimeout – otherwise a single hung process will bring your program to a halt, too.

There’s only a small extra bit of information I can add: If you make this change from a program that keeps running, SendNotifyMessage() is a better choice – there’s no timeout, and Windows will keep trying to deliver your message. Without actually hanging your process. Obviously, this doesn’t work when you terminate your program, since the data you’re sending is now non-existant. (That’s what you get for sending around pointers…)

And, since I need this from time to time and love a code sample, here’s the actual code. [Formatting deliberate to squeeze on page]

    char* pRegistryKey  = 
        "SYSTEM\CurrentControlSet\Control\Session Manager\Environment";
    char* pVariableName = "SOMEVAR";
    char* pValue        = "Another String bites the dust!";
    HKEY keyHandle;

if( RegOpenKeyEx( HKEY_LOCAL_MACHINE, 
                  pRegistryKey, 0, 
                  KEY_SET_VALUE, &keyHandle ) 
    != ERROR_SUCCESS ) 
 { return 1; }

if( RegSetValueEx( keyHandle, 
                   pVariableName, 0, 
                   REG_SZ, (BYTE*)pValue, 
                   (DWORD)strlen( pValue ) + 1) 
     != ERROR_SUCCESS )
{ return 2; }

RegCloseKey( keyHandle );

DWORD returnValue;
SendMessageTimeout( HWND_BROADCAST, 
                    WM_SETTINGCHANGE, 
                    0, (LPARAM)"Environment", 
                    SMTO_ABORTIFHUNG, 5000, 
                    &returnValue);

I’m not going to cover all the details of how to create the map – Googles documentation does a quite good job at that – especially if you supplement it with the community documentation effort at Mapki. However, there are a two things that took me some time to discover – so I’m sharing them here. Not only might it help you, it also becomes part of my outboard brain.

Custom Icons

Yes, the documentation touches on custom icons. It’s almost useful. However, when you just want Googles default icon in a different color, things become too tricky for the simplicity of the task. So here’s the quick way:

First, you obviously need the original icon to modify. It lives at http://www.google.com/mapfiles/marker.png, and its shadow is named shadow50.png. You could probably just refer to their files if you use them unmodified. I decided to copy them to my webserver instead, in case I ever want to change them. Creating the green marker from the red one is left as an exercise to the reader, but how to create a new icon?

Google gives you a list of seven properties to set – here’s the code to do it properly. (You could’ve easily found out for yourself, but if you’re like me, you appreciate a good copy&paste opportunity…;)

var baseIcon = new GIcon();
baseIcon.shadow = "icons/shadow50.png";
baseIcon.iconSize = new GSize(20, 34);
baseIcon.shadowSize = new GSize(37, 34);
baseIcon.image = "icons/marker_red.png";
baseIcon.iconAnchor = new GPoint(6, 20);
baseIcon.infoWindowAnchor = new GPoint(6, 20);

var greenIcon = new GIcon( baseIcon );
greenIcon.image = "icons/marker_green.png";


Note the cloning at the bottom – saves a total of six lines of code. Always a win.

Creating Markers

I’m creating markers from an XML data file, since keeping all Starbucks locations in my web page really didn’t seem like an option. It’s fairly forward, with one little caveat – JavaScript scoping rules. In case you, like me, say “Huh?” at this point, here’s the straightforward (but wrong!) code:

// Broken Code !!!

var markers = xmlDoc....// Get markers
for (var i = 0; i < markers.length; i++) {
... // skipped all the XML parsing
var marker = new GMarker(point,icon);
GEvent.addListener(marker, "click", function() {
marker.openInfoWindowHtml(html);
});
}


So why is this wrong? The listener is bound to the same variable all the time, “marker”. Once the loop is finished, marker is set to the last marker in the array, no matter which actual marker you click on. It’s an interesting little quirk that can be solved by putting each marker in a different variable. Since I don’t want to think up all the names, let’s get this solved by scoping – we put the actual creation and addition of the event into a separate function.

function createMarker(point,html,icon) {
var marker = new GMarker(point,icon);
GEvent.addListener(marker, "click", function() {
marker.openInfoWindowHtml(html);
});
return marker;
}


Since “marker” now leaves scope whenever the function returns, each call creates a new marker variable to be bound to. (In my opinion, declaring a variable inside a loop should do the same thing, but JavaScript disagrees with me.)

For those of you not wanting to follow all the links, let me give you some quick background: The Starbucks Challenge challenges the marketing message Starbucks is sending regarding fair trade coffee. According to them, you can always get a french pressed cup of fair trade coffee in any of their stores, if you just ask. Challenge participants then post their results, and tag it on del.icio.us. The map then gets the data from there.

While the using the the Google Map API itself certainly had hurdles – which are going to be covered in another article – the main difficulty to figure out at the outset was: “How do I get longitude and latitude for any store?”. Sending people a GPS unit was out of the question, but thankfully Starbucks itself is helping. Their link to the store driving directions helpfully contains exactly the data we need. Getting the page containing the link is easy, and each store is uniquely identified by a store key – perfect for tagging.

Getting the link out of the page is not so fun. To get this done in any reasonable amount of time, I had to go with a scripting language. I picked my current personal favorite, Python. It’s easy to write and has plenty of support for web tasks. Reading the page, for example, is a one-liner.

How to extract the link, though? The first idea that came to mind was writing an XPATH query. However, the state of XML parsing for Python is a sorry one. There are quite a few competing packages, all with their ups and downs. Thankfully, the point was quickly rendered moot – the web page is not well-formed XHTML, and all the candidates choke on that.

So what we need is a package to parse the mess that qualifies as HTML. Thankfully, there’s a package to do just that: BeautifulSoup. It takes the tag soup of any HTML page and renders it navigable. Lots of heuristics to cover all the mistakes most HTML parsers allow. Let me give you the two salient pieces of code:

1. Extracting the address of the store. This is stored in two <span> elements with the IDs of bAd2 and bAd3. Extracting that is a simple two-liner:

   soup = BeautifulSoup( webpage )
addr = soup( 'span', { 'id' : re.compile('bAd[23]') } )


   Yes, that’s it. No more work.
2. The actual location is similarly easy. It is a link element thoughtfully classified as a “DriveLink”. (Thank you, Starbucks!). All you need to do is get the soup of all DriveLinks, and look at the href attribute of the first one:

   driveLink = soup( 'a', { 'id' : 'DriveLink' } )
href = driveLink[0]['href']


   Judicious application of a bit of regular expressions then yields longitude and latitude.

And that’s all there is to this, at the core. Since I like playing with code, there’s a bit more to it, of course. If the link doesn’t provide geolocation any more, it falls back to Yahoo’s Geocode API – just in case Starbucks decides not to be helpful any more. And as to not unduly hammer their server, the results of the Store Key lookup are of course cached. And it’s all nicely wrapped up as a module. If I ever get to clean up the code, I’ll release it here. If you need it before, just ask.