An anony­mous com­menter sug­gested I use CIE L*a*b* instead of HSV. I’ll let the results speak for themselves:


And that’s with­out any tweaks, fancy dis­tance met­rics, or push­ing out­side the region when the vec­tor is too short. Who­ever you were, thanks for the sug­ges­tion! Also, thanks to Madeleine Ball, whose sug­ges­tion I mis­un­der­stood, but is basi­cally the same.

Update: with some vec­tor exten­sion code (slightly more con­trast when col­ors are really close). The code is in xchat-gnome SVN for every­one to play with now.