We present real-world lessons learned conducting a usability benchmark on a large, complex software application using the standardized, summated usability metric (SUM). We discuss nuances to the SUM equation and provide insight on key decisions including selecting tasks, setting target metric values, and translating z-scores into percentages when communicating results.