Striker Graham, on loan from Sheffield Wednesday for the rest of the season, sealed a victory for what was only Huddersfield's third win in 13 matches.
Huddersfield were riding high in the top two until they were paired with Premiership champions Chelsea in the FA Cup third round, but have slumped since then.
Now Colchester, who go to Stamford Bridge in the fifth round next week, will hope they have not fallen victim to the same curse.
Colchester defender Garry Richards was harshly sent off for a professional foul just after half-time before Graham sealed the vital win in the 68th minute.
Huddersfield had taken an early lead through Jonathan Worthington and the goal was a stroke of good fortune and Town were lucky to be ahead at the break.
Colchester top-scorer Chris Iwelumo rapped the foot of the post in the second minute and then had an effort ruled out for offside after Mark Yeates' deflected shot came back off the woodwork.
Huddersfield took the lead in the 14th minute when Gary Taylor-Fletcher let fly after swapping passes with David Mirfin and the ball took a wicked deflection off Worthington and left keeper Aidan Davison stranded.
Colchester hit back and had the ball in the net again but Richard Garcia's finish was cancelled out for a foul on keeper Phil Senior.
Having had two goals disallowed and having hit the woodwork twice, Colchester could not believe they could be behind 1-0 at half-time but their hopes of getting back in the game were hit five minutes after the re-start.
Richards found himself the wrong side of Graham and wrestled him back. Referee Graham Salisbury saw Richards as the last man and brandished a straight red card.
Huddersfield lost against 10-man Tranmere last week but this time made the advantage and their good fortune count.
Graham ensured the three points when he kept his cool and slotted past Davison after Andy Booth's knock down.
Colchester never looked likely to force their way back into the game, though Greg Halford stabbed the ball against the post in a goalmouth scramble late on.