Mick Wadsworth's Terriers are in deep trouble in the Second Division relegation zone after being beaten 2-0 at his former club Colchester United.
The defeat put a nine-point cushion between the two teams as Colchester climbed further towards safety.
In a deceptive blustery wind the game had been fairly evenly matched in the first half with few chances created.
But referee Paul Armstrong sent off the visitors' Kevin Sharp after 40 minutes and it changed the pattern of the game.
In the first half Huddersfield had the ball for the majority of the time and always looked likely to open the scoring, but with Huddersfield down to ten men for the second half the home team dominated and took all three precious points thanks to goals from Kem Izzet and Scott McGleish.
There had been few clear threats on goal in the first 45 which was scrappy and very disappointing for a crowd approaching 4,000.
Huddersfield used their possession well and launched in long balls to veteran striker Andy Booth, still a danger in the air.
But his colleagues couldn't capitalise on his knockdowns, while Colchester's striking duo of McGleish and Dean Morgan had little opportunity against a visiting defence marshalled well by Adie Moses and Nat Brown.
Colchester had some good attacks down the right wing where Sam Stockley was on fire. Twice he turned Sharp inside and out, but when Sharp was sent off it was a surprise to most of those in the ground.
Sharp responded to some barracking from fans close to the touchline and seemed to make hand gestures.
The linesman drew the referee's attention to the incident and called play back but it was a surprise when a red card was produced.
Before the end of the half Phil Senior in the Huddersfield goal, had to save well when he palmed up a powerful angled drive from Morgan.
Earlier the keeper had been helpless when Karl Duguid placed his shot past him from inside the penalty box but Moses kicked it off the line.
Huddersfield re-organised at half time and manager Wadsworth, who had been booed by both sets of supporters, left Booth up front on his own and packed his defence.
Gradually Colchester began to make their extra man pay, although it was not until the 74th minute that they broke the deadlock through Izzet, with McGleish quickly adding a second.
Colchester had other chances before the end but Huddersfield held out. The best opportunity was a tremendous 30-yarder from Thomas Pinault which flashed past the post.