Both sides had a man sent off in the first half in what was a hard-fought game of few clear-cut chances.
But just when it seemed the game was going to end in stalemate, midfielder Holdsworth appeared unmarked to head home only his third goal of the season in the 76th minute.
Substitute Joe Skarz had only been on the field four minutes when he pulled clear down the left and picked out Holdsworth for a powerful finish which went in off the far post.
In the first half, Huddersfield were denied a penalty when Andy Booth was pulled back by Paul Huntington.
The game was littered with cynical fouls and Leeds skipper Jonathon Douglas was red carded in the 39th minute for his second bookable offence.
He was booked seven minutes earlier for a trip on Jon Worthington and went for tugging at Michael Collins.
Huddersfield's advantage did not last long as midfielder Worthington, who was also booked, was sent off for his second yellow card on the stroke of half-time.
Worthington appeared to have been pushed in the back as he challenged keeper Casper Ankergren.
Worthington, one of the smallest men on the field, squared up to the tallest - Leeds defender Lubomir Michalik - who aimed his head at Worthington's face. Michalik was booked as well.
Early in the second half, Collins smashed a shot against the bar and, after Holdsworth's goal, Skarz almost scored with a delicate chip which landed on the roof of the net.
Leeds, who were without 20-goal top scorer Jermaine Beckford, never threatened and Town held out despite five minutes of stoppage time.