Willem Alberts, Bismarck du Plessis and Francois Hougaard claimed early tries for the Springboks as England were overrun at Coca-Cola Park.
Youngs crossed twice as England put themselves in sight of an unlikely victory after trailing 28-10, but JP Pietersen ended the comeback.
The Leicester scrum-half told Sky Sports: "The first 20 mins killed us unfortunately and we were always playing catch-up. Great spirit by the boys but at the end it didn't count for much.
"Now we've lost the series but we've got one more big effort next week. It was a great effort and great heart from the boys but we can't give them the first 20 minutes like that and be playing catch-up in a game like this."
Youngs' international and club team-mate Toby Flood added: "We need to make fewer errors. In the first half, every time we conceded we kicked off and then we'd be back in our own 22 via a penalty or a good kick that goes a few yards further at altitude.
"We need to control that better and we can't make those errors, that's simply what killed us."
Head coach Stuart Lancaster was "proud and frustrated" at the outcome.
"It was tight. The boys are frustrated but we've got to bottle that frustration and take it into next week," he said.
"We were disappointed with the start, to give them 12 points before we got going. We know we've got to be better in all the little details that make the difference in winning at international level.
"We got the try and that gave us the confidence, we were hanging on by our fingernails in the first half at times. But we grew, the scrum got better and we started to win the collisions and move the ball around.
"Before half-time was the time when we needed to get a foothold in the game."
Gilbert has released a new line of rugby cleats. The Gilbert Virtuo 8S is part of the exciting new product. Check it out.
The Barbarians are one of the top invitational rugby sides with a long history and classic rugby jersey.
The Nike Tiempo is a solid rugby cleat and one of few styles still made from full-grain natural leather.
The Lions get ready for their matches with this green training jersey. It's what the players wear. Get in the Gear!
A cool looking all black rugby cleat with the high performance adidas is known for. Get in the Gear!
Wear the crest of the British and Irish Lions on your t-shirt. A great look for the summer.
$29.99
The Lions are ready to get their Australia tour underway. They arrive in Perth on Monday.
The New Zealand All Blacks training jersey for 2013/14. Get in the Gear!
The USA Rugby Pro Alternate rugby jersey is perfect for any fan of the Eagles. Get yours to wear during the summer Test matches.
The NEW All Blacks 2013/14 jersey has arrived at World Rugby Shop. Dare to wear the colors of the All Blacks.