Mallinder pulled no punches in his verdict on his side's 41-22 defeat at the Stade Pierre Antoine.
The Saints, who won the Heineken Cup in 2000 and were beaten finalists last season, suffered their fourth defeat in this season's competition against a Castres side which had not won a game for two months.
"We wanted to play rugby and move a big Castres side around and I though we did that very well," Mallinder said.
"But this was always going to be a big challenge and it is always going to be hard when you have to defend for 20 minutes with 14 men.
"It was too much for us but I was happy with our attacking play, but we obviously have work to do on our set-piece and particularly our restarts which were a problem for us all game."
Fly-half Steve Myler and flanker Tom Wood were both sin-binned and Mallinder added: "When you have two yellow cards you have to ask why?
"I don't want to question the referees but we have had the same problem against Munster and we have to look at why.
"The yellow card for Steve Myler and a high tackle was a close one but we just aren't getting those decisions at the moment."
Northampton host Castres in the return game next weekend but have nothing to play for with their Heineken Cup dream now over.
"We are out of the competition and that is hard and we are playing for pride now," Mallinder added.
"Next week is all about that and we will work hard and go out and play hard like we always do in all competitions. That won't change.
"We had some young players out there and they would have learned a lot from the experience they have had out here."
Gilbert has released a new line of rugby cleats. The Gilbert Virtuo 8S is part of the exciting new product. Check it out.
The entire All Blacks apparel line has been updated for 2013/14. Check out the New Zealand All Blacks polo.
The Nike Tiempo is a solid rugby cleat and one of few styles still made from full-grain natural leather.
The Gilbert Blitz 8S rugby cleat is a great cleat at a great price of $69.99. Get a new pair of cleats today.
A cool looking all black rugby cleat with the high performance adidas is known for. Get in the Gear!
Wear the crest of the British and Irish Lions on your t-shirt. A great look for the summer.
$29.99
The All Blacks Performance t-shirt is black with hints of blue from the training jersey. Very Cool.
The New Zealand All Blacks training jersey for 2013/14. Get in the Gear!
The USA Rugby Pro Alternate rugby jersey is perfect for any fan of the Eagles. Get yours to wear during the summer Test matches.
The NEW All Blacks 2013/14 jersey has arrived at World Rugby Shop. Dare to wear the colors of the All Blacks.