Telfer made the comments to a national newspaper just a few days before the Calcutta Cup clash at Twickenham, their Six Nations opener.
It is a familiar accusation from England's Six Nations opponents and no-nonsense Leicester prop Cole said: "You know it's coming so it's boring in that regard because it's the standard go-to.
"People say 'We're arrogant, we're this, that and the other'. You can believe it if you want but I don't think we are.
"Yes we're confident in ourselves, yes we back ourselves and what we do, and we're confident going onto the field we can win games, but I wouldn't say we cross the line of arrogance.
"We understand where we are and how we're perceived. Our motivation comes from how we want to progress as a side.
"If we want to be the best side in the world we have to win this game. It starts this week.
"We have continue what we did in the autumn into this game. That is the motivation, to go out and win.
"We set the bar quite high and people will expect us to win because we beat New Zealand and New Zealand beat Scotland in the autumn.
"But we know that is not the case. The expectation is different now and we have to manage that as a team. Scotland are a dangerous team to play against.
"They've got nothing to lose. They've got an interim coaching staff and have been written off, much as we were last year, and we finished second.
"Every time I've played Scotland it's been close. They will come down and try and batter us physically."
Gilbert has released a new line of rugby cleats. The Gilbert Virtuo 8S is part of the exciting new product. Check it out.
The entire All Blacks apparel line has been updated for 2013/14. Check out the New Zealand All Blacks polo.
The Nike Tiempo is a solid rugby cleat and one of few styles still made from full-grain natural leather.
The Gilbert Blitz 8S rugby cleat is a great cleat at a great price of $69.99. Get a new pair of cleats today.
A cool looking all black rugby cleat with the high performance adidas is known for. Get in the Gear!
Wear the crest of the British and Irish Lions on your t-shirt. A great look for the summer.
$29.99
The All Blacks Performance t-shirt is black with hints of blue from the training jersey. Very Cool.
The New Zealand All Blacks training jersey for 2013/14. Get in the Gear!
The USA Rugby Pro Alternate rugby jersey is perfect for any fan of the Eagles. Get yours to wear during the summer Test matches.
The NEW All Blacks 2013/14 jersey has arrived at World Rugby Shop. Dare to wear the colors of the All Blacks.