I disagree with the sentiment that toughness is solely an interior issue.
Perimeter defenders need to be more physical off-ball. Can't let people make open cuts - need to get a body on them. Force them to really fight through space. Our best defenders, Ross, Carroll, Powell, CoJo (in some order) - none of them are very physical. CoJo might be the most physical player we have on the perimeter.
On-ball is much harder to be physical in this era because of the fouls. But after the shot/pass release you need to establish your presence.