day-dreaming isn't intrinsically bad. People do need time to think about stuff, and have their mind drift from topic to topic. Some modern teaching practices advocate deliberate "brain breaks" for students.
The issue with phones isn't so much that students are sometimes off task, but rather that the phone consumes their attention entirely. It uses up the students' useful concentration as well as their 'rest' time.
Perhaps this is what Musk means when it says that empathy is bad.
If your goal is to maximise your own personal wealth, then empathy really is a hindrance. But I put to you that maximising money is not a goal worth devoting a lifetime to. And perhaps not scamming dumb people is a valuable way to act regardless.
(This is essentially what motivated the Quakers to push for set prices for goods rather than constant bartering. They believed that dump people should still be able to go to a shop and buy stuff without getting ripped off. I'd say that moral position has made the world a slightly better place.)