In British English this is a common expression (not an idiom).
We are all poorer for his death. (ie, we have all lost something as a result of his death)
Today, few people have manners and we are all the poorer for it.
If you refuse to go to university, you will be the poorer for it.
In general, the idea is that some event or situation makes us "more poor" than before, not usually in a financial sense but in terms of status, culture, or quality and richness of life.