I certainly wouldn’t say incredibly stupid at all. But it’s definitely an additional risk, and one that needs to be understood.
But you’re correct in considering physical security first. The problem is that people often assume they have more physical security than they actually do.
And master passwords? Well, they’re nice, but they too have their limitations.