I go quite a way along with Mrs Too's answer but I had always heard the following when I was small from my dad...
Back when HE was a lad whic'll be early 1099s, the only transport to use when going on a holiday was a bus.
So, a busman ie a bus driver going by bus on his own free time was just doing what he did for the other 50 weeks of the year.
No holiday at all - which explains the expression.
Early 1099s? Must be time for bed! I meant Early 1900s...