That sounds reasonable, but I think it's wrong. Needless to say, I haven't read any scholarly literature on this, but instead conducted a thought experiment, which is much less work. Here's the thought experiment: If it were really true that government provision of X (prisons, water, education, whatever) is cheaper, then we would expect the same to be true of other stuff, like TVs, haircuts, or peaches. If that were true, then we can imagine that for some nonessential goods and services, government decides to actually make a small profit and compete in the market with private companies. We should expect government to make healthy profits overall. In a few years' time, government could abolish taxes.
Yeah, that would be great. But it seems that's not the world we live in.
To end on a positive note, here's a funny post on how to translate book blurbs into plain English.