Is it worth doing a podcast if nobody listens?

No.

Big Podcast is the book for podcasters who can’t quite figure out what they’re doing wrong (and are ready to do things right).