With a fresh install of 1.4.1.4 I have noticed that when I create a new page but do not publish it, the page still is appearing on the site as a link.

Did I miss something or is this a bug?

If you are logged in with appropiate user rights you always see unpublished or protected items.

Ah ok that was slightly confusing.

One further thing though, while testing I found that the page is still accessible by anyone if they have the url, is that by design?

There is a good explaination of this in the troubleshooting guide

And in short, yes that is by design. As my collegue said pleaase see the troubleshooting for more info.