I have quite a few albums I want to keep unpublished, but when I click the archive link, they are all available.
For example, for the moment I only have 5 public images, but the archive shows all of them.
How come, and can I maybe just remove the link to the archive page? How do I do that?

Just solved it: it is not good enough to have albums "unpublished", you also need to "unpublish" all SUBalbums.

Cheers

Remember that the logged in admin with appropiate rights sees all unpublished items as well.
The state is actually inherited so if an unpublished album contains subalbums or images they don't show up for normal visitors.