The refresh only works for the direct images in an album, not including subalbums. If you have a lot subalbums with a lot images that otherwise could kill your server.
Nothing kills the server. I have over 10000 photos and reading the metadata of all at once is not an issue.
But OK, if you think so. How about if the mini-button would not exist where it does not work anyway?
If your server is powerfull, great. Don't speak for all. It does work for the album and its direct images. Have to look at the code if it is meant to cover subalbums and anything unexpectedly broke (I guess it is not and code that way).