It does not need to be removed. It is as secure as your control panel, which is actually *more* powerful since it has the capability to add and delete images.
Dan is right, setup and upgrade really [b]can't do anything harmful[/b] to your installation or your server.
Regardless, they are password-protected and secure, so you don't need to worry about them! You're welcome to remove them if you really want to, but there's no reason even that would make anything more secure... so it's actually NOT BAD!

Trust me, I know what I'm doing.
Of course, there should be a detection that checks if the tables are already present and stable just so people know they've already installed and don't think they need to delete any setup files... ;-) I'll add that to the list.