If a drive is not being used it will have no effect on your system other
than taking up a small amount of RAM for the driver.
However you should check that the drives that are flagged as removable are
not set to provide an auto-insert notification.  Look in the device details
in device manager.
Also make sure that any process running in the background, such as your
anti-virus,  is not trying to access non-existent drives.
If your system is slow when doing a specific task, such as opening a Save As
dialogue window, then that could indicate the system thinks that some drives
are available when they aren't.