I'm not sure what you mean by out of place. Can you take a picture of what you see and explain why it's not right? The button looks fine to me on the home page.
As for the ordering of the plugins, that won't affect how the button looks or where it renders, so that's not something to keep tinkering with.
Finally, if the image is staying when you login, that likely means you've enabled the System - Cache plugin and turned on "Browser Caching". That should no be enabled or it will cause problems like you describe.
Hope that helps,
Alex