Wow, sorry, yes, I lost this post.
Sorry for the troubles your having. Yes, the best guess we have is the large number of friends you have is causing a slowdown in the loading of the module. Definitely not sure about the logging out thing, as that's very strange.
As for possible fixes, for now, I don't have any. Facebook (last month) announced a change to how Invites can be implemented. We're looking into this possibility for a future release because the way were were doing it has a lot of workarounds for Facebook's older implementation. The Invite module is one of our most popular features, so we want to get something better in general, and hopefully that will help you as well. Unfortunately, we won't have something new out until later in April (that's the plan for now at least).
Hope this helps. If you expect users on your site to have thousands of friends, you may want to leave the module disabled for now.
Good luck, and thanks for all your feedback!