We'd love to help you narrow things down. Joomla has a built-in profiling tool to help narrow down where sluggishness is happening. To enable, go to the Global Configuration area -> System and set Debug Mode to "yes". Then, at the bottom of each page stats will be shown.
Let us know the number of queries, time taken and RAM used to generate a page before and after JFBConnect is enabled. Try that for a few pages, like your home page and preferrably a 'simple' interior page which should have less going on.
That will help us narrow down what may be problematic. We've never heard of such a performance issue and JFBConnect is used on thousands upon thousands of sites. There isn't a simple magic bullet I can provide because we'll need to understand why your site is different first.
Thanks,
Alex