There are a few changes we have planeed in the upcoming 4.2 release to minimize some of the communication with Facebook. Unfortunately, some communication will always be required. We load the Facebook Javascript API on every page because for Open Graph functionality (describing your page in case someone shares it on Facebook), the ability to login on any page, or showing parts of your site in a FB Page all require it.
However, one change you can try now that may speed things up is a quick edit to the /plugins/system/jfbcsystem/jfbcsystem.php file. Search for "status" and you'll find a line like:
Change the status:true to status:false. That can improve load times a bit, depending on your site, the user and a few other factors. The 4.2 release will dynamically set that to true or false depending on a few configuration options you have set in JFBConnect.
Hope that helps get you started. If it still seems slow after that, feel free to post a page or 2 on your site that seems slow, and we'll let you know if we have any other suggestions that may help.
Thanks,
Alex