Matt,
There's a lot of features that rely on the Javascript library loading. Things such as the auto-login functionality, Page Tab or Canvas integration, Open Graph actions, and I'm sure a few others I'm missing. So, even if you don't have a 'button' or other visible widget, the Facebook Javascript can still be necessary.
With that said, frankly, it shouldn't take that long to load. First, it should be cached by your browser at least for a little bit, which should make it not have to load on each page load. Also, the script is set to load after the page has been delivered to the user. So, even if the download of that file takes a little extra, it shouldn't prevent the page from being viewed. Finally, we initialize the JS library with the barest minimum of features required for what you have enabled. That doesn't mean it's always speedy, but it is as fast as we think we can make it load.
With all that said, there is probably a few lines of code we could help you add to JFBConnect which could prevent it on that page. The problem is that you'd have to make that change whenever you upgrade and it would prevent other features, as mentioned above.
Let us know if that helps or you want us to look into the code required.
Thanks,
Alex