Frank,
That code is in the /plugins/system/jfbcsystem.php file. You can remove it if you really know what you're doing, however, I wouldn't recommend it. That code is required for any of the following features:
* Facebook Login
* Automatic login of users
* Any social widgets (Like, Comment, Fan module, etc)
* Facebook Requests/Invites
* Page Tab / Canvas support
Basically, if you're using any of JFBConnect's features on the page you want to remove it from, those features will no longer work. Additionally, the Facebook Javascript file is something that gets cached in the user's browser. While it is executed on each page (which may take 50milli-seconds or so), it should not be re-downloaded on each page, which is where most of the delay would be introduced.
I hope that helps,
Alex