We're working on implementing an option for using the OAuth redirect methods of Facebook instead of the popups. If all goes well, it will be in the v5.2.2 release of JFBConnect, due out next week.
There isn't any documentation of the low-level authentication processes of JFBConnect. You can look at the /components/com_jfbconnect/libraries/provider/xyz.php files for google and twitter to see how the OAuth process is done there, if you're interested in tinkering on your own.
If we can't get the feature implemented in v5.2.2, we'll let you know how far we made it and what issues we ran into, in case you want to take it from there. If it's not in that release, it should be in the v5.3, due out in early 2014.
Thanks,
Alex