Unfortunately, we've done some testing, and my earlier statement was incorrect. I don't think you're going to like my response here though..
We did many comparisons of times where the pop-up comes up Vs. when the AJAX window comes up. It seems as though Facebook Connect (not JFBC) uses the AJAX window if you're already logged into Facebook (through another tab). In this case, you're merely 'connecting' your account, not logging into Facebook.
If you aren't logged in, it will show the pop-up window asking for you to log in.
We did comparisons using the facebook.sourcecoast.com site and there is NO difference in the output HTML between when the AJAX shows up and when the popup does. This means that it's Facebooks code which is causing the difference.
You can also see this same thing occur on Facebook's sample Connect website (which is what we test most of our visual issues against). If you close your browser, ensure you're logged out of Facebook, and click the login button, you'll get a popup. If you log into Facebook in the background, and hit the login button, you'll see the AJAX window.
<!-- m --><a class="postlink" href="
www.somethingtoputhere.com/therunaround/">
www.somethingtoputhere.com/therunaround/
If you find any more information on the topic, we'd be happy to look into it. However, this is really out of our hands.