Hi,
I am not a Source Coast team member (just a client), but here goes my input on this.
To invite friends, I use the Request Diaglog as explained here:
developers.facebook.com/docs/reference/dialogs/requests/.
At first, the request dialog would not work. My symptoms were not the same as yours, but maybe the source of the problem is the same.
Joomla uses a MooTools release that is not compatible with FB's Javascript SDK. To solve this, you must prevent Joomla to load its default MooTools framework, and load the most recent release of MooTools (both "core" and "more"). You can get the framework from
mootools.net/ - but be sure to NOT include compatibility when you build and download it.
Now I had this problem with just plain Joomla. JomSocial may be adding some more variables to the equation, but my tip to you is look for incompatibilities with MooTools and/or JQuery...