That's a good question. Yes, you're correct, there's no way to currently skip inviting FB friends. We're working on the Invite module currently to update it for our next major release of JFBConnect. If it's possible to add more parameters into it to allow skipping things like you ask, we'll do what we can. We can only expose features that Facebook allows us to though (through their API), so I can't guarantee this is possible. Additionally, this would be a 'fixed' setting for all users, not just specific users.
This is because Facebook doesn't check your friend list to ensure that all friends have already been sent invites, and will just show them all.
Hope this helps!