We do a bit of testing with the standard Joomla captcha and know many sites that use it along with JFBConnect. There shouldn't be any issues. I just tried to register on your site, but either I already had an account or the captcha is disabled.
Does the captcha work correctly for normal registration? What is the exact error message you're seeing about characters being wrong?
Finally, if you could enable the plugin and delete my account, that would help me to test further to see what's happening.
Thanks,
Alex