So here's the exact behavior I am seeing from my tests:
1. The first time a comment is made on the page and if the 'Also post on Facebook' is checked, we are not getting the notification that the comment was created. Also, the comment doesn't show up on the page until you refresh.
2. If the that setting was unchecked for the first time the comment is made on the page, we will get the notification and the points will be awarded.
3. Any subsequent comments (whether the setting is checked or not), we will also receive the notification and the points will be awarded.
So, it seems to only be on the first comment, but I'll keep looking into it.