No, you shouldn't need to create another application at all. I'm not sure why the comments would be showing up. We're still seeing the comments for our apps, so the tool does seem to still be working.
Looking at one of your sites in Autotune (there are many), I'm not seeing the Open Graph tags I'd expect in the HTML of your page. The Open Graph tags are what 'ties' some of the social features from your site to the tools available on Facebook. My guess is that you need to go into the Social -> Open Graph tab and make sure that Open Graph is enabled.
In the v5.0 release, Open Graph is enabled by default, just so you know.
After you enable it, it may take a few days for Facebook to 'see' the new tags we added and properly populate the information in your Moderation Tool area.
Hope that helps,
Alex