Glad to hear you got the popup going.
We had some time to look into the template files you sent for the other login module you're using and have some ideas. We didn't test anything directly, so there are some caveats, but hopefully we can get you something that at least gets you started.
In the social.php file, the following line is what needs to be updated:
<span class="loginBtn submitBtn" style="width:auto"><span class="btnIco <?php echo $oauth->alias?>Ico"> </span><?php echo JText::_("LOGIN_WITH_".strtoupper($oauth->alias)); ?></span>
The biggest unknown is how the AJAX Login module hooks into that code to turn it into a clickable button using Javascript. The easiest thing to simply do is replace that whole line with:
<a href="javascript:void(0)" onclick="jfbc.login.login_custom();"><img src="/linktoimage.jpg" /></a>
That will display your own custom image in that place and should definitely work. Again, you can take a picture of the current login button and just use that. There may some minor styling needed to center things, which we can help with as well.
If you want their exact styles for hovering and such, it's a little more complex as we need to make sure that our login functionality is always called, not theirs. You can try the below, which may work, but also may not fully override their Javascript:
<span class="loginBtn" style="width:auto"><a href="javascript:void(0)" onclick="jfbc.login.login_custom();"><span class="btnIco facebookIco"> </span>Login with Facebook</a></span>
That *should* work. The main things to note are the removal of the submitBtn class tag and the addition of our anchor link.
Try that and see how it works out for you. It may not work, but will hopefully get us closer to the best state. Again, the first option should definitely work, but may not look exactly how you want. The 2nd may have functional issues and will need to be tested thoroughly in a few browsers.
Keep us posted on how that goes, and good luck,
Alex