Was My Unknown Great-Great-Grandfather a Philanderer?

Looking at my Family Tree DNA matches, trying to track down some Irish cousins and ancestors, I'm coming to the conclusion that the unknown father of my great-grandfather may have been a 19th century Lethario.

Why do I think that? When I look for matches that do not match my mother, of course, they must be on my father's side. Then I look for people with Irish ancestors that don't match my Kelly cousins. Now, this doesn't necessarily mean that they are related to this great-great-grandfather, but it does narrow it down. They could be related on my Doran line or they could be related on my Kelly line but just don't match my Kelly cousins. However, these are the only matches that have common matches like this.

Looking for common matches to my Irish DNA matches that don't match my Kellys, most of them have common matches numbering in the dozens to over 100. As a comparison, when I look for common matches with my aunt on my mother's side, who I am closely related to, I have 48 common matches with her. Looking for common matches with my mother, I have 199 common matches. When I look for common matches with 2nd to 5th cousins with Irish ancestry, I have 75-140 matches in common! Many of these in-common matches are relatively close cousins up to 5th cousins and not speculative. If I look at common matches with my known Kelly cousins, I have 5-7 matches in common with them.

What could cause this? It is theoretically possible that the line of the unknown great-great-grandfather or even my Doran ancestors had a large number of descendants and a large percentage of them have been tested. How likely is this? Given that my Panther and Menke relatives on my mother's side had very large families and the number of matches on these lines showing up is small in comparison, I don't give this theory much credit. In addition, the Doran and Kelly families I've encountered had relatively few children that passed their genes on to later generations.

I am left with the thought that the man that fathered my great-grandfather with my out-of-wedlock great-great-grandmother may have had a habit of having his way with the ladies. If this theory is correct, it's also probably pretty likely I will have a hard time tracking him down since most of the people descended from him will also not know who he is.

What do you think?

--Matt