I think you might be thinking too hard here.
For starters, you have Key Repeat and Mouse Repeat both enabled by default when you launch the game. That's technically fine, you would just have to turn it off when not needed, instead of turning it on when needed. The Repeater Target, when you first launch, is going to be every single window -- so every window will broadcast to every other window.
But then you have your Activate Repeat mapped key (Shift+Alt+R) that toggles Repeater and changes the Repeater Target -- for those in the Miners ATG. (It'll also change the broadcasting icon in the Click Bar between gray and green arrows, but only in one window -- so the icon color in most windows will not match the actual state). The first time you press it, each of your Miners will turn on broadcasting to other Miners, and the second time you press it each of your Miners will turn off broadcasting and still be only to other Miners next time it is turned on by any method.
And then you have configured 6 Hotkeys in your Repeater Profile. I recommend un-setting them as they do not sync with the Click Bar icon and probably further contribute to your confusion. Some of them will toggle Repeater on from one window, some turn it on from all, some just do keyboard or mouse and some do both.
Now... I DON'T follow how you actually want it to work. I follow that you want miners to broadcast to other miners. Here's where you lost me though:
For the new repeating action i was trying to setup for a new group,
5 chars of the miners repeating
4 chars of the non miners
6 other chars
are used to makeup the new group (already an exsiting char set with different window layouts).
if i add as you suggest all the chars to the miners action group that need the repeating function for the new group, 4 of the non miners would then be repeating while using the mining setup as they would be activated by the same hotkey.
What is the problem with being activated by the same hotkey exactly? You have sub-sets of characters that, if Repeater is enabled, you want to broadcast between themselves... is that correct? So if you're playing a miner and you have broadcasting on, you want it to send to all miners and exclude the "other chars"? If you're playing a "other char" you want it to broadcast to the "other chars" and exclude the miners?
This doesn't sound incredibly complex to me. You would want to undo 99% of the repeater-related changes you made, go back into Steps 1 and 2 of Activate Repeat...
In step 1, you can add more than one Repeater Target Action, and you can turn Repeater on from every single window if you so desire. The Repeater Target for a Miner can be set to other Miners, as you have it now, but if you have another ATG that is mutually exclusive (i.e. "other chars" are not "Miners") you can just add another Repeater Target Action.. Then in Step 2 you have a useless Repeater Target Action because it set the Repeater Target to what it already is -- Miners to Miners; the idea is SUPPOSED to be to restore the original Repeater behavior by setting the Repeater Target back to "Window: All w/o Current". That way if you need to broadcast to ALL windows instead of just Miners, your Repeater Target isn't busted.

And instead of turning on broadcasting from just the Miners ATG, I would either turn it on for all windows or the current window. This way the one toggle would work for most cases.
Finally, you can always have alternative toggles. If you use the Mapped Key Wizard (available from the Wizards menu) there is a "Broadcasting Toggler" type of Mapped Key you can generate, which will help you set up any sort of custom toggle Hotkeys you need (broadcasting to/from different ATGs, ec)