Yeah, been there. I use round buttons in a couple of places and the same issue crops up. Because everything is a square, even the round button (with a transparent edge), then the placement has resulted in a small amount of overlap between each button, but I made sure that was in the transparent areas and so it is not an issue.
This is it with the borders set to 1 so you can see the overlaps.
I used to have them spaced slightly further apart, but decided to bring them closer as I don't normally click on the edges, and further apart meant more mouse travel.
If I was really being pedantic, to get a triangle, in a custom XML, I would try using multiple smaller square buttons, that happened to result in a triangular area. These would need to be named so that they were not mapped to ISBoxer (i.e. not named btn1, btn2, btn3, etc). I would place another button somewhere that would be named so that is would be controlled via an ISBoxer button set. The smaller buttons would all need to use script to trigger this other button (which could be anywhere, and 1px by 1px. If I wanted an image, then I would use yet another button positioned underneath the smaller buttons (they are effectively layered as defined in the XML file so gui items defined last are on top).
No idea if it would all work, and it is probably a load of work for little gain.
/e Thought I'd chuck in part of a menu where I layer buttons.
The background buttons display the image. The little squares (normally these don't show) are layered on top, and as each is selected, it runs a mapped key which does some ATG manipulation and updates the image of the button behind (it changes the background image to one with a "highlight" ring around the selected item which you can see in the first 3 selected). In order for me to be able to configure this in ISBoxer, the button behind needed to be presented (i.e. named btnX), which does make the view of the button set configuration a little odd as there a bunch of buttons that don't do anything, but it works. I could have had some things happening in script and some things not, but in this case I wanted to see if I could make ISBoxer do most of the work, and only have the custom template dealing with the button definition.
There is also a bunch of other funky stuff which normally sets the correct ring highlights for when the menu is loaded, but this test character set does not have the characters assigned to ATG's for the initial selection.
n.b The colour difference in the background images is something to do with my printscrn, and saving as the png file format. It is actually displaying the same image and looks uniform to me during normal usage.