You want to permanently remove the Standard Menu from your Character Set in your ISBoxer settings. This is the "green squares" and was assigned via the last page of the Quick Setup Wizard. On your next time through the wizard, just un-check the box on the Hotkeys page that says standard broadcasting of 1 through =. To remove it from your existing Character Set, select it in the top left pane under Character Sets; then with that selected, look in the bottom left pane under Menus and where it says Standard, right click that and select Delete, then do an Export to Inner Space.
Regarding the window size, Trove is an OpenGL game and will therefore not fully operate exactly the same way as Direct3D games do as far as your Window Layout; ISBoxer doesn't produce the same quality of GUI overlay, either. Standard Window Layouts with small windows and instant swapping will not work right because OpenGL does not scale the display to the size of the window like Direct3D does. This is probably why you matched the size of the game windows, which is the right call here. You CAN use
DxNothing with Video FX to simulate a fully functional Window Layout though. MiRai is currently working on a video demonstrating the process for that.
Finally, the camera. I see that the mouselook movement in your video is choppy. From what it looks like in the video, I'd say the game is using a fairly typical method of allowing you to move the mouse, and then re-centering it to prevent it from leaving the game window while the cursor is hidden. That should actually be fine, so I'll have to do some digging to find out why it is not operating as desired with Trove (i.e. pushing it back resulting in the choppy movement). It will take some time, I don't have an ETA -- it could take days or more, but it's also possible that it's easy to track down and solve. I'll keep you updated.