Presumably you have Diablo 3 set to Windowed Fullscreen as recommended in this sticky post:
Tips: Resolutions,Windowed modes,Text,Swapping SpeedThe only way to get the in-game resolution option working is to switch the game to Fullscreen mode instead.
Otherwise, the resolution is based on your Window Layout, and you can change your Window Layout to adjust your resolution. Assuming you have Instant Swapping enabled in your Window Layout, then this involves opening up the Regions editor for your Window Layout, adding another Region to represent the resolution you want (i.e. size it to not be 1920x1080), and then setting it to the Swap Group's "Reset Region".
In step by step format:
1. In the top left pane in ISBoxer under Window Layouts, select yours
2. In the bottom right pane, observe the "Use Instant Swapping" checkbox; if this is NOT CHECKED then to change your resolutions you would just resize individual regions instead of adding a new one as I'm instructing. I'm assuming this is already checked.
3. In the bottom left pane, select "Regions" to bring up the Regions editor in the bottom right pane
4. In the bottom right pane, find your main window region (the 1920x1080 one), right click it and select Duplicate Region. A new one with a higher number should appear on top of it.
5. Resize the new region as desired. You can put in the exact numbers for the resolution you want, in the properties box on the right; or you can click and drag.
6. In the properties box on the right, still within the "Regions" tab, at the bottom is a "Swap Group" option, click the + to expand it
7. One of the 4 options underneath "Swap Group" is labelled "'Reset' Region". Change this to your newly-added Region, the highest numbered one. (i.e. if you just added and resized one labelled "4", set this to "Region 4")
8. Export to Inner Space, and re-launch your team
The new Region will then be used to set up the resolution for each instance, prior to scaling it into its Home or the Active region.