I assume you are using an Other Window Source, for video fx.
http://www.isboxer.com/wiki/Other_Window_SourceThe scaling will work just the same, in that if the viewer is a proportional size of the source then the view will be scaled down. i.e. if the source is 1920x1080 and you multiply it by 0.2, then you get 384x216, so you have a VideoFX viewer 1/5 the size of the original.
I find the Other Window Source is a bit more tricky to edit (I don't know where the origin info is stored) and has a couple of idiosyncrasies. When you create the source, you should specify the dimensions of the source window that you want to grab. If you know the offset from 0,0 and the size of the window it helps. If you are just cutting down a maximised window then this is easier as this is just the monitor dimensions. When you add the viewer, initially you specify the same size as the source (if you just want the full screen scaled size). Once the Source and Viewer have been added, hit Apply, and then change the size of the viewer to your scaled size, hit Apply again, and it should display correctly.
I could be doing something wrong of course, but this is how it appears to work for me.