Peace,
I'm not quite sure I've got this right, but in your example you only want to have a single viewport in the LayoutDocumentPane at any time.
As far as I know, there isn't a way to stop a user moving windows around, into the document pane. So what I would do is probably download the source files for AvalonDock and create new type of LayoutPane (e.g. LayoutSingleDocumentPane) which could be used to host a single child, and doesn't allow the drag and drop operations.
I don't have an example of this, but I'm sure it would be possible to do.
FallLine