I was not able to reproduce this in a clean install of X-Plane 11.11. I replayed part of a flight and toggled the movie recording on at the same time. No strange camera behavior happened. I stopped recording the movie and viewed the .avi file in the Output folder, but again it looked totally normal.
Check for add ons that could be causing this, or additional steps. If you find it is reproducible with more information and no add ons, you can file a bug report here. Include a log.txt and be as detailed as possible.