Right click on the XAML file, and click "Open With..":

 

Choose Source Code (Text) Editor, and click "Set as Default":

Next time when you double click on XAML file, it will be opened without design, and faster.

 

Thanks to colleagues from ISE