How do I install Ubuntu?

[Gaijin]

You probably didn't setup bootloader correctly, so that it points to Windows hdd and not your ExHDD, this happens the most of the times.

There is a way to fix this, it's that I just don't know it, Solidus might help you here better....
Otherwise just run the installation again and once you get to choose where you want to install it, you will have an "Bootloader" checkbox, just check it, works for me everytime...

Currently I'm on 11.04 and it's running form the USB (8gb), once I plug it out, it boots into windows normaly, and it doesn't matter in which port I put the usb...

edit: This might help you if you start a live session and try the steps...
http://techhamlet.com/2010/02/edit-grub-boot-loader/

another edit:

Yo see the "Device for Boot Loader installation", there you need to choose your USB device...