You can at any point change from horizontal to vertical/portrait mode. Simply grab a hold of your phone - and move it up to the vertical position.
See the gif below:
NB: You need to hold the handle at an angle in order for vertical mode to work. Holding the handle pointing straight up will cause two of the motor axis to align, and effectively turn the stabilizer into a 2-axis stabilizer instead of a 3-axis one.
To change back to horizontal position, simply reverse the process. Grab a hold of your phone and move it down to the horizontal position.