There is a 3-day lock penalty for canceling your trade after 5 minutes on a regular trade slot or prior to a 4 hour posting period.In addition, there is no penalty if at least one item in the slot has sold.
There is no penalty for canceling at any time for an upgraded trade slot.