That's your problem...from the OmniGraffle help for "Connecting objects with the Line Tool":
Click where you would like the source of the line; this can be an object (to which the line will stay attached) or an empty spot on the canvas (so the beginning of the line remains unattached).
Click as many empty spots on the canvas as you like to give the line midpoints.
End the line at an object by clicking the object (to which the line will stay attached), or end the line at an empty spot on the canvas by double-clicking there (this can be changed in the Drawing Tools preferences).