I have worked around this problem by
- Tagging all objects with a value indicating their intended rank grouping
- Selecting sets of objects via that tag value
- Using System Events to click the "Same" radio button
This achieves assigning the shapes to their proper rank groupings, still using entirely Applescript.
However, the next problem comes when I ask the canvas to do the Layout. If I first select the "Minimize Crossings" mode, the applications locks up, consuming 100% of the processor, and never performs the layout. I am leaping to the conclusion that it is hitting an infinite loop in the layout code caused by some conflicting constraints due to the object ranks.