I'm not sure why you think not having multiple windows open is more Mac-like!
There is no setting I'm aware of to prevent OF from opening new windows when you double-click on an action or project, follow a Growl notification, or use the dock or menu bar to go to a context. However, you can use my handy script to close all but the front window when they accumulate.
Thread on closing excess windows