I might use some variant of
parent task is root task of containing project
Code:
tell application id "com.omnigroup.omnifocus"
tell content of front document window of front document
set lstTasks to value of selected trees
set oTask to first item of lstTasks
tell oTask
set blnIsInGroup to (parent task is not root task of containing project)
end tell
end tell
end tell