[whpalmer4]

The files are embedded in your database (which lives at Library/Application Support/OmniFocus/OmniFocus.ofocus in your home directory) but you would be unwise to try deleting them via the Finder. Instead, sort the Attachment List window by clicking on the column headers, then go to an individual attachment's action or project by double-clicking on one of the rows. Delete the attachment by editing the note. Repeat as desired.

It's a little surprising to me that you're seeing it take 30-60 seconds to bring up a new window, as my database is even larger (though fewer attachments), yet switching views takes less than 10 seconds. Some additional things you might try:

collapse all the notes, via Edit->Toggle All Notes

use folders to divide up the pile, and focus on the folder in which you are working by selecting it in the sidebar and then right-clicking and selecting Focus

Basic idea is to remove as much as possible from the current view.