This is a very late response, but I think the latest preview release fixes this issue:
The accessibility API is very aggressive in the data it collects in some cases… in particular when you do a hit test in text area with accessibility API it scans the entire text storage for all link and file attachment attributes. This is an expensive operation in a large TaskPaper document. I couldn’t find any way to fix except to use private API to override that behavior and return nil for both link and attachment collections. Not pretty, but I “think” it will solve the problem for you.