Possible read/write locking issue?

TaskPaper
1

I believe there is a bug in TaskPaper’s file read/write code that causes deadlocks when other applications are also editing a file

Over the years I have periodically switched between TaskPaper and other approaches to organizing my life. I am currently in a TaskPaper phase. I keep my TaskPaper files inside folders that “live” inside nvUltra (a successor to nvAlt that Brett Terpstra and I are creating), and sometimes edit them in nvUltra, sometimes in TaskPaper, and sometimes both (depending on what I need to do.) . This generally works well, until it doesn’t…

In nvUltra (and MultiMarkdown Composer), I use Apple’s NSFileCoordinator to control access to reading/writing files and to “play nice” with the OS and other applications. Aside from the usual glitches during initial development, this has worked really well, and allows me to edit files in nvUltra while also working with them in other applications. Edits in one application appear in the the other, allowing me to use the best tool for whatever I am doing. The main tricks were properly handling multi-threading to handle it when changes were happening frequently.

I am frequently running into problems when editing a file in nvUltra and TaskPaper at the same time, however. If I edit a file in nvUltra and another app (for example, MultiMarkdown Composer), there are no issues. I can jump back and forth no problem, and make changes in either app. But when I open the file in TaskPaper, and edit in nvUltra, TP will frequently lock up, which then causes nvUltra to freeze since it is waiting for the NSFileCoordinator to complete read/write tasks. If I force quit TaskPaper, then nvUltra starts working again, and all is well. I can even replicate the same thing by opening a file in nvUltra/TP/MMDC simultaneously. All is well until TP freezes, then the file stops updating with my edits from nvUltra. Killing TP results in nvUltra/MMDC working properly again. If I do the same thing with only nvUltra/MMDC, then the problem does not occur.

I think the likelihood of the lockup occurring is related to how frequently the file is saved. nvUltra’s save frequency can be changed (with every keystroke, with every word, with every sentence, etc.) Composer relies more on manual saving or waiting for macOS’ automatic periodic saving to occur. The freezes happen much more often when I am editing in nvUltra as compared to MultiMarkdown Composer. If I manually trigger a save in Composer very frequently while typing, I can get TP to lock up more easily. But if the edits only occur on the default macOS schedule, I don’t notice problems (though my testing of this is limited.)

Of course it is possible there is a bug in my code somewhere. The fact that I cannot replicate this when I take TaskPaper out of the equation makes this less likely in my opinion, but it does not entirely exclude it. However, the frequency with which I experience this problem when using TP and the fact that I never see it with other applications is relatively strong evidence of a problem in TP’s read/write code in this situation.

I would love to see this resolved. I’m happy to help troubleshoot or provide support for a solution if you desire. Being able to safely work with a file even when while it is edited from another program is a great functionality that I make use of all the time, and it’s frustrating to have to stop what I am doing to force quit TaskPaper periodically when this happens.

Thanks!

Fletcher

2

I wonder if the extended attributes required by TaskPaper outlines for retention of folding state are a constraint here ?

They may be be the point at which it becomes slightly reductive to think of TaskPaper outlines simply as plain text files, and potentially over-complex to want to edit them in a flat text editor while also editing them in the originating outline editor.

A kind of category clash, perhaps ?

Possibly better to just raise a dialog warning in the flat editor to the effect that outline editing is in progress elsewhere ?

3

And/or let the flat editor offer to give focus to the outline editor,

rather than risk overwriting the user’s folding state ?

4

The issue is that my application has no way of knowing whether the file it is editing is also being edited by TaskPaper or any other application (nor should it need to know). So I can’t tell the user, “Don’t touch this file. Another application is also using it, and might not play nice.”

Assuming I have not made an error in my own coding (which is not 100% guaranteed), my app seems to be properly using the correct methods to safely read/write to the disk on file and works fine in my testing under other circumstances. When TaskPaper is also reading (not even editing) that same file, my app stops working because it is waiting on TaskPaper to relinquish access to my file coordinator. In the days of Dropbox, iCloud, etc. it is not unreasonable to assume that the underlying file for a document may be changed by another process, and in this case that doesn’t work.

Whether changes to the underlying file cause an unexpected change in the UI state (e.g. folding) is a separate issue, and a consequence a user should possibly be prepared to accept if a file is being edited in multiple locations (I’ll defer to Jesse how he wants this aspect of the UX to be). But even these effects can likely be minimized under most circumstances (but again a separate problem and not one I have attempted to solve in my own hierarchical editor view.)

My point is simply that it seems that TaskPaper prevents other applications from working properly if a file is opened in multiple applications at the same time. This does not seem to be a necessary thing (e.g. locking a critical database to a single process at a time), and therefore seems to be a bug.

5

Jesse will give you a fuller picture, but we should probably hesitate to conflate the correct methods for plain text files with those which are correct for outline files in which some of the data is encoded as extended attributes.

Blocking may be appropriate if it prevents another app from over-writing TaskPaper’s folding state while outline editing is in progress.

In addition, editing TaskPaper outlines in a flat text editor may well bring disagreeable surprises to users who are accustomed to outlines preserving their folding state between saves.

6

The type of file is irrelevant. NSFileCoordinator’s don’t care what type of document they are acting on. The type of file may certainly change what behavior you think is appropriate to take when the underlying file changes, but it doesn’t change the macOS/iOS system calls that should be used.

Regardless, the current behavior is that if the underlying text file is edited (it may take more than one change/save, but it happens pretty quickly), then TaskPaper and the other editor get into a NSFileCoordinator deadlock and both apps lock up, preventing the user from doing anything in either app.

It is certainly acceptable (though I would argue not optimal) to decide that TaskPaper will not update it’s NSDocument (or equivalent) when the underlying file changes and require the user to choose which version of the file to keep. I would argue this negates one of the major benefits of using plain text as a document format. But locking the UI (spinning beachball, etc.) is still not the optimal response.

As for extended attributes, I am happy to consider trying to preserve those when saving updates to files edited in nvUltra. I don’t need them in my app, but they don’t hurt anything in my app either. A few quick Google searches don’t show a simple way to do that “automatically” when updating a file via NSFileCoordinator, but maybe I’m searching the wrong terms. If there are any pointers/sample code to simply preserve existing xattr when updating the file I can look into it.

7

Irrelevant to the question of whether it is sensible to allow a flat text editor to be editing a file concurrently with an outline editor ?

You think so ?

8

Irrelevant to the mechanics of reading/writing files from the file system.

9

But not irrelevant to the user’s work and data.

10

Right – which is why I said that it’s acceptable for TaskPaper to pop an alert, “The underlying file changed. Do you want to stick with what is currently displayed or to update from the file?” (or whatever.)

But I don’t think locking up the app is a good choice. It’s a choice, sure, but not a good one.

We’re getting off track though – as a user, I certainly have my preferences about how TaskPaper responds when the file is modified (I think it should incorporate the changes made, just like most other apps in the modern document syncing/collaborative/iCloud/Dropbox era), but that is entirely up to Jesse/other devs. That is not what I am writing about.

My concern/bug report is that when the file is changed, TaskPaper (seemingly) causes both apps to lock up. I’m assuming you’re not suggesting that beachballing the app is the desired TaskPaper behavior under these circumstances. And if you agree with that, then the more productive line of reasoning here is to figure out what is going wrong and how to fix it.

I have tested my app as far as I can in this – this issue does not occur when I substitute TextEdit for TaskPaper, so it’s not simply a NSFileCoordinator issue, and doesn’t seem to be a mistake on my part. The reason I am writing is that further diagnosis will require some knowledge of TaskPaper’s code base and possibly running it in a debugging environment which means help from its dev(s).

11

I notice, FWIW, that Sublime doesn’t seem to struggle with opening an outline that is being edited in TaskPaper.

Sublime does, of course:

  • display the outline as a flat unfolded file,
  • and destroys the outline folding in the user’s TaskPaper session (on any save).

Users may vary in their enthusiasm for the proposition that TaskPaper should be rewritten so that their outline folding data can be more conveniently destroyed, mid-session, from another flat text editor …

12

I didn’t say that nvUltra had trouble when a file was being edited in TaskPaper, I said that TaskPaper had trouble when the file was edited in nvUltra. To the best of my knowledge, nvUltra properly handles coordinated file reading, so it is able to manage updating the document window when the file changes. I have not experienced any issues in this direction. The problems occur when nvUltra saves a file, and TP causes a deadlock when reading the changed file.

In addition, Sublime Text is a cross-platform app – I would suspect it does not use standard macOS NSFileCoordinators to handle file reading, so these deadlocks would not occur. But I do not know this for a fact. Possible additional evidence for this is the fact that ST does not update documents whose underlying files change while it is in the background – it only updates the document once the app is brought to the foreground. I believe the NSDocument object does not check foreground status when handling file changes (TextEdit does not, and MultiMarkdown Composer does not, which I built on an NSDocument model, which includes NSFileCoordinator functionality for read/write), but I suppose it would be possible to add this behavior if one desired.

Again – not relevant to the discussion at hand which is about deadlocking, not how TaskPaper chooses to handle whether to merge changes to the file or reject them. I am not asking that TaskPaper be “rewritten”, but rather that a bug be fixed – TP should not lock up when the underlying file changes, regardless of whether the choice is made that it accepts the changes or rejects them.

13

Same pattern with TextEdit, BBEdit, and Textastic:

  • Neither blocked nor blocking,
  • but a bad idea anyway – they destroy TaskPaper fold state on save.
14

I think the issue is related to frequency of saves. If the file is saved several times relatively quickly, it can trigger deadlocks within read code (this happened to me early on when I had some multithreading that was not quite handled correctly – it lead to deadlocks within my file read code that were not triggered otherwise. After I fixed them, it could handle updates as quickly as they happen. – Or, more precisely, it can catch up without deadlocking if it takes longer to read and update the file than the updates are taking.)

15

If your testing framework starts rapid-fire flat-file saves on a file that TaskPaper is editing as a foldable outline, with nodepath filtering and tagging, you risk:

  • not only repeatedly destroying TaskPaper’s outline folding and nodepath filtering data, but also
  • repeatedly depriving TaskPaper of the time it needs to rebuild a much more sophisticated parse and data model than a flat-file requires.

If a user of your flat-file editor attempts to open an outline with a .taskpaper extension, or with TaskPaper extended attributes, due diligence and protection of their data might (at the very least) suggest something analogous to:

  • Using runningApplicationWithProcessIdentifier to check whether the TaskPaper outliner is running
  • if it is running, check whether the AXDocument attribute of (at the very least) its front window shows the same file path as that selected by your user,
  • warn the user that outline’s folding and filtering data would be destroyed if it were opened in your flat-file editor,
  • and offer to switch focus to the outliner.

A flat-file editor may simply not be a useful instrument for editing outlines with folding state (and filter state) extended attributes.

A prudent default might be that your flat-file editor should offer to open TaskPaper outlines in the TaskPaper outliner.

What would you do with a docx file ?

16

PS the Cambridge dictionary is helpful here:

ultra definition: 1. a person who has extreme political or religious opinions

I wonder if this is the right decade in which to tie marketing to the image of ultras – whether to left or to right ? (‘Alt’ was already getting contaminated, alas)

Perhaps nvSolid, nvUseful, or nvModest ?

17

I give up.

I came here simply to report a bug in TaskPaper, which is that it craps out if the underlying file is modified under certain circumstances.

Fix it. Don’t fix it. Up to you. If any of my users have trouble, I’ll point them to this thread.

One of the things I used to love about Hog Bay products was the responsiveness and professionalism I saw when dealing with Jesse. I guess all things change.

18

My response is that of a user – nothing to do with Hog Bay.

Not clear to me from testing with other applications that there actually is a bug, but it is clear that editing a TaskPaper outline in a flat-file editor inevitably entails some data loss and is unlikely to be a sensible thing to encourage.

Better for the flat editor to open the file in TaskPaper.

1 Like
19

Quite possible! I know there has been such a bug before, though I did also rewrite all that code following the docs as closely as I could.

My test case for this is TextEdit… I open same file in TaskPaper and TextEdit and switch between the two as fast as I can… that still seems to be working for me on 10.15.2 without issue.

Are you able to reproduce this problem with TaskPaper and TextEdit?

2 Likes
20

complexpoint – When I click on your forum username, it shows that you are a member of the group “moderators”, of which you and Jesse are listed as the only members. This implies more of a relationship (in the forums at least) than simply just a user.

If I have misunderstood and you have no special standing with Hog Bay and are simply trolling, then my apologies to Jesse for allowing my frustrations with you to bleed over into Hog Bay.

Either way, at this point I have made my concerns about TaskPaper clear in this thread, and see no benefit in discussing further here. If Jesse has any questions, technical or otherwise, or wants a beta copy of nvUltra in order to test this to better and see what is going on, I’m happy to help.