Feature request for a keyboard shortcut to fully expand the sidebar

Hello Jesse,

I am still, on occasion, experiencing the bug where projects with subprojects in the sidebar don’t remember their open state after I close and later reopen TaskPaper. It might be related to Dropbox updating my TaskPaper file, but I haven’t been able to track down the reproducible cause—it just happens now and then (often enough to be irritating).

Until this bug is fixed, I realized that a new feature could make me happy in the interim. If this would be something easy for you to implement, then you can resolve my last irritation with TaskPaper.

What I am looking for: a keyboard shortcut (or menu command) that, when activated, would expand all of the projects/subprojects in the sidebar. Something like “Expand All Completely” but specifically for the sidebar.

Any chance this could be implemented into a future update?

Jim

Am I right in thinking that Dropbox syncing still discards macOS-specific metadata from files ?

Hi Rob,

I’m not sure. What type of metadata does OS X store for text files? Where would I find it? I can look into the DB behavior if I know what I am looking for.

FYI—When using our Mail to TaskPaper script in Keyboard Maestro, I can get the subprojects to collapse. So, this doesn’t seem to be Dropbox specific.

Limited test results with our script:

  • If TaskPaper is closed, and I execute our script, then my subprojects in the sidebar maintain their state.
  • If TaskPaper is open, and I execute our script, then my subprojects in the sidebar are collapsed.

I haven’t done extensive testing on the above, but a few run-throughs have consistently produced the same results. Noting it here in case it helps Jesse or you.

Jim

Hi Jim,

If we use the shell xattr command,

(see man xattr in the Terminal)

on a TaskPaper file:

 xattr /Users/houthakker/Notes/notes2018-01-06.taskpaper

Then we can see the attached metadata, including the storage of fold state and other settings.

com.taskpaper.editor.defaultRestorableState.houthakker
com.taskpaper.outline.metadata

I think that Dropbox syncing syncs the file but not the macOS-specific metadata of this kind.

Better check with Jesse whether this is sensible, but I guess it might be possible to use Hazel to:

  1. Watch a folder for changes in TaskPaper files
  2. If a changed file has TP3 metadata, add or update a record of its path and metadata in a JSON (or sqlite) file
  3. If its TP3 metadata has gone missing, restore it (if found) from the JSON/sqlite store)

The rudiments of a JavaScript for Automation road towards this might include:

(() => {

    ObjC.import('sys/xattr');
    
    // return $.getxattr
    
    return $.setxattr

})();

You can find arguments for these and other related xattr functions in an article on the nshipster.com blog.

Rob

(What I am not sure about, however, (without experimenting) is whether this would just trigger another Dropbox sync, shedding the metadata again, and descending into a loop)

Thanks for the info Rob. I will dig into Dropbox changing or losing the metadata tonight.

That said, I believe that Dropbox is not the cause of the issue, or at least, not the sole cause.

With Dropbox off (which means that it shouldn’t’ be affecting our tests), I am able to reliably reproduce the issue with our Mail script.

I made a very short video showing this in action:

So, my first question would be: does our Mail script do things differently depending on whether TaskPaper is open?

Well spotted – that will do it too – that is indeed part of the trade-off between treating .taskpaper files as application files - handled by TaskPaper as a sort of dual channel plain text + metadata ‘monad’ - and treating them as just plain text text files - highly scriptable but not retaining memory of GUI subtleties like folding state.

We should check with Jesse, but I don’t think that Birch is itself likely to contain code for reading / writing the additional metadata channel.

Over the weekend I’ll give some thought to writing a generic inject/bind function, to inject plain text transformations in through the metadata envelope (without losing it) when writing JXA scripts.

Pinging @jessegrosjean to get his input.

@Jim @complexpoint

I’m away from my development computer this weekend and need to take a look there to really remember what’s going on. Unfortunately I don’t think there’s any way to expand/collapse items in the sidebar view using the scripting API. @complexpoint It may be possible through AppleScript User Interface Automation?

I’m not sure, but I think the sidebar state isn’t stored as an xattr. The main editor view expand/collapse state is, but I don’t think the sidebar is. Instead I think that’s stored using Apple’s user interface preservation API’s, the code that allows apps to quite and then reload in the same state next time app is started. But I really need to look at the code to really see what’s happening.

I’ll try to look at things early next week… if I don’t respond with more info here please post me a reminder! :slight_smile:

Jesse

1 Like

Will do Jesse—thank you and have a great weekend!!

We can certainly get references to the the cells of the sidebar outline row – we don’t seem to see any actions like click defined for them through the System Events scripting API, but we do see their position and size, so things like Keyboard Maestro should be able to find and click them.

I’ll experiment tomorrow.

A JXA route to inspecting them might look something like:

(() => {
    'use strict';

    // GENERIC FUNCTIONS -----------------------------------------------------

    // bindMay (>>=) :: Maybe a -> (a -> Maybe b) -> Maybe b
    const bindMay = (mb, mf) =>
        mb.nothing ? mb : mf(mb.just);

    // just :: a -> Just a
    const just = x => ({
        nothing: false,
        just: x
    });

    // nothing :: () -> Nothing
    const nothing = (optionalMsg) => ({
        nothing: true,
        msg: optionalMsg
    });

    // show :: Int -> a -> Indented String
    // show :: a -> String
    const show = (...x) =>
        JSON.stringify.apply(
            null, x.length > 1 ? [x[1], null, x[0]] : x
        );

    // UI SCRIPTING REFERENCE TO TASKAPER 3 SIDEBAR OUTLINE CELLS ------------
    const
        se = Application('System Events'),
        procs = se.applicationProcesses.where({
            name: 'TaskPaper'
        }),
        mbSideRows = bindMay(
            bindMay(
                bindMay(
                    procs.length > 0 ? (
                        just(procs.at(0))
                    ) : nothing('TaskPaper not running.'),
                    proc => {
                        const ws = proc.windows;
                        return ws.length > 0 ? (
                            just(ws.at(0)
                                .splitterGroups.at(0)
                                .scrollAreas)
                        ) : nothing('No TaskPaper windows open.')
                    }
                ),
                areas => areas.length > 1 ? (
                    just(areas.at(0))
                ) : nothing('Sidebar not open.')
            ),
            area => just(area.outlines.at(0)
                .rows)
        );

    // MAYBE LIST OF SIDEBAR OUTLINE ROW CELL TEXTS -------------------------------
    return show(2,
        mbSideRows.nothing ? (
            mbSideRows.msg
        ) : mbSideRows.just()
        .map(
            x => x.uiElements.at(0)
            .uiElements.at(0)
            .name()
        )
    );
})();

Did have a look today – what I haven’t found a route to (through System Event UI scripting) is the Show/Hide (visible+clickable on hover) control to the right of the (Projects / Searches / Tags) headings

(and the file attributes do indeed seem to hold only the fold-state of the text body - not that of the sidebar)

Off topic but storing taskpaper files in Dropbox is not a great experience due to this ‘meta data’ issue. Document gets completely removed from the Recent Documents list, the only way I have to keep track of my documents is using the macOS colour labels, which do not get removed by the Dropbox application. So I can’t really say what is going on because I read the colour labels are tiny bits of meta data. I’m on High Sierra but this behavior dates back to Yosemite.

In principle iCloud should do better with these extended attributes than Dropbox.

I’m not an iCloud user myself, and the picture does seem to be mixed (some xattrs apparently do get dropped),

https://mjtsai.com/blog/2018/01/08/icloud-drive-can-strip-metadata-from-your-documents/

but Apple’s marketers seem to feel that the goal is to retain them.

PS

While on this topic, it occurs to me that any editing of a .taskpaper file in a 3rd party app on iOS might also shed xattrs.

@complexpoint — when I use xattr in the Terminal, I get:

com.apple.lastuseddate#PS
 com.apple.quarantine
 com.dropbox.attributes
 com.taskpaper.editor.defaultRestorableState.jim
 com.taskpaper.outline.metadata

I am not sure what to do with the above. At the moment, I don’t have another Mac to test with, so I am not sure if I can test if the metadata is synchronized or not.

Here is your reminder @jessegrosjean :slight_smile:

I’m not seeing that at all. My TaskPaper files are all listed in the recent documents menu. Weird.

So I edit something on my iPhone using Editorial and it syncs back to my Mac and Taskpaper has ‘forgotten’ the document, not only closes it but also missing from the Recent Documents. I do have the System Preference set properly as explained on these forums, I don’t know how macOS handles application state.

Hi Jim, those extended attribute keys, shown by xattr, turn out to store the folding state of the paragraphs themselves, but not, it seems, of the sidebar. (You can see the value attached to each key by adding the -l switch to the xattr command line (see man attr)

It appears that the the sidebar state is preserved by the app, as Jesse suggested, through a UI preservation API.

Jesse will know, but perhaps this is the process:

https://developer.apple.com/documentation/uikit/view_controllers/preserving_your_app_s_ui_across_launches/about_the_ui_preservation_process?language=objc

My impression is that this is a form of application-state memory, rather than document-state memory.
I wonder if the unexpectedly closed sidebar project trees arise when you re-open an earlier document (and the app) after a session with a different document, with which the sidebar tree had not been opened ?

Ok, I’ve looked into this a bit more.

First this info is stored in TaskPaper’s system preferences plist file under the key NSOutlineView Items Sidebar. Associated with that key is a list of item identifiers. Some are hard coded group names such as “projects”, “searches”. And some are content hashes of particular outline items in your outline that are expanded.

So it’s not stored in apple’s user interface preservation system, or in my xattr system… stored in TaskPaper’s user defaults.

This means a few things. First there’s a single user defaults key, so the sidebar expanded state is only stored for the last document that changes it. Since some keys “projects”, etc are shared across all documents it means all documents should open with the same group state. But since content hashes are not shared across documents it means that heading expansion state will only be restored for the last document that wrote to that key.

It also means that if you edit the outline branch of an expanded item then that will change the content hash for that item. And unless the state is reserved to the defaults then that item that was edited won’t be restored as expanded on the next launch.

Unfortunately I don’t have a lot of control over when those values are written to user defaults… right now I’m just setting the NSOutlineView’s autosavename and the system does the rest. But there doesn’t seem to be a good way to invalidate the default. From what I can tell it will only save new defaults when you explicitly expand/collapse an item.

I’m not sure what the best end solution is… I tried a few quick hacks to solve but they all had other tradeoffs. I think earlier you said an “Expand All in Sidebar” menu it would help you out… does that still seem useful? I could add that pretty easily.

In the new system that I’m working on with WriteRoom 4 I’m making the sidebar scriptable, so eventually you’ll have more control, but as always that’s taking more time then expected!

1 Like

I’d be delighted if I had that as a workaround solution.

I also just noticed that if you hold down “Option” when expanding items in the sidebar it will do an “Expand All”. So for example while holding down “Option” click the “Projects” group “Hide/Show” button and it will expand all in that group.

2 Likes