Should TaskPaper try to be smart about detecting leading spaces and turning them into tabs?


#1

Continuing the discussion from Multiple spaces are replaced by tabs:

Right now I have some buggy code that tries to automatically convert from leading space indentation to tab indentation when reading a file into TaskPaper. The reasoning behind this is:

  1. Lots of text editor default to using leading spaces for indentation.

  2. If such a file is opened in TaskPaper without conversion then it will visually look like there’s a hierarchy because of the leading spaces, but none of the hierarchy commands will work (such as expand collapse), because there will be no hierarchy in TaskPaper’s model. Visually you would have some indication that there isn’t a hierarchy, since the guide lines and leading bullets would all be left aligned, but still I’m sure it would be confusing.

The problem with this behavior (besides the fact that’s it’s buggy) is that it means if you open a file in TaskPaper, and then save it, spaces get converted into tabs even if you make no edits. So that behavior is a bit weird too.

I guess the “ideal” behavior would be to detect and remember the indentation method. Tabs, two spaces, four spaces, etc… and then when saving encode the indentation using the same method. That would be idea from the individual users perspective, but it would make parsing TaskPaper files more complex, since parsers would have to take into consideration multiple indentation methods.

Programmers and scripters (@complexpoint, @macdrifter, @mattgemmell) what do you think:

  1. Only consider leading tabs when parsing for indentation… just read file in and be done with it?

  2. Consider both leading tabs and spaces when parsing for indentation… but always save using tabs?

  3. Consider both leading tabs and spaces when parsing for indentation… and save out using detected method?

Note These a similar question relating to line endings. Right now TaskPaper recognizes multiple types of line endings, but always saves using \n.


Latest TaskPaper 3.2.1 preview has a small fix that I couldn't explain in the release notes
#2

I suppose that ideally the file’s existing formatting would be preserved. There are issues with exactly how you “detect” what they’re using, and what to do in the event of multiple different methods being used in the same file (do you preserve per-line?).

Having said that, normalisation is a good thing, and I imagine that it’s a minority of users who are creating or editing the files outside of TaskPaper itself (a vocal minority, certainly, since they’ll be scripters and devs and hackers, but still a minority).

For what it’s worth, TaskPaperRuby normalises the content when saving, but accepts mixed input using a default of either four spaces (configurable) or one tab per indentation level. When saving, indentation will always be in tabs. Similarly, it defaults to Unix line breaks, but can be configured to use Windows or even classic Mac breaks too. In each case, these are normalised file-wide when saving.

The ideal from an arbitrary user’s point of view would presumably be to preserve what they have. If it were me making the decision, though, I’d normalise on tabs and Unix line breaks.


#3

I agree with @mattgemmell, normalize is important, specially for those who script!

And indeed I would choose Tabs and Unix line breaks.


#4

Middle ground for me – your option 2 – I would personally prefer TaskPaper 3 to:

  1. Assume that if someone opens their file in TaskPaper 3, they do want it to be read / interpreted generously
  2. Assume that if they save it from TaskPaper 3, they know it will be saved in the canonical tab-indented (and unix line-delimited) format

#5

Thanks for the feedback. So I’ll keep my current approach, but fix the bugs!


#6

How much work would it be for TaskPaper to first determine if some of this spaces are presented and then ask the user with a popup what he wants to do. If TaskPaper changes the file, then have the ability to simply “undo” the changes.


#7

I’ve implemented choice 2 in the latest preview. It works by:

  1. Find the shortest span of leading spaces in leading whitespace.

  2. Treat that as tab size, replace all those spans with tabs.

  3. Remove any left over extra spaces in leading whitespace.

A simple way to test behavior is to just copy and paste text, because that runs through the same process.


#8

Probably not that much work… but “not that much work” always turns into a day for me. I think I’m just going to leave the current fixed behavior in place for now. I’ll come back to this later if it becomes a bigger issue down the road.


#9

I just wanted to say thank you for this feature. This solves a problem I’ve been having for some time, namely, syncing taskpaper files via Simplenote, which converts tabs to spaces in a way that has always been difficult to work around. This means that I can finally edit files in nvALT and Taskpaper and not lose all formatting.

Cheers!


#10

I wonder if this introduced a new problem. I previously might want to append something to the end of a line of text from another line of text. To do this I would select the text including the leading space, copy it to the clipboard, and paste it to the end of the desired line. The leading space would be gone. I checked and it actually is stripped off on the clipboard. This is confusing when what you select and copy isn’t on the clipboard.


#11

I meant to say that this worked some time before this version and now does not.


#12

I’m not sure that I understand the details of what you are trying to do… but I think this might be related to the “Smart Copy/Paste” setting? Make sure that’s set in the document that you are working in and see if that fixes the issue. If not can you please give a detailed step by step scenario of the behavior that you expect in an example document. Thanks!


#13

Look at:

for movie with details, note file, and two taskpaper files.


#14

Thanks, I see the problem now. It’s related to:

Basically when you paste in text I first run it through TaskPaper’s parser to create “items” and then I paste those items into the document. The parser now discards those leading spaces, encoding them in item level. But when you paste an item into another items text the item level isn’t used… a bit hard to explain but end result is spaces get lost.

I need to think about a solution, in the meantime I’m going to move these posts over to that topic.


#15

Hi,

Personally, I really appreciate TaskPaper’s ability to detect leading spaces and converting them internally to item levels, for the reasons you have cites (i.e. most plain text editors, especially those for programming, will do the opposite).

Where things become problematic for me is when TaskPaper automatically saves the same file with tabs instead of spaces, because the file is no longer valid YAML (even it were before being opend by TaskPaper).

Let me explain a bit more in detail…

In fact, TaskPaper’s on disk format for basic task usage is currently almost valid YAML, as long as the user is willing to forego some features. The almost comes from the TAB character (forbidden in YAML). Let me try to explain a bit more in detail…

My use case involves transitioning my TODO files (in multiple software projects) to a very simply structured version in YAML format (TODO.yml).

Once they are structured, they become more readily eligible for automated processing by software, such as by a simple scripts for placing a copy of the DONE items in the “Changes” file, or what not.

The reason I have chosen YAML over some other options should be obvious : wide support (like XML and JSON), much more human-friendly (than XML and JSON), ubiquity, simplicity, ease of processing, …

Granted, TaskPaper’s on disk format is great, too; and it readily ticks almost all the boxes above, except that is specific to the “task/todo management” functional domain, and therefore 3rd party software support is naturally limited to that particular domain.

However, had the format been a valid incarnation of an existing more generic data format (such as YAML), then all sorts of other possibilities would open up automagically across the board, while preserving its human-friendlieness.

All of a sudden :

  • The taskpaper format would become readily parsable by existing YAML libraries of pretty much all modern computer languages;

  • It would be possible to use many existing plain text editors (Atom, Sublime, BBEdit, Eclipse, …) in YAML mode, with syntax highlighting and such delights.


As you may have guessed, ideally, I would really love TaskPaper’s on-disk format to evolve in that direction one day: i.e. a functional format which in valid YAML, also with support of the existing great features such as attributes (@).

However, I also realize that something like this is not simply done as said. Developing it is one thing (which wil probably not be that big of deal), but the more delicate part would probably be the decision for whether or not to produce a v2 of the format in the first place…

Therefore, I keep my expectations low and only ask for a small change descriibed below…

REQUESTED FEATURE :

Give the user the option to save files with spaces instead of TABS.

Ideally the said user preference would be persistent (i.e. set via the preferences menu), and would like :

  • :white_check_mark: Save files with SPACES instead of TABS

And the above is checked, one would be able to change the number of spaces that would correspond to a tab :

  • 1 tab (level indentation) => 2 spaces

This way, the file produced will be valid YAML, as long as the user is willing to abstain from some features (such as the ‘@’). – BTW, such data will still be present in the file as opaque text values and may be extracted by 3rd party software and scripts after parsing the file as YAML, if at all needed.


Using a simple one liner to translate the tabs into spaces and vice-versa is certainly possible today, and that’s what I have been doing; but it’s so easy to forget to do that each and every time the file has been saved by TaskPaper…

Apart from this pain point, I really like being able to use a specific UI for managing tasks and TaskPaper really stands out from the rest of the bunch, in the mac space…

Cheers,
Ayk


TaskPaper Extensions Wiki
#16

One approach to that is, of course, to use the API to add that functionality, and embed it in something like Keyboard Maestro or LaunchBar etc


#17

A first draft might look something like the script below:

(See https://guide.taskpaper.com/using-taskpaper/using-scripts.html)

(() => {
    'use strict';

    // Save active TaskPaper document as space-indented file
    // Rob Trew 2018

    // Ver 0.02
    // (adjusted a type signature)

    // OPTIONS
    const
        // e.g. 4 spaces or 2 spaces
        strIndent = '    ',

        // Initial path for Save As dialog:
        strDefaultSavePath = '~/Desktop/outline.txt';

    // TASKPAPER 3 CONTEXT --------------------------------
    const tpJSContext = (editor, options) => {

        const tp3Main = () =>
            indentedTextFromNodes(
                options.indentUnit,
                x => x.bodyString,
                editor.outline.root.children
            );

        // indentedTextFromNodes :: String -> 
        //      (TP3Node -> String) -> [TP3Node] -> String
        const indentedTextFromNodes = (strUnit, fnText, nodes) => {
            const go = indent => node =>
                indent + fnText(node) +
                (node.hasChildren ? (
                    '\n' + node.children
                    .map(go(strUnit + indent))
                    .join('\n')
                ) : '');
            return nodes.map(go('')).join('\n');
        };

        // MAIN ---
        return tp3Main();
    };

    // JXA CONTEXT-----------------------------------------

    const jxaMain = () => {
        const
            ds = Application('TaskPaper')
            .documents,
            lrSavedTo = bindLR(
                bindLR(
                    ds.length > 0 ? (
                        Right(ds.at(0))
                    ) : Left('No documents open'),
                    d => Right(d.evaluate({
                        script: tpJSContext.toString(),
                        withOptions: {
                            indentUnit: strIndent
                        }
                    }))
                ),
                strOutline => bindLR(
                    confirmSavePathLR(strDefaultSavePath),
                    fp => (
                        writeFile(fp, strOutline),
                        Right(fp)
                    )
                )
            );
        return lrSavedTo.Left || lrSavedTo.Right;
    };

    // GENERIC --------------------------------------------

    // Left :: a -> Either a b
    const Left = x => ({
        type: 'Either',
        Left: x
    });

    // Right :: b -> Either a b
    const Right = x => ({
        type: 'Either',
        Right: x
    });

    // Tuple (,) :: a -> b -> (a, b)
    const Tuple = (a, b) => ({
        type: 'Tuple',
        '0': a,
        '1': b,
        length: 2
    });

    // bindLR (>>=) :: Either a -> (a -> Either b) -> Either b
    const bindLR = (m, mf) =>
        m.Right !== undefined ? (
            mf(m.Right)
        ) : m;

    // Split a filename into directory and file. combine is the inverse.
    // splitFileName :: FilePath -> (String, String)
    const splitFileName = strPath =>
        strPath !== '' ? (
            strPath[strPath.length - 1] !== '/' ? (() => {
                const
                    xs = strPath.split('/'),
                    stem = xs.slice(0, -1);
                return stem.length > 0 ? (
                    Tuple(stem.join('/') + '/', xs.slice(-1)[0])
                ) : Tuple('./', xs.slice(-1)[0]);
            })() : Tuple(strPath, '')
        ) : Tuple('./', '');

    // JXA FileSystem

    // standardSEAdditions :: () -> Application
    const standardSEAdditions = () =>
        Object.assign(Application('System Events'), {
            includeStandardAdditions: true
        });

    // doesDirectoryExist :: FilePath -> IO Bool
    const doesDirectoryExist = strPath => {
        const ref = Ref();
        return $.NSFileManager.defaultManager
            .fileExistsAtPathIsDirectory(
                $(strPath)
                .stringByStandardizingPath, ref
            ) && ref[0] === 1;
    };

    // writeFile :: FilePath -> String -> IO ()
    const writeFile = (strPath, strText) =>
        $.NSString.alloc.initWithUTF8String(strText)
        .writeToFileAtomicallyEncodingError(
            $(strPath)
            .stringByStandardizingPath, false,
            $.NSUTF8StringEncoding, null
        );

    // confirmSavePathLR :: FilePath -> Either Message FilePath
    const confirmSavePathLR = fp => {
        const
            tpl = splitFileName(fp),
            fldr = tpl[0],
            fname = tpl[1];
        return (() => {
            const sa = standardSEAdditions();
            sa.activate();
            try {
                return Right(
                    sa.chooseFileName({
                        withPrompt: "Save As:",
                        defaultName: fname,
                        defaultLocation: Path(ObjC.unwrap(
                            $(doesDirectoryExist(fldr) ? (
                                fldr
                            ) : '~')
                            .stringByExpandingTildeInPath
                        ))
                    })
                    .toString()
                );
            } catch (e) {
                return Left(e.message)
            };
        })();
    };

    // MAIN ---
    return jxaMain();
})();