Bug Where Bike Deletes Text When Saving

fssbob · August 3, 2022, 3:19am

Ugh–I’ve just discovered a rather nasty Bike bug that’s caused me to lose a fair amount of data.

I frequently “clean” my clipboard before pasting by running a script that contains the following command:

set the clipboard to (get the clipboard as text)

I’ve been using this for years, and paste text into all kinds of apps using it.

When I paste text into Bike after running this script, everything appears to be fine. But when I later reopen the document, I find that Bike has deleted the entire pasted text passage except for the first character.

This should be easy to duplicate–happens every time.

Thanks.

(v 1.3.1)

complexpoint · August 3, 2022, 3:56am

That kind of clipboard operation:

(over-writing a complex Bike clipboard, containing three different types of pasteboard item – Bike XML, OPML, and plain text – with a simple plain-text-only clipboard)

may actually be a bit more destructive than is intended.

In what sense are you thinking of it as “cleaning” ?

complexpoint · August 3, 2022, 4:13am

The problem turns out to involve, amongst other things, a bug or fossil in AppleScript behaviour, which creates a non-compliant (\r delimited rather than \n delimited) plain text sequence from your clipboard-rewriting incantation.

(It looks like Bike may need to allow for the pasting of unexpected \r delimited outlines)

If we copy a simple outline like this:

We get a Bike clipboard with three types of paste-board items: Bike XML, OPML, and Plain Text:

Expand disclosure triangle to view a representation of the Bike clipboard

{
  "com.hogbaysoftware.bike.xml as string": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<html>\n  <head>\n    <meta charset=\"utf-8\"/>\n  </head>\n  <body>\n    <ul id=\"EKfuLIfK\">\n      <li id=\"my\">\n        <p>Alpha</p>\n        <ul>\n          <li id=\"eG\">\n            <p>Beta</p>\n          </li>\n          <li id=\"UB\">\n            <p>Gamma</p>\n          </li>\n          <li id=\"PR\">\n            <p>Delta</p>\n          </li>\n        </ul>\n      </li>\n    </ul>\n  </body>\n</html>\n",
  "org.opml.opml as string": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<opml>\n  <head>\n    <meta charset=\"utf-8\"/>\n  </head>\n  <body id=\"EKfuLIfK\">\n    <outline id=\"my\" text=\"Alpha\">\n      <outline id=\"eG\" text=\"Beta\"/>\n      <outline id=\"UB\" text=\"Gamma\"/>\n      <outline id=\"PR\" text=\"Delta\"/>\n    </outline>\n  </body>\n</opml>\n",
  "public.plain-text as string": "Alpha\n\tBeta\n\tGamma\n\tDelta",
  "public.utf8-plain-text as string": "Alpha\n\tBeta\n\tGamma\n\tDelta",
  "public.data as string": "Alpha\n\tBeta\n\tGamma\n\tDelta",
  "com.hogbaysoftware.bike.references as string": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<html>\n  <head>\n    <meta charset=\"utf-8\"/>\n  </head>\n  <body>\n    <ul id=\"EKfuLIfK\">\n      <li id=\"my\">\n        <unknown/>\n      </li>\n    </ul>\n  </body>\n</html>\n"
}

Whereas if we copy four lines with tab-indents from a standards-compliant plain text editor we get just \n delimited plain text:

{
  "public.utf8-plain-text as string": "Alpha\n\tBeta\n\tGamma\n\tDelta",
}

Look, however, at what that AppleScript incantation overwrites the Bike clipboard with:

{
  "public.utf16-plain-text as string": "A\u0000l\u0000p\u0000h\u0000a\u0000\r\u0000\t\u0000B\u0000e\u0000t\u0000a\u0000\r\u0000\t\u0000G\u0000a\u0000m\u0000m\u0000a\u0000\r\u0000\t\u0000D\u0000e\u0000l\u0000t\u0000a\u0000",
  "public.utf8-plain-text as string": "Alpha\r\tBeta\r\tGamma\r\tDelta",
  "com.apple.traditional-mac-plain-text as string": "Alpha\r\tBeta\r\tGamma\r\tDelta",
  "dyn.ah62d4rv4gk81g7d3ru as string": "\u0001\u0000\u0000\u0000\u0000\u0000\u0010\u0000\u000e\u0000\u0003\u0000\u0000\u0000\f\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
}

Bike XML has been lost, and the public.utf8-plain-text pasteboard item, which should still be \n delimited, has now been overwritten with \r delimiters.

(Not sure if this is the kind of cleaning you had intended, but it’s not what macOS expects, these days).

Possibly worth flagging to Apple, though maintenance of AppleScript seems to have moved into a sunset phase now.

complexpoint · August 3, 2022, 4:32am

In the meanwhile, if you can tell us more about what you need from “cleaning”, we can probably write you a script which does what you want without overwriting \n with \r, and perhaps without overwriting other things too.

fssbob · August 3, 2022, 4:36am

Got it. Looks like I was using an AppleScript fossil. The result worked in all the apps I’ve used it with, but eventually it was going to break something. Thanks for the quick analysis. I guess Jesse can decide if it’s worth handling.

I’m going to replace the script with a small, free app I’ve found: Pure Paste. On first blush it seems to do what I want in a more up-to-date way.

This may not be possible, but it would be wonderful if Bike had a generic way of catching and giving a warning any time something unexpected is encountered. A situation like this where there isn’t any visual clue that something is wrong but the text doesn’t get saved is pretty painful.

complexpoint · August 3, 2022, 4:51am

I don’t know about Pure Paste, but this JS script should also do it, if I have understood what you are after.

You could test in Script Editor, and then assign it to a keystroke in FastScripts or in a Keyboard Maestro Execute a JavaScript for Automation action.

Expand disclosure triangle to view JS Source

(() => {
    "use strict";

    ObjC.import("AppKit");

    // Reducing a Bike clipboard to plain text
    // without pollution by \r delimiters.
    // (A problem with Applescript's `get clipboard as text`)


    // Rob Trew @2022
    // Ver 0.01

    // USE
    // This is a JavaScript for Automation script
    // for Script Editor testing, set the language selector
    // at top left to JavaScript rather than AppleScript.

    // main :: IO ()
    const main = () =>
        either(
            alert("Reducing clipboard to plain text")
        )(
            copyText
        )(
            clipTextLR()
        );

    // ----------------------- JXA -----------------------

    // alert :: String => String -> IO String
    const alert = title =>
        s => {
            const sa = Object.assign(
                Application("System Events"), {
                    includeStandardAdditions: true
                });

            return (
                sa.activate(),
                sa.displayDialog(s, {
                    withTitle: title,
                    buttons: ["OK"],
                    defaultButton: "OK"
                }),
                s
            );
        };


    // clipTextLR :: () -> Either String String
    const clipTextLR = () => {
    // Either a message, (if no clip text is found),
    // or the string contents of the clipboard.
        const
            v = ObjC.unwrap(
                $.NSPasteboard.generalPasteboard
                .stringForType($.NSPasteboardTypeString)
            );

        return Boolean(v) && 0 < v.length ? (
            Right(v)
        ) : Left("No utf8-plain-text found in clipboard.");
    };


    // copyText :: String -> IO String
    const copyText = s => {
        const pb = $.NSPasteboard.generalPasteboard;

        return (
            pb.clearContents,
            pb.setStringForType(
                $(s),
                $.NSPasteboardTypeString
            ),
            s
        );
    };


    // --------------------- GENERIC ---------------------

    // Left :: a -> Either a b
    const Left = x => ({
        type: "Either",
        Left: x
    });

    // Right :: b -> Either a b
    const Right = x => ({
        type: "Either",
        Right: x
    });


    // either :: (a -> c) -> (b -> c) -> Either a b -> c
    const either = fl =>
        // Application of the function fl to the
        // contents of any Left value in e, or
        // the application of fr to its Right value.
        fr => e => e.Left ? (
            fl(e.Left)
        ) : fr(e.Right);

    // MAIN ---
    return main();
})();

Here’s a Keyboard Maestro macro which I use in the context of pasting code into web-site fields. It seems to work fine with Bike, despite defaulting to using space indentation, and it may be worth experimenting with, or adapting:

Paste as plain text.kmmacros.zip (2.1 KB)

jessegrosjean · August 3, 2022, 10:52am

Thanks for report and detective work. I’ll get this fixed soon.

jessegrosjean · August 3, 2022, 7:46pm

Just to verify we are on the same page… when I run this script even before saving the document I see weird issues. For example trying to move caret through pasted text is slow because there is an extra hidden character between each visible character? At least that’s what I see.

complexpoint · August 3, 2022, 9:55pm

Yes, ditto here:

macOS 11.68
Bike 1.4 Preview (67)

And if we select that pasted text in Bike, and copy it, the plain text pasteBoard item contains this kind of thing:

"public.utf8-plain-text as string": 
"A\u0000l\u0000p\u0000h\u0000a\u0000\r\u0000\t\u0000B\u0000e\u0000t\u0000a\u0000\r\u0000\t\u0000G\u0000a\u0000m\u0000m\u0000a\u0000\r\u0000\t\u0000D\u0000e\u0000l\u0000t\u0000a\u0000"

complexpoint · August 3, 2022, 10:06pm

Is it possible that when the public.utf8-plain-text pasteboard content turns out to be defective (unexpectedly \r delimited), the public.utf16-plain-text pasteboard item gets pasted instead ?

This is the kind of clipboard configuration which that AppleScript snippet creates:

{
  "public.utf16-plain-text as string": "A\u0000l\u0000p\u0000h\u0000a\u0000\r\u0000\t\u0000B\u0000e\u0000t\u0000a\u0000\r\u0000\t\u0000G\u0000a\u0000m\u0000m\u0000a\u0000\r\u0000\t\u0000D\u0000e\u0000l\u0000t\u0000a\u0000",
  "public.utf8-plain-text as string": "Alpha\r\tBeta\r\tGamma\r\tDelta",
  "com.apple.traditional-mac-plain-text as string": "Alpha\r\tBeta\r\tGamma\r\tDelta",
  "dyn.ah62d4rv4gk81g7d3ru as string": "\u0001\u0000\u0000\u0000\u0000\u0000\u0010\u0000\u000e\u0000\u0003\u0000\u0000\u0000\f\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
}

fssbob · August 3, 2022, 10:31pm

I see the hidden characters if I copy the text from Bike to BBEdit. But I don’t see them if I copy the text the script produces DIRECTLY to BBEdit.

jessegrosjean · August 4, 2022, 11:31am

I think I’ve pretty much figured out what is going on:

Strings can have null characters. It doesn’t happen often, but it’s valid for it to happen. Generally everything keeps working on this case … except
If a string has null characters then when you pass to a C based API the string will be truncated at the first null character because that’s how C strings are terminated… with a null byte.
I think other apps don’t have a problem with your AppleScript because they are preferring other types from the pasteboard that don’t have the null bytes in this case… (I’ve changed Bike to do this too). Or alternatively maybe other apps are cleaning the string by removing the null bytes before processing… but I can’t imagine that all apps do that.

I think the way that I’ll fix this in Bike is:

Change my preferred pasteboard types so that I’ll get one without null bytes in this case.
Still allow/expect that null bytes might get into Bike and accept that since they are valid in utf8.
Filter out null bytes before saving to XML based file format since null bytes are invalid in XML and since the XML API is C based so those strings would get truncated.

More information on related problem can be found here:

fssbob · August 4, 2022, 6:19pm

Makes sense to me. Thanks for figuring this out.

Is there still an issue with the AppleScript output terminating lines with a \r?

Bob

jessegrosjean · August 4, 2022, 6:42pm

I’m fixing that too, but it was simpler for me to understand

jessegrosjean · August 10, 2022, 5:15pm

I think this is fixed in latest preview: Bike 1.4 Preview (68-69) "Rich Text"

fssbob · August 10, 2022, 6:45pm

Great! Thanks.

I’m still not going back to that AppleScript though.