Searches are not working after upgrading (3.2)

Hi there,

I’m not sure why, but from my searches only the “Calls” one is still working. The other ones are not shown anymore. I’ve just upgraded and to be honest, I do not see if I’ve done something wrong?

My searches look like:

  • Now! @search(not and (@today or @now))
  • Due within 7 days @search(not and (@now or @today or @DUE <=[d] today +7d or @DUE <=[d] today))
  • SingleItems @search(//(not @type=project)/descendant-or-self::)
  • Calls @search(not @done and @call)
  • Next Actions :@search(project *//((not @done) and (not @search))[0])

Any Idea?

Best
Ben

+1

my search

Prio @search(not @done and (@today or @due <=[d] today +14d or @due <=[d] today))

is shown as done in searches due to the “@done” and therefore not shown in the sidebar.

Within the value of the search tags, you’ll need a backslash before every opening or closing parenthesis.

1 Like

As @mattgemmell says you now need to escape both ( and ) characters in your searches by placing a \ character right before them.

So for example this search:

Now! @search(not and (@today or @now))

Needs to become:

Now! @search(not and \(@today or @now\))

Update

I was just looking at escaping ()'s in this example. But now I realize this search has another problem. not and is invalid search syntax. I expect there is supposed to be a @done after it like this:

Now! @search(not @done and \(@today or @now\))

Thanks,
got all to working except this one:

Next Actions :@search(project *//\(\(not @done\) and \(not @search\)\)[0])

any idea if there is one more backslash missing or one to much?

You just need a space between the colon and the @ symbol.

2 Likes

That seems less than ideal–usually in this sort of thing one would escape the parens if they wanted the literal version, not the semantic version. Was this intentional?

Edit: never mind, just stumbled across Proposal, require that both ( and ) are always escaped in tag values?

I feel like this change made sense in the proposals there, but the effect on saved search strings is bad enough to possibly outweigh the good.

One reason is what I say above: in search strings, those parens are semantic, not literal, but \ universally means “literal”. The other is that they’re the “normal” case in any reasonably complex search, and escaping should generally address the 20% case instead of the 80% case.

I’m also not sure how often a non-power user might run into nested parens in a tag–the @done case aside–but I’m certain that one would run into this in a search.

I do get the implementation issues you’re wrestling with, and understand that this is all a side effect of “everything is an attribute on an object,” but the end-user experience is damaged here for searches in particular. I don’t have an easy answer for that, but don’t look forward to “escape soup” and fussy quoting issues. :frowning:

Two layers – in this case the semantic parentheses are those of the the enclosing @search() tag

(without escaping, the point at which the search tag terminates could be undefined)

Yeah, as implemented I’m sure this is technically correct. OTOH, from a user POV I’m not sure they’d think of the @search() tag this way. I certainly don’t–for me, the @search() tag is implementation-specific boilerplate, and the semantic part is the contents.

I think the advantage of the current solution is that it doesn’t involve any special-casing. OTOH, since I think @search was already a special case to begin with in terms of design (it has nothing to do with the tasks themselves–it’s document metadata) the consistency is actually problematic there.

Parentheses don’t, of course, need escaping in search-panel use of item paths – only in this particular form of labelled storage.

If you prefer, there are other ways of attaching names to searches and displaying menus and/or attaching keyboard shortcuts – not at all hard to sketch whatever kind of script you want.

(I don’t personally find the escaping particularly distracting, and I prefer this consistent approach to the previous arrangement in which only one kind of parenthesis (I already forget whether opening or closing) was escaped).

That I’ll agree with–I hadn’t remembered saved searches requiring that (and the above didn’t have it to begin with) but now I remember that decision. Only escaping the right paren is definitely a little weird.

I have to admit that with all due respect to Jesse (I love the product, really, honest) I feel like handling parenthetical depth in an expression parser is kind of a solved problem, though, and it’s a little odd to have to do either. But you’re right, at least this way is fully consistent.

I will counterpoint by saying I do find it distracting in the corrected expressions above, though, for whatever that’s worth. I find similar escaping in bash cli distracting too, so no shocker there.

1 Like

Perhaps the issue is that there are two different expression parsers at work here (tag pattern recogniser, and peg-based itemPath expression parser). To completely rebuild the fast tag parsing (currently by regex) around the special case of saved searches might well come at the cost of overall performance, which is good and fast at the moment.

These things always depend on the use case, but perhaps it might look, in retrospect, like the tail wagging the dog ? :slight_smile:

My thinking on this is that the core problem is that putting searches inside tag values is an imperfect solution. I think the current tag/value syntax rules are correct now. It’s just that saved searches are pushing the limits of what tags are good at. There’s a reason for it:

  1. Wanted saved searches.
  2. Didn’t want to major new syntax element.
  3. Using tags is a hack, but it allows saved searches with minimal disruption.

Eventually I think the solution will be to build in some default saved searches (editable through preferences) that apply to all documents (so the documents themselves won’t have to contain the searches). Then only super geeks who want custom searches per document will need to worry about embedding saved searches in there documents.

1 Like

Still not getting my due dates to work

Due within 7 days @search(not and (@today or @due <=[d] today +7d or @due <=[d] today))

Shows me nothing but I’ve escaped the parens as described

What am I missing

The problem isn’t with escaped ()'s, instead it’s just a syntax problem. not and isn’t valid. Maybe you meant to put a @done after the not?

Weird that 4 of my 7 seraches had dropped their initial “@done” - that was in fact the problem. Unclear as to why it happened, but its definitely user error. Sorry for bother you. Should have read the patterns closer.

Thanks for the help.

Yeah, that’s odd, especially since I just noticed that the initial poster had the same problem. I think “maybe” what happened is that:

  1. Updated to new version.

  2. Tag value syntax was broken in new version so @search wasn’t recognized as a tag.

  3. But as a result @done become a recognized tag… since it was no longer in the search value.

  4. That made the search have a @done strikethrough.

  5. Maybe you saw that and Command-D to remove that done strikethrough… which removed that @done.

  6. And then when you fixed the syntax by escaping the ()'s that done was gone.

1 Like

I agree with your most basic thesis re: saved searches pushing the envelopes of the tag paradigm into something not ideal. FWIW, I do appreciate you having implemented them–“saving” searches via Keyboard Maestro or whatever is a janky solution compared to something native.

Couple of things to consider:

First:

My experience is the more free-form a task system is, the more vital it is that you can do easy searches since you end up rolling your own experience on it. Yours is very, very free-form. Having saved searches at all was the difference between me buying TP3 and not, and I’m not that much of a power user. I just use GTD, like most “prosumer” productivity folks and that implies needing them.

I mean, in general, doesn’t plain-text tracking end up attracting mostly advanced users? I’d assume the rest flock to the pretty to-do app, similar to how Markdown is likely mostly used by professional writers (novels, docs, whatever), etc.

Second:

A more nuts-and-bolts thing. I figured out why the necessary quoting was bugging me: It means I can’t develop the search in the search bar and then copy it into a saved search. Either quoting scheme screws with that. I have to go requote it, and then that ends up introducing errors.

I’m sure you can appreciate that the search bar is essentially the REPL sandbox for complex saved search development, so that’s actually a major downside. I’d love to see either the search bar support the escaped syntax or for a saved search scheme that allows a verbatim copy to be used.

1 Like

This at least I think should be pretty easy to address with a specialized “Copy as Saved Search” command. I would put it in as an option when you right click the search field text. Do you think that would help and make sense?

1 Like

Yeah, that’d work. Since selecting the saved search re-populates the search bar with the unquoted version, that also gives a workflow for modification. Good call!