Experimental Physics and
Industrial Control System

Andrew Johnson via Core-talk <core-talk at aps.anl.gov> · Tue, 17 Aug 2021 17:48:58 -0000

I take that back, this isn't a bug so much as a request that we be more
liberal than the JSON-5 spec permits. I might be open to that, but first
we should make the error easier to understand. Here's what's happening:

The values of info() and field() entries in a database file are supposed
to be valid JSON-5 values, and are currently parsed as such.

Looking at the JSON spec https://datatracker.ietf.org/doc/html/rfc7159
and the diagrams on https://www.json.org/json-en.html that spec doesn't
actually allow *any* unescaped control characters inside string values.
The spec says:

   A string begins and ends with
   quotation marks.  All Unicode characters may be placed within the
   quotation marks, except for the characters that must be escaped:
   quotation mark, reverse solidus, and the control characters (U+0000
   through U+001F).

Thus all control characters are supposed to be escaped according to the
older JSON rules.

The JSON-5 spec at https://spec.json5.org/#strings has similar language,
although its BNF also confusingly allows an unescaped SourceCharacter
from the ECMAScript language spec at https://262.ecma-
international.org/5.1/#sec-6 specification, which is defined as "any
Unicode code unit". I think that can be ignored though.

Both of our JSON parser lexers (dbLex.l and yajl_lex.c) follow those
strict specifications when it comes to the set of characters allowed
inside strings. I could change them to allow unescaped tab characters
inside strings (please comment if you have an opinion about that either
way), but I will have to fix both parsers to do that.

I agree that the error messages you got aren't particularly helpful. The
character being complained about is the initial double-quote at the
start of the string – the lexer couldn't match the whole string because
of the illegal character between the quotes, so it back-tracked to the
very start of the it and complained about the quote itself. This also
explains the "funny" result with the BEL character, which isn't
currently legal anywhere in a .db file.

To give a more friendly error message here I can add error-matching
patterns that recognize anything that looks like a string to a human but
doesn't to the strict lexer, then tell the user what's wrong with their
string. The first part should be relatively straightforward to code,
although I'm not sure I want to write code that could analyze any kind
of broken string and explain why it isn't a legal JSON string.

-- 
You received this bug notification because you are a member of EPICS
Core Developers, which is subscribed to EPICS Base.
Matching subscriptions: epics-core-list-subscription
https://bugs.launchpad.net/bugs/1940283

Title:
  Misleading Error: Invalid character '"'

Status in EPICS Base:
  New

Bug description:
  When using a tab character in a field, EPICS 7 prints a misleading
  error message, not hinting at the tab but at the closing(?) quote.

  Example 1:  field(DESC,  "a	b")
  Example 2:  info(TAB,  "a	b")

  Error: Invalid character '"'
   at or before """ in path "."  file "..." line ...

  BTW: Is there a reason why I cannot use a tab in my (info) fields?

To manage notifications about this bug go to:
https://bugs.launchpad.net/epics-base/+bug/1940283/+subscriptions

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System