Re: V4 database definition files

[Benjamin Franksen <benjamin.franksen@bessy.de>]

On Friday 28 October 2005 00:54, Andrew Johnson wrote:
> Rees, NP (Nick) wrote:
> > 2. My trivial example would probably not be like Kay's, I would put
> > all the information in attributes, not as free-form data. For
> > example:
>
> Your use of an attribute for the field value doesn't permit a field
> to be a structure containing sub-fields, which themselves may be
> structures.  This is a major requirement in V4, and the only way to
> do that in XML would be for the field value to be text instead of an
> attribute. This adds a little more line noise (punctuation) to your
> example:
>
> <record type="ai" name="my:record">
>      <field name="INP">#C1 S2@</field>  <!-- would change for V4-->
>      <field name="DTYP">Xycom 566</field>
>      <!-- And, for instance: -->
>      <dct_data>
>        <position x="1234" y="4321"/>
>        <color foreground="brown" background="yellow">
>      </dct_data>
>      <archive name="fast" group="mygroup" rate="monitor"/>
> </record>

Hm. Convincing argument.

> > However, the bonus for me is
> > that I can define extra tags that I know will be ignored by systems
> > that don't understand them as long as they validate. My problem is
> > really that the info item just is too simple to be of much use.
>
> What are the requirements that we're not meeting right now?  I'm
> happy to discuss and change the info definition to make it more
> useful, if you let me know what you want to be able to do with it.

Nick listed a about 10 different points. I added numerical RDB keys.

> > 3. As to Kay's "well, why not XML" point, I agree that this is
> > often a reason, but at heart this is because we are now accepting
> > that we don't have to write parsers anymore and/or use lex and
> > yacc. Overall, I believe this is a good thing. In my experience
> > this turns into a bigger advantage when these files essentially
> > become interface definitions between isolated components, and we
> > have a schema which automatically validates the interface. This
> > eliminates a lot of discussion late on in a large, distributed
> > project. So, I don't expect users to come into my office to thank
> > me for using XML. I don't expect my users to even know I'm using
> > XML.
>
> At APS we probably have at least 6 different ways in which DB files
> get generated, using VDCT, GDCT, JDCT, Perl scripts, shell scripts
> and hand editing.  The latter is still very common, and not just
> among the main core developers.  

We are using Capfast, hand editing (for simple db templates, containing 
at most a hand full of records) and nowadays sometimes VDCT. Plus 
substitution files generated mostly from RDB via Perl scripts.

I think one must consider creation and editing separately. It may be 
faster and easier to make some small change in a text editor, even for 
a complex db file. It is not a good idea to use a text editor to create 
a large and complex database (or template) containing many links 
between records. We still use mostly Capfast for this, but merely due 
to (perceived?) problems with VDCT's graphical representation.

The main point for using a text editor is making incremental changes, 
including bulk changes of many properties at once. Once VDCT can handle 
this gracefully there will be much less need for a text editor. Thus a 
more verbose representation such as XML is obviously bound to be is not 
as critical.

> I expect developers to want to carry 
> on using their favorite tools to generate application databases, and
> other people's experience (not just Kay's) has shown that XML is not
> a good choice for files that humans have to create by hand or read.

You probably don't want to port DCT or GDCT to V4 file format, or do 
you?

> On a related note, there are currently no special tools for creating
> DBD files, and I very much doubt that the frequency with which new
> record types get created would justify writing a special tool for
> this purpose. I do not want to make developers write DBD files by
> hand in an XML format, and the parser for our current DBD file format
> can be written in a reasonably small number of lines of Perl

This is not a good argument, IMO. In know these Perl parsers. Have 
written a few of them myself. They usually rely a lot on regular 
expressions, only approximating the (often unspefified) grammar in 
question and therefore almost always unreliable in corner cases, taht 
is, as soon as you write the file in a way taht respects teh grammar 
(if there is one) but that has not been anticipated by the writer of 
the "parser".

> I did look at the possibilities of using XML some time back, and I
> don't see it as impossible for someone to create an additional DB
> parser for iocCore to allow XML databases to be loaded.  However
> there is at least one reason that I think justifies the inclusion of
> another parser; it involves conversions between strings and
> structures for Channel Access clients: In V4, all CA traffic will be
> passed over the network in the native type of the server channel,
> thus conversions from the data type the client presents and the
> server's type must take place in the CA client library.  We also want
> to allow clients to be able to read or write whole structures, and
> thus we have to have a parser in the CA client library that can
> understand a hierarchical data structure value when expressed as a
> string, and convert it into a structure for onward transmission to
> the server.  My current design for the V4 DB file format is intended
> to make use of the same parser, so you can use exactly the same
> syntax for the value of a field in the DB file that you do from a
> command-line CA utility.  I really don't want the string format we
> use there to be XML:
>
> % caput anj:mystruct "<struct><field name="a">10</field><field
> name="b">20</field></struct>"

Now, this is indeed quite a strong argument.

Anyway, even if XML is rejected as the standard format for V4, couldn't 
we at least learn from it and make it extensible? I.e. specify our own 
format in such a way that tools by default ignore any kind of 
tag/structure they don't know about? This would mean specifying the 
language in two levels: an inner (core) language -- similar to XML but 
with a different syntax -- and an outer language that specifies certain 
kinds of "tags"? It would address the most important point of Nick's 
arguments. It's just an idea, at the moment, but I think the currently 
proposed format is quite regular so there are ample opportunities for 
abstraction (into a "core" layer, which could then be extended).

Ben