Simon Burge wrote:
The patch below removes the XML conversion support from proplib when
it's built for the kernel, for which I can see no obvious use. With

Hmm, I was just now thinking of having a driver initialize a set of
properties using dictionary_internalize from a constant string of
XML, which might be a possible use - though I could certainly do it
without, it's just more code. I don't know if that's the sort of
use that was contemplated in the design.
-Chap

In article <20060507155425.GA3820 (AT) thoreau (DOT) thistledown.com.au>,
Simon Burge <simonb (AT) wasabisystems (DOT) comwrote:
>The patch below removes the XML conversion support from proplib when
>it's built for the kernel, for which I can see no obvious use. With
>this patch, we get rid of a good chunk of the kernel bloat that new
>proplib adds. For an evbppc/WALNUT kernel the sizes before proplib,
>with normal proplib, and with proplib with no XML import/export
>functionality are:
>
> text data bss dec hex filename
>3837324 452840 232444 4522608 450270 netbsd.devprop
>3856516 452968 234288 4543772 45551c netbsd.proplib
>3843576 452880 233888 4530344 4520a8 netbsd.no-xml-proplib
>
>Anyone have any problems with applying this patch?

I think having the choice to compile it in or not is good (for size challenged
machines), but I can see a use of the XML code in the kernel such as
passing configuration properties between the kernel and userland via ioctl.

christos

Sun, 7 May 2006 16:40:50 +0000 (UTC), christos (AT) astron (DOT) com (Christos
Zoulas) wrote:

I think having the choice to compile it in or not is good (for size challenged
machines), but I can see a use of the XML code in the kernel such as
passing configuration properties between the kernel and userland via ioctl.

Is passing XML to the kernel really the best way to do that? I confess
that my first instinct would be to parse the XML at user level, and use
some sort of TLV representation for the kernel interface.

M. Bellovin, http://www.cs.columbia.edu/~smb

Steven M. Bellovin wrote:
that my first instinct would be to parse the XML at user level, and use
some sort of TLV representation for the kernel interface.

Well, as long as there's support for SME representation that can be
passed to the kernel, in common code that doesn't have to be reinvented
by each driver author who wants to do something similar. IIUC, the 'XML'
parser in proplib is just a lightweight parser for enough "XML" to
represent proplib objects maybe it's good enough?

Perhaps the best way to split it off is to put it in another file
pulled in by an autoconfig attribute like propxml; that way any
drivers that rely on it can just have that attribute and if you
configure one in, you get the support automagically.
-Chap

>that my first instinct would be to parse the XML at user level, and use
>some sort of TLV representation for the kernel interface.

Well, as long as there's support for SME representation that can be
passed to the kernel, in common code that doesn't have to be reinvented
by each driver author who wants to do something similar. IIUC, the 'XML'
parser in proplib is just a lightweight parser for enough "XML" to
represent proplib objects maybe it's good enough?

Am I being particuarly oblivious in that I'm not understanding what
exactly proplib's _intended_ use is? From the various commits and
rototillery, it involves something with device drivers and other
goo - but what exactly that use is eludes me. (The manpage and
includes aren't illuminative, either.)

# Tom Spindler 2006-05-08:
>that my first instinct would be to parse the XML at user level, and use
>some sort of TLV representation for the kernel interface.

Well, as long as there's support for SME representation that can be
passed to the kernel, in common code that doesn't have to be reinvented
by each driver author who wants to do something similar. IIUC, the 'XML'
parser in proplib is just a lightweight parser for enough "XML" to
represent proplib objects maybe it's good enough?

Am I being particuarly oblivious in that I'm not understanding what
exactly proplib's _intended_ use is? From the various commits and
rototillery, it involves something with device drivers and other
goo - but what exactly that use is eludes me. (The manpage and
includes aren't illuminative, either.)

As I understand it, proplib provides framework to canonically
encode/decode primitive (number, bool, string) and aggregate
(array, lookup table/dictionary) data types to/from a string.
This can of course be used in a variety of applications.

For device drivers in particular, the model usage is that you
can setup device properties ({name, value} pairs) in
device_register() and the driver can pick it up in a generic
way.

For instance, in your board-specific device_register()
implementation you can store ethernet MAC address provided to
you by the bootloader as device property for say emac(4), which
can later on read it without having to care about its (MD)
origin.

HTH & please correct me if I'm wrong.
-- Jachym

Sun, May 07, 2006 at 04:40:50PM +0000, Christos Zoulas wrote:
In article <20060507155425.GA3820 (AT) thoreau (DOT) thistledown.com.au>,
Simon Burge <simonb (AT) wasabisystems (DOT) comwrote:
>The patch below removes the XML conversion support from proplib when
>it's built for the kernel, for which I can see no obvious use. With
>this patch, we get rid of a good chunk of the kernel bloat that new
>proplib adds. For an evbppc/WALNUT kernel the sizes before proplib,
>with normal proplib, and with proplib with no XML import/export
>functionality are:
>
> text data bss dec hex filename
>3837324 452840 232444 4522608 450270 netbsd.devprop
>3856516 452968 234288 4543772 45551c netbsd.proplib
>3843576 452880 233888 4530344 4520a8 netbsd.no-xml-proplib
>
>Anyone have any problems with applying this patch?

I think having the choice to compile it in or not is good (for size challenged
machines), but I can see a use of the XML code in the kernel such as
passing configuration properties between the kernel and userland via ioctl.

The problem with droping the "bloat" is that we then have no way to pass
property lists in and out of the kernel. So whatever features come to
relie on this won't work for the shrunken kernels.

Also, someone may opt for other parameter passing methods, just so they
would continue to work with a "shrunken" kernel. We then would end up
with TW systems doing the same thing, which I think would be worse.

Take care,

Bill

PGP SIGNATURE
Version: GnuPG v1.2.3 (NetBSD)

o4dUce+8vrNu4nIHfospos8=
=4HFx
PGP SIGNATURE

Mon, May 08, 2006 at 07:40:04PM -0700, Garrett D'Amore wrote:

I'd be interested in seeing a better break down of the "bloat". We're
only talking ~20k here. Yes I know every bit is precious, but how much
can we prune this and *retain* _any_ common format for data passing?

I have a thought. Maybe we should start counting calories. So, those
committing changes required in all kernels that increase the size of
those kernels would need to find an equal amount of space elsewhere to
whack, or get Extra Special Permission from core.

Jason, for example, is an extremely smart guy, and I am pretty sure he
could find 20K elsewhere in the kernel to kill off in a relatively short
span of time. :-)

Thor

May 7, 2006, at 8:54 AM, Simon Burge wrote:

The patch below removes the XML conversion support from proplib when
it's built for the kernel, for which I can see no obvious use.

of the key points is to have this intermediate format that can be
used to pass between the kernel and user space. In order to do that,
the kernel needs to have the ability to generate and parse the XML.
-- thorpej

Jason Thorpe wrote:
of the key points is to have this intermediate format that can be
used to pass between the kernel and user space. In order to do that,
the kernel needs to have the ability to generate and parse the XML.

Did there seem to be any possible value in my compromise suggestion
yesterday of putting the "XML" code in a separate file that's associated
with an autoconfig attribute like propxml?

Then any drivers or features that actually rely on the intermediate
format would be marked in 'files' with the propxml attribute, and
building a kernel to include any of them would automagically suck
in the extra code with no need to make a manual decision about an
option, and at the same time you'd have clear documentation in 'files'
about who relies on the functionality and who doesn't.

I suppose you'd want it unconditionally if you were including LKM
support, just in case you later load a module that wants it.
-Chap

I wrote:
Did there seem to be any possible value in my compromise suggestion
yesterday of putting the "XML" code in a separate file that's associated
with an autoconfig attribute like propxml?

Hmm, never mind - looking again at the proposal, I suppose if proplib
is to be used in staple functionality like mount, there really wouldn't
be much point in making the representation part optional.

Plus, as it gets worked into existing operations like mount, it is
likely to replace some of their code that exists now, and so the
net memory impact may decrease as it comes to be more widely used.

Hmm, could proplib contribute to the business of retaining persistent
attributes for a devfs?
-Chap

Tue, 9 May 2006, Chapman Flack wrote:

Hmm, never mind - looking again at the proposal

is this public?

Plus, as it gets worked into existing operations like mount, it is
likely to replace some of their code that exists now, and so the
net memory impact may decrease as it comes to be more widely used.

I would be interested in seeing an implementation example for this, is
anybody (Jason?) working on anything that would serve as an introduction?

and unrelatedly, why proplib(3) and not prop(3) ?

iain

Edward B. DREGER wrote:
GDPerhaps instead of XML we could use some binary "fixed format" kind
GDof representation.
maybe something YAML-esque?

In defense of Jason's bikeshed -
- it's using, IIUC, a strict subset of XML just adequate to export its
own stuff, so the only reason for it to set off anybody's XML-alarm
would be the spelling of its tokens, not the inherent complexity of
its parser.
- that presumably means that you couldn't expect proplib to parse
arbitrary output from XML tools, but you could expect existing
general purpose XML tools to do useful things with proplib output
if you wanted to.
- the choice was based, IIUC, on prior art from Apple. Even if that
has no direct technical benefit (this kind of device-dependent stuff
might never be exchanged between Ss), it still offers a kind of
conceptual harmony to people who might administer both NetBSD and SX
machines.
- it's already painted and will hold bicycles. core@'s already pictured
having those bikes finally out of the kitchen.

As someone already asked earlier in this thread, what's the maximum
difference in memory footprint to be expected from replacing the current
proplib parser with any other parser?
-Chap

May 11, 2006, at 7:02 AM, Chapman Flack wrote:

In defense of Jason's bikeshed -
- it's using, IIUC, a strict subset of XML just adequate to export its
own stuff, so the only reason for it to set off anybody's XML-alarm
would be the spelling of its tokens, not the inherent complexity of
its parser.

That is correct. It is not a general XML parser. It parses only
"property lists".
- that presumably means that you couldn't expect proplib to parse
arbitrary output from XML tools, but you could expect existing
general purpose XML tools to do useful things with proplib output
if you wanted to.

Again, correct.
- the choice was based, IIUC, on prior art from Apple. Even if that
has no direct technical benefit (this kind of device-dependent stuff
might never be exchanged between Ss), it still offers a kind of
conceptual harmony to people who might administer both NetBSD and SX
machines.

Not just conceptual harmony for those who admin both (S X admins
don't fiddle with plists manually very often, if at all). Basically,
I saw Apple XML plists as prior art for a file format that was
extremely flexible and human editable. If I wanted to do it in some
other markup language, I would have had to define another DTD-
analogue, and honestly I didn't see the point in divergence for
divergence's sake. Besides, would a general YAML parser be smaller
than my specialized plist parser?

Sure, Apple has other file formats for plists (like a very compact
and fast-to-parse binary plist format), but they're not human
readable / editable with standard text tools. I could certainly
entertain the idea of being able to externalize to / internalize from
that format but it is also desirable to keep the text form at the
same time, for other reasons.
-- thorpej

Garrett D'Amore wrote:
Is there some reason that the exchange across kernel boundaries should
be done in a human editable format? The only thing I can think of is to
allow the kernel to directly read configuration data from a filesystem
that was previously hand-edited.

That, and that it's easy to do, Jason only had to implement one
serialized form, and it requires nothing of userland code program
can be dumb as a post and still get readable config data from somewhere
and pass it to a kernel service.

A binary format could make the lexer smaller in the kernel, but is
really sort of moving the work around; now there needs to be userland
code to create the binary format, somebody needs to implement it in a
library, and everybody needs to link to it.

As I see it, now that proplib is there, some further ideas about it
can be handled incrementally in the usual way. For instance, I noticed
(and suggested to Jason) some ways the header size of the objects
could be economized, which could be a big win if you have lots of
small objects but it's nothing that needs to be done immediately,
it can be reworked at any time without visible API change, and I could
even submit a patch for it myself one day if nobody beats me to it.

Changing the serialized form used by the kernel would be more visible,
but if, say, user code by convention used the user proplib for all its
manipulations, and to serialize everything in 'kernel form' before
passing it to the kernel, that would limit the visibility of later
changing the 'kernel form'.

And whether that kind of future change is worth making could be decided
after some actual benchmarking of the code, to find just what realistic
tasks it's too slow for, and by how much, and how much better the binary
form would be. The bigger the need, the more likely someone takes the
time to code it.
-Chap

May 11, 2006, at 3:41 PM, Garrett D'Amore wrote:

A binary format was what I had in mind when I made my suggestion
that we
*might* want to consider an alternative to the current
:-) If
apple has a nice one ready to use, then I can see no reason to invent
another one.

I will investigate this some more.

Is there some reason that the exchange across kernel boundaries should
be done in a human editable format? The only thing I can think of
is to
allow the kernel to directly read configuration data from a filesystem
that was previously hand-edited.

of the reasons for doing this whole exercise is to make LKMs
easier to support. Right now, you have to compile cfdata tables with
the C compiler. Dumb. I'd rather just load a plist into the kernel
and let it digest that. Now, it's convenient for drivers to have XML
plists, but if the kernel lacks a way to gobble them up, then they
need to be converted to binary plists. Fine, the LKM loader could do
that, but that's an extra step in userland that would not have
otherwise been required.

You could just have both parsers in the kernel, but that kind of
defeats the purpose, eh?

But, I perceive that the drawback to such might be larger kernels,
larger data sizes (assuming a binary representation would be more
compact) and longer run times encoding/decoding the data, in what
*might* be some time sensitive loops. (As an example, I imagine using
"packed" plists as a way to exchange generalized event notifications
across a kernel/user boundary. If you have a number of these
happening
a second, you'll want it to be *fast*. In fact, Solaris does
precisely
this with their sysevent architecture using packed "nvlists", which
are
similar in some ways to plists.)

If I had a vote, I'd vote for a compact binary representation in the
kernel (and across the kernel boundary), and tools in userland to help
manipulate/convert them. (I'm not sure I have a vote, but I can
express
my opinion, can't I? :-)

The tool to convert them would simply be proplib itself -- a flag to
decide that output format to use, and auto-detection on ingest.

Simple enough, if it really buys much. For prop_data_t, yah you
avoid the base64 step, and for prop_string_t, you avoid the encode step.

I will investigate the Apple binary plist format a little more and
follow up.
-- thorpej

Hello,

Thu, May 11, 2006 at 03:01:24PM -0700, Jason Thorpe wrote:
Sure, Apple has other file formats for plists (like a very compact and
fast-to-parse binary plist format), but they're not human readable / editable
with standard text tools. I could certainly entertain the idea of being able
to externalize to / internalize from that format but it is also desirable to
keep the text form at the same time, for other reasons.

What about the old NeXTSTEP (is it NeXTSTEP?) text format? If human
readability and editability is a concern, I think it is much more
human-readable and editable. (Imagine using property lists for the kernel
command line - entering a property list in the boot loader would be much
easier in the NeXTSTEP format than in the XML format.)

For reference: XML property lists:

NeXTSTEP text property lists:

For those using the Windowmaker window manager - the text format is used
as the format for its configuration files in GNUstep/Defaults.

Pavel

May 11, 2006, at 5:16 PM, Jed Davis wrote:
, as long as we're getting out the bikeshed paint buckets:
json-04.txt
JSN can't represent data blobs (i.e. has no <data></dataanalogue).
-- thorpej

May 11, 2006, at 10:48 PM, Pavel Cahyna wrote:

What about the old NeXTSTEP (is it NeXTSTEP?) text format? If human
readability and editability is a concern, I think it is much more
human-readable and editable. (Imagine using property lists for the
kernel
command line - entering a property list in the boot loader would be
much
easier in the NeXTSTEP format than in the XML format.)

The old-style plist format used by Step cannot represent
numbers. From the reference you cite:

ASCII property lists support the four primary property list data
types: NSString (java.lang.String in Java), NSData, NSArray, and
NSDictionary. Additionally, NSDate and NSNumber objects can be
archived, but will become strings when unarchived.
-- thorpej

Fri, May 12, 2006 at 10:50:15AM -0700, Jason Thorpe wrote:

May 11, 2006, at 10:48 PM, Pavel Cahyna wrote:

>What about the old NeXTSTEP (is it NeXTSTEP?) text format? If human
>readability and editability is a concern, I think it is much more
>human-readable and editable. (Imagine using property lists for the
>kernel
>command line - entering a property list in the boot loader would be
>much
>easier in the NeXTSTEP format than in the XML format.)

The old-style plist format used by Step cannot represent
numbers. From the reference you cite:

ASCII property lists support the four primary property list data
types: NSString (java.lang.String in Java), NSData, NSArray, and
NSDictionary. Additionally, NSDate and NSNumber objects can be
archived, but will become strings when unarchived.

Supporting numbers would need just a small modification in the format.
Let's interpret unquoted string 123 as a number, and a quoted one "123" as
a string.

Pavel

May 20, 2006, at 4:19 AM, Daniel Carosone wrote:
Like, say, XDR?
IIRC, XDR is not self-describing; you have to know what is coming.
-- thorpej