I'm ruminating a bit here, but bear with me.
I'm in the process of thinking about moving a news server from one
machine to another, and want to keep the uptime as high as possible.
course, I could have it stop accepting new messages, flush all the
files and rsync it over. But with 30M article files and rather large
history files, that's going to take a while. I can do the initial
rsync first while it's running as normal, then switch it to
non-spooling mode, and then do the rest, but it's still going to take
quite a bit of time.
So today's random thought is -- would it be possible to run an
instance of inn that would just accept whatever it was fed? That is,
instead of (I've cheated a bit here and looked in the source code)
having ARTassignnumbers() do it's normal thing, could it just accept
the Xref header assigned by the master news server?
If that were possible, then all the articles on the standby server
would have the same article numbers as the master server. I could
feed the standby server these messages at leisure, either with an
ad-hoc thing or via a regular news feed. Then, when the time comes,
I'd just, well, switch the servers. Nobody would know the difference.
This idea seems so simple that I'm sure there are reasons this
wouldn't work. :-)
(The other thing this could be used for is that people could set up
"replicated" news servers, and users could connect to the nearest
server for greater speed, and could switch servers at will without
having to do anything but alter the server name. news.us.gmane.org,
news.au.gmane.org)

Lars Magne Ingebrigtsen <larsi (AT) gnus (DOT) orgwrites:

So today's random thought is -- would it be possible to run an
instance of inn that would just accept whatever it was fed? That is,
instead of (I've cheated a bit here and looked in the source code)
having ARTassignnumbers() do it's normal thing, could it just accept
the Xref header assigned by the master news server?

If that were possible, then all the articles on the standby server
would have the same article numbers as the master server. I could
feed the standby server these messages at leisure, either with an
ad-hoc thing or via a regular news feed. Then, when the time comes,
I'd just, well, switch the servers. Nobody would know the difference.

Yup, that's exactly what Xref slaving is for. Note that the two systems
have to have identical active files, and you'll need to fiddle with
artcutoff to be sure that the server doesn't reject the really old
messages.

(The other thing this could be used for is that people could set up
"replicated" news servers, and users could connect to the nearest
server for greater speed, and could switch servers at will without
having to do anything but alter the server name. news.us.gmane.org,
news.au.gmane.org)

Yeah, people have been doing this with Xref slaving for quite a while.
There are some fiddly bits (you want the feeds to be as near real-time as
possible, since otherwise people who move between servers will see odd
article holes, and you also generally want to feed only one stream of
articles to the slaves, since otherwise articles may arrive out of order
and create holes that confuse clients), but it by and large works okay.

Russ Allbery <rra (AT) stanford (DOT) eduwrites:

Yup, that's exactly what Xref slaving is for.

I really should read the manual more and think less. :-)

Note that the two systems have to have identical active files, and
you'll need to fiddle with artcutoff to be sure that the server
doesn't reject the really old messages.

Right. I'll be setting up a slave server when I get the hardware
next week I guess I need to start by writing a script that makes
a "feed" from the entire existing spool to boot-strap the slave
server? And "newgroup messages", too. Well, that should be pretty
trivial to write, but does such a script already exists, by any
chance?

Lars Magne Ingebrigtsen <larsi (AT) gnus (DOT) orgwrites:

Right. I'll be setting up a slave server when I get the hardware next
week I guess I need to start by writing a script that makes a "feed"
from the entire existing spool to boot-strap the slave server?

Right.

And "newgroup messages", too.

Note that the newgroup has to be received and processed before any
messages in that group are processed or things won't work the way that you
want.

Well, that should be pretty trivial to write, but does such a script
already exists, by any chance?

URL: <http://www.eyrie.org/~eagle/faqs/inn.html#S6.4>

Subject: 6.4. Feed all articles on a server to another server

To feed all articles on an existing server to another one, regardless of
how they're stored on the server, first tell the new server to accept
articles regardless of how old they are (otherwise, INN will reject
articles older than artcutoff in inn.conf):

ctlinnd param c 0

You may also want to set xrefslave to true in inn.conf and then restart
INN on the new server if you want to keep the same article numbers as you
had on the old server.

Next, make sure that the old server is listed in incoming.conf of the new
server, and reload incoming.conf with ctlinnd to pick up that change.
Also make sure that the new server carries exactly the same set of
newsgroups as the old server.

Then try these commands (a variation on commands posted by Katsuhiro
Kondou to inn-workers) on the old server:

cd pathdb
perl -ne 'chomp; ($a,$b,$_) = split " "; print "$_\n" if $_' history \
| tr . / pathoutgoing/list
innxmit server list

where pathdb is the path to the directory containing the history file
(usually ~news/db), pathoutgoing is the path to the outgoing spool
directory (usually ~news/spool/outgoing), and server is the name of the
new news server to which you're feeding the articles.

When done, set xrefslave to false in inn.conf again if you changed it and
then either restart INN on the new server (necessary if you changed
xrefslave) or use another ctlinnd param command to set the cutoff value
back to what's specified in inn.conf.

Please note that when using xrefslave, this method requires that all of
the articles in your spool have Xref headers. Current versions of INN
will always add an Xref header, but very old versions (earlier 1.x
versions) will only add an Xref header to crossposted articles. If you're
trying to import such a spool, you'll need to modify all of those articles
to add an Xref header.

Russ Allbery <rra (AT) stanford (DOT) eduwrites:

Yup, that's exactly what Xref slaving is for.

I really should read the manual more and think less. :-)

Note that the two systems have to have identical active files, and
you'll need to fiddle with artcutoff to be sure that the server
doesn't reject the really old messages.

Right. I'll be setting up a slave server when I get the hardware
next week I guess I need to start by writing a script that makes
a "feed" from the entire existing spool to boot-strap the slave
server? And "newgroup messages", too. Well, that should be pretty
trivial to write, but does such a script already exists, by any
chance?

Russ Allbery <rra (AT) stanford (DOT) eduwrites:

Note that the newgroup has to be received and processed before any
messages in that group are processed or things won't work the way
that you want.

Hm if I rejuggle the feed file created by just taking all the
tokens from the history file to have all the control messages first,
then I should probably get the right result? (Assuming that I
haven't expired the newgroup messages, that is.)

Hm, I see that I don't have the control.{newgroup,rmgroup} groups, but
if I create those two groups, and issue newgroup messages for all the
groups that I have now, and then have the tokens for these two groups
first in the feed, then that might probably be the easiest method.
Sort of.

URL: <http://www.eyrie.org/~eagle/faqs/inn.html#S6.4>

Subject: 6.4. Feed all articles on a server to another server

Cool.

Todd <tco2 (AT) cornell (DOT) eduwrites:

If you are expiring news then there is no need to feed articles
over. Just set up the xrefslave and wait for the unmigrated
articles to expire.

I'm not expiring anything. :-) See http://gmane.org/ for an
explanation.

(And sorry for the double-posting earlier; I messed up.)

Lars Magne Ingebrigtsen <larsi (AT) gnus (DOT) orgwrites:
Russ Allbery <rra (AT) stanford (DOT) eduwrites:

>Note that the newgroup has to be received and processed before any
>messages in that group are processed or things won't work the way that
>you want.

Hm if I rejuggle the feed file created by just taking all the tokens
from the history file to have all the control messages first, then I
should probably get the right result? (Assuming that I haven't expired
the newgroup messages, that is.)

Yeah, or you can just sync up the active file first and then it doesn't
really matter. (It only matters for going forward.)

Hm, I see that I don't have the control.{newgroup,rmgroup} groups, but
if I create those two groups, and issue newgroup messages for all the
groups that I have now, and then have the tokens for these two groups
first in the feed, then that might probably be the easiest method. Sort
of.

I usually just copy over the active file from the current server, run a
script that resets all the numbers to 0000000001 0000000000, and then do
the feed. (Although these days, just modifying the active file doesn't
create all the groups in overview properly; tdx-util -F may be needed as
well. I haven't poked at that stuff in a long time.)

Alternately, you could just generate ctlinnd newgroup commands from the
active file and do it that way; it should work, just be a bit slower.

Russ Allbery <rra (AT) stanford (DOT) eduwrites:

Yeah, or you can just sync up the active file first and then it doesn't
really matter. (It only matters for going forward.)

Yup. I was thinking more about getting a, er, methodology in place
for setting up (possible) mirrors. If I could have the admins of
these other servers first unpack a (static) tar file of configuration
files, they would then get all the groups/articles in place via the
feed. It might simplify things a bit in the long run.

Lars Magne Ingebrigtsen <larsi (AT) gnus (DOT) orgwrites:
Russ Allbery <rra (AT) stanford (DOT) eduwrites:

>Yeah, or you can just sync up the active file first and then it doesn't
>really matter. (It only matters for going forward.)

Yup. I was thinking more about getting a, er, methodology in place
for setting up (possible) mirrors. If I could have the admins of
these other servers first unpack a (static) tar file of configuration
files, they would then get all the groups/articles in place via the
feed. It might simplify things a bit in the long run.

, yeah, good point.

Have we fixed it at this point so that checkgroups can actually make the
group changes and not just send out a mail telling people what to do? I
thought we had, but I haven't checked recently. If so, a single
checkgroups would be easier than all the newgroups.

Russ Allbery <rra (AT) stanford (DOT) eduwrites:

Have we fixed it at this point so that checkgroups can actually make the
group changes and not just send out a mail telling people what to do? I
thought we had, but I haven't checked recently. If so, a single
checkgroups would be easier than all the newgroups.

Hm, yes But I've already scripted something up to generate all
the newgroup messages. :-)