Sat, Apr 01, 2006 at 03:51:59PM -0500, stan wrote:
I'm in the (hopefully) final stages of setting up my new
Amanda machine.
I want to use a combination of vtapes, and physical tapes
in a RAIT. At the moment, I have 10 vtapes set up but I
may reduce that to 5. My physical tape is a Ultrim 3, which
has a marketing capacity of 400G. I have 1.2T of space to use
for vtapes. I want to use 25 physical tapes in rotation.
Based upon some suggestions from anthony.worrall (AT) reading (DOT) ac.uk,
I have modified his script that sets the changer position
to a given portion, and then copies the disklabel from the phys.
tape to the vtape. Here is what I wound up with for that
#!/bin/sh
$1
DEV=/dev/nst0
AMMT=/opt/amanda/sbin/ammt
AMDD=/opt/amanda/sbin/amdd
AMDUMP=/opt/amanda/sbin/amdump
AMTAPE=/opt/amanda/sbin/amtape
$AMMT -t $VTAPE rewind
$AMMT -t $DEV rewind
$AMDD if=$DEV of=$VTAPE bs=32k count=1 2>&1 /dev/null
$AMTAPE DailyDump slot $1
$AMDUMP DailyDump
I believe that I need to set up 3 configs for this to work correctly.
They are:
1. The primary config (DailyDump) in my case, used for normal backups.
2. A tape only config (tape_only in my case). This will be used for labeling
the physical atpes, and for recovering from physical tapes, after
the vtape half of the mirror is gone.
3. A disk only (disk_only in my case). Used for recovering from vtapes
without having to mount the physical tape. Potentially while a backup
is still running.
Now, it seems to me that these 3 configs will need to share a good bit
of stuff. The amanda.conf's will be different (although I'm using the
include directive to bring in many common pieces). I would assume that
the disklist could be linked to each config directory. Which leaves (at
least) the following or discussion:
infofile (I think this should be common)
logdir (I'm uncertain of this0.
indexdir (this pretty much has to be common.
Never tried, submitted for your consideration and comments (you == list).
All three configs identical (actually links) except for a file
listing the device for that config (RAIT, FILE, or /dev) which
as you suggest is sourced (included) into amanda.conf.
Same tape list suggests also 25 vtapes, which you create and label.
25 vtapes, if filled with data would overwhelm your 1.2 TB available
space (never thought I'd type 1.2 TB was overwhelmed ;) so a separate
non-amanda system is needed to keep them trimmed. Something like,
before each amdump run execute the script to sort the vtapes by
date of last use AND eliminate from the list those that have less
than some trigger amount of data. When running normally that should
give you a fairly consistant number of vtapes (?3, 4, 5, 6?). If
the list is longer than some number (your choice) pick off the oldest
above that number and have the script delete all the data files leaving
the tape label and other necessary files. That should free up the space
for the days amdump run.
What should I do about tapelist?
Same for all (links) but if you try to recover old stuff from vtape
you get errors.
Also, I'm thinking that dumpcycle ,
runspercycle , and tapecycle will need to be the same everywhere.
a 4, 20, and 25 in my case.
I don't get these numbers. 4 day dump cycle? is that 4 week?
20 runs is ?4 days ?4 weeks ?
Sure you don't mean 7 days per dumpcycle (time between level 0's),
5 runs (weekdays) per dumpcycle, and 25 tapes in rotation?
If the above, I'd also suggest listing tapecycle as 20 even if you
actually rotate 25 tapes. If one goes bad you will still be able
to use the next one rather than finding a blank one to label as
a new tape in a crunch.
jl
Finally, what should I do about changerfile? I'm using chg-multi.
Thanks for any advice on this somewhat convoluted setup.

Sun, Apr 02, 2006 at 10:05:11AM -0400, Jon LaBadie wrote:
Sat, Apr 01, 2006 at 03:51:59PM -0500, stan wrote:

Never tried, submitted for your consideration and comments (you == list).

All three configs identical (actually links) except for a file
listing the device for that config (RAIT, FILE, or /dev) which
as you suggest is sourced (included) into amanda.conf.

Same tape list suggests also 25 vtapes, which you create and label.

25 vtapes, if filled with data would overwhelm your 1.2 TB available
space (never thought I'd type 1.2 TB was overwhelmed ;) so a separate
non-amanda system is needed to keep them trimmed. Something like,
before each amdump run execute the script to sort the vtapes by
date of last use AND eliminate from the list those that have less
than some trigger amount of data. When running normally that should
give you a fairly consistant number of vtapes (?3, 4, 5, 6?). If
the list is longer than some number (your choice) pick off the oldest
above that number and have the script delete all the data files leaving
the tape label and other necessary files. That should free up the space
for the days amdump run.

The missing vtapes would still be recorded, so to recover them,
I would have to use teh tape_only config. Makes sense to me.

What should I do about tapelist?

Same for all (links) but if you try to recover old stuff from vtape
you get errors.

K, that makes sense.

Also, I'm thinking that dumpcycle ,
runspercycle , and tapecycle will need to be the same everywhere.
a 4, 20, and 25 in my case.

I don't get these numbers. 4 day dump cycle? is that 4 week?
20 runs is ?4 days ?4 weeks ?

Sorry. here is what I have:

dumpcycle 4 weeks
runspercycle 20
tapecycle 25 tapes

EG, 25 tapes in rotation. 5 runs per week, and tapes enough
for 4 weeks before I need to reuse a physical tape.

Sure you don't mean 7 days per dumpcycle (time between level 0's),
5 runs (weekdays) per dumpcycle, and 25 tapes in rotation?

Maybe that would be better. I guess I'm depending on the promotion
of dumps that are not "due", teh way I have it, right?

If the above, I'd also suggest listing tapecycle as 20 even if you
actually rotate 25 tapes. If one goes bad you will still be able
to use the next one rather than finding a blank one to label as
a new tape in a crunch.

I don;t like that as it defeats some of the "protection" of inserting
the wrong tape, and overwriting.

Finally, what should I do about changerfile? I'm using chg-multi.

So, this won't be a problem for the tape_only, and disk_only configs?

Thanks for the help on this!

Sun, Apr 02, 2006 at 10:05:11AM -0400, Jon LaBadie wrote:
Sat, Apr 01, 2006 at 03:51:59PM -0500, stan wrote:
I'm in the (hopefully) final stages of setting up my new
Amanda machine.

Never tried, submitted for your consideration and comments (you == list).

All three configs identical (actually links) except for a file
listing the device for that config (RAIT, FILE, or /dev) which
as you suggest is sourced (included) into amanda.conf.

Same tape list suggests also 25 vtapes, which you create and label.

more thought.

The only thing I don't rally like about this is the amount of "wrapper"
functionality that I need to set up around the Amanda system. EG the script
taht I presently have to copy disk labels to vtapes, and one to keep
the vtape disk space under control.

Sunday 02 April 2006 10:46, stan wrote:
Sun, Apr 02, 2006 at 10:05:11AM -0400, Jon LaBadie wrote:
>Sat, Apr 01, 2006 at 03:51:59PM -0500, stan wrote:
>I'm in the (hopefully) final stages of setting up my new
>Amanda machine.
>>
>Never tried, submitted for your consideration and comments (you ==
>list).
>>
>All three configs identical (actually links) except for a file
>listing the device for that config (RAIT, FILE, or /dev) which
>as you suggest is sourced (included) into amanda.conf.
>>
>Same tape list suggests also 25 vtapes, which you create and label.
>
more thought.
>
>The only thing I don't rally like about this is the amount of
"wrapper" functionality that I need to set up around the Amanda
system. EG the script taht I presently have to copy disk labels to
vtapes, and one to keep the vtape disk space under control.

To which I'd have to reply, that the wrapper script thats big enough to
do the job isn't too big. I currently have the amdump wrapped in
another script of about 3.5kb, and it calls another script when amdump
is finished thats about 6.5kb. I could probably clean up the trash &
make the comments more concise & cut the whole thing by 30%, but it
works, and I have a strong inclination to leave whats working well
alone. And it is working well for me.

Sun, Apr 02, 2006 at 10:46:12AM -0400, stan wrote:
Sun, Apr 02, 2006 at 10:05:11AM -0400, Jon LaBadie wrote:
Sat, Apr 01, 2006 at 03:51:59PM -0500, stan wrote:
I'm in the (hopefully) final stages of setting up my new
Amanda machine.

Never tried, submitted for your consideration and comments (you == list).

All three configs identical (actually links) except for a file
listing the device for that config (RAIT, FILE, or /dev) which
as you suggest is sourced (included) into amanda.conf.

Same tape list suggests also 25 vtapes, which you create and label.

more thought.

The only thing I don't rally like about this is the amount of "wrapper"
functionality that I need to set up around the Amanda system. EG the script
taht I presently have to copy disk labels to vtapes, and one to keep
the vtape disk space under control.

Don't understand the need for a label wrapper if you have 25 each vtape
and ptape.

Sunday 02 April 2006 12:14, stan wrote:

I put this back on the list Stan, I hope you don't mind.

Sun, Apr 02, 2006 at 10:53:01AM -0500, Gene Heskett wrote:
>Sunday 02 April 2006 10:46, stan wrote:
>Sun, Apr 02, 2006 at 10:05:11AM -0400, Jon LaBadie wrote:
>>Sat, Apr 01, 2006 at 03:51:59PM -0500, stan wrote:
>>I'm in the (hopefully) final stages of setting up my new
>>Amanda machine.
>>>
>>Never tried, submitted for your consideration and comments (you
>>== list).
>>>
>>All three configs identical (actually links) except for a file
>>listing the device for that config (RAIT, FILE, or /dev) which
>>as you suggest is sourced (included) into amanda.conf.
>>>
>>Same tape list suggests also 25 vtapes, which you create and
>>label.
>>
>more thought.
>>
>>The only thing I don't rally like about this is the amount of
>"wrapper" functionality that I need to set up around the Amanda
>system. EG the script taht I presently have to copy disk labels to
>vtapes, and one to keep the vtape disk space under control.
>>
>To which I'd have to reply, that the wrapper script thats big enough
>to do the job isn't too big. I currently have the amdump wrapped in
>another script of about 3.5kb, and it calls another script when
>amdump is finished thats about 6.5kb. I could probably clean up the
>trash & make the comments more concise & cut the whole thing by 30%,
>but it works, and I have a strong inclination to leave whats working
>well alone. And it is working well for me.
>
>Well, I wasn't really complaining, just pointing out that if what
>I am tryign to do is of general interst, perhaps it, or some hooks
>to help support it, could be added in future versions of Amanda.
>
>I suspect that more and more people will want to integrate _some_
>vtape functionality into thier Amanda systems as time goes on, but
>I can't see people wanting to give up the security of physical tapes
>either, gievn the choice.
>
>BTW, what (in broad general concept) do your wrapper scripts do?
>Just curios, maybe they do something that I would find interesting.

The amdump wrapper isn't terribly complex, it runs amcheck once to force
the 'loading' of the correct (v)tape & then fires off the amdump run.

When thats finished and amdump has quit, releasing all the file locks,
it then runs a sub-script that does some housekeeping of its own
temporary files, then generates a tarball of the $config directory, and
a tarball of the indices & data directory such that the indices so
captured are fully complete and uptodate as of this backup, reflecting
the complete history of the backup for the last tapecycle days, and
appends that to the (v)tape.

Then it falls back to the first script and does a run of amverify. It
doesn't see the last 2 files of course but does its checking over the
amdump generated files. I get 3 emails from it per session, one from
the 2 scripts, one from amdump and one from amverify.

If I was still running tapes, I'd disable the amverify run as that would
double the wear rate, and tape drives & tapes have a short enough life
as it is. I was running this back when I was using tapes, and have
added the amverify run since switching to vtapes since in terms of hard
drive life, its virtually free. The HD is a WD 200GB pata drive, now
about 2 years old, spinning 24/7/365, and smartd has found nothing to
complain about yet.

That knocking sound in the background? Me, knocking on wood of
course :)

Sun, Apr 02, 2006 at 11:43:17AM -0500, Gene Heskett wrote:
Sunday 02 April 2006 12:14, stan wrote:
I put this back on the list Stan, I hope you don't mind.
I just discoverd that mutt was _not_ sending it to the list.
My mistake, thanks for correcting it.

Sun, Apr 02, 2006 at 12:18:41PM -0400, Jon LaBadie wrote:
Sun, Apr 02, 2006 at 10:43:30AM -0400, stan wrote:
Sun, Apr 02, 2006 at 10:05:11AM -0400, Jon LaBadie wrote:
Sat, Apr 01, 2006 at 03:51:59PM -0500, stan wrote:

Sorry. here is what I have:

dumpcycle 4 weeks
runspercycle 20
tapecycle 25 tapes

EG, 25 tapes in rotation. 5 runs per week, and tapes enough
for 4 weeks before I need to reuse a physical tape.

That would mean some DLEs, sans promotion, would get only
one level 0 a month. You would have either one or at most
two level 0's in the entire 25 tapes for those DLEs. Suppose
a tape containing that level 0 went bad or the host was down
just as that single level 0 was overwritten on the next cycle.
Then you might have no level 0.

That is true. But in practice promotion has prevented that from
occuring in the past.

Stiil you have given good advice, and I will change my settings
as you suggest, thatnks for the advice.

Sure you don't mean 7 days per dumpcycle (time between level 0's),
5 runs (weekdays) per dumpcycle, and 25 tapes in rotation?

Maybe that would be better. I guess I'm depending on the promotion
of dumps that are not "due", teh way I have it, right?

If the above, I'd also suggest listing tapecycle as 20 even if you
actually rotate 25 tapes. If one goes bad you will still be able
to use the next one rather than finding a blank one to label as
a new tape in a crunch.

I don;t like that as it defeats some of the "protection" of inserting
the wrong tape, and overwriting.

But only for the 21 - 25th least recently used tapes (oldest dumps).
What ever floats your boat is fine.

Well, actually, given that all 25 tapes have been labled, and run
through a minimum of 1 cycles. My experience has been that insert any,
but the correct atpe, from the set of 25 will prevent it from being
overwriten.

Thanks.

BTW, one more question.

Having the changer settings in all 3 configs will not create an issue,
will it?

Sun, Apr 02, 2006 at 12:13:29PM -0400, Jon LaBadie wrote:
Sun, Apr 02, 2006 at 10:46:12AM -0400, stan wrote:
Sun, Apr 02, 2006 at 10:05:11AM -0400, Jon LaBadie wrote:
Sat, Apr 01, 2006 at 03:51:59PM -0500, stan wrote:
I'm in the (hopefully) final stages of setting up my new
Amanda machine.

Never tried, submitted for your consideration and comments (you == list).

All three configs identical (actually links) except for a file
listing the device for that config (RAIT, FILE, or /dev) which
as you suggest is sourced (included) into amanda.conf.

Same tape list suggests also 25 vtapes, which you create and label.

more thought.

The only thing I don't rally like about this is the amount of "wrapper"
functionality that I need to set up around the Amanda system. EG the script
taht I presently have to copy disk labels to vtapes, and one to keep
the vtape disk space under control.

Don't understand the need for a label wrapper if you have 25 each vtape
and ptape.

Mmm, intersting.

Different than where I was headed, but given that I'm going to control
disk space on that volume group "manually" (with a wrapper script),
then I suppose I could do that. less thing to deal with. And I could
declare the vtapes to be the same size as the physical tapes.

Having someone else look at things, often puts a new prespectivr
on things.

I think I will do as you sugest here.

Thanks.