[NSLUG] USB tick for backup okay?

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

[NSLUG] USB tick for backup okay?

Mike Spencer

I've been using a USB stick for regular backups of ~/.

Is this a good idea? Are there potential durability problems, re-write
limits  or something else I haven't thought of that make this a bad
or deprecated practice?


- Mike

--
Michael Spencer                  Nova Scotia, Canada       .~.
                                                           /V\
[hidden email]                                     /( )\
http://home.tallships.ca/mspencer/                        ^^-^^
_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] USB tick for backup okay?

billdavidson
How often do you fo a test restore?  How frequent are the backups?

I think if fsck of the backup media is good, and you can restore a random file, and if the backup sets are reasonably fresh, you have no worries.  I would not rely on USB stivks for long term archival storage, but for backups that are regularly refreshed and verified they should be fine.

_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] USB tick for backup okay?

Dave Flogeras
In reply to this post by Mike Spencer
On Tue, Mar 27, 2018 at 2:16 AM, Mike Spencer <[hidden email]> wrote:

I've been using a USB stick for regular backups of ~/.

Is this a good idea? Are there potential durability problems, re-write
limits  or something else I haven't thought of that make this a bad
or deprecated practice?

Depending on the value of your data (think irreplaceable photos), you could even flip-flop between two sticks and keep one at a friends house/in your car/on your person.  That way if the worst happened (theft, fire) you might still have a copy.  It also has the benefit that if one failed for any reason, you still have the other.  Sort of a handraulic, sneakernet RAID.

Another thing to ponder, if you are using the stick with FAT filesys, you would definitely want to only backup using a tar format (versus cp or rsync).  FAT has lower resolution time-stamps, and cannot backup all the Unix permissions/ownership properly.

my 0.02

_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

[NSLUG] Re: USB stick for backup okay?

Mike Spencer
In reply to this post by billdavidson

I asked about suitability of USB sticks for backups.

[hidden email] replied:

> How often do you [d]o a test restore?

Not often.  Once in a while I browse some recent files and see that
they are as expected.  I guess I should random sample or cmp more
often.

> How frequent are the backups?

Every 3 or 4 days up to daily.

> I think if fsck of the backup media is good,...

Ha, yes.  The importance of backups was brought home to me when there
was a minor irregularity with a HD partition used for storing audio
and video, other big files.  So I ran fsck.  And all the contents of
my home directory vanished.  That kinda undermined my faith in fsck.
I have O'Reilly books that explain file systems in detail but haven't
spent the hours to beat it all up so I'm pretty unclear on fsck
details.

I've forgotten the details of the original problem and fsck command or
responses used.  Happily I had a backup only a week old and the other
partitions weren't affected.  Retired the HD.

> ...and you can restore a random file...

Yes.

> ...and if the backup sets are reasonably fresh, you have no worries.
> I would not rely on USB sticks for long term archival storage, but
> for backups that are regularly refreshed and verified they should be
> fine.

Ah, there's a point.  I keep lots of text files in ~/man, ~/txt/* and
~/Mail/Archive, images in ~/gr/* that never change.  So they're
effectively long-term archives.  How could "regularly refreshed" be
implemented for them?  I assume nothing like that happens by itself.

Then Dave Flogeras <[hidden email]> wrote:

> Depending on the value of your data (think irreplaceable photos), you could
> even flip-flop between two sticks and keep one [off-site].

Yeah, I've been thinking about doing that.

> It also has the benefit that if one failed for any reason, you still
> have the other.

Yes.  Good point.

> Another thing to ponder, if you are using the stick with FAT
> filesys, you would definitely want to only backup using a tar format
> (versus cp or rsync).

Long ago, I cluelessly tried to do a backup on a FAT stick.  Someone
here offered a friendly clue reminding me that it was normal for Foo,
foo and FOO to be the same file under FAT.  I created an ext2 fs on
the stick.  (Ext2 because I'm running an old kernel that doesn't grok
ext4.  When main box finally gets an upgrade, so will the backup
stick.)  Now, initial backup with cp, then a script to do rsync.

So, maybe I should periodically alter the backup script to add the
--checksum switch to rsync?  AIUI, that would force rsync to verify
matching MD4 checksums for a file on my HD and the backup copy
I made (say) last year.  That would force full read of all 2.9G in
/mnt/usb/backup, catch and update any defective files, but reading
wouldn't "refresh" the files found to be okay at the moment, would it?

Thanks, all.  Further comment/pointers welcome.

- Mike

--
    Just the other day I was thinking I should take a problem to one of
    them old guys who know everything.  Then I realized:  they're all dead
    and *I* am one of the old guys who know everything.  But dang!  I
    don't know everything yet!
_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] Re: USB stick for backup okay?

billdavidson
I can't get rid of the image of a USB tick brrowing its head into the side of my computer, sucking out the data, possibly leaving behind a virus!


On Mar 28, 2018 2:21 AM, [hidden email] wrote:


Not often.  Once in a while I browse some recent files and see that
they are as expected.  I guess I should random sample or cmp more
often.

Don't browse, move a file and restore it from backup.  (I used to be the backup guy at the place I worked, a law firm, where documents really, really matter.  I regularly moved some random file away from its expected location and restored from (tape) backup.  This accomplished two things: it proved the backup was complete and consistent, and that I knew how to restore so I wouldn't be flustered in a data outage.)

Ah, there's a point.  I keep lots of text files in ~/man, ~/txt/* and
~/Mail/Archive, images in ~/gr/* that never change.  So they're
effectively long-term archives.


For long term archives I would use optical media.

  How could "regularly refreshed" be
implemented for them?  I assume nothing like that happens by itself.


By "refreshed" I just meant new backups.  Typically one takes periodic full backups (~/*) and more frequent incremental or differential (things that are new or have changed (since last full or since last partial)).  Since we typically don't have an infinite supply of media this means some sort of rotation schedule.



_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] Re: USB stick for backup okay?

Dop Ganger
In reply to this post by Mike Spencer
On Wed, 28 Mar 2018, Mike Spencer wrote:

> Ah, there's a point.  I keep lots of text files in ~/man, ~/txt/* and
> ~/Mail/Archive, images in ~/gr/* that never change.  So they're
> effectively long-term archives.  How could "regularly refreshed" be
> implemented for them?  I assume nothing like that happens by itself.

Given a usb stick on /dev/sdc1, verify the entire stick
intermittently with:

dd if=/dev/sdc of=/dev/null bs=1M

It won't protect you against bit rot, but it'll protect you to some extent
against the stick itself going bad.

If you're using multiple USB sticks something like Parchive might be worth
looking at to reduce risk of loss.

Cheers... Dop.
_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] Re: USB stick for backup okay?

Michael Crawford-2
I asked my Mom to send a specific stick to my brother-in-law should I
die or become incapacitated.

One day I left town without informing her of my plans.  She sent ALL
of my sticks to him.

Upon receiving them, Stanley copied them all to his hard drive.

I later asked him to return all those sticks.  He didn't know where
they were but said "That's OK, I have them all imaged on my box".

Every last one of those images were corrupt.  He and my sister were
never able to find the actual sticks.  Eventually my time of morning
expired.  I resolved to get on with life without all that precious
data.

I have since gotten far more diligent about backups.  Each of my boxes
has a directory for local backups, my server has a directory for
off-site backups and I have a stick for on-site but air-gapped
backups.

Philosophically,

M. David Crawford
_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] Re: USB stick for backup okay?

Daniel MacKay-2
All media — spinning magnetic, ssd, tape, floppy optical — fail.  Some fail more predictably than others - the two worst I’ve worked with were a horrible thing called a ZIP drive, and 8mm tapes in Exabyte brand drives. Both of those had high failure rates and VERY poor compatibility: that is, you’d have an Exabyte drive fail, take your spare off the shelf and discover that it couldn’t read any of your backup tapes.

In terms of philosophy, this is an oldie - twenty years old? or more? - but a goodie:

        http://www.taobackup.com/

It doesn’t matter what medium you use, if you follow the Tao of Backup.

Practically, this is what I do:

My Macs get Time Machine’d to a local hard disk. Time Machine does (something like) a week’s worth of once an hour, then a month’s worth of once-a-day, then one a week forever. I have two Time Machine disks; Norval has the other one backing up his Macs at his office; every month or so we swap drives so if a drive gets destroyed by a flood or fire, there’s an older offsite one.

My production Unix machines all back up, via rsync, to an antique Solaris box in my basement running rsync (again, for physical separation.)  Every night the backup machine does a ZFS snapshot of the backup volume. A script deletes those snapshots with a “Towers of Hanoi” algorithm: there’s a backup a day old, two days old, four days, eight days, 16 days etc.
_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] Re: USB stick for backup okay?

Paul Russell
I still have the punch cards of a COBOL program that I wrote in high school in '82. They make great bookmarks and, in a pinch, I can reconstruct the program. Now all I need to do is find something to read them....

On Wed, Mar 28, 2018 at 4:00 PM, Daniel MacKay <[hidden email]> wrote:
All media — spinning magnetic, ssd, tape, floppy optical — fail.  Some fail more predictably than others - the two worst I’ve worked with were a horrible thing called a ZIP drive, and 8mm tapes in Exabyte brand drives. Both of those had high failure rates and VERY poor compatibility: that is, you’d have an Exabyte drive fail, take your spare off the shelf and discover that it couldn’t read any of your backup tapes.

In terms of philosophy, this is an oldie - twenty years old? or more? - but a goodie:

        http://www.taobackup.com/

It doesn’t matter what medium you use, if you follow the Tao of Backup.

Practically, this is what I do:

My Macs get Time Machine’d to a local hard disk. Time Machine does (something like) a week’s worth of once an hour, then a month’s worth of once-a-day, then one a week forever. I have two Time Machine disks; Norval has the other one backing up his Macs at his office; every month or so we swap drives so if a drive gets destroyed by a flood or fire, there’s an older offsite one.

My production Unix machines all back up, via rsync, to an antique Solaris box in my basement running rsync (again, for physical separation.)  Every night the backup machine does a ZFS snapshot of the backup volume. A script deletes those snapshots with a “Towers of Hanoi” algorithm: there’s a backup a day old, two days old, four days, eight days, 16 days etc.
_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug



--
Paul Russell
Sackville Volunteer Of The Year, 2017
Parade Marshall, 2018 Sackville Celebrate Canada Parade
President, Sackville Cobequid PC Association
(902) 830-5043

_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] Re: USB stick for backup okay?

D G Teed-2
In reply to this post by Mike Spencer



Long ago, I cluelessly tried to do a backup on a FAT stick.  Someone
here offered a friendly clue reminding me that it was normal for Foo,
foo and FOO to be the same file under FAT.  I created an ext2 fs on
the stick.  (Ext2 because I'm running an old kernel that doesn't grok
ext4.  When main box finally gets an upgrade, so will the backup
stick.)  Now, initial backup with cp, then a script to do rsync.

So, maybe I should periodically alter the backup script to add the
--checksum switch to rsync?  AIUI, that would force rsync to verify
matching MD4 checksums for a file on my HD and the backup copy
I made (say) last year.  That would force full read of all 2.9G in
/mnt/usb/backup, catch and update any defective files, but reading
wouldn't "refresh" the files found to be okay at the moment, would it?

Thanks, all.  Further comment/pointers welcome.



You could make a manifest of files that have changed since a period of time
using find with mtime.  Then feed that list of files to tar with -T pointing to to a file having a
manifest of filenames with paths.  Store the tar file on USB or whatever.  This
strategy avoids the USB FAT file name limitations.

In the past I've used faubackup but for awhile it wasn't maintained much.  Not sure now the
status on that tool.  There are probably similar scripted tools developed.

Another project that would be fun to try is making a software RAID out of
a set of USB sticks.  That would be JBOS.

For my backups of stuff that will never change like family photos, I used
Blu-ray and MDISC.  I have it on a network storage device as well, but in case that
became dead/ransomwared, etc., there is no way the problem could spread to
a Bluray disc.  MDISC are supposed to last far longer than conventional
optical media which can deteriorate at roughly 20 years or more.  When Bluray
goes away as a common media format, I'll need to see what's next.

Single drive NAS solutions are not too pricey and are OK for a second copy
of something you already have.  They can be mounted with CIFS from Linux.



_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] Re: USB stick for backup okay?

Dave Flogeras
In reply to this post by Mike Spencer


On Wed, Mar 28, 2018 at 2:21 AM, Mike Spencer <[hidden email]> wrote:

Long ago, I cluelessly tried to do a backup on a FAT stick.  Someone
here offered a friendly clue reminding me that it was normal for Foo,
foo and FOO to be the same file under FAT.  I created an ext2 fs on
the stick.  (Ext2 because I'm running an old kernel that doesn't grok
ext4.  When main box finally gets an upgrade, so will the backup
stick.)  Now, initial backup with cp, then a script to do rsync.

So, maybe I should periodically alter the backup script to add the
--checksum switch to rsync?  AIUI, that would force rsync to verify
matching MD4 checksums for a file on my HD and the backup copy
I made (say) last year.  That would force full read of all 2.9G in
/mnt/usb/backup, catch and update any defective files, but reading
wouldn't "refresh" the files found to be okay at the moment, would it?

Thanks, all.  Further comment/pointers welcome.


This is also worth looking at.  I'm sure I've posted before, but in case of new people, here we go again.



_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] Re: USB stick for backup okay?

Daniel MacKay-2

On 2018-03-28, at 20:42 , Dave Flogeras <[hidden email]> wrote:
> Poor man's time-machine:

very clever!
_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] Re: USB stick for backup okay?

Richard Bonner
In reply to this post by Daniel MacKay-2

On Wed, 28 Mar 2018, Daniel MacKay wrote:

> All media spinning magnetic, ssd, tape, floppy, optical fail.

***   As can solid-state media. I had a bag full of failed flashdrives.  )-:

    I gave them all to Alan at A1 Laptop. He uses the cases to recover
working flashdrive circuit boards.

  Richard
_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: [NSLUG] Re: USB stick for backup okay?

Rory-9
In reply to this post by Dave Flogeras

That's some powerful rsync-fu, I like it.

Just to throw in my $0.02 worth:

I've tried various schemes throughout my digital life. In the end, I've been won over by the magic of hard links and online backups that can be easily browsed in-place with nothing more than ls or a file manager.

I really like Apple's TimeMachine (not something I'd say about many Apple products), it 'just works' and is easy enough for all family members to use. Sadly, I can't quite get to a fully FOSS household (yet) so TimeMachine wins for the Mac. For all the other bits (primarily Linux + a smattering of NetBSD and OpenBSD) I use BackInTime. It offers pretty much the same results as TimeMachine and 'just works'.

With both Linux and Mac I have backups that I can browse easily. I can pick any date and go to a complete filesystem exactly as it was that day using any file utility I choose. This is very confidence inspiring compared with large tarballs or non-portable formats.

For storage I have a NAS (FWIW: a Thecus N5500, tossed the original firmware, installed Debian) with 5 drives in RAID6 + LVM. This is the primary backup box, media server, print server and dumping ground for shared files. I have two other portable drives that I rotate periodically. I use them to backup the most critical bits off the NAS, there's always one at home and another at the office. So far I've never needed to recover from the portable drives in spite of burning through a couple of disks in the NAS (RAID is beautiful when it works).

This is a lot of mechanical disks to feed and care for but the $$ per GB works for me. I outgrew optical media long ago; having backups spread across multiple removable disks just got annoying and the DVD stacks started to really add up.

On 2018-03-28 08:42 PM, Dave Flogeras wrote:


On Wed, Mar 28, 2018 at 2:21 AM, Mike Spencer <[hidden email]> wrote:

Long ago, I cluelessly tried to do a backup on a FAT stick.  Someone
here offered a friendly clue reminding me that it was normal for Foo,
foo and FOO to be the same file under FAT.  I created an ext2 fs on
the stick.  (Ext2 because I'm running an old kernel that doesn't grok
ext4.  When main box finally gets an upgrade, so will the backup
stick.)  Now, initial backup with cp, then a script to do rsync.

So, maybe I should periodically alter the backup script to add the
--checksum switch to rsync?  AIUI, that would force rsync to verify
matching MD4 checksums for a file on my HD and the backup copy
I made (say) last year.  That would force full read of all 2.9G in
/mnt/usb/backup, catch and update any defective files, but reading
wouldn't "refresh" the files found to be okay at the moment, would it?

Thanks, all.  Further comment/pointers welcome.


This is also worth looking at.  I'm sure I've posted before, but in case of new people, here we go again.




_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug


_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: USB stick for backup okay?

Mike Spencer

Good discussion.  Thank you all.

Executive summary: My USB stick backup is okay as far as it goes.  But
I need to have a more organized plan for verification and for insurance
against the unexpected.

So I'll be reviewing your responses -- too many to ack here in detail
-- and may have questions later.

Digression: Still turning up problems with the IBM P4 box that's my
main computer. Shipped with IDE drive but has 2 plugs on mobo for SATA
drives.  After the IDE HD failure in 2016, replacement drive was SATA.
Appeared to work fine.  But then found that large writes to HD (say,
copy 1G from USB to HD) bogged the CPU down.  Also, playing a DVD
video with mplayer causes the clock to fall behind ca. 6 sec. per
minute of video.

Now I find that growisofs starts, then gives the system a nervous
breakdown -- repeated DVD drive noises, CPU sluggish, repeated I/O
errors.  Required a shutdown to recover control.  I'm guessing it's a
hardware bug related to the other weirdness mentioned above.

Doing a cp of ~/ over LAN to a newish laptop (tedious) and growisofs on
the laptop worked as expected.  I really must grovel through the
bother of replacing the SATA drive with a good, little-used IDE drive
I now have.

Tnx,
- Mike

--
Michael Spencer                  Nova Scotia, Canada       .~.
                                                           /V\
[hidden email]                                     /( )\
http://home.tallships.ca/mspencer/                        ^^-^^
_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: USB stick for backup okay?

Dop Ganger
On Tue, 3 Apr 2018, Mike Spencer wrote:

> Digression: Still turning up problems with the IBM P4 box that's my
> main computer. Shipped with IDE drive but has 2 plugs on mobo for SATA
> drives.  After the IDE HD failure in 2016, replacement drive was SATA.
> Appeared to work fine.  But then found that large writes to HD (say,
> copy 1G from USB to HD) bogged the CPU down.  Also, playing a DVD
> video with mplayer causes the clock to fall behind ca. 6 sec. per
> minute of video.

An easy thing to check is whether you have multiple devices sharing an
interrupt, which is nominally OK but slow interrupt handlers can lead to
it not being OK. Run

watch -n .1 cat /proc/interrupts

in one terminal window then perform a slow operation in another and see if
most of the interrupts are pegging one IRQ. A quick fix might be to switch
which socket the SATA drive is plugged in to, or move USB devices around.

Cheers... Dop.
_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug
Reply | Threaded
Open this post in threaded view
|

Re: USB stick for backup okay?

Mike Spencer

Dop wrote:

> On Tue, 3 Apr 2018, Mike Spencer wrote:
>
>> Digression: Still turning up problems with the IBM P4 box that's my
>> main computer. Shipped with IDE drive but has 2 plugs on mobo for
>> SATA drives.  After the IDE HD failure in 2016, replacement drive
>> was SATA.  Appeared to work fine.  But then found that large writes
>> to HD (say, copy 1G from USB to HD) bogged the CPU down.  Also,
>> playing a DVD video with mplayer causes the clock to fall behind
>> ca. 6 sec. per minute of video.
>
> An easy thing to check is whether you have multiple devices sharing
> an interrupt, which is nominally OK but slow interrupt handlers can
> lead to it not being OK. Run
>
> watch -n .1 cat /proc/interrupts
>
> in one terminal window then perform a slow operation in another and
> see if most of the interrupts are pegging one IRQ. A quick fix might
> be to switch which socket the SATA drive is plugged in to, or move
> USB devices around.

Hah.  Thanks for that.  watch(1) is a new one for me.

I've been distracted by other things but I did try it while playing a
DVD.  Didn't see anything that didn't make sense.  But whatever that
DVD/clock-slow thing is, it's probably very different from the
big-write bog-down.

Int 0, labeled "timer" ticks all the time, apparently 100 times per
second.  (With -n 1, it updated by 101 every screen update.)  I don't
actually know what that is.  Maybe a hack similar to watch(1) would
reveal that when a DVD is playing, that rate changes?  Unsure what
such an observation would explain, though.

More later.


Tnx,
- Mike

--
Michael Spencer                  Nova Scotia, Canada       .~.
                                                           /V\
[hidden email]                                     /( )\
http://home.tallships.ca/mspencer/                        ^^-^^
_______________________________________________
nSLUG mailing list
[hidden email]
http://nslug.ns.ca/mailman/listinfo/nslug