Skip to main content

Backup

Thinking about this because the raw video from Kiersten's shoots is LARGE. I don't anticipate going back and re-editing these projects but it would be nice to keep the full quality versions somewhere. To that end I think I'd like to re-encode the raw videos to a more compressed format to keep on spinning storage, and offload the full quality files to an offline medium. Bluray discs could easily hold the raw footage and not require tons of hardware to keep up with. Some sort of convention for keeping track of where the original data lives should be devised.

I think I'll create a JSON document with a common title like COLD_STORAGE.json in directories where some original data has been offloaded to physical media. This could contain the file listing and a reference to the disc(s) where the data is. Then if it gets to the point of needing a more robust solution those could be scraped from the filesystem into some DB application

The Master Plan

for already compressed stuff e.g. original videos:

  1. figure out how to optimally distribute the input files into directories of a specific capacity - disc capacity minus some percentage for parity data - 5% seems like a huge amount in a promising way
    • fpart can do this?
    • for a 25GB BD-R ~23.7 GB data, 1.185 GB parity
    • need to figure out the actual size to have a full disc after mkisofs
    • iso format - level 3, rockridge extension, joliet extension (only for windows, shrug), padding (default)
      • NO LEADING DOTS
  2. store the mapping for each file from source directory to archive directory in a database - sqlite is easiest and very compatible
  3. generate parity for the contents of each directory with par2 to a target size of 5%
  4. mkisofs for each directory of data and parity
  5. burn iso to each disc
  6. store the sqlite db in the place where the original files were - this lives on warm backup

Giving this a try with a bunch of photos including our wedding photos.

  1. copied some photos down from the nas up to about 20GB
  2. created parity for the image files and excluded the XMP files (those are small and non-critical) with this command:
    • par2 create -rg1 -m2048 parity_1g.par2 -- $(find ./ -not -iname "*.xmp" -printf "%p "
    • this gives a target size of 1G of parity data which is probably overkill, but I haven't crammed as many files as I could possibly fit for this test
  3. dragged all folders and the parity files to brasero to create a .iso - that kinda sucked use mkisofs -iso-level 3 -rational-rock -preparer "Jack Case - jackacase@gmail.com" -V "Wedding Photo Archive" -v -o wedding_photo_archive.iso photo_archive
  4. burning now - the brasero gui locks up but the log file is continuing
    • the log file showed that it's using growisofs under the hood, so I could just use that directly on the command line.
  5. Burn successful. ejected disc and reloaded, ran par2 verify and everything was clean :D

No comments to display

Back to top