OpenNAS
Features
-
DHCP/mDNS to locate server on configuration OR show IP on LCD.
- Bubba3 has solved this. Just enter "b3" in a browser
-
SMB Network Access
-
Sync between local USB disks
-
Sync between remote USB disks
-
MiniDLNA
-
Auto mount of USB disks
Future features
- Basic firewall
- Act as Media Renderer which output directly to amp
- Automatic photo transfer (over USB) with automatic duplicate rejection.
Phases
- Single LAN NAS only
- Add firewall / pass through
Hardware
- ALIX 2D13 System Board OR Mini-ITX
- 3 LAN, 2-4 USB
- 3 LED
- Single Push button
- I2C to LCD display
- CF card
- USB disks
Example hardware.
Views
- Network setup
- Pass through (2 LAN & firewall) / Local (1 LAN & no firewall)
- External ethernet setup
- Internal ethernet setup
- DHCPD setup
- Remote SSH account setup
- Local user setup
- Remote login setup
- Firewall setup (later)
Cloud Storage Price Comparison
- $0.10 per GB-month of provisioned storage
- $0.120 per GB transfer
(server instance cost is not included)
tarsnap backup service
- Storage: $0.30 / GB-month
- Bandwidth: $0.30 / GB
- Storage: $10 / unlimited for personal use
- Bandwidth: free
$ 1800 / year for 1 TB
Cost example:
1 TB or storage and 100 GB transfer / month
Service | : Cost / month |
---|---|
EC2 | $ 112 |
tarsnap | $ 330 |
cyphertite | $ 10 (personal use) |
rsync.net | $ 150 |
Use Case Analysis
Disk categories
- Disk Type:
Primary
/Backup
- Ownership:
Yours
/Partners
- Location:
Local
/Remote
(0+)
Common functions & Rules
- Setup
Local
settings. How to access internet etc. not related to disk. - Setup a link to a
Remote
location.- A link (referring to myself) is sent through mail which is copied to the local UI.
- A bi-directional link is setup which involves device to device connection service.
- You can only have a single
Primary
disk but zero or moreSecondary
disks associated with thePrimary
Create
a localYours
&Local
disk. It may bePrimary
orBackup
Create
a localPartners
&Remote
&Backup
diskFile
Accessto
Local&
Primary&
Yours` disk through Samba, AFS and NFSSync
fromLocal
&Primary
&Yours
->Yours
&Secondary
Recovery
fromYours
&Secondary
->Local
&Primary
&Yours
Options
Sync
may beIncremental
orNon-incremental
Sync
may beManual
orPeriodic
File
Accessusers to
Primary` diskSecondary
may beEncrypted
orNon-encrypted
Recovery
may specify a specific increment to recover fromRecovery
may specify a subset of files to recoverSync
through company firewall in reverse?
Status Information
Backup
disks must present latest sync actions.Sync Actions
should present time of sync, data transferred and time to sync.- All disks must present disk free (and disk used)
EHCI bug in OpenBSD
I still have problems when mounting a USB disk for a long time on OpenBSD. This is currently the only showstopper for this project.
The following script (test.sh
) is used to test it.
#!/bin/sh
echo "FORMAT:"
/root/format_usbdisk.sh sd0
echo "MOUNT directly after format: "+`date`
mount_ffs /dev/sd0a /mnt/matteus
time ls -l /mnt/matteus
sleep 3600
echo "MOUNT after 1 hour: "+`date`
time ls -l /mnt/matteus
sleep 7200
echo "MOUNT after 3 hour: "+`date`
time ls -l /mnt/matteus
sleep 14400
echo "MOUNT after 7 seconds: "+`date`
time ls -l /mnt/matteus
I also make use of format_usbdisk.sh
.
#!/bin/sh
# Automates the following process so BE CAREFUL !!!
# fdisk -iy sd0
# Writing MBR at offset 0.
# disklabel -E sd0
# Label editor (enter '?' for help at any prompt)
# > z
# > a
# partition: [a]
# offset: [64]
# size: [15165296]
# FS type: [4.2BSD]
# Rounding size to bsize (32 sectors): 15165280
# > w
# > q
# No label changes.
# newfs sd0a
# TODO: Check that the disk is removable
DISK=$1
LOGFILE="/tmp/format_usbdisk.log"
if [ $# -ne 1 -o -z "$1" ]; then
echo "Usage: ./format_usbdisk.sh <dev>"; exit
fi
echo "Disk on $DISK will be formatted to FFS2. Wait until all blocks are written ..."
echo
fdisk -iy $DISK 2>&1 > $LOGFILE
disklabel -E $DISK 2>&1 >> $LOGFILE << __EOT__
z
a
w
q
__EOT__
newfs "${DISK}a"
echo
Views
Disk view
- Each disk is listed by location.
- Sync status and sync setup is accessed through the corresponding backup disk.
- Unconfigured disks show up as such.
- A location setup cover access between local and remote devices.
Open items
- Compare functionality with Hammer. Is this a better option?
Scripts
- [[format-usbdisk.sh]]
Hardware with internal HDD
If we should build a OpenNAS system with a internal HDD I think the following components could be suitable.
The system may also work as a very capable firewall.
- apu1d - 116 euro
- 19" DualRack System for PC Engines ALIX, APU boards - 70 euro
- Embedded Box for AMD APU1D and APU1D4 for WLAN and HDD - 35 euro
- Mean Well, 12V, 2.1A, 25W Single Output Switching Power Supply - 14 euro
- 8GB Transcend SD Card - 10 euro
Total: 175 euro
An internal 3.5 SATA HDD (e.g. 1-2 TB) is needed to complete the system.
AWS S3 Glacier
It seems like AWS S3 Glacier Deep Archive is one option for a long term remote cloud based backup copy.
The cheapest (as of 2023) options for cloud storage is about 4 USB / month / TB. Glacier deep archive is 1 USB / month / TB but with a lot of other fees for access retrieval etc. So it can mainly be used as insurance backup if all other backups fails. See Backup to Amazon S3 Glacier Deep Archive - Cost Analysis for a deeper cost analysis.
So Glacier could be used for the following.
- Store large files that don't individually change (e.g. media and photos)
- Only use as long term storage which possibly never should be retrieved
- Dont use for small files
- Dont use for large files that change content often
So a script that store media files incrementally towards glacier could be one option.
- Keep a metadata database for all files stored to glacier
- Check new files in folders that should be incrementally backed up
- Never delete files on glacier only update metadata for deleted files
- Dont to deduplication on files
- Implement a retrieval function which collects all files at a certain date using metadata file
Store files with folder structure as rsync incremental backups (but without the hard links)
<data for backup>/full local path/filename
The following technologies could be used.
- Golang or Crystal for CLI app language
- Use LMDB as metadata DB
- Store LMDB both in Glacier and faster storage like S3
Investigation Tasks
- Setup OpenMDNSD to allow configuration view to show up under http://opennas.local.
- Naming convention of disks (Location, Primary/Backup, Local/Remote). Disk metadata storage (file). Unique identifier for Location and disks
- How to enable disk encryption
- How to recover a specific version of a file system / set of files
- Automatic detection of USB disks
- Design configuration / execution process. Single configuration file generate changes in target files. New configuration is deployed through with or without reboot
- How to handle SSH connection through firewall. Should we assume that device work as bridge towards internet
- Define minimal firewall configuration setup
References
- Do-It-Yourself Backup System Using Rsync
- Different thin client hardware
- Mark’s OpenBSD Router Guide
- OpenBSD mini-server
- FreeNAS - Rsync Tasks
- Amahi
- pfSense
- Waiter - A java DNS-SD library
- Crystal Fonts LCD displays
- Pico LCD
- Kinsky UpNP Control Point
- Bytemin Appliance
- Excito - Bubba servers
- B3 to rsync.net, B3 to extenal via rsync
- zumastor and tux3
- Embedding NetBSD: VOIP Applications
- Fujitsu Futro S400 and fattening Futro S400
- Tunna klienter förenklar och sparar pengar
- The secret server
- Running OpenBSD off a USB Stick
- Arch ARM
- AdTrap
- http://www.csync.org/
- flashrd+nsh OpenBSD Appliances
- duplicity and Create secure remote backups using Duplicity – Tutorial
- Bonjour clients for different platforms
- HP t5720 - Thin Client Hardware Reference Manual. Note that the flash disk is connected to a standard IDE connector. See Upgrade Guide. Use e.g. an IDE to CF converter.
- Why OpenBSD and then Dragonfly
- AllWinner A10
- Plug: the brain of your devices
- PEFS FreeBSD disk encryption and Secure Backups for Lazy Developer
- arkos.io - Nice web UX
- resflash
- arp(8) - Wake on LAN from OpenBSD via arp(8)
- https://rack-matrix.com/ - Customizable 1U racks
- Olimex A20-SOM-EVB - A20 with SATA
- restic