OpenNAS
Features
-
DHCP/mDNS to locate server on configuration OR show IP on LCD.
- Bubba3 has solved this. Just enter "b3" in a browser
-
SMB Network Access
-
Sync between local USB disks
-
Sync between remote USB disks
-
MiniDLNA
-
Auto mount of USB disks
Future features
- Basic firewall
- Act as Media Renderer which output directly to amp
- Automatic photo transfer (over USB) with automatic duplicate rejection.
Phases
- Single LAN NAS only
- Add firewall / pass through
Hardware
- ALIX 2D13 System Board OR Mini-ITX
- 3 LAN, 2-4 USB
- 3 LED
- Single Push button
- I2C to LCD display
- CF card
- USB disks
Example hardware.
Views
- Network setup
- Pass through (2 LAN & firewall) / Local (1 LAN & no firewall)
- External ethernet setup
- Internal ethernet setup
- DHCPD setup
- Remote SSH account setup
- Local user setup
- Remote login setup
- Firewall setup (later)
Cloud Storage Price Comparison
- $0.10 per GB-month of provisioned storage
- $0.120 per GB transfer
(server instance cost is not included)
tarsnap backup service
- Storage: $0.30 / GB-month
- Bandwidth: $0.30 / GB
- Storage: $10 / unlimited for personal use
- Bandwidth: free
$ 1800 / year for 1 TB
Cost example:
1 TB or storage and 100 GB transfer / month
| Service | : Cost / month |
|---|---|
| EC2 | $ 112 |
| tarsnap | $ 330 |
| cyphertite | $ 10 (personal use) |
| rsync.net | $ 150 |
Use Case Analysis
Disk categories
- Disk Type:
Primary/Backup - Ownership:
Yours/Partners - Location:
Local/Remote(0+)
Common functions & Rules
- Setup
Localsettings. How to access internet etc. not related to disk. - Setup a link to a
Remotelocation.- A link (referring to myself) is sent through mail which is copied to the local UI.
- A bi-directional link is setup which involves device to device connection service.
- You can only have a single
Primarydisk but zero or moreSecondarydisks associated with thePrimary Createa localYours&Localdisk. It may bePrimaryorBackupCreatea localPartners&Remote&BackupdiskFileAccesstoLocal&Primary&Yours` disk through Samba, AFS and NFSSyncfromLocal&Primary&Yours->Yours&SecondaryRecoveryfromYours&Secondary->Local&Primary&Yours
Options
Syncmay beIncrementalorNon-incrementalSyncmay beManualorPeriodicFileAccessusers toPrimary` diskSecondarymay beEncryptedorNon-encryptedRecoverymay specify a specific increment to recover fromRecoverymay specify a subset of files to recoverSyncthrough company firewall in reverse?
Status Information
Backupdisks must present latest sync actions.Sync Actionsshould present time of sync, data transferred and time to sync.- All disks must present disk free (and disk used)
EHCI bug in OpenBSD
I still have problems when mounting a USB disk for a long time on OpenBSD. This is currently the only showstopper for this project.
The following script (test.sh) is used to test it.
#!/bin/sh
echo "FORMAT:"
/root/format_usbdisk.sh sd0
echo "MOUNT directly after format: "+`date`
mount_ffs /dev/sd0a /mnt/matteus
time ls -l /mnt/matteus
sleep 3600
echo "MOUNT after 1 hour: "+`date`
time ls -l /mnt/matteus
sleep 7200
echo "MOUNT after 3 hour: "+`date`
time ls -l /mnt/matteus
sleep 14400
echo "MOUNT after 7 seconds: "+`date`
time ls -l /mnt/matteus
I also make use of format_usbdisk.sh.
#!/bin/sh
# Automates the following process so BE CAREFUL !!!
# fdisk -iy sd0
# Writing MBR at offset 0.
# disklabel -E sd0
# Label editor (enter '?' for help at any prompt)
# > z
# > a
# partition: [a]
# offset: [64]
# size: [15165296]
# FS type: [4.2BSD]
# Rounding size to bsize (32 sectors): 15165280
# > w
# > q
# No label changes.
# newfs sd0a
# TODO: Check that the disk is removable
DISK=$1
LOGFILE="/tmp/format_usbdisk.log"
if [ $# -ne 1 -o -z "$1" ]; then
echo "Usage: ./format_usbdisk.sh <dev>"; exit
fi
echo "Disk on $DISK will be formatted to FFS2. Wait until all blocks are written ..."
echo
fdisk -iy $DISK 2>&1 > $LOGFILE
disklabel -E $DISK 2>&1 >> $LOGFILE << __EOT__
z
a
w
q
__EOT__
newfs "${DISK}a"
echo
Views
Disk view
- Each disk is listed by location.
- Sync status and sync setup is accessed through the corresponding backup disk.
- Unconfigured disks show up as such.
- A location setup cover access between local and remote devices.
Open items
- Compare functionality with Hammer. Is this a better option?
Scripts
- [[format-usbdisk.sh]]
Hardware with internal HDD
If we should build a OpenNAS system with a internal HDD I think the following components could be suitable.
The system may also work as a very capable firewall.
- apu1d - 116 euro
- 19" DualRack System for PC Engines ALIX, APU boards - 70 euro
- Embedded Box for AMD APU1D and APU1D4 for WLAN and HDD - 35 euro
- Mean Well, 12V, 2.1A, 25W Single Output Switching Power Supply - 14 euro
- 8GB Transcend SD Card - 10 euro
Total: 175 euro
An internal 3.5 SATA HDD (e.g. 1-2 TB) is needed to complete the system.
AWS S3 Glacier
It seems like AWS S3 Glacier Deep Archive is one option for a long term remote cloud based backup copy.
The cheapest (as of 2023) options for cloud storage is about 4 USB / month / TB. Glacier deep archive is 1 USB / month / TB but with a lot of other fees for access retrieval etc. So it can mainly be used as insurance backup if all other backups fails. See Backup to Amazon S3 Glacier Deep Archive - Cost Analysis for a deeper cost analysis.
So Glacier could be used for the following.
- Store large files that don't individually change (e.g. media and photos)
- Only use as long term storage which possibly never should be retrieved
- Dont use for small files
- Dont use for large files that change content often
So a script that store media files incrementally towards glacier could be one option.
- Keep a metadata database for all files stored to glacier
- Check new files in folders that should be incrementally backed up
- Never delete files on glacier only update metadata for deleted files
- Dont to deduplication on files
- Implement a retrieval function which collects all files at a certain date using metadata file
Store files with folder structure as rsync incremental backups (but without the hard links)
<data for backup>/full local path/filename
The following technologies could be used.
- Golang or Crystal for CLI app language
- Use LMDB as metadata DB
- Store LMDB both in Glacier and faster storage like S3
Investigation Tasks
- Setup OpenMDNSD to allow configuration view to show up under http://opennas.local.
- Naming convention of disks (Location, Primary/Backup, Local/Remote). Disk metadata storage (file). Unique identifier for Location and disks
- How to enable disk encryption
- How to recover a specific version of a file system / set of files
- Automatic detection of USB disks
- Design configuration / execution process. Single configuration file generate changes in target files. New configuration is deployed through with or without reboot
- How to handle SSH connection through firewall. Should we assume that device work as bridge towards internet
- Define minimal firewall configuration setup
References
- Do-It-Yourself Backup System Using Rsync
- Different thin client hardware
- Mark’s OpenBSD Router Guide
- OpenBSD mini-server
- FreeNAS - Rsync Tasks
- Amahi
- pfSense
- Waiter - A java DNS-SD library
- Crystal Fonts LCD displays
- Pico LCD
- Kinsky UpNP Control Point
- Bytemin Appliance
- Excito - Bubba servers
- B3 to rsync.net, B3 to extenal via rsync
- zumastor and tux3
- Embedding NetBSD: VOIP Applications
- Fujitsu Futro S400 and fattening Futro S400
- Tunna klienter förenklar och sparar pengar
- The secret server
- Running OpenBSD off a USB Stick
- Arch ARM
- AdTrap
- http://www.csync.org/
- flashrd+nsh OpenBSD Appliances
- duplicity and Create secure remote backups using Duplicity – Tutorial
- Bonjour clients for different platforms
- HP t5720 - Thin Client Hardware Reference Manual. Note that the flash disk is connected to a standard IDE connector. See Upgrade Guide. Use e.g. an IDE to CF converter.
- Why OpenBSD and then Dragonfly
- AllWinner A10
- Plug: the brain of your devices
- PEFS FreeBSD disk encryption and Secure Backups for Lazy Developer
- arkos.io - Nice web UX
- resflash
- arp(8) - Wake on LAN from OpenBSD via arp(8)
- https://rack-matrix.com/ - Customizable 1U racks
- Olimex A20-SOM-EVB - A20 with SATA
- restic