Backups - Revision history

Paul: /* Writing new software */

2024-09-24T10:07:25Z

Writing new software

← Older revision		Revision as of 10:07, 24 September 2024
Line 25:		Line 25:
	* Copy files from X to Y		* Copy files from X to Y
	* Database (SQLite? DuckDB?) to hold metadata (map hashes to file paths)		* Database (SQLite? DuckDB?) to hold metadata (map hashes to file paths)
	* SHA-~~512~~ for deduplication		* File size and SHA-3 for deduplication

	Thoughts:		Thoughts:

2024-09-24T08:41:27Z

Writing new software

← Older revision		Revision as of 08:41, 24 September 2024
Line 41:		Line 41:
	Security:		Security:

			* Assumption is that you trust the backup target
	* If you want encryption at rest, use LUKS etc. to encrypt the underlying device		* If you want encryption at rest, use LUKS etc. to encrypt the underlying device
	* If you want encryption in transit, use SSH or TLS		* If you want encryption in transit, use SSH or TLS

2024-09-09T09:47:52Z

Writing new software

← Older revision		Revision as of 09:47, 9 September 2024
Line 37:		Line 37:
	* Can we compress the files? Can this be done in parallel?		* Can we compress the files? Can this be done in parallel?
	* What is the fastest collision-resistant algorithm for file comparison?		* What is the fastest collision-resistant algorithm for file comparison?
			* How do we stop two backup processes running simultaneously?

	Security:		Security:

2024-09-09T09:47:08Z

Writing new software

← Older revision		Revision as of 09:47, 9 September 2024
Line 30:		Line 30:

	* Can the hash checking of files be done in parallel?		* Can the hash checking of files be done in parallel?
			* Should we compare file size first, and only calculate hashes if the sizes are the same?
	* What metadata should we store about each file?		* What metadata should we store about each file?
	* How to restore files?		* How to restore files?

2024-08-07T10:02:39Z

Writing new software

← Older revision		Revision as of 10:02, 7 August 2024
Line 25:		Line 25:
	* Copy files from X to Y		* Copy files from X to Y
	* Database (SQLite? DuckDB?) to hold metadata (map hashes to file paths)		* Database (SQLite? DuckDB?) to hold metadata (map hashes to file paths)
	* ~~MD5~~ for deduplication		* SHA-512 for deduplication

	Thoughts:		Thoughts:
Line 35:		Line 35:
	* Deduplication at the file level - could also do this with chunks of files at a later date?		* Deduplication at the file level - could also do this with chunks of files at a later date?
	* Can we compress the files? Can this be done in parallel?		* Can we compress the files? Can this be done in parallel?
			* What is the fastest collision-resistant algorithm for file comparison?

	Security:		Security:

2022-12-27T13:02:22Z

← Older revision		Revision as of 13:02, 27 December 2022
Line 47:		Line 47:
	* [https://jpmens.net/2019/04/15/i-mirror-my-github-repositories-to-gitea/ I mirror my Github repositories to Gitea]		* [https://jpmens.net/2019/04/15/i-mirror-my-github-repositories-to-gitea/ I mirror my Github repositories to Gitea]
	* [https://github.com/PyGithub/PyGithub PyGithub]		* [https://github.com/PyGithub/PyGithub PyGithub]

			[[Category:Software]]
			[[Category:Open Source Software]]

2022-12-19T19:16:00Z

Writing new software

← Older revision		Revision as of 19:16, 19 December 2022
Line 24:		Line 24:
	* Exclude file (one per line)		* Exclude file (one per line)
	* Copy files from X to Y		* Copy files from X to Y
	* Database (SQLite?) to hold metadata (map hashes to file paths)		* Database (SQLite? DuckDB?) to hold metadata (map hashes to file paths)
	* MD5 for deduplication		* MD5 for deduplication

2022-12-19T19:15:32Z

Writing new software

← Older revision		Revision as of 19:15, 19 December 2022
Line 16:		Line 16:

	== Writing new software ==		== Writing new software ==

			Powerful Command-Line Applications in Go - may be useful

	MVP:		MVP:

2022-12-19T15:33:43Z

Writing new software

← Older revision		Revision as of 15:33, 19 December 2022
Line 32:		Line 32:
	* How to prune files?		* How to prune files?
	* Deduplication at the file level - could also do this with chunks of files at a later date?		* Deduplication at the file level - could also do this with chunks of files at a later date?
			* Can we compress the files? Can this be done in parallel?

	Security:		Security:

2022-12-19T15:31:16Z

Writing new software

← Older revision		Revision as of 15:31, 19 December 2022
Line 31:		Line 31:
	* How to restore files?		* How to restore files?
	* How to prune files?		* How to prune files?
			* Deduplication at the file level - could also do this with chunks of files at a later date?

	Security:		Security: