Merge branch 'main' of gitlab.stud.idi.ntnu.no:raphaesl/dcsg2003

oblig2/oblig2.md

@@ -444,7 +444,108 @@ This verify that memcache is indeed working as excpected.
 
 ### 1) Backup VM
 
+First, we create our new backup VM: "backup". We also attach our newly created openstack volume to it. As for SSH, we decide to use an alternative solution to the one proposed in the assignment. We give the manager ssh key to the backup VM, so that it might contact the database.
+
 ### 3) Backup Script
+Then to make a working backup solution. There are four components to our system. On manager, you have the script "backupDatabase.sh". On the backup VM, there is "backup.sh", and "restore.sh". And on database, there is "checkDbStatus.sh", wich is a script that was written for a previous assignment. The purpose of the script is to check the status of cockroach, and restart it if it's down.
+
+The first step in the backup process is to run backupDatabase.sh on manager. This can safely be done with the database running and backup VM shut off. We use crontab to automaticly backup each morning 08:00, when our bookface has the least visitors.
+
+backupDatabase.sh will first exit bookface production, and reroute visitors to our fallback webserver. Then, the backup VM is started using an openstack command. The script will then repeatedly attempt an SSH connection towards backup until it succeeds. Once in, it will run "backup.sh" and exit. Once this script is done, the backup is complete. The backup VM is turned back off, and production is reinstated.
+
+```
+backupDatabase.sh
+#!bin/bash
+source /home/ubuntu/DCSG2003_V23_group45-openrc.sh
+source /home/ubuntu/dcsg2003/configuration/base.sh
+
+name="backup"
+ip=$(openstack server list --name $name | tail -2 | head -1 | cut -d'|' -f5  | grep -o -P '(?<=192.168.).*(?= )' | awk '{print "192.168."$0}' | xargs)
+
+discord_log "Initiating daily backup..."
+
+# Turn production off
+bash /home/ubuntu/dcsg2003/configuration/manager/production.sh down
+
+# Start backup server
+openstack server start $name
+
+# Wait for valid ssh connection
+until ssh root@$ip exit
+do
+    sleep 5
+done
+
+# Execute backup script
+ssh root@$ip bash /home/ubuntu/dcsg2003/configuration/backupServer/backup.sh
+
+# Turn off backup server
+openstack server stop $name
+
+# Restore Production status
+bash /home/ubuntu/dcsg2003/configuration/manager/production.sh up
+```
+
+Once the "backup.sh" script is run on the backup server, it will first mount the volume. Then it will make an SSH connection towards the database VM, use it to kill the cockroach process. With the database down, it will start copying all the content of /bf_data to a local temporary folder.
+Once the download is complete, "checkDbStatus.sh" is run via SSH on the database VM, wich will detect the DB as down, and immediatly restart it. Meanwhile, the copied files are compressed into a tarball, wich is then moved to the backup volume. We also have a mechanism in place that deletes all but the three newest backups on every backup. This frees up a lot of space, but makes our backups have a three day expiration date.
+
+Once the backup is done, the script exits. The script on manager will continue, and the VM will be shut down until it is run again the next morning.
+
+```
+backup.sh
+#!/bin/bash
+source /home/ubuntu/dcsg2003/configuration/base.sh
+
+
+dbIp="192.168.130.246"
+ssh -o "StrictHostKeyChecking no" ubuntu@$dbIp exit
+
+# Manager turns on backup-server.
+
+# Mount volume to backup
+mount /dev/vdb /backup
+
+# Deletes all but the 3 newests backups
+ls -t bfdata* | tail -n +4 | xargs rm -f
+
+# Turn off cockroach database
+pid=$(ssh root@$dbIp ps aux | grep "cockroach" | grep -v "grep" | awk '{print $2}' | head -1)
+ssh root@$dbIp kill $pid
+
+# Delete temporary folder
+rm -r /tmp/bfdata
+
+# Copy bfdata to /tmp/bfdata
+scp -r root@$dbIp:/bfdata /tmp/bfdata/
+
+if [ $? -ne 0 ]
+then
+discord_log "There was an error when copying database files."
+ssh root@$dbIp bash /home/ubuntu/dcsg2003/configuration/database/checkDbStatus.sh
+exit
+done
+
+# Turn on cockroach database
+ssh root@$dbIp bash /home/ubuntu/dcsg2003/configuration/database/checkDbStatus.sh
+
+
+# Archive bfdata to zip
+tar cvzf bfdata_backup_$(date +'%d-%m-%y_%H-%M').tgz -C /tmp/ bfdata/
+if [ $? -ne 0 ]
+then
+discord_log "There was an error when compressing database files."
+exit
+done
+
+# Copy zip file to volume /backup
+cp ./bfdata_backup_* /backup
+if [ $? -ne 0 ]
+then
+discord_log "There was an error when sending the backup tarball to volume storage. Check storate capacity."
+exit
+done
+```
+
 
 ### Manager back up list