Updating the Backup Scripts

So, it’s been awhile since I actually looked at the code for my Backup Scripts.. I had been using them religiously for quite some time, but when I got rid of the server cluster that I had, I stopped using them.

Recently, I setup a server in the Azure cloud to run this website and a few others on. I figured, what a perfect use for the scripts!

Upon firing them up, I found several (no, really, pretty much every bit of code) pieces of the script needed overhauling as errors were thrown all over the place.

First thing I started with is the home-backup script. I used a really handy website called ShellCheck that assisted in the ‘QA’ of the code.

I think, it’s about time to start versioning the script, instead of just making changes whenever. I’m not sure what version it’ll start with, but compared to the original, there will be a lot of improvements.

First, I’m working on enabling the ability to call flags on the command, for example: “./home-backup -d /some/backup/dir -u some_user”. Which will make it easy to setup single user backups, instead of backing up everything.

Second, Error handling. Right now, the script doesn’t have any error handling at all. Since it’s a backup script, the only risk is, the script fails and no backup is taken. I’m building error handling around the backup destination, so that if the directory doesn’t exist, it will prompt you to create the directory. I”ll likely work into the script an automatic creation when using a flag.

Now, the mysql-backup script, will mimic the home-backup because they both go through the same paces.

So, the home-restore script.. This is one I haven’t really dug into yet. Obviously, the first thing that needs to be done is error handling and a “rollback” feature. Right now, it expects that all conditions are met, and just goes. This one, may take quite a bit longer to be “ready”.

For now, I’ve put the WebUI portion of this project on the shelf. Somewhere down the line, I might pick it backup and actually make it happen. But, for now, it’s on the shelf.

If you’ve got any bash experience, and wish to lend a hand, please by all means contact me!

Thanks for reading!

Dan

Adding on to backup bash scripts

So, ages ago, I wrote up a couple of bash scripts that automated the backup process on a server. It’s all text based, so you’d have to be logged into a terminal or be ssh’d into the box.

Today, I got to thinking, I should give it a pretty web gui that makes the process easier and functions through a web gui.

It’s still really early in the through process, but I’m intrigued by the project and will start working on it soon.

Original Post: http://www.merval.org/2012/03/bash-scripts/

Github: https://github.com/merval/BackupScripts

Apple Wireless Keyboard [Review]

I bought this keyboard on Christmas Day 2013, off Amazon. It was delivered on January 2nd. Considering the Holidays, that’s pretty good. So, first thing I noticed with this keyboard is the size. It’s literally the same keyboard you get with a Macbook or Macbook Pro. I actually lined it up with my Macbook keyboard and it’s exactly the same.

The keyboard doesn’t weigh much either, Apple says it weighs in at 8 ounces. All of the weight is from the batteries (two AA batteries). But it very, very light. It also has these skid guards on the bottom of it, I’m not sure if they are suppose to prevent you from pushing it around when typing, but it doesn’t really do much other than keep the bottom of the keyboard from rubbing the desk.

Usability

With the keyboard only coming in at 12.8 x 1.4 x 7.3 inches in size, it’s pretty compact, and I have giant hands. I first came across this keyboard while working on a computer for a family friend. At first, I was taken back by how small the keyboard really is and how big my hands were compared to it. I had a bit of a hard time trying to figure out how to make my fingers work on such a small keyboard. But once I got the hang of it, it actually felt pretty nice.

The idea of taking a keyboard from a Macbook and making it into a bluetooth keyboard is pretty ingenious. The only thing I miss from the old Windows keyboard I used prior to this, is the 10 key.

This keyboard has all the buttons the Macbooks have, including controls for iTunes and screen brightness (need to have a compatible monitor).

The best part about this keyboard is that when I got it, I put batteries in and started using it. That was on January 2nd. Today is May 28th, and I just switched out the batteries. I use the computer, A LOT. So for the batteries to last nearly five months, is staggering. Well done Apple.

Cons

I’ve been ranting and raving about this keyboard for long enough… Now for the things that bug me.

I don’t use Caps Lock very often so, I didn’t catch this right away, but the button doesn’t seem to notice it’s being pressed, I sometimes have to press it two or three times before the light kicks on showing Caps Lock is one.

When I type, I don’t think of myself as one of those people who is smacking the keyboard as hard as they can. But with this keyboard, for some reason, as I type it makes this vibration sound, as if the alignment is off, just a hair. The area below the spacebar tends to have a little room that bounces off my desk as I type and makes this terrible sound. But, I usually have headphones on and don’t hear it.

I would love to be able to sync the keyboard with multiple devices, and have the power button act as switch between devices, but instead, I have to either completely disconnect the bluetooth on one device or have the device forget the keyboard to switch to another one.

 

All in all, I have really enjoyed this keyboard. The nice part is, where I work, we use Windows 7 and going from this keyboard to a Windows keyboard, isn’t as difficult as I thought it was going to be. Seeing as with OS X you have several keyboard shortcuts that don’t mesh up with Windows keyboards. But on a subconscious level, because the keyboards are so different, when I sit down at my desk at work, I don’t even miss a beat.

So, with the few cons I have, I’m giving this keyboard a 9.6 out of 10 stars.

 

The case of the MacBook LCD issue.

So, My wife’s uncle gave me a Macbook of his that worked, but the LCD panel was not working, or so I thought. I proceeded to do a full tear down on it, attempting to locate the problem, having found what I thought the issue was, a pinched display cable (which turned out to be incorrect). I took the macbook to the Apple store to see if they could assist, granted the macbook wasn’t in one piece anymore. The Apple rep explained, because I had taken the whole laptop apart, they were unable to service it and suggested I take the computer to the MacStore. Which I happily did, in hopes they would solve my problem.

We arrived at the MacStore and explained my problem. Which was that when you opened the macbook the screen would light up briefly but then turn off again once opened past at 15 degree angle. They agreed to do a diagnoses on the computer for me. A day or so later, the tech calls me and explains the LCD panel is faulty and the clutch cover was broken and needed to be replaced. All told, repairs would be over $400 dollars. I declined the repair and brought the computer home. Having been told the LCD panel was bad and the bezel (the part that goes around the panel covering the screws) was broken in half, I opted to buy a whole LCD assembly, for only $85 on ebay. I got the assembly but quickly realized the inverter cable was wrong, so without a thought, I swapped out the cable for the one on the other LCD panel. Got everything screwed together and was just about done when I decided, before I lock it all down, I had better do a test run to ensure I haven’t forgotten any cables.

I plugged in the power and hit the switch. The screen didn’t light up. I double and triple checked all my connection, ensuring everything was correct, I thought, “I wonder what will happen if I close the lid to a 15 degree angle..”. Son-of-a-bitch! The thing lights up at a 15 degree angle. How the hell could I have TWO faulty LCD panels that only work at 15 degrees? Then I investigated further. I had swapped out the inverter cable when I first got this assembly, so I started inspecting the cable, and I’ll be damned, there is a small cut in the cable and one of the wires were separated.

So, out of curiosity I started moving the cable around. Before I knew it, I had found the sweet spot, the LCD panel was completely light up and I could see everything! The LCD wasn’t BAD, the little cable was faulty. Now the MacStore had charged me $49 for the diagnostic, which after I had discovered the faulty cable, I called them and asked for a full refund of my money because their tech didn’t catch the cable issue. They provided me in-store credit. Which works out nicely, my Wife found an iPad Mini case there that she really liked.

So to wrap this whole thing up, I went on ebay and found a guy selling the exact cable I need, for $10 ($12 after shipping).

This was my second time opening this particular Macbook model, my wife and I actually have this exact macbook model (Only it’s the white model and this one is the black one) and I had opened the case up awhile back to clean out some dust and more recently replace the isight camera.

I’m actually starting to become quite comfortable cracking these macbooks open and fixing problems. The only thing I really need to get my hands on is a magnetic screw organizer. Macbooks have an insane amount of screws. And most of them are tiny and very easy to lose.

So, anyway. Hopefully by Monday or Tuesday I’ll have my inverter cable and a fully functional Macbook! (it works now but only through an external Monitor.)

When I got the macbook it had 4GB of RAM. I decided to divvy up the RAM between both macbooks and gave swapped one of the sticks in each, bringing both to 3GB. I’m hoping to upgrade my macbook to a 1TB HD. 🙂 It may be a mid-2007 macbook, but I’m gonna milk every ounce of possible out of this puppy!

Thanks for reading!

Catch ya next time!

– Dan

Open Source, Why it’s good for business.

There has been a lot of buzz recently about Windows XP’s End of Life next year, and a lot of people think of Hackers when they think of, or hear about Linux. What a lot of people fail to understand is, not all hackers are bad, the primary definition of a hacker is, “An enthusiastic and skillful computer programmer or user.”

Most hackers are people who take something and use it in a manner it wasn’t intended, or build their own version of something. This is where open source comes in, the theory behind open source is free information for all. Which is essentially what the “bad” hackers are doing, taking information and making it free to the public. But the difference is, one is perfectly legal and the other is not. Can you guess which is which?

So, What is Open Source? It’s free software/hardware that you are free to use and modify however you wish, and you can redistribute it after you’ve made your changes. Almost all versions of Linux are Open Source, there are a few out there that are Enterprise and really all you’re paying for there is the support and the name, it’s kinda like buying a pair of Nike’s, they are like every other pair of shoes, except there is a brand name attached to them.

Without giving you a crazy long history of Linux, Prior to October 5th 1991, Linux didn’t exist, Linus Torvalds wrote the original Linux kernel and still to this day, any additions must have his final say. Prior to that there was Unix which is a closed source operating system that only a few companies have the ability to modify, which is why Linus created the Linux Kernel, to provide a free open source system that anybody can modify. Since it’s inception in 1991 Linux has grown like a wildfire. More than 90% of today’s 500 fastest supercomputers run some variant of Linux, including the 10 fastest.

Why is Linux and Open Source such a good thing? Imagine that you create something, that to you is awesome and works great, but then someone else see’s where it can be improved and made even better. Now imagine that happening with thousands of volunteers all at once, across several different versions of what you created. That is what Open Source is, and it’s so great because it provides not only multiple options for people, but what you created is constantly getting better. All at no cost to you.

Which is why Open Source is so great for business, when you buy a computer with Windows or Mac on it, you get updates and security patches but when the next version comes out, you have to fork over whatever amount of money their asking for that version. Unless of course you use Linux, then it’s absolutely Free.

So every time a new upgrade comes out for your version of Linux, you can upgrade and not pay a dime for the operating system. Thanks for thousands of volunteers who tirelessly fix issues and add new features.

Did You Know!?
Linux is a leading operating system on servers and other big iron systems such as mainframe computers and supercomputers?

So, the next time you have a computer that is a little old and doesn’t run Windows or Mac as well anymore, instead of tossing it out, either donate it, or put Linux on it and give some new life to that old machine!

OwnCloud – My Review

I have to give it to OwnCloud for making the installation really easy! In less than  two minutes I had OwnCloud installed and ready to go.The GUI has a nice sleek feel to it. It doesn’t have as much functionality as I thought it would, but none the less I took it from sandbox to live the same night. Linking my Desktop and Phone to the install was incredibly easy.

Now to the things that bug me enough to make me tell you about it.

  • No built in Quota bar. This is a MUST. How am I to know where my current storage limit is?
  • The default text editor isn’t a WYSIWYG (What you see is what you get). It is the ugliest interface for typing that I have seen since Basic on a Tandy Color Computer. You have no control of font size or formatting.
  • I would like to see some sort of a built in Open Office type suite. Not everybody wants to use Office to generate a spreadsheet or word document.
  • Changing the colors is a challenge. This should be an option in the Admin section. A user should not need to climb into the Terminal and edit config files to change colors or theme settings.
  • For music to show up in the Music category and be able to stream from the web GUI it shouldn’t be a requirement to have music in a folder named “Music”.

All in all, I am happy with it. It’s a nice alternative to Dropbox or Google Drive. However, if you don’t have an ISP with a decent Upload speed you will suffer a lot.

I will get some screenshots added later tonight.

OwnCloud

You’re familiar with the term cloud right?… If not, I’m sure you have heard of Dropbox, Box, Google Driveetc. These are all Cloud services, they store your data on a server that is accessible from anywhere. So, what I’m going to do, is create my own using OwnCloud and an Ubuntu Server I have (currently it sits and runs PyTivo.. That’s a different blog post 🙂 ) (Edit: Yes I am aware of Ubuntu Enterprise Cloud Server.. I will look at installing and playing with this.)

I was watching TWiT‘s new show “Know How…” and their first episode was about rolling out your own cloud a few different ways, they shows the Tonido Plug and the Pogo Plug. Now Tonido has a software suite you can download and use but from the sounds of it, you are actually allow them to see some of your data (at least that is how it is with the Tonido plug device). I’m not so interested in allowing that to happen, not because I’m doing anything illegal but because I don’t like the idea of people willy nilly looking at my data.

Now, looking at OwnCloud you can run the software on your machine (Desktop or Server, I prefer server) and it will simply host a Web GUI that you can access your data from. OwnCloud comes with a client you install on your device and you can access your data from your server. I don’t know if there is a mobile app yet. I mainly want this setup so I can easily access content from my home server without needing to worry about a super low max limit (Dropbox currently has 4gb on my account and Box has 50gb).

I plan on toying with it and seeing what all it does and then writing a review on it. The idea of running your own cloud (If you have an ISP that gives you a nice upload speed and doesn’t limit you) is a really neat idea. You don’t have to worry about uploading to a server, where God knows who is looking at your data.

The other option is to use software to encrypt your data before you upload to these cloud services, the reason I dislike that idea is that when I want to run in and grab something quick I don’t want to have to worry about, “Does this computer/device have the software to allow me to view this?”

So, I will install and toy with OwnCloud on my Sandbox machine and see what I come up with.  If you do not have a sandbox machine. You should REALLY invest in one. Mine is basically an old computer I had laying around after an upgrade that I tossed some hardware into. You can also pick up computers pretty cheap on ebay or a local computer recycler. Free Geek is a good place to look too.

See you on the other side!

-Dan

Media Center PC

I recently put together a Media Center PC (more commonly known as HTPC, Home Theater PC) but after some reading I found that the hardware I used, which was an old Pentium 4, is on the “avoid” list.

So, I’m asking the internet this question, What is the best hardware on a budget for a MythTV Backend/Frontend Combo? Or if it will work better, Backend and front end separated.

Or, if you don’t suggest MythTV, What do you suggest?

Bash Scripts.

I run a web server that contains multiple websites and multiple SQL databases. I decided to use bash scripts to manage my backup’s and for any file restores that were needed. I know the argument will be, “Why bash scripts? there are software options that do the same thing”. The answer is, I couldn’t find one that fit my specific needs and so I decided to write something that fit my needs.

I read several forum threads of people trying to accomplish parts of what I wanted, but nobody had meshed everything together. If you are looking for a bash script that does a grandfather-father-son archive rotation of each user directory into it’s own archive and also does the same thing for databases, then these may work for you. The archives are stored in Daily, Weekly and Monthly. The rotation will do a daily backup Sunday through Friday. On Saturday a weekly backup is done giving you four weekly backups a month. The monthly backup is done on the first of the month rotating two monthly backups based on if the month is odd or even.

I hope you find these as useful as I did.

So this first script I found on the Ubuntu server Archive page.

This one archives the specified files/directories

#!/bin/bash
####################################
#
# Backup to NFS mount script with
# grandfather-father-son rotation
# of specified directory
#
####################################

# What to backup.
backup_files=”/etc /root”

# Where to backup to.
dest=”/backup/system”

# Setup variables for the archive filename.
day=$(date +%A)
hostname=$(hostname -s)

# Find which week of the month 1-4 it is.
day_num=$(date +%d)
if (( $day_num week_file=”$hostname-week1.tgz”
elif (( $day_num > 7 && $day_num week_file=”$hostname-week2.tgz”
elif (( $day_num > 14 && $day_num week_file=”$hostname-week3.tgz”
elif (( $day_num > 21 && $day_num < 32 )); then
week_file=”$hostname-week4.tgz”
fi

# Find if the Month is odd or even.
month_num=$(date +%m)
month=$(expr $month_num % 2)
if [ $month -eq 0 ]; then
month_file=”$hostname-month2.tgz”
else
month_file=”$hostname-month1.tgz”
fi

# Create archive filename.
if [ $day_num == 1 ]; then
archive_file=$month_file
elif [ $day != “Saturday” ]; then
archive_file=”$hostname-$day.tgz”
else
archive_file=$week_file
fi

# Print start status message.
echo “Backing up $backup_files to $dest/$archive_file”
date
echo

# Backup the files using tar.
tar czf $dest/$archive_file $backup_files

# Print end status message.
echo
echo “Backup finished”
date

# Long listing of files in $dest to check file sizes.
ls -lh $dest/

 

This one keeps the same concept except that instead of defining which file/directory the script grabs all the directories within /home

#!/bin/bash
####################################
#
# Backup to NFS mount script with
# grandfather-father-son rotation
# of each home directory within
# it’s own archive
#
####################################

# Where to backup to.
dest=”/backup/users”

# Setup variables for the archive filename.
day=$(date +%A)
#folder=$(backup)

# Find which week of the month 1-4 it is.
day_num=$(date +%d)
if (( $day_num week_file=”-week1.tgz”
elif (( $day_num > 7 && $day_num week_file=”-week2.tgz”
elif (( $day_num > 14 && $day_num week_file=”-week3.tgz”
elif (( $day_num > 21 && $day_num < 32 )); then
week_file=”-week4.tgz”
fi

# Find if the Month is odd or even.
month_num=$(date +%m)
month=$(expr $month_num % 2)
if [ $month -eq 0 ]; then
month_file=”-month2.tgz”
else
month_file=”-month1.tgz”
fi

# Create archive filename.
if [ $day_num == 1 ]; then
archive_file=$month_file
elif [ $day != “Saturday” ]; then
archive_file=”-$day.tgz”
else
archive_file=$week_file
fi

# Print start status message.
echo “Backing up This may take a few minutes.”

# Backup the files using tar.
for folder in $(ls /home); do
sudo -u $folder tar czf “$dest/$folder$archive_file” /home/”$folder”

# Print end status message.
echo
echo “Backup $folder complete.”
#date
done

# Long listing of files in $dest to check file sizes.
ls -lh $dest/
echo
echo “Backup is complete”
exit

 

Now, if you multiple SQL databases on your sever, this will benefit you a lot.

#!/bin/bash
####################################
#
# Backup to NFS mount script with
# grandfather-father-son rotation
# of all your SQL Databases.
#
####################################

# Where to backup to.
dest=”/backup/sql-backup”

# Setup variables for the archive filename.
day=$(date +%A)

# Find which week of the month 1-4 it is.
day_num=$(date +%d)
if (( $day_num week_file=”-week1.sql.tgz”
elif (( $day_num > 7 && $day_num week_file=”-week2.sql.tgz”
elif (( $day_num > 14 && $day_num week_file=”-week3.sql.tgz”
elif (( $day_num > 21 && $day_num < 32 )); then
week_file=”-week4.sql.tgz”
fi

# Find if the Month is odd or even.
month_num=$(date +%m)
month=$(expr $month_num % 2)
if [ $month -eq 0 ]; then
month_file=”-month2.sql.tgz”
else
month_file=”-month1.sql.tgz”
fi

# Create archive filename.
if [ $day_num == 1 ]; then
archive_file=$month_file
elif [ $day != “Saturday” ]; then
archive_file=”-$day.sql.tgz”
else
archive_file=$week_file
fi

# Backup the files.
MYSQL=’/usr/bin/mysql’

MYSQLDUMP=’/usr/bin/mysqldump’
DUMPOPTS=’–opt –hex-blob –skip-extended-insert’

user=”CHANGEME”
pass=”CHANGEME”
# Get the names of the database tables
databases=`$MYSQL -u$user -p$pass –skip-column-names -e’SHOW DATABASES’`

# Write the compressed dump for each table
for db in $databases; do
filename=`date +”$dest/$db$archive_file”`
echo “creating $filename”
$MYSQLDUMP $DUMPOPTS -u$user -p$pass –database $db
| gzip -9 > $filename

done

echo “Backup of SQL Datases Complete”
exit

 

I currently have not written a restore for the SQL backup, if you have phpMyAdmin, you can use the import function.

Okay, so we have backed up all the home directories. What if someone needs something specific from a specific backup. I know that this script doesn’t account for a failure. I’m working on that.

#!/bin/bash
##############################
#
# Script written by Dan Walker
# for Merval.Org Hosting to
# restore a specific file
# from a specific backup.
#
##############################

# Specify Backup Directory
echo -n “Where are we restoring from? (default is /backup/users): ”
while read -e inputline
do
backup_path=”$inputline”
# Display what user typed
if [ -z “${backup_path}” ]
then
echo “You didn’t type anything”
backup_path=”/backup/users”
echo “Using $backup_path”
else

if [ -n “${backup_path}” ]
then
backup_path=”$inputline”
echo “Using custom location: $backup_path”
fi
fi

# Lets ask what user to restore
echo -n “Which user?: ”
read -e user

echo -n “What are we restoring? (leave out /home/): ”
read -e source

# Now lets figure out what backup to restore
echo -n “Which backup? (Daily, Weekly or Monthly): ”
read -e choice
if [ $choice = “Daily” ];
then
echo -n “Which day? (Sunday – Friday): ”
read -e date
echo “You chose $date”
echo “Starting restore process. This may take a moment”
cd $backup_path
sudo -u $user tar -xzf $user-$date.tgz -C / home/$user/$source
echo “Restored /home/$user/$source from the $date backup”
fi

if [ $choice = “Weekly” ];
then
echo -n “Which week? (1-4): ”
read -e week
echo “You chose to restore to $week(s) ago”
echo “Starting restore process. This may take a moment”
cd $backup_path
sudo -u $user tar -xzf $user-week$week.tgz -C / home/$user/$source
echo “Restored /home/$user/$source to the back from $week week(s) ago”
fi

if [ $choice = “Monthly” ]
then
echo -n “Which Month? (1 or 2): ”
read -e month
echo “You chose to restore to $month(s) ago”
echo “Starting restore process. This may take a moment.”
cd $backup_path
sudo -u $user tar -xzf $user-month$month.tgz -C / home/$user/$source
echo “Restored /home/$user/$source from $month month(s) ago”
fi

exit
done

 

Suggestions are always appreciated!

Thanks for reading!

Been Nerdin’ Out..

So, I recently went on a bit of a nerd binge.. I found a Linksys 54gWRT Wireless Router and loaded DD-WRT on it.. I then found a Belkin F5D7231-4 and force loaded DD-WRT on it through TFTP.. I think I’m hooked LOL.. I have always enjoyed making things function in ways they weren’t meant to. I rooted my new phone, a Samsung Galaxy S 4G (still waiting for Cyangonmod… *taps foot impatiently*) and I loaded several apps on it that give me more control 🙂

I am currently in the process of making a Clone of a hard drive that I have to a larger drive using the UBD (Ultimate Boot Disk).. I’m hoping it copies the boot sector too, that would make life SO easy. I converted my main desktop to Ubuntu and rather than dual boot it, I figured build a whole other computer and have a dual monitor that I can switch over with a KVM switch (in case you don’t know what a KVM is, it allows two computers to be hooked together through a device so you can swap between the two without swapping cables) so I can have an Ubuntu and Windows Box running at the same time and I can just hop between the two as I need.

Having both of those computers down, I am using an old Mac G4 Tower to write this blog, My Macbook is in the other room and I’m too lazy to go get it. Meh.. This old device needs to be upgraded soon. I saw a G5 PPC at Free Geek awhile back for $160 dollars but didn’t have the money to get it, I so wouldn’t mind having one of those.. it was a dual 2 ghz chip, When i was there not long ago they only had the big brother to the one i have right now, with no G5s… Oh well, maybe next time.

Upon getting my little DD-WRT addiction, I also bought up several routers (which aren’t too old, but they aren’t REALLY new either) and figured I would try and sell them on Ebay or CL.. Well that didn’t pan out.. So now I have several routers just sitting.. So if you know someone who needs a router lol let me know 😉

The Belkin I put DD-WRT on, i don’t know that I will sell it, I might set it up as a wireless booter for networks that are a little out of my range. DD-WRT has a neat feature that you can connect out to a wireless access point and dupe it so you can connect to it from further away than normal (other wise known as a Signal Booster).. So i’m curious how that will work out. 🙂

I bought the belkin for 9 dollars at goodwill, made into a 600 dollar router 😉 and i bought the Linksys for 5 dollars at Value Village 😉 made them both into 600 dollar routers lol..

 

Alright, I’m gonna take off and go finish that disk clone.

Don’t forget, WWE Hell In a Cell is tonight on PPV. I will have a post on that tomorrow.

Take care!

Dan