Backing up Linux server to Windows server?

I need to back up a (virtual) Ubuntu server. The backup media (an external USB disk) is mounted on the Windows (Hyper-V Server) host. The Windows servers on the same host can simply back up using Windows Backup over SMB.

How should I go about backing up the Linux box, given that it'll end up on an NTFS-formatted disk?

Update

I'm not so sure that Samba will work -- it won't preserve symlinks, devnodes, permissions, etc. Similarly, it's not going to preserve filename cases and other odd characters.

I'd like it to be full-fidelity, so that I can use it for disaster-recovery...

From serverfault Roger Lipscombe

• Installing (and configuring) Samba will allow you to back up the Linux host just like your other Windows servers.

  From Matt

• You only need the samba client on Ubuntu - most likely this is already installed. Mount the share with:

  sudo mount -t cifs //netbiosname/sharename /media/sharename -o username=winusername,password=winpassword,iocharset=utf8,file_mode=0777,dir_mode=0777
  

  You can then perform your backup using cp, rsync, or your program of choice.

  Alternate: If you can't use samba, you may want to just create a file on the samba filesystem (or directly, since the USB Drive is on the host, you should be able to access it in your VM) then create a "loopback" filesystem on the drive to do your backup.

  dd if=/dev/zero of=/tmp/test-img bs=1024 count=10000000 # 10G filesystem
  mkfs -t ext3 -q /tmp/test-img
  mkdir /mnt/image
  mount –o loop /tmp/test.img /mnt/image
  

  From Dave Drager

• You're going to use Samba as the transport-I suggest using tar to create the actual backup. Tar will honor the symlinks and store the permissions as well.

  HERE is a good website I use for understanding tar as a backup program

  From Josh Budde

• If you make a tarball of your data first, and backup the tarball to the cifs mount like described in other answers, you'll preserve symlinks and whatnot.

  From wzzrd

removing duplicate lines from file with grep

I want to remove all lines where the second column is 05408736032.

0009300|05408736032|89|01|001|0|0|0|1|NNNNNNYNNNNNNNNN|asdf|
0009367|05408736032|89|01|001|0|0|0|1|NNNNNNYNNNNNNNNN|adff|

From serverfault

• Is it that you want to remove all lines where the second | separated field contains '05408736032'? Will all the lines be formatted the same? If so, this should output the file minus those lines (it's perl that takes the original file as the first argument and the file it's going to as the second).

  #!/usr/bin/perl
  use warnings;
  use strict;
  my  ($file1, $file2) = @ARGV;
  open my $origin_file, '<', $file1;
  open my $newfile, '>', $file2;
  while (my $line = <$origin_file>) {
      my @values = split '/|/', $line;
      print $newfile $line unless $vaules[1] = '05408736032';
  }
  close $newfile or die $!;
  close $origin_file or die $!;
  

  (I haven't tested this, so you probably want to backup the original file before you try it)

  On reading again, you may be looking to grab only lines with a unique second column. This should do that.

  #!/usr/bin/perl
  use warnings;
  use strict;
  my  ($file1, $file2) = @ARGV;
  open my $origin_file, '<', $file1;
  open my $newfile, '>', $file2;
  while (my $line = <$origin_file>) {
      my @values = split '/|/', $line;
      print $newfile $line unless defined $unique{$values[1]};
      $unique{$vaules[1]} += 1;
  }
  close $newfile or die $!;
  close $origin_file or die $!;
  

  From Cian

• You can do something like:

  for f in `cat $file`; do 
    val=`echo $f | cut -d\| -f 2`
    if [ `grep $val $file | wc -l` -lt 2 ]; then
       echo $f
    fi
  done
  

  but, like most shell scripts, it's pretty inefficient. You'd be better off doing it in perl, something like:

  @infile=<>;
  
  foreach (@infile) {
  
    @foo = split(/|/);
    if exists $found{$foo[1]} {
      $found{$foo[1]}++;
    } else {
      $found{$foo[1]}++;
    }
  
  }
  
  foreach (@infile) {
    @foo = split(/|/);
    if ($found{$foo[1]} < 2) {
      print $_;
    }
  }
  

  From pjz

• awk -F \| '{if ($2 != 05408736032) print}'
  

  Dennis Williamson : You can leave out the "if" and the "print": `awk -F \| '$2 != "05408736032"'`
  From SergeyZh

• This might do what you want:

  sort -t '|' -k 2,2 -u  foo.dat
  

  However this sorts the input according to your field, which you may not want. If you really only want to remove duplicates, your best option is Perl:

  perl -ne '$a=(split "\\|")[1]; next if $h{$a}++; print;' foo.dat
  

  From wallenborn

• Pure Bash:

  oldIFS=$IFS
  while read line
  do
      IFS=$'|'
      testline=($line)  # make an array split according to $IFS
      IFS=$oldIFS       # put it back as soon as you can or you'll be sooOOoorry
      if [[ ${testline[1]} != "05408736032" ]]
      then
          echo $line
      fi
  done < datafile
  

  From Dennis Williamson

Anyone running Cisco CMS plugin 1.2 on a modern system?

I have a Cisco 3750 that I am trying to use the web interface for. It wants me to download and install a CMS plugin version 1.2 that appears to have been built in 2004. The install of the plugin works, but when I try to run the plugin, I get:

The details: Cisco CMS v1.2 plugin, IE 8, Windows XP/Pro, Java 6 update 16.

Any hints would be appreciated.

Update: I get the same error when trying to run it in Firefox on the same system, so I'm thinking it is a Java issue.

From serverfault David Mackintosh

• Generally with Cisco and their propensity for using Java, try back-revving your installed version of Java to version 4 and try it. If version 4 doesn't work, try version 5. Ugh.

  Farseeker : You should see their internal app called C3. I was (un)lucky enough to be there at its launch, even though it was better than their old app, it's still an absolute horror that likely deserves to be on thedailywtf.com

  Farseeker : (It's a monolithic java app, seeing as how I forgot to say how that was relevant)

  David Mackintosh : Nasty java versioning FTW.
  From GregD

How do I enable mutual SSL in IIS7 with a self-signed certificate?

I've created a self-signed certificate in IIS7. Then I exported this certificate to a .pfx and then installed it on the client machine's IE browser. Then I set "Require Client Certificate" on the server's IIS configuration. When I try to visit the site with IE, a dialog box comes up for me to choose a certificate, however, there are no certs in that dialog box. When I click "OK" without choosing any certs, I get a 403 forbidden error. How can I make this work? Appreciate the help in advance.

From serverfault $pageUsers[$entry.posts[0].author].name

• Using Microsoft Management Console, add the certificate manager snap-in and choose CurrentUser. Then import the certificate into the CurrentUser->Personal store.

  Restart IE and you should now see the certificate in the list

  From Wayne

• Chances are the SSL only contains the Server Extended Key Usage (EKU) and not the client EKU...

  From Dscoduc

A route misbehaving, blocking only *DNS* to other networks

I have a PC with two NICs, one is connected to a LAN (eth0, static IP 192.168.0.254), another to a DSL modem in DMZ mode (eth1, receives public IP from modem).

Yesterday, it suddenly stopped working for accessing the Internet.

I've narrowed down the problem to this (or maybe this is just a side-effect, I'm not sure):

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.0.0     *               255.255.0.0     U     1      0        0 eth0
link-local      *               255.255.0.0     U     1000   0        0 eth0
default         <public_ip>     0.0.0.0         UG    0      0        0 eth1

Edit: By public_ip I don't mean the actual public IP assigned to this machine, but another public IP, which I guess it's the one assigned to the modem.

With the default routes as above, I can ping to IPs but I cannot resolve domains, so it seems DNS is blocked somehow or maybe it's trying to use the DNS server from eth0.

If I delete route 192.168.0.0, then instead of the public_ip it shows the FQDN. And then I can resolve domains and access the Internet just fine.

If I assign another computer as the DMZ node in the modem, it works just fine, so it has to be something with this PC. I even tried another NIC for eth0, but no dice.

Any ideas?

From serverfault Ivan

• Whats in your /etc/resolv.conf? Could it be trying resolve using something in 192.168.0.0/16?

  Ivan : namserver 127.0.0.1
  From Craig

• I'm not sure why this problem arised, one moment it was working fine, the next...

  Anyway, the problem seems to be that the modem had an IP address like 192.168.1.254/255.255.255.0 (our internal network is 192.168.0.0/255.255.0.0), and it seems the machine was trying to find 192.168.1.254 inside our LAN (whyyyy!?!?).

  I didn't notice it before because of the public IP assigned to the machine (i.e., not 192.168.1.x).

  So I changed the IP used by the modem, and it now works.

  I'm still wondering what the hell happened here. The best explanation I can find is that our ISP updated the modem's firmware without our knowledge, and this somehow interfered with how it was working before.

  From Ivan

• Based on your additional answer:

  This was happening because the 192.168.1.254/24 address used by your modem is also within the 192.168.0.0/16 address ranged used on your internal network.

  As you had no specific routing table entry for 192.168.1/24 your PC used the best route it had - the one to 192.168/16.

  From Alnitak

ssh failed logins /var/log/btmp

On a VPS where I host some blogs /var/log/btmp file is fairly old - but is at 6.2 gigs.

I assume this means a lot of failed login attempts? Is this common over the course of a year ? Bots trying to get server access?

From serverfault Hendler

• I'm not familiar with the system which keeps it's ssh login info in that file (mine is in /var/log/authlog) but yes, automated attempts to log into ssh is a common part of what I consider the "background noise" of the internet. Often changing the port ssh listens on can cut this log clutter down considerably, though it's important not to confuse that with making your server more secure from a purposeful entry attempt.

  Josh Brower : +1 for cutting down on background noise, and pointing out that it is not Security through Obscurity
  From Devin Ceartas

• If you would like to rotate that log every month you can try add the below code to /etc/logrotate.conf

  /var/log/btmp {
  monthly
  minsize 1M
  create 0600 root utmp
  rotate 1
  }
  

  From Josh Budde

How do you synchronise huge sparse files (VM disk images) between machines?

Is there a command, such as rsync, which can synchronise huge, sparse, files from one linux server to another?

It is very important that the destination file remains sparse. It may be longer (but not bigger) than the drive which contains it. Only changed blocks should be sent across the wire.

I have tried rsync, but got no joy. groups.google.com/group/mailing.unix.rsync/browse_thread/thread/94f39271980513d3

If I write a programme to do this, am I just reinventing the wheel? http://www.finalcog.com/synchronise-block-devices

Thanks,

Chris.

From serverfault

• I'm not aware of such a utility, only of the system calls that can handle it, so if you write such a utility, it might be rather helpful.

  what you actually can do is use qemu-img convert to copy the files, but it will only work if the destination FS supports sparse files

  From dyasny

• Rsync only transfers changes to each file and with --inplace should only rewrite the blocks that changed without recreating the file. From their features page.

  rsync is a file transfer program for Unix systems. rsync uses the "rsync algorithm" which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand.

  Using --inplace should work for you. This will show you progress, compress the transfer (at the default compression level), transfer the contents of the local storage directory recursively (that first trailing slash matters), make the changes to the files in place and use ssh for the transport.

  rsync -v -z -r --inplace --progress -e ssh /path/to/local/storage/ \
  user@remote.machine:/path/to/remote/storage/
  

  I often use the -a flag as well which does a few more things. It's equivalent to -rlptgoD I'll leave the exact behavior for you to look up in the man page.

  : The '-S' is for sparse files, not 'chops long lines'. From man page: -S, --sparse handle sparse files efficiently. I'll give this a try, thanks.

  wizard : Thanks I fixed that - I Was going off of something that was said in the link you gave.

  : No, unfortunately this does not solve the problem. It *does* sync the file, but it turns the sparse file at the far end into a non-sparse file. I am using ssh/rsync which comes with Ubuntu 9.04.

  : My above comment was incorrect. The problem was that rsync creates non-sparse files on its first copy. The --inplace rsync does work correctly, provided that the destination file already exists and is as long (not big) as the origin file. I now have a solution, but it requires me to check whether each file already exists on the target server. If it does, I do an --inplace, if it doesn't, I use --sparse. This is not ideal, but it works.

  : Solved - http://www.finalcog.com/rsync-vm-sparse-inplace-kvm-vmware
  From wizard

• Take a look at Zumastor Linux Storage Project it implements "snapshot" backup using binary "rsync" via the ddsnap tool.

  From the man-page:

  ddsnap provides block device replication given a block level snapshot facility capable of holding multiple simultaneous snapshots efficiently. ddsnap can generate a list of snapshot chunks that differ between two snapshots, then send that difference over the wire. On a downstream server, write the updated data to a snapshotted block device.

  From rkthkr

• Could replicating the whole file system be a solution? DRBD? http://www.drbd.org/

  : I don't think drbd is a good solution here, but the idea of rsyncing --inplace the whole fs, rather than the disk-image-files, is interesting. I'm not sure whether rsync allows this - I'll give it a try and report back...
  From James C