Basic OpenVPN setup not working

I am attempting to connect 2 win7 (x64+ x32) computers (there will be 4 in total) using OpenVPN. Right now they are on the same network but the intention is to be able to access the client remotely regardless of its location.

The Problem I am having is I am unable to ping or tracert between the two computers. They seem to be on different subnets even though I have the mask set to 255.255.255.0. The server ends up as 10.8.0.1 255.255.255.252 and the client 10.8.0.6 255.255.255.252. And a third ends up as 10.8.0.10. I don't know if this a Windows 7 problem or something I have wrong in my config. Its a very simple set up, I'm not connecting two LANs.

this is the server config (removed all the extra lines because it was too ugly)

port 1194
proto udp
dev tun
ca keys/ca.crt
cert keys/server.crt
key keys/server.key  # This file should be kept secret
dh keys/dh1024.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
client-to-client
duplicate-cn
keepalive 10 120
comp-lzo
persist-key
persist-tun
status openvpn-status.log
verb 6

this is the client config

client
dev tun
proto udp
remote thisdomainis.random.com 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca keys/ca.crt
cert keys/client.crt
key keys/client.key
ns-cert-type server
comp-lzo
verb 6

Is there anything I missed in this? keys are all correct and the vpn's connect fine, its just the subnet or route issue.

Thank You

EDIT

it seems on the server the openvpn-status.log has the routes for the client

SERVER

OpenVPN CLIENT LIST
Updated,Wed May 19 18:26:32 2010
Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
client,192.168.10.102:50517,19157,20208,Wed May 19 17:38:25 2010
ROUTING TABLE
Virtual Address,Common Name,Real Address,Last Ref
10.8.0.6,client,192.168.10.102:50517,Wed May 19 17:38:56 2010
GLOBAL STATS
Max bcast/mcast queue length,0
END

Also this is from the client.log file: Which seems to be correct

C:\WINDOWS\system32\route.exe ADD 10.8.0.0 MASK 255.255.255.0 10.8.0.5 

Another EDIT

'route print' on the server shows the route:

Destination Mask      Gateway   Interface   
10.8.0.0 255.255.255.0 10.8.0.2 10.8.0.1 

the same on the client shows

10.8.0.0 255.255.255.0 10.8.0.5 10.8.0.6  

So the routes are there.. what can the problem be?

Is there anything wrong with my configs?

Why would OpenVPN be having problems communicating?

From serverfault WalterJ89

• Hi, i have the same problem, trying to find a soluction...

  Giovanni : OK, ready for me, reinstalling openvpn, using the same parameters, sundely working!
  From Giovanni

Computers "applying computer settings" for a long time on start-up...why?

Hello. Might be a bit of a long shot but I'm stumped, along with the ICT Manager for the school I'm working it.

In one of the IT rooms when you switch a computer on it will boot through BIOS fine, but when it gets to "applying computer settings" it can hang for a long time (~15 minutes). If you unplug the computer from the network it starts up fine, gets to the login screen, then you can plug the network cable in and it will work fine.

I don't think it's anything to do with the fact we've been coming close to running out of IP addresses or a problem with our DHCP. Microsoft KB says apply the latest service pack, which we've done, and check a service.

Servers and domian controllers are S2003, Computers and Desktops are XP.

Does anybody have any thoughts on what to try?

EDIT: The logs have shown a problem with folder re-direction for mapping My Documents to the "U" drive.

From serverfault tombull89

• This happened to us once, all we needed to do was remove the affected machine from the domain, and then add it back.

  tombull89 : well, I've done that and it starts up as quick as you like and gets to the login box no problem...now it just takes ages to log in >_<

  Bart Silverstrim : For you to log in, or everyone? Could be profile issues. Or it could be a network communication issue; test the network pathway to see that there isn't a problem with the line. Check the logs to see if anything suspicious is logged. Test the hard disk to rule out any issues with files, although that probably isn't it...check on the domain controller's logs to see if anything is there. How big are the profiles, if they're roaming?

  tombull89 : everybody had problem, not just me - the profiles arn't roaming (don't ask why, decision was made before I got here) - I'm doubtful its the the line as I've moved it from the room to the systems office and it's not happy. I'm checking the logs at the moment.
  From Shrinivas

• Its usually a misconfiguratied DNS address in your TCP/IP settings. Check that the PC hasn't been configured to use a static IP address and that that the DNS address is correct.

  Farseeker : Back in my early days of playing with AD when I didn't understand DNS, this was ALWAYS the issue
  From Jason

• I've seen this happen a bunch of times and it can be a real pain to diagnose at times. Just today I had the problem with one of the managers' laptops. In this instance the problem was caused by drive mappings to targets that no longer exist. After deleting the offending shortcuts and relevant registry entries the startup time went from 10+ minutes to about 1 minute.

  tombull89 : I think it may be something to do with drive mapping as shown in the logs, but as it's mapping My Document folders I'm surprised It would have a problem. I'll check.
  From John Gardeniers

• At a high level, this is almost always called by trying to contact a server/service that's not answering. Even in 2010, Windows' networking timeouts are insanely long and likes to retry.

  The trick is finding what's causing the failed lookup. DNS is an obvious one, but would tend to affect > 1 client. GPOs are also suspect, so enabling the debug log is a good start. Look for suspicously large time stamps between lines.

  When all else fails, a sniffer will show you the failed connections. You still have to figure out why it's trying to connect there (or why there isn't accessible).

  Chris S : Windows 2010???

  tombull89 : I think he means in the year 2010; i.e. he's surpised ms hasn't reduced it. Thanks for th kb article, I'll have a read of that.
  From Mark Brackett

Does smbfs create constant inode numbers?

If I use smbfs to mount a filesystem, will the inodes always be the same for each file? Or am I in danger of the inodes changing when I mount the fs again or if the remote system reboots?

From serverfault $pageUsers[$entry.posts[0].author].name

• No. smbfs like other non-posix filesystems use a memory cache to try to maintain a stable database of inodes for files. If that cache is flushed or filled, it will loose all information of what inode corresponds to a file. That means reboots, but also can happen without a reboot, for instance check the following discussion: http://www.mail-archive.com/bug-findutils@gnu.org/msg01212.html

  From jneves

IIS 6.0 subdomains with host headers and non existent subdomains

Hey Everyone -

We have a wildcard A-Record pointing to our IP and have a number of sites running on IIS 6 with host headers and have a a wildcard SSL certificate for the domain so that each site can run under SSL.

For example: https://A.foo.com https:/B.foo.com https:/C.foo.com

Everything is working well but I noticed that when we type a non existent subdomain, say D.foo.com, it redirects to A.foo.com. Any idea why that is or how I can change that? I think we may have set up the A.foo.com site before we applied the wildcard A-record with our domain provider and before we had set up the SSL cert.

Thanks.

From serverfault Mustafakidd

• The default configuration of the default web site in IIS is to not filter on host headers. This web site will receive any requests that do not match one of the other web sites.

  In its default state IIS 6's default web site will receive all requests to the server (by IP address) whatever the host header contains.

  From Richard

Multiple logins with pam_mount means multiple (redundant) mounts ...

I've configured pam_mount.so to automagically mount a cifs share when users login; the problem is if a user logs into multiple times simultaneously, the mount command is repeated multiple times.

This so far isn't a problem but it's messy when you look at the output of a mount command.

# mount
/dev/sda1 on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
none on /dev type devtmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
//srv1/UserShares/jrisk on /home/jrisk type cifs (rw,mand)
//srv1/UserShares/jrisk on /home/jrisk type cifs (rw,mand)
//srv1/UserShares/jrisk on /home/jrisk type cifs (rw,mand)

I'm assuming I need to fiddle with either the pam.d/common-auth file or pam_mount.conf.xml to accomplish this.

How can I instruct pam_mount.so to avoid duplicate mountings?

---

[Edit]

The contents of my pam_mount.conf.xml file:

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE pam_mount SYSTEM "pam_mount.conf.xml.dtd">
<pam_mount>
  <debug enable="1" />
  <volume user="*" server="srv1" path="UserShares" mountpoint="home" fstype="cifs" />
  <cifsmount>mount -t cifs //%(SERVER)/%(VOLUME)/%(USER) %(MNTPT)/%(USER) -o "user=%(USER),uid=%(USERUID),gid=%(USERGID)%(before=\",\" OPTIONS)"</cifsmount>
  <umount>umount %(MNTPT)/%(USER)</umount>
  <mntoptions allow="nosuid,nodev,loop,encryption,fsck,nonempty,allow_root,allow_other" />
  <mntoptions require="nosuid,nodev" />
  <path>/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin</path>
  <logout wait="0" hup="0" term="0" kill="0" />
  <mkmountpoint enable="1" remove="true" />
</pam_mount>

From serverfault Jamie

• "Multiple simultaneous logins" is probably the key. More than likely, what's happening is that the second and subsequent mount commands are getting launched prior to the first mount command finishing. This seems very likely, given how slow network mount commands run. What you probably need is some sort of shared memory / state file / etc which can make sure that only one mount process will start up. Well, at least until the pam_mount author works in a long-term fix for that race condition... :)

  You might look at the pam_tally module. You could use that module to maintain a login counter per-user, and deny if the count is over 1. In the control field, then, you could set it up so that the pam_mount module is skipped if pam_tally fails. Specifically, I think maybe something like this would work:

  auth [success=ignore default=1] pam_tally.so deny=1 onerr=succeed no_lock_time no_reset
  auth optional pam_mount.so pam_mount_options
  

  ...Or something along those lines. The am_tally2 module would also work, if you need some external system to also manipulate the counter, say, when you manually unmount a filesystem or something (since pam_tally2 comes with a binary that can be used to manipulate the counts).

  dannysauer : Or you could just symlink /etc/mtab to /proc/mounts, since /proc/mounts won't have dups. ;)

  Jamie : I tried `pam_tally.so`: most likely a syntax thing but I couldn't get to work.
  From dannysauer

• The offending line was in my pam_mount.conf.xml file:

  <mkmountpoint enable="1" remove="true" />
  

  should be:

  <mkmountpoint enable="1" remove="false" />
  

  With the value set to true, the pam_mount.so module was trying to remove /home/$USER/ from the system, not, as I assumed, ./$USER in the /home/ directory.

  Jamie : I was mistaken, this didn't to correct the problem above.
  From Jamie

• I suspect that pam_mount is failing to unmount the directory. Could you please confirm if the directory remains mounted after the user logged out with a mount?

  If that's the case, the only solution I know of is using pam_script to run a umount -l /home/$USER on session close.

  Jamie : Thanks for the suggestion. I find again that I'm impressed with all the optional libraries PAM supports. This library strikes me as a little inelegant and circumventing the issue rather than addressing it.

  jneves : I agree. I ended up using pam_script handling both mount and umount in a custom script done by me.
  From jneves

• Why not use autofs?

  From Warner

How can I capture output from LFTP? (Output not written to STDOUT or STDERR?)

I would like get access to progress information from lftp. Currently, I'm using curl like so:

curl http://example.com/file -o file -L 2> download.log

This writes curl's progress information to the download.log file, which I can tail to get real-time progress.

But the same approach doesn't work with lftp, either with stdout or stderr. I end up with an empty download.log file, until the transfer is complete.

lftp -e 'get http://example.com/file;quit' 2> download.log
lftp -e 'get http://example.com/file;quit' 1> download.log

When I don't redirect output, I see progress on the screen. When I do redirect output, I stop seeing progress on the screen, but nothing shows up in download.log. After the file transfer is complete, I see the final result, like this - but nothing before:

97618627 bytes transferred in 104 seconds (913.1K/s)

Is lftp doing something unusual with its output - printing to screen without printing to stdout/stderr? Are there other ways of capturing screen output than redirecting stdout/stderr?

From serverfault jondahl

• It sounds to me like it's buffering its output. You might try the unbuffer expect script (man page).

  jondahl : That was exactly it. Thanks!
  From Dennis Williamson

• Check the xfer domain variables :

  set xfer:log 1

  set xfer:eta-period 5 # every 5 seconds

  set xfer:rate-period 20 # average rate

  will put log transfer information into ~/.lftp/transfer_log

  Not sure you can change the log file destination nevertheless

  From juj

X server for windows

Hi everyone,

I'm looking to run cygwin xterm on my machine. For this I need an X server, which one would you recommend?

It's important that it won't have too much memory leaks and that it would be fast enough to allow reasonable terminal based work on the PC.

Thank you, Maxim.

From serverfault Maxim Veksler

• How about Cygwin/X?

  Maxim Veksler : I see there are 2 alternatives Cygwin/X and Xming. Do you know which one is lighter in terms of cpu/memory consumption?

  Andrejs Cainikovs : Xming is simple and easier to install though less configurable than Cygwin/X. It also has the advantage of more active development. On the other hand, Cygwin/X is much closer port of X for Windows with all the possible features.
  From Andrejs Cainikovs

• When I was on MS Windows XP, I found Xming to be quite stable, manageable and not resource-hungry.

  From ΤΖΩΤΖΙΟΥ

A space-efficient guest filesystem for grow-as-needed virtual disks ?

A common practice is to use non-preallocated virtual disks.

Since they only grow as needed, it makes them perfect for fast backup, overallocation and creation speed.

Since file systems are usually based on physical disks they have the tendency to use the whole area available¹ in order to increase the speed² or reliability³.

I'm searching a filesystem that does the exact opposite : try to touch the minimum blocks need by an aggressive block reuse.

I would happily trade some performance for space usage.

There is already a similar question, but it is rather general. I have very specific goal : space-efficiency.

<sub>1. Like page caching uses all the free physical memory
2. Canonical example : online defragmentation
3. Canonical example : snapshotting</sub>

From serverfault Steve Schnepp

• you can also use LVM volumes, adding PVs as necessary to a VG in order to grow it, and using lvextend to grow LVs dedicated to VMs

  Ignacio Vazquez-Abrams : But none of those is a filesystem.

  dyasny : That's why the word "also" is used. This approach allows better grained space management. The actual FS is in the VM anyway, not necessarily on the host

  Steve Schnepp : @dyasny: the main purpose of the *grow-as-needed* is specifically to avoid this kind of space micro-management.

  dyasny : actually, vdsm does exactly that - lvextend as required.
  From dyasny

• If you don't mind being on the edge, you can do this with the new btrfs filesystem. The backup target device will occupy no more space than is used by the copied data.

  https://btrfs.wiki.kernel.org/index.php/Main_Page

  1. Create a partition (using LVM logical volume or regular)
  2. Format: mkfs -t btrfs
  3. Mount it: mount -t btrfs /mnt/btrfs
  4. Create a brts sub-volume: btrfsctl -S home_template /mnt/btrfs
  5. Umount /mnt/btrfs and mount the subvol: mount -t btrfs -o subvol=home_template /mnt/template
  6. Populate /mnt/template with files/folders if needed in the backup target
  7. Make a snapshot of the home_template subvol for each user. This occupies 0 space until mounted and written to. btrfsctl -s /mnt/joeblow /mnt/template 8 Mount the snapshot as needed and let the backups begin: mount -t btrfs -o subvol=joeblow /mnt/backup
  Steve Schnepp : Is it for the guest fs also ?
  From

Windows Server task manager displays much higher memory use than sum of all processes' working set size

I have a 16 GB Windows Server 2008 x64 machine mostly running SQL Server 2008. The free memory as seen in Task Manager is very low (128 MB at the moment), i.e. about 15.7 GB are used. So far, so good.

Now when I try to narrow down the process(es) using the most memory I get confused: None of the processes have more than 200MB Working Set Size as displayed in the 'Processes' tab of Task Manager. Well, maybe the Working Set Size isn't the relevant counter?

To figure that out I used a PowerShell command [1] to sum up each individual property of the process object in sort of a brute force approach - surely one of them must add up to the 15.7 GB, right? Turns out none of them does, with the closest being VirtualMemorySize (around 12.7 GB) and PeakVirtualMemorySize (around 14.7 GB). WTF?

To put it another way: Which of the numerous memory related process information is the "correct" one, i.e. counts towards the server's physical memory as displayed in the Task Manager's 'Performance' tab?

Thank you all!

[1] $erroractionpreference="silentlycontinue"; get-process | gm | where-object {$.membertype -eq "Property"} | foreach-object {$.name; (get-process | measure-object -sum $_.name ).sum / 1MB}

From serverfault Sleepless

• Adding up the memory usage of all processes will not generally produce meaningful results. That will leave two major users of memory unaccounted for, the system cache and the standby list. You cannot account for memory usage by simply adding up a list of numbers. The memory management system is far too complex for that.

  Sleepless : "The memory management system is far too complex for that." - you're right about that... I now resorted to reading the "Memory Management" Chapter from Windows Internals, 4th ed. and I now understand I was seeing things too simplistically...
  From Larry Miller

• Adding up working sets also multi-counts pages that are shared across processes, like code from system DLLs and executables started more than once.

  From afrazier

• SQL-Server uses on 64-bit AWE for locking the memory-pages and the amount is not counted in taskmanagers "The Working Set".

  TomTom : ah - no, it does not. Not if the SQL Server is 64 bit installed, and seriously only someone challenged with the basics would install a 32 bit SQL Servre on a 64 bit OS. I rather assume this is a 64 bit SQL Server, which does not use AWE.
  From Ice

• Thanks for all the insightful comments - I am still confused about which counter to look for to gauge "memory use", i.e. which of the multiple counters is the most relevant and why?

  From Sleepless

Can you have a WMI query for GPO Filter based on user's OU?

I'm wondering if there is a way to have a WMI query check the OU of the user logging on. I'd like a GPO (linked to Citrix servers OU) to apply only to users if the user is in a certain OU - this is for Citrix so the overly obvious answer of - well just link it to the OU the user is in does not apply. This also cannot be done using security groups because a long time ago those started to get used as Distribution Groups also and now too many are widely inaccurate. Lastly I need to apply this to the entire GPO as there are more than just group policy preferences included so I can't use the item-level targeting feature either. But my OUs are accurate so I'd like to use those if I can. I'd like a WMI query filter to say, apply GPO if user is member of OU 'x'

that doable?

From serverfault Jordan Weinstein

• Group policies are applied to OUs. Filtering can be applied via groups or wmi queries. In your case the best way to solve your problem would be to create another group that contains the users you want to affect via this policy. It is possible to get the information you want via wmi but it's not trivial. See Mapping Active Directory Classes

  From Jim B

• And if dealing with 'Mapping Active Directory Classes' scares you, there is another way. If you can live with a bit of a lag between user-moves and when the GPO applies to them, creating a batch process that maintains a series of groups that indicate which users are in which OU's. For instance.

  grp.ou-members.acct.receivable

  Would contain all the users in the /Acct/Receivable OU. Then you could use a group filter instead of a complex WMI filter. It wouldn't be real-time, unless you have some Identity Management System hooks you can exercise to make it so, but it would do the job. These kind of groups are fairly easy to create and maintain with PowerShell scripting.

  From sysadmin1138

Installing SSL Certificate for use in IIS7, installation "works", but cert listing disappears

Windows Server 2008 R2, IIS7. We have an SSL cert from Go Daddy. It's a wildcard cert, so it will work across subdomains (e.g. *.domain.com). I followed the instructions located at http://help.godaddy.com/topic/742/article/4801 for installing the certificate. I get to the IIS step, where I:

• Click on "Security Certificates" feature when the server is selected in the left pane
• Click on "Complete Certificate Request"
• Navigate to the .crt file on the file system
• Give it a "friendly" name, click finish

The cert gets listed on the main pane now of this "Server Certificates" panel. But, if I refresh the page, or navigate away and come back, it's gone. And the cert is not listed as a viable binding when trying to bind a site to https.

This seems like a pretty straight forward process, but clearly I'm missing something here. Any ideas?

EDIT: I found this post, which seems to imply this behavior happens when you try to use the intermediate certificate. When I downloaded the files from GoDaddy, there were 2 in a zip file. 1 was the gd_iis_intermediates, the other was named for the domain. I installed the domain one (extension .crt). There didn't seem to be any other option - installing the other from IIS gives an error "Cannot find the certificate request that is associated with this certificate file. A certificate request must be completed on the computer where the request was created".

That being said, there doesn't appear to be any other download I can use.

There was also mention, in the comments (and elsewhere after googling) of "exporting" the cert as a pfx, and installing that. But I can't figure out how to export it - even through certmgr.msc.

I should also mention this cert is installed on another computer running IIS6 (this IIS7 installation is meant to be a failover, plus the primary while we upgrade the IIS6 to IIS7). But I can't figure out how to export it from that computer either.

From serverfault Matt

• Try exporting the certificate from the IIS6 server using these instructions: http://www.sslshopper.com/move-or-copy-an-ssl-certificate-from-a-windows-server-to-another-windows-server.html

  That will ensure that the certificate has a private key.

  Matt : The option to export the private key is grayed out, saying it was marked as "unexportable"

  Matt : Come to think of it, the fact it was marked as unexportable is probably why this certificate didn't get migrated over during the msdeploy migration of the server ... hmm

  Robert : If you aren't able to find a server where the certificate is exportable, you will need to generate a new CSR and have GoDaddy reissue/re-key it to get a new matching certificate.
  From Robert

• I'm in the same situation but this is first time we went with Go Daddy and we don't have an existing certificate to export.

  Any help appreciated as we are stuck.

  From West

• The certificate was not exportable, so I was unable to use Roberts suggestion. Ultimately, I had to rekey the certificate at the Go Daddy account management page, and install it on both servers again. Some of the options during the wizard for the install on IIS6 were grayed out for me, and my initial attempt on that server failed. I ended up installing the certificate on the new server (IIS7), and then exporting that certificate in a .pfx format, and then importing that version into the IIS6 installation. At which point everything started working.

  From Matt

• I've found the problem can be reproduced when the leaf certificate has been installed under Intermediate Certification Authorities. Removing it (and leaving any real intermediate, if applicable) then completing the wizard corrects the problem.

  From Ryan Fox

• I ran into this issue as well. Rekeying the cert resolved the issue, but the reason was that I was using a UCC cert, and the SARs had been changed AFTER the cert had last been re-keyed. Re-keying the cert again resolved the issue. I spent 2 hours on the phone with a tech there before I found that out <:(

  From rotard

VPS host can't send email to Google and Yahoo Mail

Hi, I got a new VPS setup and I'm wondering why I can't send emails to yahoo and gmail. Here's the error in /var/log/maillog:

00:43:00 mylamp sendmail[32507]: o45Gh0nc032505: to=, ctladdr= (48/48), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=120405, relay=alt4.gmail-smtp-in.l.google.com. [74.125.79.27], dsn=4.0.0, stat=Deferred: Connection refused by alt4.gmail-smtp-in.l.google.com

What seems to be the problem?

From serverfault mandeler

• Could be your IP got black-listed by a previous owner of your VPS's IP. Common for an un-managed VPS to be compromised by a spammer or other blackhat activity in the wild.

  Edit: Also, make sure your reverse DNS (rDNS) is set up properly, as big SMTP servers will need this to verify you are who you say you are.

  TomTom : THat pretty much sums it up. Use something like http://www.mxtoolbox.com/blacklists.aspx to check whether your server is OK or the IP is burned ;)
  From mtkoan

• We would agree with mtkoan and TomTom...verify that you are not:

  Blacklisted, have correct MX Records and rDNS and lastly, you can use our site to run SMTP Diagnostics which heck for Open Relay, HELO response and more.

  http://mxtoolbox.com

  Let us know if you have any other additional issues or questions.

  Thank you,

  @MxToolBox

  From MxToolBox Support

Does a machine have to be domain joined in order to use Direct Access

Does a machine (e.g., mobile laptop) have to be domain joined in order to use Direct Access (DA)? Or does DA allow the user to provide credentials just like VPN does when setting up DA?

From serverfault Philipp Schmid

• "DirectAccess clients must be members of an Active Directory domain."

  Sources:

  • MS DirectAccess Technical Overview Doc: http://www.microsoft.com/downloads/details.aspx?FamilyID=64966e88-1377-4d1a-be86-ab77014495f4&displaylang=en
  • http://blogs.techrepublic.com.com/10things/?p=1371
  Philipp Schmid : Thx. Does it have to be the same domain as the server providing DA? E.g. can consultants use DA from their machine which might be joined to their consulting companies AD rather than the customer's AD?

  Shaji : The DA and the clients should belong to the same domain.

  iainlbc : Yes they would need to be on the same domain. You might be able to provide access to resources on both domains/networks using one DA Server if there was an existing VPN/trusted sites configuration between the two locations in place, however I am unable to verify or prove that.
  From iainlbc

How can i track all queries in my MySQL server?

Hey everybody!

I'm trying to count every SELECTS, UPDATES, INSERTs querys in my MySQL server. The main idea is to say, at the end of the day, "we had x querys today, being x1 selects, x2 inserts, etc".

How can i do it?

Thanks!!

From serverfault santiago.basulto

• Edit my.cnf and add

  log=/path/to/whatever/file/you.want

  and restart MySQL.

  You will find all the queries logged in the file you specify.

  From lrosa

• Current MySQL versions support two different kinds of query logs:

  First, the general query log tracks all statements executed on the MySQL server, including SELECTs. But it's slow, inefficient, and can easily kill the performance of a production database, so be careful with it.

  Depending on which version of MySQL you're running, you should use either the '--log' option or the '--general-log' and '--general_log_file' options, together, to enable the general query log and control where it writes to. The docs have more details:

  • http://dev.mysql.com/doc/refman/5.1/en/query-log.html
  • http://dev.mysql.com/doc/refman/5.1/en/log-tables.html

  In case the general query log's runtime penalty is too high, you might consider whether the binary log (binlog) could meet your needs, instead. The binlog tracks any statements that can update data or schema, but not read-only statements like SELECT. But the binlog is really fast and disk-efficient, and it can also be used for replication, backups/restores, etc.

  The basic option to enable the binlog is '--log-bin', and there are some other related options that control exactly how it works. The 'mysqlbinlog' utility (part of the core MySQL server installation) can parse the binlog's binary format back into SQL statements, which you can pipe to whatever script or program you use to summarize the queries (e.g., mysqlbinlog <BIN_LOG_FILENAME> | <YOUR_SCRIPT_NAME>. See the relevant doc pages for more info:

  • (binlog docs) http://dev.mysql.com/doc/refman/5.0/en/binary-log.html
  • (mysqlbinlog translation utility docs) http://dev.mysql.com/doc/refman/5.0/en/mysqlbinlog.html
  From Ryan B. Lynch

• logs are all fine and dandy - answers mentioned by Irosa and Ryan are great, but you can do it completly passively - without enabling logging. take a look at mk-query-digest from maatkit. it can analyze not only mysql logs but also traffic captured with tcpdump. and 'for free' it will give you nice breakdown of query types/execution time etc.

  From pQd

• MySQL already maintains internal counters:

  mysqladmin ext | grep -e 'Com_\(update\|select\|insert\)'

  no need for log files/tables/tcpdumping/maatkit.

  santiago.basulto : Thanks man, great answer. I've been looking for the meaning of Com_ satus variables, but i can't find it. What those exactly means?

  ggiroux : http://dev.mysql.com/doc/refman/5.1/en/server-status-variables.html#statvar_Com_xxx -- in short, com_select counts select queries, com_update the updates, etc.
  From ggiroux

• Besides the two mentioned solutions (MySQL query log and mk-query-digest) you might want to take a look at MySQL Proxy which can be used to log the queries sent to your mysqld.

  From joschi

Show full process name in top

I'm running a rails stack on ubuntu.

When I ps -AF, I get a descriptive process name set by the apache module like

00:00:43 Rails: /var/www...

which is really helpful in diagnosing load issues.

But when I top, the same process show up simply as

ruby

Is there any way to get the ps -AF process name in top?

From serverfault Ben K.

• While top is running, you can press c to toggle between showing the process name and the command line. To remember the toggle state for next time, press W to save the current configuration to ~/.toprc.

  Ben K. : thanks, this makes my life so much easier!
  From Phil Ross

• When running top type c to toggle command line/process.

  From Iain

• This is more of a general suggestion, than an answer:

  Try out htop. It shows the full process name by default and I think it's much easier to use.

  From Trey

Server Clustering (Django, Apache, Nginx, Postgres)

I have a project deployed with django, Apache, Nginx and Postgres. The project has requirement of live data viewable to customers. The projects main points are: 1. Devices in field send data to server(devices are also like website users) after login. 2. There is background import process which imports the uploaded data in postgres. 3. The webusers of the system use this data and can send commands to the devices, which devices read when they login. 4. There are also background analysis routines running on the data.

All the above mentioned setup and system is deployed on one amazon EC2 cloud machine. The project currently supports over 600 devices and 400 users. But as the number of devices are increasing with time the performance of the server is going down.

We want to extend this project so that it can support more and more devices. My initial thinking is, We will create one more server like current one and divide the devices amongst these to servers. But Again We need a central user and device managment point though django admin.

Any Ideas? What are the best possible ways to create a scalable architecture? How can I create a Postgres Cluster and Use it with Django, if possible?

From serverfault

• Your question is short on details and long on hand-waving, but it sounds like your initial thinking is a pretty sound start. Your app sounds pretty similar to the Zenoss monitoring suite, which uses essentially the same load-distribution architecture to scale up: Multiple monitoring hosts sharing the data collection workload, with a single admin interface, and a database on either the admin host or a separate system.

  If your bottleneck is at point #1 (devices sending data to your server), splitting those tasks across a second machine should carve out some room for load growth. The biggest implementation obstacle is usually how to manage tasks across multiple Django servers. Celery, a distribued task queue engine, is probably the best option at the moment. It was originally designed around Django, which is good for you, and it has very active and helpful community of developers and users.

  • http://celeryproject.org/

  If points #2 and #4 are your current limitation, though, you're probably talking about database scalability. This is just a hard problem, in general: There is no code-transparent, load-neutral, and cheap way to scale up database capacity.

  If you only need to get more database "read" IO capacity, replication will probably do the trick. Postgres supports replication using an external tool called Slony-I. The is single-master replication, with multiple read-only "slave" hosts that get fed copies of statements executed on the master. All of your app's writes (UPDATE, INSERT, DELETE...) go through the single master host, but you distribute your reads (SELECT...) across the master and all of the slaves.

  • http://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling

  The code modifications needed for distributed reads are usually pretty straightforward. Django recently added support for replicated databases, which I haven't used, but it's supposed to be pretty good.

  • http://code.djangoproject.com/wiki/MultipleDatabaseSupport

  If you need more database write IO capacity, sharding will probably work. Each host keeps a separate, unique chunk of each database table. The DB clients use a deterministic function decides where any given record should reside, so the load distribution is effectively stateless and can scale up to huge numbers of DB servers. Django's new multi-database support (same link as above) also supports sharding. You'll need some code changes, the pain should be limited.

  Also, I want to mention Memcached, which seems to be part of just about every highly scalable web application on the Internet, today (Facebook, Google, Twitter...). A good caching implementation can cut your database requirements to a fraction of their original size, by converting expensive, slow DB lookups into cheap, fast cache lookups. Django has supported Memcached integration for quite a while, now.

  • http://docs.djangoproject.com/en/dev/topics/cache/
  • http://memcached.org/

  I realize none of this is too specific, but it should give you a pretty good starting place for working out the details. Good luck with your project.

  From Ryan B. Lynch

• First you must realize, where is your botellneck? Application layer problem? Data layer access? What is your access pattern? Mostly reads? Or maybe mostly writes?

  For application layer:

  • adding more application servers
  • some actions can be put into job queue without user waiting for finish (eg. commands to devices)

  For data layer there are some ways, that you can follow:

  • Think about your workload? Could you reduce some queries? Could you change your schema? Maybe adding some denormalization (precomputing statistics, aggregating data). For very large tables you probably could add vertical partitioning
  • For read scaling you could use replication, as Ryan B. Lynch
  • Caching with memcached or something similar. But remember: "There are only two hard things in Computer Science: cache invalidation and naming things."
  • I don't reccomend sharding (horizontal partitioning), because managing sharded database is painfull. Here is nice article about sharding.
  • Split your data into different data backends. Here is nice article describing the idea.
  From ms

ServerName wildcards in Apache name-based virtual hosts?

On our LAN I've set up several 'fake' TLDs in the DNS server, with the intention of using them for Apache name-based virtual hosting. I'd like to combine this with mass-virtual-hosting (i.e. VirtualDocumentRoot) on an Ubuntu 10.04 LAMP server.

However, I can't get it to select the right vhost!

Here is a summary of the Apache config:

NameVirtualHost 10.10.0.205

<VirtualHost 10.10.0.205>
   ServerName *.test
   VirtualDocumentRoot /var/www/%-3.0.%-2/test/%1/
   CustomLog /var/log/apache2/access.log vhost_combined
</VirtualHost>

<VirtualHost 10.10.0.205>
   ServerName *.dev
   VirtualDocumentRoot /var/www/%-3.0.%-2/dev/%1/
   CustomLog /var/log/apache2/access.log vhost_combined
</VirtualHost>

A hostname such as www.domain.com.dev, correctly resolves to 10.10.0.205, but always selects the top vhost, instead of the bottom one, which matches more closely.

I was under the impression that Apache would first try to match the ServerName before defaulting to the top vhost for a given IP. What am I doing wrong? Or is this not possible and must I use another IP for each TLD?

apachectl -S outputs (trimmed):

10.10.0.205:*          is a NameVirtualHost
    default server *.test
    port * namevhost *.test
    port * namevhost *.dev

From serverfault Martijn Heemels

• Use ServerAlias, rather than ServerName alone:

  ServerName somename.dev

  ServerAlias *.dev

  Martijn Heemels : Thanks for the quick response. Works like a charm!
  From Mo

FTP Load Balancer

I need an EC2 instance to balance all incoming FTP connections to a list of FTP servers (EC2 instances too). This list will be changed dynamically due to the load of the FTP servers (launch a new FTP server when the FTP servers are overloaded or shutdown a FTP server when the load is low).

What you recommend? a FTP proxy? DNS server? Load balancer?

Note: The FTP servers must support Passive Mode

From serverfault inakiabt

• I haven't had the opportunity (or more so the need) yet to use Amazon's Elastic Load Balancing but I would put it out there as one for you to investigate.

  If it covers your needs then, just as the rest of their services, it takes away from you having to figure out all the pieces they already take care of (maintenance, redundancy of the balancer itself so you don't have single point of failure, etc).

  http://aws.amazon.com/elasticloadbalancing/

  inakiabt : Seems it is not supported: http://developer.amazonwebservices.com/connect/thread.jspa?messageID=139143 http://developer.amazonwebservices.com/connect/thread.jspa?messageID=142839

  ManiacZX : That is too bad, I was hoping down the line to look into if it could handle SIP traffic, from that second post it sounds like that might be held back by the same issues needing more like ip hash load balancing.
  From ManiacZX

• I'd make it simple and use DNS. Get the list of FTP server IPs and add A records for a sub domain like ftp.example.com. Everyone using the domain should get a fairly random server and the load should generally be evenly distributed.

  FTP is probably one of the hardest things to load balance. Unless you have a real need, I'd stick with DNS.

  If you're uploading and downloading files under 5G, you might be better served by using S3.

  From Seth

Adding ftp records as A or CNAME

Is it better to use a A record or CNAME record for FTP.

From serverfault Saif Bechan

• FTP doesn't care. In the end as long as it points to an IP address it's fine.

  From Ignacio Vazquez-Abrams

• For Future Beeing its better to use an CNAME so when you change the Server or the IP Address you only need the Change the A-Record for the IP Adress or the Cname if you switch servers.

  CNAME ftp.mycompany.com server1.mycompany.com A server1.mycompany.com 123.123.123.123

  Saif Bechan : +1 thank you this is a handy tip
  From daBONDi

Why can't European users access my site?

Hello, My site has been running just fine for the past couple of years, but all of a sudden, two days ago, European users have been experiencing serious connection problems to the site. I really want to fix this for them, but what's the best way to figure out what the issue is? I have absolutely no connection problems to the site on my end, nor do Asian or other American users it seems. Using just-ping.com, some European servers come back with some packet losses. I tried doing some traceroutes from European servers to my own, but they all seemed to work just fine.

I'd at least like to be able to tell users that if the problem does not lie with my server, then it at least lies somewhere out of my control. I really want to figure out what the choke point is though. Is there another way I might be able to find out why they can't seem to connect to the site? Just looking for any other ideas from people that have had a similar experience.

From serverfault japancheese

• I would first make sure they they are resolving the IP correctly. If that works, you will want a couple traceroutes from the clients (Or at least their public IPs). You can then give that information to your ISP and they should be able to find out what is wrong. It does sound like it is probably not your server.

  Maybe just-traceroute will be able to show you where the packet loss is happening or places where there are big jumps in latency.

  Izzy : In this case, the *pathping* command would probably be more useful than just a trace
  From Kyle Brandt

• Are they all coming from the same country? This is a long shot, but there are regional differences in Antivirus/Firewall software. Eg: Many UK banking users are given Trusteer Rapport with their banking software. Many Finnish users use F-Secure. Might also be related to browsers: Norwegians use Opera more than other countries, etc.

  bortzmeyer : The OP seemed to assume that it was a network problem, not an application one.
  From username

• Use traceroute.org to perform traceroutes from various places all over the world You'll get a better idea of what is going on. Asking traceroutes from clients is hopeless: they won't understand or send you a screenshot (yes, I've seen that) or forget to include important information.

  Per Kyle Brandt's suggestion, use the name or the IP address of your service, to see if it makes a difference.

  From bortzmeyer