Thursday, April 14, 2011

What component do I need to monitor my internet traffic on my PC?

I would like to be able to see and monitor my internet data (http/emule/email) on my own PC using Windows XP. I am thinking of something like WireShark but I would like to control it programmatically.

I would be using C or C++.

How can I do this?

From stackoverflow
  • WireShark uses winpcap to do it's thing.

    Winpcap comes with a C interface.

    : is it the only method ?
  • winpcap is probably the most well known choice, but you could also write a Layered Service Provider. There's not a whole lot of documentation, but a good place to start is the article on msdn: http://www.microsoft.com/msj/0599/LayeredService/LayeredService.aspx

    This has some advantages vs layer 2 packet parsing, but also some disadvantages. You'll need to evaluate where in the stack you want to live.

    edit: Obviously, when I say pcap -- I really mean any similar approach. Obviously, winpcap was not the first driver/library combo to provide this sort of information.

  • Have a look at the code of the tool trafficWatcher. It uses WinPCap to distinguish between LAN and internet traffic.

    The sourcecode is available here.

What is the correct way of reading from a TCP socket in C/C++?

Here's my code:

// Not all headers are relevant to the code snippet.
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <cstdlib>
#include <cstring>
#include <unistd.h>

char *buffer;
stringstream readStream;
bool readData = true;

while (readData)
{
 cout << "Receiving chunk... ";

 // Read a bit at a time, eventually "end" string will be received.
 bzero(buffer, BUFFER_SIZE);
 int readResult = read(socketFileDescriptor, buffer, BUFFER_SIZE);
 if (readResult < 0)
 {
  THROW_VIMRID_EX("Could not read from socket.");
 }

 // Concatenate the received data to the existing data.
 readStream << buffer;

 // Continue reading while end is not found.
 readData = readStream.str().find("end;") == string::npos;

 cout << "Done (length: " << readStream.str().length() << ")" << endl;
}

It's a little bit of C and C++ as you can tell. The BUFFER_SIZE is 256 - should I just increase the size? If so, what to? Does it matter?

I know that if "end" is not received for what ever reason, this will be an endless loop, which is bad - so if you could suggest a better way, please also do so.

From stackoverflow
  • Where are you allocating memory for your buffer? The line where you invoke bzero invokes undefined behavior since buffer does not point to any valid region of memory.

    char *buffer = new char[ BUFFER_SIZE ];
    // do processing
    
    // don't forget to release
    delete[] buffer;
    
  • Several pointers:

    You need to handle a return value of 0, which tells you that the remote host closed the socket.

    For nonblocking sockets, you also need to check an error return value (-1) and make sure that errno isn't EINPROGRESS, which is expected.

    You definitely need better error handling - you're potentially leaking the buffer pointer to by 'buffer'. Which, I noticed, you don't allocate anywhere in this code snippet.

    Someone else made a good point about how your buffer isn't a null terminated C string if your read() fills the entire buffer. That is indeed a problem, and a serious one.

    Your buffer size is a bit small, but should work as long as you don't try to read more than 256 bytes, or whatever you allocate for it.

    If you're worried about getting into an infinite loop when the remote host sends you a malformed message (a potential denial of service attack) then you should use select() with a timeout on the socket to check for readability, and only read if data is available, and bail out if select() times out.

    Something like this might work for you:

    fd_set read_set;
    struct timeval timeout;
    
    timeout.tv_sec = 60; // Time out after a minute
    timeout.tv_usec = 0;
    
    FD_ZERO(&read_set);
    FD_SET(socketFileDescriptor, &read_set);
    
    int r=select(socketFileDescriptor+1, &read_set, NULL, NULL, &timeout);
    
    if( r<0 ) {
        // Handle the error
    }
    
    if( r==0 ) {
        // Timeout - handle that. You could try waiting again, close the socket...
    }
    
    if( r>0 ) {
        // The socket is ready for reading - call read() on it.
    }
    

    Depending on the volume of data you expect to receive, the way you scan the entire message repeatedly for the "end;" token is very inefficient. This is better done with a state machine (the states being 'e'->'n'->'d'->';') so that you only look at each incoming character once.

    And seriously, you should consider finding a library to do all this for you. It's not easy getting it right.

  • Without knowing your full application it is hard to say what the best way to approach the problem is, but a common technique is to use a header which starts with a fixed length field, which denotes the length of the rest of your message.

    Assume that your header consist only of a 4 byte integer which denotes the length of the rest of your message. Then simply do the following.

    // This assumes buffer is at least x bytes long,
    // and that the socket is blocking.
    void ReadXBytes(int socket, unsigned int x, void* buffer)
    {
        int bytesRead = 0;
        int result;
        while (bytesRead < x)
        {
            result = read(socket, buffer + bytesRead, x - bytesRead)
            if (result < 1 )
            {
                // Throw your error.
            }
    
            bytesRead += result;
        }
    }
    

    Then later in the code

    unsigned int length = 0;
    char* buffer = 0;
    // we assume that sizeof(length) will return 4 here.
    ReadXBytes(socketFileDescriptor, sizeof(length), (void*)(&length));
    buffer = new char[length];
    ReadXBytes(socketFileDescriptor, length, (void*)buffer);
    
    // Then process the data as needed.
    
    delete [] buffer;
    

    This makes a few assumptions:

    • ints are the same size on the sender and receiver.
    • Endianess is the same on both the sender and receiver.
    • You have control of the protocol on both sides
    • When you send a message you can calculate the length up front.

    Since it is common to want to explicitly know the size of the integer you are sending across the network define them in a header file and use them explicitly such as:

    // These typedefs will vary across different platforms
    // such as linux, win32, OS/X etc, but the idea
    // is that a Int8 is always 8 bits, and a UInt32 is always
    // 32 bits regardless of the platform you are on.
    // These vary from compiler to compiler, so you have to 
    // look them up in the compiler documentation.
    typedef char Int8;
    typedef short int Int16;
    typedef int Int32;
    
    typedef unsigned char UInt8;
    typedef unsigned short int UInt16;
    typedef unsigned int UInt32;
    

    This would change the above to:

    UInt32 length = 0;
    char* buffer = 0;
    
    ReadXBytes(socketFileDescriptor, sizeof(length), (void*)(&length));
    buffer = new char[length];
    ReadXBytes(socketFileDescriptor, length, (void*)buffer);
    
    // process
    
    delete [] buffer;
    

    I hope this helps.

    grieve : Ori Pessach's comment is a good complimentary answer to this one.
  • If you actually create the buffer as per dirks suggestion, then:

      int readResult = read(socketFileDescriptor, buffer, BUFFER_SIZE);
    

    may completely fill the buffer, possibly overwriting the terminating zero character which you depend on when extracting to a stringstream. You need:

      int readResult = read(socketFileDescriptor, buffer, BUFFER_SIZE - 1 );
    
  • This is an article that I always refer to when working with sockets..

    THE WORLD OF SELECT()

    It will show you how to reliably use 'select()' and contains some other useful links at the bottom for further info on sockets.

  • 1) Others (especially dirkgently) have noted that buffer needs to be allocated some memory space. For smallish values of N (say, N <= 4096), you can also allocate it on the stack:

    #define BUFFER_SIZE 4096
    char buffer[BUFFER_SIZE]
    

    This saves you the worry of ensuring that you delete[] the buffer should an exception be thrown.

    But remember that stacks are finite in size (so are heaps, but stacks are finiter), so you don't want to put too much there.

    2) On a -1 return code, you should not simply return immediately (throwing an exception immediately is even more sketchy.) There are certain normal conditions that you need to handle, if your code is to be anything more than a short homework assignment. For example, EAGAIN may be returned in errno if no data is currently available on a non-blocking socket. Have a look at the man page for read(2).

    nbolton : Good point, its not good to leave open socket handles lying around; will consider throwing afterward.
    Dan Breslau : Actually, I didn't address the open socket handle, because it's not opened in the snippet that you posted. But I'm glad you thought of it :-)
  • I have found this guide to be most helpful.

Migrating to WCF from ASP.Net 2.0 web services

I currently have a suite of .Net web services that are developed in ASP.Net 2.0. They don't utilize any WS-E extensions, rather they implement security at the application level. They are fairly straightforward data retrieval/update features. I am interested in re-factoring these web services gradually over time into WCF services, mostly to future-proof them against the deprecation of old framework versions. My main concern is the amount of re-coding that will need to be done in the client applications to re-connect with these services.

  1. Has anyone gone down this road already?
  2. Was it worth it?
  3. Can you recommend any reading materials that I can use to continue my research in this area?
From stackoverflow
  • Your concerns are valid. I've been taking the approach of 'if it ain't broke, don't fix it' and leaving existing 'legacy' web services alone until they need work. For new development we have been adopting WCF the solution to communication problems big and small. The most tangible benefit is the ease of integration if you have a nice clean object model. The most tangible negative is that (relatively speaking) there are much more configuration headaches, especially when you start trying to secure things.

    I find it interesting you mentioned the WS-* spec as that is where we had the most problems integrating WCF with a Java system. The security features are just plain missing right now, and we ended up using Microsoft.Web.Services3 (WSE3) for that one piece of the .NET side of things.

    I would highly recommend reading through these articles even if you don't do security with WCF right now, you probably will have to at some point.

    John Saunders : Could you elaborate on which security features are missing from WCF? I didn't know it was missing any at all.
    slf : Specifically when dealing with UsernameToken on a custom binding we couldn't find a way to set MustUnderstand to false. After a couple of days of troubleshooting WCF we found WSE3 supported this out of the box and used it simply out of respect to the project schedule.
  • It is quite possible that your clients won't need to change at all.

    On the server side, I've recently seen an ASMX service that was turned into a WCF service simply by placing [ServiceContract] on the [WebService] class, and [OperationContract] on the [WebMethod] methods, and by turning the non-primitive types it returned into [DataContract] and [DataMember].

Selecting a good dictionary key

I have an object that I want to use to look up other objects. I will be using a Dictionary<TKey, TValue>().

The key object has two strings that uniquely identify it, say KeyObj.Str1 and KeyObj.Str2.

What do you recommend that I use as the key for the dictionary?

1: The concatenation of the strings.

Dictionary<String, TValue>();
Key = KeyObj.Str1:KeyObj.Str2; ("somestring:anotherstring")

2: A unique integer for each object to identify it?

Dictionary<int, TValue>();
KeyObj.ID = _nextID++;
Key = KeyObj.ID;

3: A reference to the object.

Dictionary<KeyObj, TValue>();
Key = KeyObj;

Option 3 would be the easiest, but it seems like it would be inefficient to index a dictionary based on reference values.

If the key object contained a single unique string, the obvious choice would be use that, but having two strings that are only unique in combination makes it more difficult.

From stackoverflow
  • You don't need to use a new class as the dictionary key. Using a new struct instead as it will be much more lightweight... And have it consist of those two string values obviously.

    Josh G : The class that I am using is more complicated that I described in the example... I simplified it to keep it clear. I don't want to make it a struct.
  • If performance is a major consideration, you could consider using a hashvalue of the two strings. But then your 'value' field would have to contain both the keys and the value.

    I have a reference to another SO question, I just have to find it.

    http://stackoverflow.com/questions/658065/is-it-faster-to-search-for-a-large-string-in-a-db-by-its-hashcode/658187

    But that question is more DB oriented. And the performance is considered for thousands of iterations.

  • I would say option 1.

  • Concatenating them is probably the best idea. You can expose a property in the KeyObj object that does the concatenation so you don't have to perform it each time you're accessing the dictionary value.

    Edit:

    I apparently misread the question. I think what you really want to do is a mix of 1 and 3, you can override Equals() and GetHashCode() to use the strings that uniquely identify the object (just make sure they are immutable!)

    public override Equals(object obj) 
    {
       if (obj == null || !(obj is KeyObj))
          return false;
       KeyObj other = (KeyObj)obj;
       if (this.Key1 == other.Key1 && this.Key2 == other.Key2)
         return true;
       return false;
    }
    
    public override GetHashCode()
    {
        return (this.Key1 + this.Key2).GetHashCode();
    }
    

    Then you can use the 3rd option you suggested:

    Dictionary<KeyObj, ValueObj>...
    
  • what about using the KeyObj.GetHashCode()?

    Josh G : This sounds promising...
    Groo : According to MSDN: The default implementation of the GetHashCode method does not guarantee unique return values for different objects.
    Groo : (therefore the actual question here was how to implement it)
  • Any of them are valid, but I'm assuming you'd want to be able to quickly find these objects based on one of the two strings, so using an int as the key would mean you'd still have to scan the values to find the object you wanted.

    Are the strings both unique, or only when combined? If they're both unique, and you're willing to trade a bit of space, you could do:

    dict.Add(KeyObj.Str1, KeyObj);
    dict.Add(KeyObj.Str2, KeyObj);
    

    and have two references to the object in the dictionary, using each unique string as a key. Or, you could always just combine the strings if they're only unique together, and it'll use the hashcode internally to look them up.

    Josh G : They are only unique when combined.
  • Concatenated strings should work best.

    IF you know that their combination is unique, then that is what you should choose -- remember that Hash code is usually unique, but not always.

  • Remember that a dictionary is a glorified hash table, so the key (no pun intended) is to use a key that will result in very few (if any) collisions with another key. I'd lean toward #3, but that's assuming the KeyObj type has a good hash value generator.

    Groo : I wouldn't say so, because all keys must be unique in a dictionary.
    Josh G : Does the Dictionary class implicitly use KeyObj.GetHashCode() for comparing the reference objects?
    Groo : It actually uses a default implementation of EqualityComparer (if you don't specify one). It uses the GetHashCode result to speed up the search (by creating several buckets), but at the end it uses the Equals method to make sure that they are identical.
  • You could use option 3 if you can override GetHashCode() and Equals() appropriately, i.e. something like this:

        public override int GetHashCode()
        {
            return str1.GetHashCode() ^ str2.GetHashCode();
        }
    
        public override bool Equals(object obj)
        {
            if (!obj is KeyObj)
            {
                return false;
            }
    
            KeyObj key = (KeyObj)obj;
            return this.str1.Equals(key.str1) && this.str2.Equals(key.str2);
        }
    
    Josh G : This is a good option for making #3 work. Thanks.

Unable to decode unicode string in Python 2.4

This is in python 2.4. Here is my situation. I pull a string from a database, and it contains an umlauted 'o' (\xf6). At this point if I run type(value) it returns str. I then attempt to run .decode('utf-8'), and I get an error ('utf8' codec can't decode bytes in position 1-4).

Really my goal here is just to successfully make type(value) return unicode. I found an earlier question that had some useful information, but the example from the picked answer doesn't seem to run for me. Is there something I am doing wrong here?

Here is some code to reproduce:

Name = 'w\xc3\xb6rner'.decode('utf-8')
file.write('Name: %s - %s\n' %(Name, type(Name)))

I never actually get to the write statement, because it fails on the first statement.

Thank you for your help.

Edit:

I verified that the DB's charset is utf8. So in my code to reproduce I changed '\xf6' to '\xc3\xb6', and the failure still occurs. Is there a difference between 'utf-8' and 'utf8'?

The tip on using codecs to write to a file is handy (I'll definitely use it), but in this scenario I am only writing to a log file for debugging purposes.

From stackoverflow
  • You need to use "ISO-8859-1":

    Name = 'w\xf6rner'.decode('iso-8859-1')
    file.write('Name: %s - %s\n' %(Name, type(Name)))
    

    utf-8 uses 2 bytes for escaping anything outside ascii, but here it's just 1 byte, so iso-8859-1 is probably correct.

  • Your string is not in UTF8 encoding. If you want to 'decode' string to unicode, your string must be in encoding you specified by parameter. I tried this and it works perfectly:

    print 'w\xf6rner'.decode('cp1250')
    

    EDIT

    For writing unicode strings to the file you can use codecs module:

    import codecs
    f = codecs.open("yourfile.txt", "w", "utf8")
    f.write( ... )
    

    It is handy to specify encoding of the input/output and using 'unicode' string throughout your code without bothering of different encodings.

  • It's obviously 1-byte encoding. 'รถ' in UTF-8 is '\xc3\xb6'.

    The encoding might be:

    • ISO-8859-1
    • ISO-8859-2
    • ISO-8859-13
    • ISO-8859-15
    • Win-1250
    • Win-1252
  • So in my code to reproduce I changed '\xf6' to '\xc3\xb6', and the failure still occurs

    Not in the first line it doesn't:

    >>> 'w\xc3\xb6rner'.decode('utf-8')
    u'w\xf6rner'
    

    The second line will error out though:

    >>> file.write('Name: %s - %s\n' %(Name, type(Name)))
    UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 7: ordinal not in range(128)
    

    Which is entirely what you'd expect, trying to write non-ASCII Unicode characters to a byte stream. If you use Jiri's suggestion of a codecs-wrapped stream you can write Unicode directly, otherwise you will have to re-encode the Unicode string into bytes manually.

    Better, for logging purposes, would be simply to spit out a repr() of the variable. Then you don't have to worry about Unicode characters being in there, or newlines or other unwanted characters:

    name= 'w\xc3\xb6rner'.decode('utf-8')
    file.write('Name: %r\n' % name)
    
    Name: u'w\xf6rner'
    

How to get whole text of an Element in xml.minidom?

I want to get the whole text of an Element to parse some xhtml:

<div id='asd'>
  <pre>skdsk</pre>
</div>

begin E = div element on the above example, I want to get

<pre>skdsk</pre>

How?

From stackoverflow
  • Strictly speaking:

    from xml.dom.minidom import parse, parseString
    tree = parseString("<div id='asd'><pre>skdsk</pre></div>")
    root = tree.firstChild
    node = root.childNodes[0]
    print node.toxml()
    

    In practice, though, I'd recommend looking at the http://www.crummy.com/software/BeautifulSoup/ library. Finding the right childNode in an xhtml document, and skipping "whitespace nodes" is a pain. BeautifulSoup is a robust html/xhtml parser with fantastic tree-search capacilities.

    Edit: The example above compresses the HTML into one string. If you use the HTML as in the question, the line breaks and so-forth will generate "whitespace" nodes, so the node you want won't be at childNodes[0].

Localization Problem Decimal Seperator

Is it possible that .NET uses on server A ',' as decimal seperator and on another server B '.'? + How can you detect this?

When converting strings to doubles, on server A everything works fine, but on server B we have problems.

Example:

server A : 20,4 --> 20.4 server B : 20,4 --> 204

We would need to detect this so that on both servers things keep on working.

thx, Lieven Cardoen

From stackoverflow
  • Sounds like the locale is being set correctly on one server but not the other.

    There are a few ways you could do this, but you might want to set the locale for the current thread in your App/ASP.NET page using Thread.CurrentThread.CurrentCulture and Thread.CurrentThread.CurrentUICulture.

    To set this for the entire application, you'd do this in your web.config:

    <configuration>
      <system.web>
        <globalization culture="en-US" uiCulture="en-US" />
    ...
    
    Lieven Cardoen : Can you set this for the overall asp.net application instead of for each thread separately?
    Eric Petroelje : Yup! Updated my answer.
    Eric Petroelje : I guess in your case it might be either "nl-BE" or "fr-BE", but you get the idea.
  • I use

    String.Format(System.Globalization.CultureInfo.InvariantCulture, ....)
    

    in those cases where you do not want to have culture specific output. (For example, when you write serialize data for other applications etc.) This way no matter what Culture is currently set, the output will always be the same.