Tuesday, May 3, 2011

How to output list of floats to a binary file in Python

I have a list of floating-point values in Python:

floats = [3.14, 2.7, 0.0, -1.0, 1.1]

I would like to write these values out to a binary file using IEEE 32-bit encoding. What is the best way to do this in Python? My list actually contains about 200 MB of data, so something "not too slow" would be best.

Since there are 5 values, I just want a 20-byte file as output.

From stackoverflow
  • have a look at struct.pack_into

  • See: Python's struct module

    import struct
    s = struct.pack('f'*len(floats), *floats)
    f = open('file','wb')
    f.write(s)
    f.close()
    
    Frank Krueger : Actually, that's what I'm doing now. I just thought there had to be a better way.
    Alex Martelli : And there IS a better way -- the array module. See http://docs.python.org/library/array.html for all the details.
  • struct.pack() looks like what you need.

    http://docs.python.org/library/struct.html

  • The array module in the standard library may be more suitable for this task than the struct module which everybody is suggesting. Performance with 200 MB of data should be substantially better with array.

  • Alex is absolutely right, it's more efficient to do it this way:

    from array import array
    output_file = open('file', 'wb')
    float_array = array('d', [3.14, 2.7, 0.0, -1.0, 1.1])
    float_array.tofile(output_file)
    output_file.close()
    

    And then read the array like that:

    input_file = open('file', 'r')
    float_array = array('d')
    float_array.fromstring(input_file.read())
    

    array.array objects also have a .fromfile method which can be used for reading the file, if you know the count of items in advance (e.g. from the file size, or some other mechanism)

  • I'm not sure how NumPy will compare performance-wise for your application, but it may be worth investigating.

    Using NumPy:

    from numpy import array
    a = array(floats,'float32')
    output_file = open('file', 'wb')
    a.tofile(output_file)
    output_file.close()
    

    results in a 20 byte file as well.

0 comments:

Post a Comment