Sunday, May 1, 2011

Working with large CSV files

When working with large CSV files I sometimes get this:
Traceback (most recent call last):
  File "stats.py", line 12, in
    for i, row in enumerate(csv.reader(open(filename))):
_csv.Error: field larger than field limit (131072)

This means the CSV file contains more content between delimiters that allowed by default. To raise this limit use:

import sys
import csv
csv.field_size_limit(sys.maxint)

No comments: