Chunk size in python large size

def rows(f, chunksize=1024, sep='|'):
    """
    Read a file where the row separator is '|' lazily.

    Usage:

    >>> with open('big.csv') as f:
    >>>     for r in rows(f):
    >>>         process(r)
    """
    row = ''
    while (chunk := f.read(chunksize)) != '':   # End of file
        while (i := chunk.find(sep)) != -1:     # No separator found
            yield row + chunk[:i]
            chunk = chunk[i+1:]
            row = ''
        row += chunk
    yield row

Source: stackoverflow.com

Similar codes

23 votes368 views2 answers
Flask or django
Python Flask
0 vote169 views1 answer
True Positive, True Negative, False Positive, False Negative in scikit learn
Python
0 vote165 views1 answer
Instead of: firstName = "John" lastName = "Henry" city = "Manchester"
Python
0 vote203 views1 answer
Python counter infinite series
Python
11 votes158 views4 answers
Python execute shell command and get output
Python
1 vote190 views1 answer
Reshape wide to long in pandas
Python
0 vote177 views1 answer
Rqg3
Python
1 vote182 views1 answer
Speech enhancement technique
Python
0 vote206 views2 answers
Flask heroku
Python Flask
0 vote166 views1 answer
Draw a bow tie in python
Python