Chunk size in python large size

def rows(f, chunksize=1024, sep='|'):
    """
    Read a file where the row separator is '|' lazily.

    Usage:

    >>> with open('big.csv') as f:
    >>>     for r in rows(f):
    >>>         process(r)
    """
    row = ''
    while (chunk := f.read(chunksize)) != '':   # End of file
        while (i := chunk.find(sep)) != -1:     # No separator found
            yield row + chunk[:i]
            chunk = chunk[i+1:]
            row = ''
        row += chunk
    yield row

Source: stackoverflow.com

Similar codes

1 vote144 views1 answer
Python pandas apply function to one column
Python
0 vote129 views1 answer
Find index of values greater than python
Python
10 votes183 views3 answers
How to insert sound in python
Python
0 vote171 views1 answer
Pandas save csv list as columns
Python
40 votes145 views2 answers
Sorting values in dictionary in python
Python
1 vote136 views1 answer
Anagrams string python
Python
0 vote154 views1 answer
Python copy file create intermediate directories
Python
1 vote155 views1 answer
How to write a dict in pytohn
Python
7 votes145 views1 answer
Cryptography python
Python
5 votes151 views1 answer
Python how long since date
Python