/Python/Remove duplicate rows in csv file python

Remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Source: stackoverflow.com

Similar codes

0 vote140 views1 answer
Django auth user model
Python
0 vote149 views1 answer
How to set text in QComboBox pyqt5
Python
1 vote154 views2 answers
{:.1%} print one decimal pandas
Python
0 vote133 views1 answer
RuntimeError: input must have 3 dimensions, got 4 site:stackoverflow.com
Python
0 vote141 views1 answer
Wie printe ich in python
Python
0 vote145 views1 answer
Value = int(line.rsplit(':', 1)[1].strip()) ValueError : invalid literal for int() with base 10: '' grass gis
Python
0 vote131 views1 answer
Python merge two byte files
Python
1 vote142 views1 answer
How to use datetime to tell your age in python
Python
0 vote142 views2 answers
Random array in python
Python
0 vote145 views1 answer
Get most recurring element in a list python
Python