/Python/Remove duplicate rows in csv file python

Remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Source: stackoverflow.com

Similar codes

0 vote123 views1 answer
Sort eigenvalues and eigenvectors python
Python
4 votes139 views3 answers
Python first day of last month
Python
1 vote86 views1 answer
Make each element in a list occur once python
Python
0 vote128 views1 answer
Pandas count freq of each value
Python
1 vote124 views1 answer
Spacy matcher syntax
Python
1 vote133 views1 answer
Show aruco marker axis opencv python
Python
0 vote134 views2 answers
Loop through groupby pandas
Python
7 votes144 views2 answers
Pandas read_csv ignore unnamed columns
Python
0 vote135 views1 answer
Check python version colab
Python
0 vote122 views1 answer
Uppy tus
Python