/Python/Remove duplicate rows in csv file python

Remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Source: stackoverflow.com

Similar codes

2 votes183 views8 answers
No
Python
0 vote177 views1 answer
Change increment in for loop python
Python
1 vote184 views1 answer
R2 score sklearn
Python
0 vote237 views2 answers
Python dict for k v
Python
158 votes288 views3 answers
Traversing dictionary in python
Python
1 vote185 views1 answer
How to convert 24 hours to 12 hours in python
Python
3 votes174 views1 answer
Pandas loc for list
Python
1 vote188 views1 answer
Python invert binary tree
Python
2 votes173 views2 answers
First day of the month python
Python
25 votes172 views6 answers
Uninstall all packages python
Python