/Python/Remove duplicate rows in csv file python

Remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Source: stackoverflow.com

Similar codes

0 vote179 views1 answer
Python online practice test
Python
6 votes177 views2 answers
Add python kernel to Jupyter
Python
2 votes167 views1 answer
Converting a string to a dictionary in python
Python
0 vote233 views1 answer
Flask on gevent over https
Python Flask
1 vote184 views1 answer
Plt imshow python
Python
1 vote146 views1 answer
Iterate over pandas series
Python
-1 vote180 views1 answer
How to remove last character from a list in python
Python
2 votes189 views1 answer
How to find geometric mean in python
Python
12 votes183 views5 answers
Python find intersection of two lists
Python
0 vote201 views2 answers
How to run turtle in python
Python