/Python/Remove duplicate rows in csv file python

Remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Source: stackoverflow.com

Similar codes

1 vote250 views1 answer
Image delete in django from the folder
Python Django
7 votes249 views4 answers
Print multiple lines python
Python
0 vote211 views1 answer
Pandas iloc include header
Python
21 votes230 views6 answers
Convert datetime to date pandas
Python
0 vote195 views2 answers
When excel is loaded into python, numeric datatype changes to float
Python
2 votes241 views1 answer
Django filter multiple fields
Python Django
0 vote201 views1 answer
Calculate angle between 3 points python
Python
0 vote243 views1 answer
Creating base models django
Python Django
15 votes207 views3 answers
How to delete in python
Python
44 votes290 views3 answers
Online matplotlib compiler
Python