/Python/Remove duplicate rows in csv file python

Remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Source: stackoverflow.com

Similar codes

0 vote151 views1 answer
Warning: this is a development server. do not use it in a production deployment.
Python
-1 vote171 views1 answer
Py string find regex pos
Python
0 vote186 views1 answer
Get source selenium python
Python
3 votes168 views1 answer
Isaplha in python
Python
0 vote167 views1 answer
List of imported modules in python
Python
0 vote184 views2 answers
Flask conditional according to urrl
Python
0 vote167 views1 answer
Updating file multiple times in pandas
Python
0 vote291 views1 answer
Shared a local host django
Python Django
0 vote163 views2 answers
Fatal error: Python.h: No such file or directory 45 | #include <Python.h> | ^~~~~~~~~~ compilation terminated. error: command 'gcc' failed with exit status 1
Python
0 vote162 views1 answer
Python advanced programming exercises
Python