/Python/Remove duplicate rows in csv file python

Remove duplicate rows in csv file python

# credit to the Stack Overflow user in the source link

with open('file_with_duplicates.csv','r') as in_file, open('ouput.csv','w') as out_file:
  
    seen = set() # set for fast O(1) amortized lookup
    
    for line in in_file:
        if line in seen: 
          continue # skip duplicate

        seen.add(line)
        out_file.write(line)

Source: stackoverflow.com

Similar codes

1 vote110 views1 answer
How to define dtype of each column before actually reading csv file
Python
1 vote102 views2 answers
Pandas filter by dictionary
Python
3 votes107 views2 answers
Apply lambda function to multiple columns pandas
Python
8 votes105 views1 answer
Positive lookahead regex python
Python
0 vote120 views1 answer
Check even or odd in single line
Python
1 vote101 views1 answer
Python edit string variable
Python
6 votes95 views2 answers
Read data from yaml file in python
Python
1 vote135 views1 answer
Managin media django
Python Django
105 votes155 views6 answers
Python json to dict
Python
1 vote95 views1 answer
Não nulo pandas
Python