/Python/Python delete duplicate lines in file

Python delete duplicate lines in file

with open("file.txt", "r") as txt_file:
  new_data = list(set(txt_file))
  return new_data

Txt file duplicate line remover python

lines_seen = set() # holds lines already seen

with open("file.txt", "r+") as f:
    d = f.readlines()
    f.seek(0)
    for i in d:
        if i not in lines_seen:
            f.write(i)
            lines_seen.add(i)
    f.truncate()

Source: stackoverflow.com

Similar codes

11 votes173 views4 answers
Get count of values in column pandas
Python
6 votes211 views5 answers
Django logout
Python Django
1 vote150 views1 answer
Install quick-mailer
Python
1 vote158 views1 answer
Flask form errors
Python
1 vote172 views1 answer
Get all keys and values from dictionary python
Python
1 vote206 views1 answer
Import "django.core.urlresolvers" could not be resolved
Python Django
3 votes166 views2 answers
Apply lambda function to multiple columns pandas
Python
0 vote181 views1 answer
Cannot access modules from neighbouring directories jupyter notebook
Python
3 votes156 views1 answer
How to change the window colour in pygame
Python
0 vote178 views1 answer
Program to find the largest of three numbers in python
Python