/Python/Python delete duplicate lines in file

Python delete duplicate lines in file

with open("file.txt", "r") as txt_file:
  new_data = list(set(txt_file))
  return new_data

Txt file duplicate line remover python

lines_seen = set() # holds lines already seen

with open("file.txt", "r+") as f:
    d = f.readlines()
    f.seek(0)
    for i in d:
        if i not in lines_seen:
            f.write(i)
            lines_seen.add(i)
    f.truncate()

Source: stackoverflow.com

Similar codes

0 vote110 views1 answer
No module named 'fastai'
Python
10 votes98 views1 answer
Print first dictionary keys python
Python
2 votes103 views2 answers
How to take input in python3 separated by space
Python
18 votes140 views5 answers
Python continue
Python
0 vote120 views1 answer
How to delete all elements of a list in python
Python
0 vote99 views1 answer
Changing axis labels matplotlib
Python
0 vote137 views1 answer
Complex arrays python
Python
0 vote108 views1 answer
Updateview
Python
1 vote118 views2 answers
Change xticks python
Python
0 vote129 views1 answer
Python logging to syslog linux
Python