/Python/Python delete duplicate lines in file

Python delete duplicate lines in file

with open("file.txt", "r") as txt_file:
  new_data = list(set(txt_file))
  return new_data

Txt file duplicate line remover python

lines_seen = set() # holds lines already seen

with open("file.txt", "r+") as f:
    d = f.readlines()
    f.seek(0)
    for i in d:
        if i not in lines_seen:
            f.write(i)
            lines_seen.add(i)
    f.truncate()

Source: stackoverflow.com

Similar codes

7 votes130 views1 answer
Python download form web
Python
1 vote126 views2 answers
TypeError: can’t multiply sequence by non-int of type ‘float’
Python
0 vote136 views1 answer
Minimum of two columns in pandas
Python
0 vote130 views1 answer
Pd sample every class
Python
0 vote105 views1 answer
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_3_1:0", shape=(None, None, 71), dtype=float32) at layer "input_3". The following previous layers were accessed without issue: []
Python
1 vote127 views1 answer
Can't convert np.ndarray of type numpy.object_.
Python
0 vote118 views1 answer
How to count number from 1 to 10 in python
Python
2 votes131 views1 answer
Suppres tensorflow warnings
Python
0 vote136 views1 answer
Save python dic
Python
0 vote120 views1 answer
Python Record live streams (TS FILES)
Python