Python Program to Remove Duplicates from List

Write a Python Program to Remove all the duplicates from the given list. The set won’t allow duplicates, so we can convert the list to set, and then convert it back to the list will remove the list duplicates.

dupList = [1, 2, 3, 2, 4, 8, 9, 1, 7, 6, 4, 5]
print("List Items = ", dupList)

uniqSet = set(dupList)
uniqList = list(uniqSet)
   
print("List Items after removing Duplicates = ", uniqList)

Remove Duplicates in a List using set output.

Python Program to Remove Duplicate Items from List using for loop

This Python program allows entering the list size and items. The for loop will iterate the dupList items. The if statement with not in operator checks whether the value is not present in the uniqList. If True, append that value to uniqList.

dupList = []

listNumber = int(input("Enter the Total List Items = "))
for i in range(1, listNumber + 1):
    listValue = int(input("Enter the %d List Item = " %i))
    dupList.append(listValue)

print("List Items = ", dupList)

uniqList = []

for val in dupList:
    if val not in uniqList:
        uniqList.append(val)
   
print("List Items after removing Duplicates = ", uniqList)

Using list comprehension

In this example, we used list comprehension to remove duplicate items from the list. This code is the same as the above example, but we used the list comprehension concept.

dupList = [1, 2, 5, 8, 1, 9, 11, 5, 22, 6, 2, 8, 14]

print("List Items = ", dupList)

uniqList = []
[uniqList.append(i) for i in dupList if i not in uniqList]

print("List Items after removing Duplicates = ", uniqList)

List Items =  [1, 2, 5, 8, 1, 9, 11, 5, 22, 6, 2, 8, 14]
List Items after removing Duplicates =  [1, 2, 5, 8, 9, 11, 22, 6, 14]

Python Program to Remove List Duplicates using collections fromkeys

In this example, we imported the OrderedDict from collections and used the fromkeys function to remove duplicates. Don’t forget to convert the result to a list.

from collections import OrderedDict

dupList = [8, 1, 9, 2, 8, 4, 9, 11, 5, 22, 6, 4, 8]

print("List Items = ", dupList)

uniqList = OrderedDict.fromkeys(dupList)

print("List Items after removing Duplicates = ", list(uniqList))

The output of the using OrderedDict from collections.

Remove List Duplicates using unique function

Both the numpy and pandas module has the unique function to remove duplicates, so we used the same and converted the result to a list. To convert the result, we used the tolist() function.

import numpy as np
import pandas as pd

dupList = [1, 2, 2, 4, 1, 5, 6, 8, 6, 8, 9, 7, 4]
print("List Items = ", dupList)

uniqList = np.unique(dupList).tolist()
print("List Items after removing Duplicates = ", uniqList)

uniqList2 = pd.unique(dupList).tolist()
print("List Items after removing Duplicates = ", uniqList2)

numpy unique function output

List Items =  [1, 2, 2, 4, 1, 5, 6, 8, 6, 8, 9, 7, 4]
List Items after removing Duplicates =  [1, 2, 4, 5, 6, 7, 8, 9]
List Items after removing Duplicates =  [1, 2, 4, 5, 6, 8, 9, 7]

Using enumerate function

This example uses the enumerate function to remove Duplicates from the List.

from collections import OrderedDict

dupList = [1, 2, 3, 2, 4, 1, 5, 6, 5, 8, 7, 9, 8]

print("List Items = ", dupList)

uniqList = [val for x, val in enumerate(dupList) if val not in dupList[:x]]

print("List Items after removing Duplicates = ", uniqList)

The output of the enumerate example is shown below.

List Items =  [1, 2, 3, 2, 4, 1, 5, 6, 5, 8, 7, 9, 8]
List Items after removing Duplicates =  [1, 2, 3, 4, 5, 6, 8, 7, 9]