Remove duplicates but keep order
Deduplicate a list while preserving the first occurrence order, for hashable values or by key.
- Python
- GreenSnippets
- MIT
- Executed with Python 3.11
def unique(items):
return list(dict.fromkeys(items))
def unique_by(items, key):
seen = set()
result = []
for item in items:
k = key(item)
if k not in seen:
seen.add(k)
result.append(item)
return resultExample / expected output
unique([3, 1, 3, 2, 1]) # [3, 1, 2]How it works
list(set(x)) loses order. Since Python 3.7 dictionaries preserve insertion order, so dict.fromkeys is a one-liner for hashable items. The keyed version handles unhashable items such as dicts by tracking a derived key.
Related snippets
- Python Tested
Group items by a key
Collect a list of records into a dictionary of lists keyed by a field or function result.
- Python Tested
Flatten a nested list
Flatten arbitrarily nested lists and tuples into a single flat iterator, leaving strings intact.
- Python Tested
Token bucket rate limiter
Allow bursts but cap the sustained request rate with a small thread-safe token bucket.
- Python Tested
Time a block of code with a context manager
Measure elapsed wall-clock time of a with-block using time.perf_counter.