Computer SciencePython Programming

Knowledge guide

Python split(), Unique Tokens, and Case-Sensitive Sorting

Learn how whitespace tokenization, equality-based deduplication, and string ordering interact, and decide when normalization is appropriate.

Tokenization chooses the units you process

Python's str.split() returns a list of strings. With no separator argument, runs of whitespace act as one delimiter, and leading or trailing whitespace produces no empty boundary items. An explicit separator behaves differently:

text = "  Red\tblue  Red  "
print(text.split())       # ['Red', 'blue', 'Red']
print("a,,b,".split(",")) # ['a', '', 'b', '']

Whitespace splitting is not linguistic word recognition. Punctuation remains attached, so bird and bird, are different tokens. Decide whether punctuation matters before choosing a cleaning rule.

Equality decides what counts as a duplicate

A list preserves the order in which you add tokens. A membership test can keep only the first occurrence:

unique = []
for token in ["Map", "map", "Map", "apple"]:
    if token not in unique:
        unique.append(token)
print(unique)  # ['Map', 'map', 'apple']

Strings compare case-sensitively: Map and map are distinct. Converting text with casefold() is a deliberate change to the definition of uniqueness, not an automatic improvement. Keep the original case when it carries meaning or the exercise requires it.

Sorting is separate from deduplication

sorted(unique) creates a sorted list; unique.sort() changes the existing list and returns None. Do not write unique = unique.sort().

For the example above, sorted(unique) is ['Map', 'apple', 'map']. Python compares strings lexicographically using Unicode code points. This is deterministic string order, not a universal dictionary order for every language. sorted(unique, key=str.casefold) instead orders using a case-folded key while retaining the original strings. It does not itself merge case variants.

A reusable check

Check three decisions separately: how text is split, which tokens count as equal, and how the final result is ordered. An empty input should produce an empty list; repeated tokens should appear once under the chosen equality rule; changing the sorting key should change order without dropping data. These checks reveal silent changes to case or punctuation more clearly than merely counting the output items.

Related question

Apply this knowledge

Use the concept guide to understand the reasoning, then return to the complete question and worked answer.

.split Python: Find the Unique Words in romeo.txt

Sources

These references support the core concepts and interpretation boundaries explained above.