Duplicate Detector

2025-12-19

This page shows how to find duplicate lines in any text stream using a single Perl one-liner. It covers the core idiom, common variations, and real world log parsing.

perl -ne 'print if $seen{$_}++'

That command prints every line that shows up more than once.

The trick is the post-increment operator. The first time Perl sees a line, $seen{$_} evaluates to 0 (false), so the command skips the print. The second time, the value is 1 (true), so Perl prints the line. A third or fourth appearance still evaluates as true, so it prints again.

If you only want to catch the second occurrence, we will walk through that next.

Part 1: HOW IT WORKS

print if $seen{$_}++

Break it down:

PIECE           WHAT IT DOES
------------    ------------------------------------------
$seen{$_}       Hash lookup - has this line been seen?
++              Post-increment - add 1 AFTER returning value
$seen{$_}++     Returns OLD value (0 first time), then increments
print if ...    Print if the condition is true (non-zero)

On the first encounter, $seen{$_} holds undef, which Perl treats as 0 in numeric context. The expression returns 0 and then bumps the stored value to 1. The condition fails, so nothing prints.

On the second encounter, $seen{$_} already holds 1. The expression returns 1 and then bumps the stored value to 2. The condition passes, so Perl prints the line.

      .--.
     |o_o |
     |:_/ |
    //   \ \
   (|     | )
  /'\_   _/`\
  \___)=(___/

Part 2: VARIATIONS

Print duplicates only once (not every repeat):

perl -ne 'print if $seen{$_}++ == 1'

The == 1 clause tells Perl to print only on the second occurrence.

Print unique lines only (no duplicates at all):

perl -ne 'print unless $seen{$_}++'

This flips the logic. Perl prints the first time it sees a line and skips every repeat.

Print lines that appear exactly N times (requires two passes):

perl -ne '$c{$_}++; END { print for grep { $c{$_} == 3 } keys %c }'

Part 3: FIRST VS LAST OCCURRENCE

Print the first occurrence of each line:

perl -ne 'print unless $seen{$_}++'

Print the last occurrence of each line:

perl -ne '$last{$_} = $_; END { print values %last }'

This approach overwrites the stored value each time, so only the final line survives.

That approach drops the original order. If you want the last occurrence while keeping the original sequence, use this instead:

perl -ne '$last{$_} = $.; END { print sort { $last{$a} <=> $last{$b} } keys %last }'

The trick here is to store the line number ($.) instead of the line itself, then sort by those numbers at the end.

Part 4: CASE INSENSITIVE

Ignore case when detecting duplicates:

perl -ne 'print if $seen{lc $_}++'

The lc function lowercases the hash key. "Hello" and "HELLO" now share the same bucket.

You can also normalize whitespace at the same time:

perl -ne '$k = lc; $k =~ s/\s+/ /g; print if $seen{$k}++'

Part 5: FIELD-BASED DUPLICATES

Detect duplicates on a specific column:

perl -ane 'print if $seen{$F[0]}++'

The -a flag splits each input line into the @F array. This version checks only the first field.

Find duplicate IPs in a log:

perl -ane 'print if $seen{$F[0]}++' access.log

Find duplicate usernames in /etc/passwd:

perl -F: -ane 'print if $seen{$F[0]}++' /etc/passwd

Part 6: COUNTING DUPLICATES

Count how many times each line appears:

perl -ne '$c{$_}++; END { print "$c{$_}: $_" for keys %c }'

The output looks like this:

3: this line appeared three times
1: this line appeared once
5: this line appeared five times

Sort the results by frequency:

perl -ne '$c{$_}++; END { print "$c{$_}: $_" for sort { $c{$b} <=> $c{$a} } keys %c }'

The most frequent lines now appear first.

Part 7: ADJACENT DUPLICATES

The uniq command only removes adjacent duplicates:

perl -ne 'print unless $_ eq $last; $last = $_'

The current line must match the previous line to be skipped.

This approach runs faster because it avoids a hash, but it only catches consecutive repeats:

aaa
aaa    <- removed
bbb
aaa    <- NOT removed, not adjacent

Part 8: REAL WORLD EXAMPLES

Find duplicate lines in a config file:

perl -ne 'print "$.: $_" if $seen{$_}++' config.ini

The $. variable adds line numbers so you can locate them quickly.

Find duplicate hostnames in /etc/hosts:

perl -ane 'print if $seen{$F[1]}++' /etc/hosts

This checks the hostname field, which sits in the second column.

Find duplicate SSH public keys:

perl -ne 'print if $seen{(split)[1]}++' ~/.ssh/authorized_keys

The key string sits in the second field, so this catches repeated entries.

Find duplicate cron jobs:

crontab -l | perl -ne 'print if $seen{$_}++'

Part 9: MEMORY CONSIDERATIONS

The hash stores every unique line in memory. For huge files with many unique lines, this approach eats RAM quickly.

For massive files, consider this pipeline instead:

sort file.txt | uniq -d

An external sort handles files larger than available memory, though it drops the original order.

You can also process only the tail end if you only care about recent duplicates:

tail -10000 huge.log | perl -ne 'print if $seen{$_}++'

Part 10: THE FAMILY

These patterns share the same core idiom:

perl -ne 'print if $seen{$_}++'          # All duplicates
perl -ne 'print if $seen{$_}++ == 1'     # Each duplicate once
perl -ne 'print unless $seen{$_}++'      # Unique lines only
perl -ne '$c{$_}++; END{print for grep{$c{$_}>1}keys%c}'  # Dupes, one each

The post-increment idiom powers every one of them.

Part 11: WHY POST-INCREMENT

Why use $seen{$_}++ instead of ++$seen{$_}?

Pre-increment returns the new value:

++$seen{$_}    # Returns 1 on first encounter (true!)

Post-increment returns the old value:

$seen{$_}++    # Returns 0 on first encounter (false!)

With pre-increment, every line prints because the bump happens before the comparison. Post-increment is what makes the logic work.

Part 12: COMBINING WITH OTHER PATTERNS

Find duplicates that also match a pattern:

perl -ne 'print if /error/i && $seen{$_}++'

This catches only repeated error lines.

Find duplicates across multiple files:

perl -ne 'print "$ARGV: $_" if $seen{$_}++' *.log

The $ARGV variable shows which file contained the duplicate.

Find duplicates within a time window (log files):

perl -ane '
    $t = $F[0]; 
    %seen = () if $t ne $last_t; 
    print if $seen{$_}++; 
    $last_t = $t
' timestamped.log

This resets the seen hash whenever the timestamp column changes.

                $seen{$_}++
                    |
               +----+----+
               |         |
            first     again
               |         |
             skip      print

     The post-increment trick

perl.gg