Magic Argv

2025-12-20

Here's the scenario. You're a sysadmin. Logs rotate nightly and get compressed. Your directory looks like this:

access.log
access.log.1
access.log.2.gz
access.log.3.gz
access.log.4.gz

Some compressed, some not. You need to grep through all of them.

The ugly way:

cat access.log access.log.1
zcat access.log.*.gz

Two commands. Or write a wrapper script. Or...

perl -e '
    s/^([^'"'"']+\.gz)\z/gzcat '"'"'$1'"'"' |/ for @ARGV;
    print while <>;
' access.log*

One command handles both. The diamond operator sees pipes and plain files, doesn't care which is which.

Part 1: THE TRICK

Before the diamond operator runs, we rewrite @ARGV:

s/^([^']+\.gz)\z/gzcat '$1' |/ for @ARGV;

Files ending in .gz become:

access.log.2.gz  ->  gzcat 'access.log.2.gz' |

That trailing pipe tells Perl "run this command and read its output." The diamond operator sees a pipe, executes gzcat, and gives you decompressed data.

Plain files pass through unchanged. The regex doesn't match them.

Part 2: HOW PIPES IN @ARGV WORK

The diamond operator has magic behaviors:

ARGV ENTRY          PERL DOES
----------------    ----------------------------------
filename            open(ARGV, '<', 'filename')
command |           open(ARGV, '-|', 'command')
| command           open(ARGV, '|-', 'command')

A trailing pipe means "read from this command." A leading pipe means "write to this command."

This is old-school two-argument open behavior. Usually dangerous. Here, we're exploiting it deliberately.

      .--.
     |o_o |
     |:_/ |
    //   \ \
   (|     | )
  /'\_   _/`\
  \___)=(___/

Part 3: THE REGEX EXPLAINED

s/^([^']+\.gz)\z/gzcat '$1' |/

PIECE       MEANING
--------    ------------------------------------------
^           Start of string
(           Begin capture
[^']+       One or more chars that aren't single quotes
\.gz        Literal .gz
)           End capture
\z          End of string
gzcat '$1'  Replacement: gzcat command with filename
|           Trailing pipe - read from command

Why [^']+ instead of .+?

Security. If someone passes a filename with a single quote in it, they could inject commands. The [^'] prevents that.

evil'file;rm -rf /.gz    # Won't match - has quote

Part 4: MULTIPLE COMPRESSION FORMATS

Handle .gz and .Z files:

s/^([^']+\.(?:gz|Z))\z/gzcat '$1' |/ for @ARGV;

Handle .bz2 too:

s/^([^']+\.bz2)\z/bzcat '$1' |/ for @ARGV;
s/^([^']+\.(?:gz|Z))\z/gzcat '$1' |/ for @ARGV;

Two substitutions, handles three formats.

Or combined:

for (@ARGV) {
    s/^([^']+\.(?:gz|Z))\z/gzcat '$1' |/;
    s/^([^']+\.bz2)\z/bzcat '$1' |/;
    s/^([^']+\.xz)\z/xzcat '$1' |/;
}

Part 5: PRACTICAL SCRIPT

Make it a reusable tool:

#!/usr/bin/env perl
# zperl - run perl with transparent decompression

for (@ARGV) {
    next if /^\s*-/;  # Skip flags
    s/^([^']+\.(?:gz|Z))\z/gzcat '$1' |/;
    s/^([^']+\.bz2)\z/bzcat '$1' |/;
    s/^([^']+\.xz)\z/xzcat '$1' |/;
    s/^([^']+\.zst)\z/zstdcat '$1' |/;
}

# Hand off to regular perl with modified @ARGV
exec 'perl', @ARGV;

Now:

zperl -ne 'print if /error/' access.log*

Handles all compression formats transparently.

Part 6: GREP ALL LOGS

Search through rotated logs:

perl -e '
    s/([^'"'"']+\.gz)$/gzcat '"'"'$1'"'"' |/ for @ARGV;
    print "$ARGV: $_" while <>;
' /var/log/syslog* | grep -i error

The $ARGV variable tells you which file (or pipe) you're reading from.

Or pure Perl:

perl -e '
    s/([^'"'"']+\.gz)$/gzcat '"'"'$1'"'"' |/ for @ARGV;
    print "$ARGV: $_" if /error/i while <>;
' /var/log/syslog*

Part 7: THE QUOTE NIGHTMARE

Those ugly quote escapes:

'"'"'

This is shell escaping. Inside single quotes, you can't use single quotes. So you:

1. End single quote: '
2. Add escaped single quote: "'"
3. Resume single quote: '

Result: '"'"' produces a literal single quote.

In a script file, it's cleaner:

s/^([^']+\.gz)\z/gzcat '$1' |/ for @ARGV;

No shell escaping needed.

Part 8: SECURITY NOTES

This trick uses two-argument open semantics. That's usually bad.

The [^']+ in the regex protects against quote injection. But there are other special characters:

>file       # Opens for writing
|command    # Pipe to command
command|    # Pipe from command

If someone controls your filenames, they control your system.

Safe version:

for (@ARGV) {
    if (/\.gz$/ && -f $_) {  # Must be a real file
        $_ = "gzcat \Q$_\E |";  # \Q quotes metacharacters
    }
}

The -f checks it's a regular file. The \Q escapes special chars.

Part 9: MODERN ALTERNATIVE

PerlIO::gzip does this cleanly:

perl -MPerlIO::gzip -ne 'print if /error/' access.log.gz

But it requires installing a module. The @ARGV trick works with stock Perl.

For serious work, use the module. For quick hacks on random servers where you can't install anything? Magic @ARGV.

Part 10: OTHER @ARGV TRICKS

Set default files:

@ARGV = glob('/var/log/messages*') unless @ARGV;
print while <>;

Expands to all message logs if no args given.

Filter @ARGV:

@ARGV = grep { -f $_ && -r $_ } @ARGV;

Only process readable regular files.

Add STDIN explicitly:

push @ARGV, '-' unless @ARGV;

The dash means STDIN. Now your script works with pipes.

Part 11: THE FULL MONTY

Putting it all together:

#!/usr/bin/env perl
use strict;
use warnings;

# Default to common log locations if no args
@ARGV = glob('/var/log/syslog*') unless @ARGV;

# Handle compressed files
for (@ARGV) {
    next unless -f;  # Skip non-files
    s/^([^']+\.gz)\z/gzcat '$1' |/;
    s/^([^']+\.bz2)\z/bzcat '$1' |/;
    s/^([^']+\.xz)\z/xzcat '$1' |/;
}

# Add STDIN if nothing left
@ARGV = ('-') unless @ARGV;

# Process
while (<>) {
    print "$ARGV:$.: $_" if /error/i;
}

                @ARGV
                  |
          +-------+-------+
          |       |       |
        file    file.gz   -
          |       |       |
        open    gzcat   STDIN
          |       |       |
          +-------+-------+
                  |
                 <>

     Transparent decompression