export LC_ALL=C
awk '!a[$1]++' access.log|head
If access.log is large enough, awk will fail.
When this happens, one can split access.log into pieces, process separately then recombine.
But that's more or less what sort(1) does with large files, creating temporary files in $TMPDIR or other user-specified directory after -T if using GNU sort.
There was a way to eliminate duplicate lines from an unordered list using k/q, without using temporary files but I stopped using it after Kx, Inc. was sold off and I started using musl exclusively. q requires glibc.
For example, something like
#!/bin/sh
# usage: $0 file
echo "k).Q.fs[l:0::\`:$1];l:?:l;\`:$1 0:l"|exec q >null;
Can this be done in ngn k.
The other approach I use to avoid temporary files is to just put the list in an SQL database, add a UNIQUE constraint, and update the database.
When this happens, one can split access.log into pieces, process separately then recombine.
But that's more or less what sort(1) does with large files, creating temporary files in $TMPDIR or other user-specified directory after -T if using GNU sort.
There was a way to eliminate duplicate lines from an unordered list using k/q, without using temporary files but I stopped using it after Kx, Inc. was sold off and I started using musl exclusively. q requires glibc.
For example, something like
Can this be done in ngn k.The other approach I use to avoid temporary files is to just put the list in an SQL database, add a UNIQUE constraint, and update the database.