textmode
Joined 10 years ago
Karma
234
Submissions
319
Member since
Jun 15
View on Hacker News
Recent submissions
Comment · 2 years ago
correction:
sed -n '/pattern/=' file|yy092|sed -nf/dev/stdin file
Comment · 2 years ago
Correction:
/* remove HTTP headers from multiple gzip or single zip from stdin */
int fileno (FILE *);
int setenv (const char *, const char *, int);
#define jmp (yy_start) = 1 + 2 *
int x;
%option nounput noinput noyywrap
%%
HTTP\/[\40-\176]+\x0d\x0a x++;
[\40-\176]+:[\40-\176]+\r\n if(!x)fwrite(yytext,1,yyleng,yyout);
\x0D\x0A if(!x)fwrite(yytext,1,yyleng,yyout);x=0;
%%
int main()
{
yylex();
exit(0);
}
Retrieve hostnames, IP addresses and (if available) sitemap URLs from latest Common Crawl.
ftp -4 https://data.commoncrawl.org/crawl-data/CC-MAIN-2023-50/robotstxt.paths.gz # <-- 180K
gzip -dc robotstxt.paths.gz \
|head -5 \
|sed 's>.*>GET /& HTTP/1.1[]Host: data.commoncrawl.org[]Connection: >;
$!s/$/keep-alive[]/;$s/$/close[]/' \
|tr [] '\r\n' \
|openssl s_client -quiet -connect data.commoncrawl.org:443 \
|yy054 \
|zegrep -a '(^Sitemap:)|(^Host:)|(^WARC-Target-URI:)|(^WARC-IP-Address:)' > 1.txt
exec cat 1.txtComment · 2 years ago
Usage example:
Download NetBSD 1.0 in a single TCP connection.
y="GET /pub/NetBSD-archive/NetBSD-1.0/source/src10/"
z="Host: archive.netbsd.org"
sed '$!s>.*>'"$y"'& HTTP/1.1[]'"$z"'[]Connection: keep-alive[]>;
$s>.*>'"$y"'& HTTP/1.0[]'"$z"'[]>' << eof \
|tr '[]' '\r\n' \
|openssl s_client -quiet -connect 151.101.129.6:443 -servername archive.netbsd.org > http+gzip
src10.aa
src10.ab
src10.ac
src10.ad
src10.ae
src10.af
src10.ag
src10.ah
src10.ai
src10.aj
src10.ak
src10.al
src10.am
src10.an
src10.ao
src10.ap
src10.aq
src10.ar
src10.as
src10.at
src10.au
src10.av
src10.aw
src10.ax
src10.ay
src10.az
src10.ba
src10.bb
src10.bc
src10.bd
src10.be
src10.bf
eof
yy054 < http+gzip|tar tvzf /dev/stdin
Include an argv[1] will print HTTP headers only
yy054 print < http+gzip
yy054 x < http+gzipComment · 2 years ago
Normally I use yy030 but I have been experimenting with this instead.
Seems to be slightly faster and smaller than similar programs from html-xml-utils.
https://www.w3.org/Tools/HTML-XML-utils/man1/
Compile:
links -no-connect -dump https://news.ycombinator.com/item?id=38727772 \
|sed '1,4d;77,$d;s/[ ]\{6\}//' \
|flex -8Cem;cc -O3 -std=c89 -W -Wall -pipe lex.yy.c -static -o yy044
strip -s yy044
# NB. not a real cookie
curl -H "cookie=user=santa&K7RGzmUtAoKv9OIRMfQ9bfwYpiDEuypp" -siA "" \
https://news.ycombinator.com \
|host=news.ycombinator.com/ yy044 r \
|sed -n 's/&/\&/g;/vote/p'Comment · 4 years ago
/* chunked transfer decoding */
#define echo do{if(fwrite(yytext,(size_t)yyleng,1,yyout)){}}while(0)
#define jmp (yy_start) = 1 + 2 *
int fileno (FILE *);
int ischunked,chunksize,count;
xa "\15"|"\12"
xb "\15\12"
xc "HTTP/0.9"|"HTTP/1.0"|"HTTP/1.1"
xd [Cc][Hh][Uu][Nn][Kk][Ee][Dd]
xe [0-9a-fA-F]+\r\n
xf [0-9a-fA-F]*\r\n
%option noyywrap nounput noinput
%s xb xc xd xe xf
%%
^{xc} echo;ischunked=0;jmp xc;
<xc>^transfer-encoding: echo;jmp xb;
<xb>\r\n\r\n echo;jmp xe;
<xb>{xd} echo;ischunked=1;
<xe>{xf}|{xe} {
count=0;
if(ischunked==1)
{chunksize=strtol(yytext,NULL,16);
jmp xd;};
};
<xd>{xb} jmp xf;
<xd>. {
count++;
if(count==chunksize)jmp xe;
echo;
};
<xf>^[A-Fa-f0-9]+{xa}
<xf>{xa}+[A-Fa-f0-9]+{xa}
<xf>{xb}[A-Fa-f0-9]+{xb}
%%
int main(){ yylex();exit(0);}Comment · 5 years ago
Below is a short script that downloads and makes a PDF from the image files. No browser required.
The script uses a feature of HTTP/1.1 called pipelining; proponents of HTTP/2 and HTTP/3 want people to believe it has problems because it does not fit their commercialised web business model. As demonstrated by the script below, it has no problems. It's a feature that simply does not suit the online ad industry-funded business model with its gigantic corporate browser, bloated conglomeration web pages and incessant data collection. Here, only 2 TCP connections are used to retrieve 141 images. Most servers are less restrictive and allow more than 100 requests per TCP connection. Pipelining works great. Much more efficient than browsers which open hundreds of connections. IMHO.
(export Connection=keep-alive
x1=http://www.minimizedistraction.com/img/vrg_google_doc_final_vrs03-
x2(){ seq -f "$x1%g.jpg" $1 $2;};
x3(){ yy025|nc -vvn 173.236.175.199 80;};
x2 1 100|x3;
x2 101 200|x3;
)|exec yy056|exec od -An -tx1 -vw99999|exec tr -d '\40'|exec sed 's/ffd9ffd8/ffd9\
ffd8/g'|exec sed -n /ffd8/p|exec split -l1;
for x in x??;do xxd -p -r < $x > $x.jpg;rm $x;done;
convert x??.jpg 1.pdf 2>/dev/null;rm x??.jpg
ls -l ./1.pdf