fetch: correct env var name for --crl option to work

Reviewed by: grembo
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D46973

fetch(1): suppress "Not Modified" in quiet mode

"fetch -qi" should skip printing "Not Modified" for successful
http(s) request. Still print it by default (v_level == 1).

MFC after: 1 week

usr.bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

R

usr.bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

fetch: make -S argument accept values > 2GB

Use strtoll() to parse the argument of the -S option.

FreeBSD has supported 64 bit file offsets for more than 25 years on
all architectures and off_t is

fetch: make -S argument accept values > 2GB

Use strtoll() to parse the argument of the -S option.

FreeBSD has supported 64 bit file offsets for more than 25 years on
all architectures and off_t is a 64 bit integer type for that reason.

While strtol() returns a 64 bit value on 64 LP64 architectures, it
is limit to 32 bit on e.g. i386. The strtoll() function returns a 64
but result on all supported architectures and therefore supports the
possible file lengths and file offsets on 32 bit archtectures.

Reported by: antoine
MFC after: 3 days

show more ...

fetch(1): more fixes for soft failure handling

Fix logic error introduced in my commit
bf599c03f09dea0f7e188e002b42d782af6841c3

Also, authorization errors should not be considered as soft failures.

fetch(1): fix error in previous commit

strncmp() compares terminating zero and sizeof() includes it.
Un-obsfuscate the code and show what it is indended to do.

fetch(1): do not consider HTTP 5XX errors as soft failures

This change fixes "fetch -a" looping forever on "502 Bad gateway"
error and similar.

MFC after: 1 month

fetch(1): correct progress accounting after previous commit

MFC after: 1 month

fetch(1): process truncated transfer as soft failure

Let "fetch -a" resume truncated transfer automatically
perform another attempt if it obtained some new data in previous one
making progress.

Thi

fetch(1): process truncated transfer as soft failure

Let "fetch -a" resume truncated transfer automatically
perform another attempt if it obtained some new data in previous one
making progress.

This makes it more robust against frequent but transient network failures.
For example:

=> sqlite-src-3370200.zip doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://www.sqlite.org/2022/sqlite-src-3370200.zip
sqlite-src-3370200.zip 3% of 12 MB 45 kBps 04m24s
fetch: sqlite-src-3370200.zip appears to be truncated: 524288/13145234 bytes
sqlite-src-3370200.zip 10% of 12 MB 67 kBps 02m56s
fetch: sqlite-src-3370200.zip appears to be truncated: 1327104/13145234 bytes
sqlite-src-3370200.zip 28% of 12 MB 123 kBps 01m14s
fetch: sqlite-src-3370200.zip appears to be truncated: 3735552/13145234 bytes
sqlite-src-3370200.zip 54% of 12 MB 253 kBps 24s
fetch: sqlite-src-3370200.zip appears to be truncated: 7176192/13145234 bytes
sqlite-src-3370200.zip 62% of 12 MB 90 kBps 55s
fetch: sqlite-src-3370200.zip appears to be truncated: 8241152/13145234 bytes
sqlite-src-3370200.zip 82% of 12 MB 113 kBps 20s
fetch: sqlite-src-3370200.zip appears to be truncated: 10862592/13145234 bytes
sqlite-src-3370200.zip 12 MB 185 kBps 12s
===> Fetching all distfiles required by sqlite3-3.37.2,1 for building

MFC after: 1 month

show more ...

Merge ^/head r338392 through r338594.

Through a combination of insufficient variable initialization and
imprudent reuse of static buffers, the end-of-transfer statistics
displayed when stdout is not a tty always ended up as 0 B / 0 Bps.

Through a combination of insufficient variable initialization and
imprudent reuse of static buffers, the end-of-transfer statistics
displayed when stdout is not a tty always ended up as 0 B / 0 Bps.
Reorganize the code to use caller-provided buffers, tweak the ETA
display a bit, and reduce the visual differences between the tty and
non-tty end-of-transfer displays.

PR: 202424
Approved by: re (gjb@)

show more ...

fetch: correct env var name for --crl option to work

Reviewed by: grembo
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D46973

fetch(1): suppress "Not Modified" in quiet mode

"fetch -qi" should skip printing "Not Modified" for successful
http(s) request. Still print it by default (v_level == 1).

MFC after: 1 week

usr.bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

R

usr.bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

fetch: make -S argument accept values > 2GB

Use strtoll() to parse the argument of the -S option.

FreeBSD has supported 64 bit file offsets for more than 25 years on
all architectures and off_t is

fetch: make -S argument accept values > 2GB

Use strtoll() to parse the argument of the -S option.

FreeBSD has supported 64 bit file offsets for more than 25 years on
all architectures and off_t is a 64 bit integer type for that reason.

While strtol() returns a 64 bit value on 64 LP64 architectures, it
is limit to 32 bit on e.g. i386. The strtoll() function returns a 64
but result on all supported architectures and therefore supports the
possible file lengths and file offsets on 32 bit archtectures.

Reported by: antoine
MFC after: 3 days

show more ...

fetch(1): more fixes for soft failure handling

Fix logic error introduced in my commit
bf599c03f09dea0f7e188e002b42d782af6841c3

Also, authorization errors should not be considered as soft failures.

fetch(1): fix error in previous commit

strncmp() compares terminating zero and sizeof() includes it.
Un-obsfuscate the code and show what it is indended to do.

fetch(1): do not consider HTTP 5XX errors as soft failures

This change fixes "fetch -a" looping forever on "502 Bad gateway"
error and similar.

MFC after: 1 month

fetch(1): correct progress accounting after previous commit

MFC after: 1 month

fetch(1): process truncated transfer as soft failure

Let "fetch -a" resume truncated transfer automatically
perform another attempt if it obtained some new data in previous one
making progress.

Thi

fetch(1): process truncated transfer as soft failure

Let "fetch -a" resume truncated transfer automatically
perform another attempt if it obtained some new data in previous one
making progress.

This makes it more robust against frequent but transient network failures.
For example:

=> sqlite-src-3370200.zip doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://www.sqlite.org/2022/sqlite-src-3370200.zip
sqlite-src-3370200.zip 3% of 12 MB 45 kBps 04m24s
fetch: sqlite-src-3370200.zip appears to be truncated: 524288/13145234 bytes
sqlite-src-3370200.zip 10% of 12 MB 67 kBps 02m56s
fetch: sqlite-src-3370200.zip appears to be truncated: 1327104/13145234 bytes
sqlite-src-3370200.zip 28% of 12 MB 123 kBps 01m14s
fetch: sqlite-src-3370200.zip appears to be truncated: 3735552/13145234 bytes
sqlite-src-3370200.zip 54% of 12 MB 253 kBps 24s
fetch: sqlite-src-3370200.zip appears to be truncated: 7176192/13145234 bytes
sqlite-src-3370200.zip 62% of 12 MB 90 kBps 55s
fetch: sqlite-src-3370200.zip appears to be truncated: 8241152/13145234 bytes
sqlite-src-3370200.zip 82% of 12 MB 113 kBps 20s
fetch: sqlite-src-3370200.zip appears to be truncated: 10862592/13145234 bytes
sqlite-src-3370200.zip 12 MB 185 kBps 12s
===> Fetching all distfiles required by sqlite3-3.37.2,1 for building

MFC after: 1 month

show more ...

Merge ^/head r338392 through r338594.

Through a combination of insufficient variable initialization and
imprudent reuse of static buffers, the end-of-transfer statistics
displayed when stdout is not a tty always ended up as 0 B / 0 Bps.

Through a combination of insufficient variable initialization and
imprudent reuse of static buffers, the end-of-transfer statistics
displayed when stdout is not a tty always ended up as 0 B / 0 Bps.
Reorganize the code to use caller-provided buffers, tweak the ETA
display a bit, and reduce the visual differences between the tty and
non-tty end-of-transfer displays.

PR: 202424
Approved by: re (gjb@)

show more ...

Support If-Modified-Since for https as well as http.

PR: 224426
Submitted by: zsnafzig@edu.uwaterloo.ca
MFC after: 1 week