libexec: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

R

libexec: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

show more ...

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

show more ...

phttpget: move out of portsnap

Currently, WITHOUT_PORTSNAP forces WITHOUT_FREEBSD_UPDATE because the
latter relies on phttpget, which lives inside the portsnap build bits.

Remove the dependency bet

phttpget: move out of portsnap

Currently, WITHOUT_PORTSNAP forces WITHOUT_FREEBSD_UPDATE because the
latter relies on phttpget, which lives inside the portsnap build bits.

Remove the dependency between these two options by moving phttpget out into
^/libexec and building/installing it if either WITH_PORTSNAP or
WITH_FREEBSD_UPDATE.

Future work could remove the conditional if it's decided that users will use
it independently of either the current in-base consumers.

Reported by: swills
Reviewed by: jilles, emaste
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D26255

show more ...

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

show more ...

Cleanup unnecessary semicolons from utilities we all love.

Set SO_NOSIGPIPE on sockets used by phttpget. Without this, if
(1) phttpget is attempting to download enough files that it can't send
all the requests at once, and
(2) the remote server forcibly clo

Set SO_NOSIGPIPE on sockets used by phttpget. Without this, if
(1) phttpget is attempting to download enough files that it can't send
all the requests at once, and
(2) the remote server forcibly closes the connection, resulting in RST
packets being sent,
phttpget will receive a SIGPIPE and terminate without downloading all
of the files.

This is probably responsible for a number of hard-to-reproduce errors
with portsnap and freebsd-update.

MFC after: 3 days

show more ...

Improve conformance to the HTTP specification by using case-insensitive
comparisons for header keywords. Apparently some proxies use creative
capitalization.

Weird proxy found by: brooks
MFC after:

Improve conformance to the HTTP specification by using case-insensitive
comparisons for header keywords. Apparently some proxies use creative
capitalization.

Weird proxy found by: brooks
MFC after: 3 days

show more ...

Add support for HTTP/1.0 Persistent Connections to phttpget. Requests are
be marked as HTTP/1.1 but "Connection: Keep-Alive" is added; this convinces
HTTP/1.0 servers and proxies to hold the TCP con

Add support for HTTP/1.0 Persistent Connections to phttpget. Requests are
be marked as HTTP/1.1 but "Connection: Keep-Alive" is added; this convinces
HTTP/1.0 servers and proxies to hold the TCP connection open despite not
being able to use HTTP pipelining.

This dramatically cuts down on the number of TCP connections (and thus port
numbers) used by portsnap when talking to an HTTP/1.0 proxy (e.g., squid),
and has the side benefit of improving performance in those cases.

Tested by: simon
Approved by: re (kensmith)
MFC After: 1 week

show more ...

Add support for the HTTP_TIMEOUT environment variable (integer number
of seconds, just like in fetch(1)).

Submitted by: rdivacky

Allow the HTTP_PROXY environment variable to be (mis)spelled as
"http_proxy", since some people apparently do this and fetch(3)
allows it.

Correct handling of HTTP_PROXY_AUTH. A password may have `:'.

Approved by: cperciva
MFC after: 2 days

Merge from accidental commit to RELENG_5:
Correctly identify the host and port values on a failed
getaddrinfo lookup.

Correctly handle a TCP connection being shutdown by the server while
we're reading response headers. (Handle it as a connection-killing
error, rather than entering an infinite loop reading zero byte

Correctly handle a TCP connection being shutdown by the server while
we're reading response headers. (Handle it as a connection-killing
error, rather than entering an infinite loop reading zero bytes.)

Reported by: simon
Discovered thanks to: A not-very-transparent transparent HTTP proxy.
MFC after: 3 days

show more ...

Add HTTP proxy authentication, via the HTTP_PROXY_AUTH environment
variable.

Tested by: Emil Mikulic
X-MFC-After: 6.0-RELEASE

Style fix: "if (pointer)" -> "if (pointer != NULL)"

Remove unused variable.

Reported by: stefanf

Replace INT_MAX with OFF_MAX. I used INT_MAX originally (in ports)
because OFF_MAX wasn't defined on FreeBSD 4.x.

When parsing the HTTP_PROXY environment variable, strip a trailing /
from the port number (if any exists). This unbreaks
env HTTP_PROXY="http://localhost:3128/" portsnap fetch

While I'm here, list

When parsing the HTTP_PROXY environment variable, strip a trailing /
from the port number (if any exists). This unbreaks
env HTTP_PROXY="http://localhost:3128/" portsnap fetch

While I'm here, list both the host and the port in the error message
output if getaddrinfo() fails, since either of them could be responsible
for the failure.

show more ...

Add portsnap to the base system. This is a secure, easy to use,
fast, lightweight, and generally good way for users to keep their
ports trees up to date.

This is version 0.9.4 from the ports tree (

Add portsnap to the base system. This is a secure, easy to use,
fast, lightweight, and generally good way for users to keep their
ports trees up to date.

This is version 0.9.4 from the ports tree (sysutils/portsnap) with
the following changes:
1. The experimental pipelined http code is enabled. No seatbelts
in -CURRENT. (^_^)
2. The working directory has moved from /usr/local/portsnap to
/var/db/portsnap (as discussed on -arch two days ago).
3. Portsnap now fetches a list of mirrors (distributed as DNS SRV
records) and selects one randomly. This should help to avoid the
uneven loading which plagues the cvsup mirror network.
4. The license is now 2-clause BSD instead of 3-clause BSD.
5. Various incidental changes to make portsnap fit into the base
system's build mechanics.

X-MFC-After: 6.0-RELEASE
X-MFC-Before: 5.5-RELEASE
X-MFC-To: RELENG_6, RELENG_5, ports
discussed on: -arch and several other places
"yes please" from: simon, remko, flz, Diane Bruce
thinks this is a great idea: bsdimp
Hopes he didn't forget any files: cperciva

show more ...