NEWS: update

If not --trust-server-names is used, FTP will also get the destination
file name from the original url specified by the user instead of the
redirected url.  Closes CVE-2016-4971.

* src/ftp.c (ftp_get_listing): Add argument original_url.
(getftp): Likewise.
(ftp_loop_internal): Likewise.  Use original_url to generate the
file name if --trust-server-names is not provided.
(ftp_retrieve_glob): Likewise.
(ftp_loop): Likewise.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

.gitignore

@@ -2,6 +2,7 @@
 /.sc-start-sc_bindtextdomain
 /.sc-start-sc_prohibit_HAVE_MBRTOWC
 /.version
+/ABOUT-NLS
 /GNUmakefile
 /INSTALL
 /Makefile
@@ -25,6 +26,7 @@ testenv/*.log
 testenv/*.trs
 # build-aux/
 build-aux/.gitignore
+build-aux/ar-lib
 build-aux/compile
 build-aux/config.guess
 build-aux/config.sub

.travis.yml

@@ -0,0 +1,50 @@
+sudo: no
+
+language: c
+
+compiler:
+    - gcc
+
+notifications:
+    email:
+        recipients:
+            - darnir@gmail.com
+            - bug-wget@gnu.org
+        on_success: change
+        on_failure: always
+
+env:
+    global:
+        - PERLLIB=$PERLLIB:../../tests
+        - PERL5LIB=$PERL5LIB:../../tests
+    matrix:
+        - SSL=""
+        - SSL="openssl"
+        - SSL="gnutls"
+
+addons:
+    apt:
+        packages:
+            - automake
+            - autoconf
+            - autopoint
+            - flex
+            - texinfo
+            - pkg-config
+            - libgnutls-dev
+            - libssl-dev
+            - make
+            - libhttp-daemon-perl
+            - libio-socket-ssl-perl
+            - libidn11-dev
+            - gettext
+            - texlive
+            - python3
+            - valgrind
+            - language-pack-tr
+            - language-pack-ru
+
+script:
+    - make distclean || true
+    - ./bootstrap
+    - ./contrib/travis-ci $SSL

MAILING-LIST

@@ -23,15 +23,6 @@ Gmane archives conveniently include messages from both the current
 list, and the previous one. Messages also show up in the Gmane archives
 sooner than they do at `lists.gnu.org'.
 
-Bug Notices List
-----------------
-
-Additionally, there is the <wget-notify@addictivecode.org> mailing
-list. This is a non-discussion list that receives bug report
-notifications from the bug-tracker. To subscribe to this list, send an
-email to <wget-notify-join@addictivecode.org>, or visit
-`http://addictivecode.org/mailman/listinfo/wget-notify'.
-
 Obsolete Lists
 --------------
 

NEWS

@@ -7,6 +7,42 @@ See the end for copying conditions.
 
 Please send GNU Wget bug reports to <bug-wget@gnu.org>.
 
+* Changes in Wget 1.18
+
+* By default, on server redirects to a FTP resource, use the original
+  URL to get the local file name. Close CVE-2016-4971.  This
+  introduces a backward-incompatibility for HTTP->FTP redirects and
+  any script that relies on the old  behaviour must use
+  --trust-server-names.
+
+* Check the HSTS file is not world-writable before using it.
+
+* Parse <img srcset> attributes on a recursive download.
+
+* Fix problem with SNI server names having trailing dot(s)
+
+* New options --bind-dns-address and --dns-servers.
+
+* When Wget is built with libiconv, it now converts non-ASCII URIs to
+  the locale's codeset when it creates files.  The encoding of the
+  remote files and URIs is taken from --remote-encoding, defaulting to
+  UTF-8.  The result is that non-ASCII URIs and files downloaded via
+  HTTP/HTTPS and FTP will have names on the local filesystem that
+  correspond to their remote names.
+
+* Changes in Wget 1.17.1
+
+* Fix compile error when IPv6 is disabled or SSL is not present.
+
+* Fix HSTS memory leak.
+
+* Fix progress output in non-C locales.
+
+* Fix SIGSEGV when -N and --content-disposition are used together.
+
+* Add --check-certificate=quiet to tell wget to not print any warning about
+  invalid certificates.
+
 * Changes in Wget 1.17
 
 ** Remove FTP passive to active fallback due to privacy concerns.

README.checkout

@@ -25,10 +25,11 @@ Compiling From Repository Sources
        tests in the tests/ sub directory. Tarball distributions include an
        already-generated wget.1 manual. The command "make check" runs the
        test suite written in perl and python (see below). To execute all the tests
-       you need [24]libwww-perl and libio-socket-ssl-perl perl library. If
-       "perl -MCPAN -e 'install Bundle::LWP'" fails then you most likely don't
-       have cpan module installed. First download [25]CPAN and [26]install it.
-       Then execute "perl -MCPAN -e 'install Bundle::LWP'".
+       you need libwww-perl and libio-socket-ssl-perl perl library.
+       GNU/Linux distros likely have these two libraries packaged.
+       If not, install them directly with "perl -MCPAN -e 'install Bundle::LWP'"[24].
+       If that fails then you most likely don't have the CPAN module installed.
+       First download [25]CPAN and [26]install it, then try again the above command.
        Now "make check" should pass most of the tests in the test suite.
 
      * [45]Python, if you want to run the tests in the testenv/ subdirectory.
@@ -99,6 +100,9 @@ Compiling From Repository Sources
 
      * [47]GnuPG with GPGME is used to verify GPG-signed Metalink resources.
 
+     * [48]libcares is needed to bind DNS resolving to a given IP address.
+       The command line options --dns-servers and --bind-dns-address are
+       only available when configured with --with-cares.
 
    For those who might be confused as to what to do once they check out
    the source code, considering configure and Makefile do not yet exist at
@@ -207,3 +211,4 @@ References
   45. https://www.python.org/
   46. https://launchpad.net/libmetalink
   47. https://www.gnupg.org
+  48. http://c-ares.haxx.se/

bootstrap.conf

@@ -38,6 +38,7 @@ close
 connect
 dirname
 fcntl
+flock
 fnmatch
 futimens
 ftello

configure.ac

@@ -57,6 +57,8 @@ AC_CANONICAL_HOST
 AC_DEFINE_UNQUOTED([OS_TYPE], "$host_os",
                    [Define to be the name of the operating system.])
 
+dnl Non-verbose make
+m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])])
 
 dnl
 dnl Process External Libraries
@@ -70,7 +72,6 @@ dnl SSL: Configure SSL backend to use
 AC_ARG_WITH([ssl],
   [AS_HELP_STRING([--with-ssl={gnutls,openssl}], [specify SSL backend. GNU TLS is the default.])])
 
-
 dnl Zlib: Configure use of zlib for compression
 AC_ARG_WITH([zlib],
   [AS_HELP_STRING([--without-zlib], [disable zlib.])])
@@ -79,6 +80,9 @@ dnl Metalink: Configure use of the Metalink library
 AC_ARG_WITH([metalink],
   [AS_HELP_STRING([--with-metalink], [enable support for metalinks.])])
 
+dnl C-Ares: Configure use of the c-ares library for DNS lookup
+AC_ARG_WITH(cares, AS_HELP_STRING([--with-cares], [enable support for C-Ares DNS lookup.]), with_cares=$withval, with_cares=no)
+
 dnl
 dnl Process features
 dnl
@@ -180,7 +184,7 @@ dnl
 dnl Gettext
 dnl
 AM_GNU_GETTEXT([external],[need-ngettext])
-AM_GNU_GETTEXT_VERSION([0.17])
+AM_GNU_GETTEXT_VERSION([0.18.1])
 
 AC_PROG_RANLIB
 
@@ -742,6 +746,31 @@ AS_IF([test "X$enable_pcre" != "Xno"],[
   ])
 ])
 
+dnl
+dnl Check for libcares (resolver library)
+dnl
+
+AS_IF([test "X$with_cares" == "Xyes"],[
+  PKG_CHECK_MODULES([CARES], libcares, [
+    CFLAGS="$CARES_CFLAGS $CFLAGS"
+    AC_CHECK_HEADER(ares.h, [
+      LIBS="$CARES_LIBS $LIBS"
+      AC_DEFINE([HAVE_LIBCARES], [1], [Define if libcares is available.])
+      RESOLVER_INFO="libcares, --bind-dns-address and --dns-servers available"
+    ])
+  ], [
+    AC_CHECK_HEADER(ares.h, [
+      AC_CHECK_LIB(cares, ares_set_local_ip4, [
+        LIBS="-lcares ${LIBS}"
+        AC_DEFINE([HAVE_LIBCARES], 1, [Define if libcares is available.])
+        RESOLVER_INFO="libcares, --bind-dns-address and --dns-servers available"
+      ])
+    ])
+  ])
+], [
+  RESOLVER_INFO="libc, --bind-dns-address and --dns-servers not available"
+])
+
 
 dnl Needed by src/Makefile.am
 AM_CONDITIONAL([IRI_IS_ENABLED], [test "X$iri" != "Xno"])
@@ -776,5 +805,6 @@ AC_MSG_NOTICE([Summary of build options:
   Assertions:        $ENABLE_ASSERTION
   Valgrind:          $VALGRIND_INFO
   Metalink:          $with_metalink
+  Resolver:          $RESOLVER_INFO
   GPGME:             $have_gpg
 ])

contrib/travis-ci

@@ -0,0 +1,60 @@
+#!/bin/bash
+
+# Set a stricter bash mode
+set -e
+set -u
+
+# Set up the subset of tests this Matrix instance will run.
+
+# One instance is forked for each of the SSL library possibilities
+SSL=${1:-""}
+case $SSL in
+    "") SSL_LIB="--without-ssl";;
+    "openssl") SSL_LIB="--with-ssl=openssl";;
+    "gnutls") SSL_LIB="--with-ssl=gnutls";;
+esac
+
+# For some reason. /proc/cpuinfo reports 16 cores on Travis, while the docs
+# claim that each instance has only 2 cores. We believe the docs and force a
+# value of n+1 here.
+CORES=3
+
+# Define a large number of Warning flags for the compiler. Hopefully, someone
+# will sit and analyze the output to clean the warnings from the codebase.
+CFLAGS="-std=c89 -pedantic -O2 -Wall -Wextra -Wstrict-prototypes -Wold-style-definition -Wwrite-strings -Wshadow -Wformat -Wformat-security -Wunreachable-code -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition"
+
+# A cachefile for ./configure. Saves significant time when reconfiguring
+CACHEFILE=$PWD/config_check.cache
+
+# measure time consumed and print it at the end of the script
+START=$(date +%s)
+
+# Test different Permutations of Wget. Always under Valgrind
+#
+# Options:
+#   - All Features Enabled
+#     - C       [Default Locale]
+#     - Russian [Multibyte / Multicolumn Locale]
+#     - Turkish [Special Test due to issues with language]
+#   - IPv6 Disabled
+#     - C
+for options in "" "--disable-ipv6"; do
+  export DISTCHECK_CONFIGURE_FLAGS="-C --cache-file=$CACHEFILE --enable-assert --enable-valgrind-tests $SSL_LIB $options"
+  echo "  ./configure $DISTCHECK_CONFIGURE_FLAGS CFLAGS=\"$CFLAGS\""
+  ./configure $DISTCHECK_CONFIGURE_FLAGS CFLAGS="$CFLAGS"
+
+  if [[ -z $options ]]; then
+      LANG_ARR=("ru_RU.UTF-8" "tr_TR.UTF-8" "C")
+  else
+      LANG_ARR=("C")
+  fi
+
+  for xLCALL in "${LANG_ARR[@]}"; do
+    export TESTS_ENVIRONMENT="LC_ALL=$xLCALL VALGRIND_TESTS=1"
+    echo "    TESTS_ENVIRONMENT=\"$TESTS_ENVIRONMENT\" make distcheck CFLAGS=$CFLAGS -j$CORES"
+    make distcheck CFLAGS="$CFLAGS" -j$CORES
+  done
+done
+
+END=$(date +%s)
+echo "Duration: $((END-START))"

doc/Makefile.am

@@ -57,7 +57,10 @@ wget.pod: $(srcdir)/wget.texi version.texi
 	$(TEXI2POD) -D VERSION="$(VERSION)" $(srcdir)/wget.texi $@
 
 $(MAN): wget.pod
-	$(POD2MAN) --center="GNU Wget" --release="GNU Wget @VERSION@" --utf8 $? > $@
+	$(POD2MAN) --center="GNU Wget" --release="GNU Wget @VERSION@" --utf8 $? > $@; \
+	if [ $? != 0 ]; then \
+          $(POD2MAN) --center="GNU Wget" --release="GNU Wget @VERSION@" $? > $@; \
+	fi
 
 #wget.cat: $(MAN)
 #	nroff -man $? > $@

doc/wget.texi

@@ -571,6 +571,27 @@ the local machine.  @var{ADDRESS} may be specified as a hostname or IP
 address.  This option can be useful if your machine is bound to multiple
 IPs.
 
+@cindex bind DNS address
+@cindex client DNS address
+@cindex DNS IP address, client, DNS
+@item --bind-dns-address=@var{ADDRESS}
+[libcares only]
+This address overrides the route for DNS requests. If you ever need to
+circumvent the standard settings from /etc/resolv.conf, this option together
+with @samp{--dns-servers} is your friend.
+@var{ADDRESS} must be specified either as IPv4 or IPv6 address.
+Wget needs to be built with libcares for this option to be available.
+
+@cindex DNS server
+@cindex DNS IP address, client, DNS
+@item --dns-servers=@var{ADDRESSES}
+[libcares only]
+The given address(es) override the standard nameserver
+addresses,  e.g. as configured in /etc/resolv.conf.
+@var{ADDRESSES} may be specified either as IPv4 or IPv6 addresses,
+comma-separated.
+Wget needs to be built with libcares for this option to be available.
+
 @cindex retries
 @cindex tries
 @cindex number of tries
@@ -694,13 +715,11 @@ Without @samp{-c}, the previous example would just download the remote
 file to @file{ls-lR.Z.1}, leaving the truncated @file{ls-lR.Z} file
 alone.
 
-Beginning with Wget 1.7, if you use @samp{-c} on a non-empty file, and
-it turns out that the server does not support continued downloading,
-Wget will refuse to start the download from scratch, which would
-effectively ruin existing contents.  If you really want the download to
-start from scratch, remove the file.
+If you use @samp{-c} on a non-empty file, and the server does not support
+continued downloading, Wget will restart the download from scratch and overwrite
+the existing file entirely.
 
-Also beginning with Wget 1.7, if you use @samp{-c} on a file which is of
+Beginning with Wget 1.7, if you use @samp{-c} on a file which is of
 equal size as the one on the server, Wget will refuse to download the
 file and print an explanatory message.  The same happens when the file
 is smaller on the server than locally (presumably because it was changed
@@ -983,7 +1002,7 @@ specified in bytes (default), kilobytes (with @samp{k} suffix), or
 megabytes (with @samp{m} suffix).
 
 Note that quota will never affect downloading a single file.  So if you
-specify @samp{wget -Q10k ftp://wuarchive.wustl.edu/ls-lR.gz}, all of the
+specify @samp{wget -Q10k https://example.com/ls-lR.gz}, all of the
 @file{ls-lR.gz} will be downloaded.  The same goes even when several
 @sc{url}s are specified on the command-line.  However, quota is
 respected when retrieving either recursively, or from an input file.
@@ -1586,11 +1605,11 @@ users:
 # @r{Log in to the server.  This can be done only once.}
 wget --save-cookies cookies.txt \
      --post-data 'user=foo&password=bar' \
-     http://server.com/auth.php
+     http://example.com/auth.php
 
 # @r{Now grab the page or pages we care about.}
 wget --load-cookies cookies.txt \
-     -p http://server.com/interesting/article.php
+     -p http://example.com/interesting/article.php
 @end group
 @end example
 
@@ -1725,6 +1744,12 @@ this option to bypass the verification and proceed with the download.
 site's authenticity, or if you really don't care about the validity of
 its certificate.}  It is almost always a bad idea not to check the
 certificates when transmitting confidential or important data.
+For self-signed/internal certificates, you should download the certificate
+and verify against that instead of forcing this insecure mode.
+If you are really sure of not desiring any certificate verification, you
+can specify --check-certificate=quiet to tell wget to not print any
+warning about invalid certificates, albeit in most cases this is the
+wrong thing to do.
 
 @cindex SSL certificate
 @item --certificate=@var{file}
@@ -1772,6 +1797,18 @@ system-specified locations, chosen at OpenSSL installation time.
 Specifies a CRL file in @var{file}.  This is needed for certificates
 that have been revocated by the CAs.
 
+@cindex SSL Public Key Pin
+@item --pinnedpubkey=file/hashes
+Tells wget to use the specified public key file (or hashes) to verify the peer.
+This can be a path to a file which contains a single public key in PEM or DER
+format, or any number of base64 encoded sha256 hashes preceded by ``sha256//''
+and separated by ``;''
+
+When negotiating a TLS or SSL connection, the server sends a certificate
+indicating its identity. A public key is extracted from this certificate and if
+it does not exactly match the public key(s) provided to this option, wget will
+abort the connection before sending or receiving any data.
+
 @cindex entropy, specifying source of
 @cindex randomness, specifying source of
 @item --random-file=@var{file}
@@ -2555,11 +2592,11 @@ The @samp{-D} option allows you to specify the domains that will be
 followed, thus limiting the recursion only to the hosts that belong to
 these domains.  Obviously, this makes sense only in conjunction with
 @samp{-H}.  A typical example would be downloading the contents of
-@samp{www.server.com}, but allowing downloads from
-@samp{images.server.com}, etc.:
+@samp{www.example.com}, but allowing downloads from
+@samp{images.example.com}, etc.:
 
 @example
-wget -rH -Dserver.com http://www.server.com/
+wget -rH -Dexample.com http://www.example.com/
 @end example
 
 You can specify more than one address by separating them with a comma,
@@ -2799,7 +2836,7 @@ These links are not relative:
 @example
 <a href="/foo.gif">
 <a href="/foo/bar.gif">
-<a href="http://www.server.com/foo/bar.gif">
+<a href="http://www.example.com/foo/bar.gif">
 @end example
 
 Using this option guarantees that recursive retrieval will not span
@@ -3669,7 +3706,7 @@ same directory structure the original has, with only one try per
 document, saving the log of the activities to @file{gnulog}:
 
 @example
-wget -r http://www.gnu.org/ -o gnulog
+wget -r https://www.gnu.org/ -o gnulog
 @end example
 
 @item
@@ -3677,7 +3714,7 @@ The same as the above, but convert the links in the downloaded files to
 point to local files, so you can view the documents off-line:
 
 @example
-wget --convert-links -r http://www.gnu.org/ -o gnulog
+wget --convert-links -r https://www.gnu.org/ -o gnulog
 @end example
 
 @item
@@ -3687,22 +3724,22 @@ sheets, are also downloaded.  Also make sure the downloaded page
 references the downloaded links.
 
 @example
-wget -p --convert-links http://www.server.com/dir/page.html
+wget -p --convert-links http://www.example.com/dir/page.html
 @end example
 
-The @sc{html} page will be saved to @file{www.server.com/dir/page.html}, and
-the images, stylesheets, etc., somewhere under @file{www.server.com/},
+The @sc{html} page will be saved to @file{www.example.com/dir/page.html}, and
+the images, stylesheets, etc., somewhere under @file{www.example.com/},
 depending on where they were on the remote server.
 
 @item
-The same as the above, but without the @file{www.server.com/} directory.
+The same as the above, but without the @file{www.example.com/} directory.
 In fact, I don't want to have all those random server directories
 anyway---just save @emph{all} those files under a @file{download/}
 subdirectory of the current directory.
 
 @example
 wget -p --convert-links -nH -nd -Pdownload \
-     http://www.server.com/dir/page.html
+     http://www.example.com/dir/page.html
 @end example
 
 @item
@@ -3731,12 +3768,12 @@ wget -r -l2 -P/tmp ftp://wuarchive.wustl.edu/
 
 @item
 You want to download all the @sc{gif}s from a directory on an @sc{http}
-server.  You tried @samp{wget http://www.server.com/dir/*.gif}, but that
+server.  You tried @samp{wget http://www.example.com/dir/*.gif}, but that
 didn't work because @sc{http} retrieval does not support globbing.  In
 that case, use:
 
 @example
-wget -r -l1 --no-parent -A.gif http://www.server.com/dir/
+wget -r -l1 --no-parent -A.gif http://www.example.com/dir/
 @end example
 
 More verbose, but the effect is the same.  @samp{-r -l1} means to
@@ -3752,7 +3789,7 @@ interrupted.  Now you do not want to clobber the files already present.
 It would be:
 
 @example
-wget -nc -r http://www.gnu.org/
+wget -nc -r https://www.gnu.org/
 @end example
 
 @item
@@ -3760,7 +3797,7 @@ If you want to encode your own username and password to @sc{http} or
 @sc{ftp}, use the appropriate @sc{url} syntax (@pxref{URL Format}).
 
 @example
-wget ftp://hniksic:mypassword@@unix.server.com/.emacs
+wget ftp://hniksic:mypassword@@unix.example.com/.emacs
 @end example
 
 Note, however, that this usage is not advisable on multi-user systems
@@ -3797,7 +3834,7 @@ to recheck a site each Sunday:
 
 @example
 crontab
-0 0 * * 0 wget --mirror http://www.gnu.org/ -o /home/me/weeklog
+0 0 * * 0 wget --mirror https://www.gnu.org/ -o /home/me/weeklog
 @end example
 
 @item
@@ -3809,7 +3846,7 @@ would look like this:
 
 @example
 wget --mirror --convert-links --backup-converted  \
-     http://www.gnu.org/ -o /home/me/weeklog
+     https://www.gnu.org/ -o /home/me/weeklog
 @end example
 
 @item
@@ -3822,13 +3859,13 @@ or @samp{application/xhtml+xml} to @file{@var{name}.html}.
 @example
 wget --mirror --convert-links --backup-converted \
      --html-extension -o /home/me/weeklog        \
-     http://www.gnu.org/
+     https://www.gnu.org/
 @end example
 
 Or, with less typing:
 
 @example
-wget -m -k -K -E http://www.gnu.org/ -o /home/me/weeklog
+wget -m -k -K -E https://www.gnu.org/ -o /home/me/weeklog
 @end example
 @end itemize
 @c man end
@@ -3935,14 +3972,14 @@ username and password.
 Like all GNU utilities, the latest version of Wget can be found at the
 master GNU archive site ftp.gnu.org, and its mirrors.  For example,
 Wget @value{VERSION} can be found at
-@url{ftp://ftp.gnu.org/pub/gnu/wget/wget-@value{VERSION}.tar.gz}
+@url{https://ftp.gnu.org/pub/gnu/wget/wget-@value{VERSION}.tar.gz}
 
 @node Web Site, Mailing Lists, Distribution, Various
 @section Web Site
 @cindex web site
 
 The official web site for GNU Wget is at
-@url{http://www.gnu.org/software/wget/}. However, most useful
+@url{https//www.gnu.org/software/wget/}. However, most useful
 information resides at ``The Wget Wgiki'',
 @url{http://wget.addictivecode.org/}.
 
@@ -3956,14 +3993,14 @@ information resides at ``The Wget Wgiki'',
 The primary mailinglist for discussion, bug-reports, or questions
 about GNU Wget is at @email{bug-wget@@gnu.org}. To subscribe, send an
 email to @email{bug-wget-join@@gnu.org}, or visit
-@url{http://lists.gnu.org/mailman/listinfo/bug-wget}.
+@url{https://lists.gnu.org/mailman/listinfo/bug-wget}.
 
 You do not need to subscribe to send a message to the list; however,
 please note that unsubscribed messages are moderated, and may take a
 while before they hit the list---@strong{usually around a day}.  If
 you want your message to show up immediately, please subscribe to the
 list before posting. Archives for the list may be found at
-@url{http://lists.gnu.org/pipermail/bug-wget/}.
+@url{https://lists.gnu.org/archive/html/bug-wget/}.
 
 An NNTP/Usenettish gateway is also available via
 @uref{http://gmane.org/about.php,Gmane}. You can see the Gmane
@@ -3971,15 +4008,7 @@ archives at
 @url{http://news.gmane.org/gmane.comp.web.wget.general}. Note that the
 Gmane archives conveniently include messages from both the current
 list, and the previous one. Messages also show up in the Gmane
-archives sooner than they do at @url{lists.gnu.org}.
-
-@unnumberedsubsec Bug Notices List
-
-Additionally, there is the @email{wget-notify@@addictivecode.org} mailing
-list. This is a non-discussion list that receives bug report
-notifications from the bug-tracker. To subscribe to this list,
-send an email to @email{wget-notify-join@@addictivecode.org},
-or visit @url{http://addictivecode.org/mailman/listinfo/wget-notify}.
+archives sooner than they do at @url{https://lists.gnu.org}.
 
 @unnumberedsubsec Obsolete Lists
 
@@ -3991,7 +4020,7 @@ discussing patches to GNU Wget.
 Messages from @email{wget@@sunsite.dk} are archived at
 @itemize @tie{}
 @item
-@url{http://www.mail-archive.com/wget%40sunsite.dk/} and at
+@url{https://www.mail-archive.com/wget%40sunsite.dk/} and at
 @item
 @url{http://news.gmane.org/gmane.comp.web.wget.general} (which also
 continues to archive the current list, @email{bug-wget@@gnu.org}).
@@ -4020,7 +4049,7 @@ via IRC at @code{irc.freenode.org}, @code{#wget}. Come check it out!
 
 @c man begin BUGS
 You are welcome to submit bug reports via the GNU Wget bug tracker (see
-@url{http://wget.addictivecode.org/BugTracker}).
+@url{https://savannah.gnu.org/bugs/?func=additem&group=wget}).
 
 Before actually submitting a bug report, please try to follow a few
 simple guidelines.
@@ -4037,7 +4066,7 @@ Lists}).
 @item
 Try to repeat the bug in as simple circumstances as possible.  E.g. if
 Wget crashes while downloading @samp{wget -rl0 -kKE -t5 --no-proxy
-http://yoyodyne.com -o /tmp/log}, you should try to see if the crash is
+http://example.com -o /tmp/log}, you should try to see if the crash is
 repeatable, and if will occur with a simpler set of options.  You might
 even try to start the download at the page where the crash occurred to
 see if that page somehow triggered the crash.
@@ -4102,7 +4131,7 @@ Windows-related features might look at them.
 
 Support for building on MS-DOS via DJGPP has been contributed by Gisle
 Vanem; a port to VMS is maintained by Steven Schweda, and is available
-at @url{http://antinode.org/}.
+at @url{https://antinode.info/dec/sw/wget.html}.
 
 @node Signals,  , Portability, Various
 @section Signals
@@ -4180,23 +4209,23 @@ download an individual page.  Because of that, Wget honors RES when
 downloading recursively.  For instance, when you issue:
 
 @example
-wget -r http://www.server.com/
+wget -r http://www.example.com/
 @end example
 
-First the index of @samp{www.server.com} will be downloaded.  If Wget
+First the index of @samp{www.example.com} will be downloaded.  If Wget
 finds that it wants to download more documents from that server, it will
-request @samp{http://www.server.com/robots.txt} and, if found, use it
+request @samp{http://www.example.com/robots.txt} and, if found, use it
 for further downloads.  @file{robots.txt} is loaded only once per each
 server.
 
 Until version 1.8, Wget supported the first version of the standard,
 written by Martijn Koster in 1994 and available at
-@url{http://www.robotstxt.org/wc/norobots.html}.  As of version 1.8,
+@url{http://www.robotstxt.org/robotstxt.html}.  As of version 1.8,
 Wget has supported the additional directives specified in the internet
 draft @samp{<draft-koster-robots-00.txt>} titled ``A Method for Web
 Robots Control''.  The draft, which has as far as I know never made to
 an @sc{rfc}, is available at
-@url{http://www.robotstxt.org/wc/norobots-rfc.txt}.
+@url{http://www.robotstxt.org/norobots-rfc.txt}.
 
 This manual no longer includes the text of the Robot Exclusion Standard.
 

src/build_info.c.in

@@ -8,6 +8,10 @@ nls             defined ENABLE_NLS
 ntlm            defined ENABLE_NTLM
 opie            defined ENABLE_OPIE
 psl             defined HAVE_LIBPSL
+cares            defined HAVE_LIBCARES
+
+metalink        defined HAVE_METALINK
+gpgme           defined HAVE_GPGME
 
 ssl choice:
     openssl     defined HAVE_LIBSSL || defined HAVE_LIBSSL32

src/connect.c

@@ -369,7 +369,14 @@ connect_to_ip (const ip_address *ip, int port, const char *print)
        logprintf.  */
     int save_errno = errno;
     if (sock >= 0)
-      fd_close (sock);
+      {
+#ifdef WIN32
+	/* If the connection timed out, fd_close will hang in Gnulib's
+	   close_fd_maybe_socket, inside the call to WSAEnumNetworkEvents.  */
+	if (errno != ETIMEDOUT)
+#endif
+	  fd_close (sock);
+      }
     if (print)
       logprintf (LOG_NOTQUIET, _("failed: %s.\n"), strerror (errno));
     errno = save_errno;

src/convert.c

@@ -308,7 +308,7 @@ convert_links (const char *file, struct urlpos *links)
             char *quoted_newname = local_quote_string (newname,
                                                        link->link_css_p);
 
-            if (link->link_css_p)
+            if (link->link_css_p || link->link_noquote_html_p)
               p = replace_plain (p, link->size, fp, quoted_newname);
             else if (!link->link_refresh_p)
               p = replace_attr (p, link->size, fp, quoted_newname);
@@ -329,7 +329,7 @@ convert_links (const char *file, struct urlpos *links)
             char *newname = convert_basename (p, link);
             char *quoted_newname = local_quote_string (newname, link->link_css_p);
 
-            if (link->link_css_p)
+            if (link->link_css_p || link->link_noquote_html_p)
               p = replace_plain (p, link->size, fp, quoted_newname);
             else if (!link->link_refresh_p)
               p = replace_attr (p, link->size, fp, quoted_newname);
@@ -352,7 +352,7 @@ convert_links (const char *file, struct urlpos *links)
             char *newlink = link->url->url;
             char *quoted_newlink = html_quote_string (newlink);
 
-            if (link->link_css_p)
+            if (link->link_css_p || link->link_noquote_html_p)
               p = replace_plain (p, link->size, fp, newlink);
             else if (!link->link_refresh_p)
               p = replace_attr (p, link->size, fp, quoted_newlink);

src/convert.h

@@ -69,6 +69,7 @@ struct urlpos {
   unsigned int link_base_p  :1;     /* the url came from <base href=...> */
   unsigned int link_inline_p    :1; /* needed to render the page */
   unsigned int link_css_p   :1;     /* the url came from CSS */
+  unsigned int link_noquote_html_p :1; /* from HTML, but doesn't need " */
   unsigned int link_expect_html :1; /* expected to contain HTML */
   unsigned int link_expect_css  :1; /* expected to contain CSS */
 

src/ftp-basic.c

@@ -429,6 +429,65 @@ ip_address_to_eprt_repr (const ip_address *addr, int port, char *buf,
   buf[buflen - 1] = '\0';
 }
 
+/* Bind a port and send the appropriate PORT command to the FTP
+   server.  Use acceptport after RETR, to get the socket of data
+   connection.  */
+uerr_t
+ftp_eprt (int csock, int *local_sock)
+{
+  uerr_t err;
+  char *request, *respline;
+  ip_address addr;
+  int nwritten;
+  int port;
+  /* Must contain the argument of EPRT (of the form |af|addr|port|).
+   * 4 chars for the | separators, INET6_ADDRSTRLEN chars for addr
+   * 1 char for af (1-2) and 5 chars for port (0-65535) */
+  char bytes[4 + INET6_ADDRSTRLEN + 1 + 5 + 1];
+
+  /* Get the address of this side of the connection. */
+  if (!socket_ip_address (csock, &addr, ENDPOINT_LOCAL))
+    return FTPSYSERR;
+
+  /* Setting port to 0 lets the system choose a free port.  */
+  port = 0;
+
+  /* Bind the port.  */
+  *local_sock = bind_local (&addr, &port);
+  if (*local_sock < 0)
+    return FTPSYSERR;
+
+  /* Construct the argument of EPRT (of the form |af|addr|port|). */
+  ip_address_to_eprt_repr (&addr, port, bytes, sizeof (bytes));
+
+  /* Send PORT request.  */
+  request = ftp_request ("EPRT", bytes);
+  nwritten = fd_write (csock, request, strlen (request), -1);
+  if (nwritten < 0)
+    {
+      xfree (request);
+      fd_close (*local_sock);
+      return WRITEFAILED;
+    }
+  xfree (request);
+  /* Get appropriate response.  */
+  err = ftp_response (csock, &respline);
+  if (err != FTPOK)
+    {
+      fd_close (*local_sock);
+      return err;
+    }
+  if (*respline != '2')
+    {
+      xfree (respline);
+      fd_close (*local_sock);
+      return FTPPORTERR;
+    }
+  xfree (respline);
+  return FTPOK;
+}
+#endif
+
 #ifdef HAVE_SSL
 /*
  * The following three functions defined into this #ifdef block
@@ -542,65 +601,6 @@ bail:
 }
 #endif /* HAVE_SSL */
 
-/* Bind a port and send the appropriate PORT command to the FTP
-   server.  Use acceptport after RETR, to get the socket of data
-   connection.  */
-uerr_t
-ftp_eprt (int csock, int *local_sock)
-{
-  uerr_t err;
-  char *request, *respline;
-  ip_address addr;
-  int nwritten;
-  int port;
-  /* Must contain the argument of EPRT (of the form |af|addr|port|).
-   * 4 chars for the | separators, INET6_ADDRSTRLEN chars for addr
-   * 1 char for af (1-2) and 5 chars for port (0-65535) */
-  char bytes[4 + INET6_ADDRSTRLEN + 1 + 5 + 1];
-
-  /* Get the address of this side of the connection. */
-  if (!socket_ip_address (csock, &addr, ENDPOINT_LOCAL))
-    return FTPSYSERR;
-
-  /* Setting port to 0 lets the system choose a free port.  */
-  port = 0;
-
-  /* Bind the port.  */
-  *local_sock = bind_local (&addr, &port);
-  if (*local_sock < 0)
-    return FTPSYSERR;
-
-  /* Construct the argument of EPRT (of the form |af|addr|port|). */
-  ip_address_to_eprt_repr (&addr, port, bytes, sizeof (bytes));
-
-  /* Send PORT request.  */
-  request = ftp_request ("EPRT", bytes);
-  nwritten = fd_write (csock, request, strlen (request), -1);
-  if (nwritten < 0)
-    {
-      xfree (request);
-      fd_close (*local_sock);
-      return WRITEFAILED;
-    }
-  xfree (request);
-  /* Get appropriate response.  */
-  err = ftp_response (csock, &respline);
-  if (err != FTPOK)
-    {
-      fd_close (*local_sock);
-      return err;
-    }
-  if (*respline != '2')
-    {
-      xfree (respline);
-      fd_close (*local_sock);
-      return FTPPORTERR;
-    }
-  xfree (respline);
-  return FTPOK;
-}
-#endif
-
 /* Similar to ftp_port, but uses `PASV' to initiate the passive FTP
    transfer.  Reads the response from server and parses it.  Reads the
    host and port addresses and returns them.  */

src/ftp.c

@@ -236,7 +236,7 @@ print_length (wgint size, wgint start, bool authoritative)
   logputs (LOG_VERBOSE, !authoritative ? _(" (unauthoritative)\n") : "\n");
 }
 
-static uerr_t ftp_get_listing (struct url *, ccon *, struct fileinfo **);
+static uerr_t ftp_get_listing (struct url *, struct url *, ccon *, struct fileinfo **);
 
 static uerr_t
 get_ftp_greeting(int csock, ccon *con)
@@ -315,13 +315,14 @@ init_control_ssl_connection (int csock, struct url *u, bool *using_control_secur
    and closes the control connection in case of error.  If warc_tmp
    is non-NULL, the downloaded data will be written there as well.  */
 static uerr_t
-getftp (struct url *u, wgint passed_expected_bytes, wgint *qtyread,
+getftp (struct url *u, struct url *original_url,
+        wgint passed_expected_bytes, wgint *qtyread,
         wgint restval, ccon *con, int count, wgint *last_expected_bytes,
         FILE *warc_tmp)
 {
   int csock, dtsock, local_sock, res;
   uerr_t err = RETROK;          /* appease the compiler */
-  FILE *fp;
+  FILE *fp = NULL;
   char *respline, *tms;
   const char *user, *passwd, *tmrate;
   int cmd = con->cmd;
@@ -393,7 +394,9 @@ getftp (struct url *u, wgint passed_expected_bytes, wgint *qtyread,
   if (!(cmd & DO_LOGIN))
     {
       csock = con->csock;
+#ifdef HAVE_SSL
       using_data_security = con->st & DATA_CHANNEL_SECURITY;
+#endif
     }
   else                          /* cmd & DO_LOGIN */
     {
@@ -1186,7 +1189,7 @@ Error in server response, closing control connection.\n"));
         {
           bool exists = false;
           struct fileinfo *f;
-          uerr_t _res = ftp_get_listing (u, con, &f);
+          uerr_t _res = ftp_get_listing (u, original_url, con, &f);
           /* Set the DO_RETR command flag again, because it gets unset when
              calling ftp_get_listing() and would otherwise cause an assertion
              failure earlier on when this function gets repeatedly called
@@ -1512,8 +1515,9 @@ Error in server response, closing control connection.\n"));
             {
               fd_close (csock);
               fd_close (dtsock);
+              err = CONERROR;
               logputs (LOG_NOTQUIET, "Could not perform SSL handshake.\n");
-              return CONERROR;
+              goto exit_error;
             }
         }
       else
@@ -1523,7 +1527,8 @@ Error in server response, closing control connection.\n"));
         {
           fd_close (csock);
           fd_close (dtsock);
-          return CONERROR;
+          err = CONERROR;
+          goto exit_error;
         }
     }
 #endif
@@ -1760,6 +1765,13 @@ Error in server response, closing control connection.\n"));
     }
   } while (try_again);
   return RETRFINISHED;
+
+exit_error:
+
+  /* If fp is a regular file, close and try to remove it */
+  if (fp && !output_stream)
+    fclose (fp);
+  return err;
 }
 
 /* A one-file FTP loop.  This is the part where FTP retrieval is
@@ -1768,8 +1780,8 @@ Error in server response, closing control connection.\n"));
    This loop either gets commands from con, or (if ON_YOUR_OWN is
    set), makes them up to retrieve the file given by the URL.  */
 static uerr_t
-ftp_loop_internal (struct url *u, struct fileinfo *f, ccon *con, char **local_file,
-                   bool force_full_retrieve)
+ftp_loop_internal (struct url *u, struct url *original_url, struct fileinfo *f,
+                   ccon *con, char **local_file, bool force_full_retrieve)
 {
   int count, orig_lp;
   wgint restval, len = 0, qtyread = 0;
@@ -1794,7 +1806,7 @@ ftp_loop_internal (struct url *u, struct fileinfo *f, ccon *con, char **local_fi
     {
       /* URL-derived file.  Consider "-O file" name. */
       xfree (con->target);
-      con->target = url_file_name (u, NULL);
+      con->target = url_file_name (opt.trustservernames || !original_url ? u : original_url, NULL);
       if (!opt.output_document)
         locf = con->target;
       else
@@ -1912,8 +1924,8 @@ ftp_loop_internal (struct url *u, struct fileinfo *f, ccon *con, char **local_fi
 
       /* If we are working on a WARC record, getftp should also write
          to the warc_tmp file. */
-      err = getftp (u, len, &qtyread, restval, con, count, &last_expected_bytes,
-                    warc_tmp);
+      err = getftp (u, original_url, len, &qtyread, restval, con, count,
+                    &last_expected_bytes, warc_tmp);
 
       if (con->csock == -1)
         con->st &= ~DONE_CWD;
@@ -2081,7 +2093,8 @@ Removing file due to --delete-after in ftp_loop_internal():\n"));
 /* Return the directory listing in a reusable format.  The directory
    is specifed in u->dir.  */
 static uerr_t
-ftp_get_listing (struct url *u, ccon *con, struct fileinfo **f)
+ftp_get_listing (struct url *u, struct url *original_url, ccon *con,
+                 struct fileinfo **f)
 {
   uerr_t err;
   char *uf;                     /* url file name */
@@ -2102,7 +2115,7 @@ ftp_get_listing (struct url *u, ccon *con, struct fileinfo **f)
 
   con->target = xstrdup (lf);
   xfree (lf);
-  err = ftp_loop_internal (u, NULL, con, NULL, false);
+  err = ftp_loop_internal (u, original_url, NULL, con, NULL, false);
   lf = xstrdup (con->target);
   xfree (con->target);
   con->target = old_target;
@@ -2125,8 +2138,9 @@ ftp_get_listing (struct url *u, ccon *con, struct fileinfo **f)
   return err;
 }
 
-static uerr_t ftp_retrieve_dirs (struct url *, struct fileinfo *, ccon *);
-static uerr_t ftp_retrieve_glob (struct url *, ccon *, int);
+static uerr_t ftp_retrieve_dirs (struct url *, struct url *,
+                                 struct fileinfo *, ccon *);
+static uerr_t ftp_retrieve_glob (struct url *, struct url *, ccon *, int);
 static struct fileinfo *delelement (struct fileinfo *, struct fileinfo **);
 static void freefileinfo (struct fileinfo *f);
 
@@ -2138,7 +2152,8 @@ static void freefileinfo (struct fileinfo *f);
    If opt.recursive is set, after all files have been retrieved,
    ftp_retrieve_dirs will be called to retrieve the directories.  */
 static uerr_t
-ftp_retrieve_list (struct url *u, struct fileinfo *f, ccon *con)
+ftp_retrieve_list (struct url *u, struct url *original_url,
+                   struct fileinfo *f, ccon *con)
 {
   static int depth = 0;
   uerr_t err;
@@ -2299,7 +2314,10 @@ Already have correct symlink %s -> %s\n\n"),
           else                /* opt.retr_symlinks */
             {
               if (dlthis)
-                err = ftp_loop_internal (u, f, con, NULL, force_full_retrieve);
+                {
+                  err = ftp_loop_internal (u, original_url, f, con, NULL,
+                                           force_full_retrieve);
+                }
             } /* opt.retr_symlinks */
           break;
         case FT_DIRECTORY:
@@ -2310,7 +2328,10 @@ Already have correct symlink %s -> %s\n\n"),
         case FT_PLAINFILE:
           /* Call the retrieve loop.  */
           if (dlthis)
-            err = ftp_loop_internal (u, f, con, NULL, force_full_retrieve);
+            {
+              err = ftp_loop_internal (u, original_url, f, con, NULL,
+                                       force_full_retrieve);
+            }
           break;
         case FT_UNKNOWN:
           logprintf (LOG_NOTQUIET, _("%s: unknown/unsupported file type.\n"),
@@ -2375,7 +2396,7 @@ Already have correct symlink %s -> %s\n\n"),
   /* We do not want to call ftp_retrieve_dirs here */
   if (opt.recursive &&
       !(opt.reclevel != INFINITE_RECURSION && depth >= opt.reclevel))
-    err = ftp_retrieve_dirs (u, orig, con);
+    err = ftp_retrieve_dirs (u, original_url, orig, con);
   else if (opt.recursive)
     DEBUGP ((_("Will not retrieve dirs since depth is %d (max %d).\n"),
              depth, opt.reclevel));
@@ -2388,7 +2409,8 @@ Already have correct symlink %s -> %s\n\n"),
    ftp_retrieve_glob on each directory entry.  The function knows
    about excluded directories.  */
 static uerr_t
-ftp_retrieve_dirs (struct url *u, struct fileinfo *f, ccon *con)
+ftp_retrieve_dirs (struct url *u, struct url *original_url,
+                   struct fileinfo *f, ccon *con)
 {
   char *container = NULL;
   int container_size = 0;
@@ -2438,7 +2460,7 @@ Not descending to %s as it is excluded/not-included.\n"),
       odir = xstrdup (u->dir);  /* because url_set_dir will free
                                    u->dir. */
       url_set_dir (u, newdir);
-      ftp_retrieve_glob (u, con, GLOB_GETALL);
+      ftp_retrieve_glob (u, original_url, con, GLOB_GETALL);
       url_set_dir (u, odir);
       xfree (odir);
 
@@ -2497,14 +2519,15 @@ is_invalid_entry (struct fileinfo *f)
    GLOB_GLOBALL, use globbing; if it's GLOB_GETALL, download the whole
    directory.  */
 static uerr_t
-ftp_retrieve_glob (struct url *u, ccon *con, int action)
+ftp_retrieve_glob (struct url *u, struct url *original_url,
+                   ccon *con, int action)
 {
   struct fileinfo *f, *start;
   uerr_t res;
 
   con->cmd |= LEAVE_PENDING;
 
-  res = ftp_get_listing (u, con, &start);
+  res = ftp_get_listing (u, original_url, con, &start);
   if (res != RETROK)
     return res;
   /* First: weed out that do not conform the global rules given in
@@ -2600,7 +2623,7 @@ ftp_retrieve_glob (struct url *u, ccon *con, int action)
   if (start)
     {
       /* Just get everything.  */
-      res = ftp_retrieve_list (u, start, con);
+      res = ftp_retrieve_list (u, original_url, start, con);
     }
   else
     {
@@ -2616,7 +2639,7 @@ ftp_retrieve_glob (struct url *u, ccon *con, int action)
         {
           /* Let's try retrieving it anyway.  */
           con->st |= ON_YOUR_OWN;
-          res = ftp_loop_internal (u, NULL, con, NULL, false);
+          res = ftp_loop_internal (u, original_url, NULL, con, NULL, false);
           return res;
         }
 
@@ -2636,8 +2659,8 @@ ftp_retrieve_glob (struct url *u, ccon *con, int action)
    of URL.  Inherently, its capabilities are limited on what can be
    encoded into a URL.  */
 uerr_t
-ftp_loop (struct url *u, char **local_file, int *dt, struct url *proxy,
-          bool recursive, bool glob)
+ftp_loop (struct url *u, struct url *original_url, char **local_file, int *dt,
+          struct url *proxy, bool recursive, bool glob)
 {
   ccon con;                     /* FTP connection */
   uerr_t res;
@@ -2658,16 +2681,17 @@ ftp_loop (struct url *u, char **local_file, int *dt, struct url *proxy,
   if (!*u->file && !recursive)
     {
       struct fileinfo *f;
-      res = ftp_get_listing (u, &con, &f);
+      res = ftp_get_listing (u, original_url, &con, &f);
 
       if (res == RETROK)
         {
           if (opt.htmlify && !opt.spider)
             {
+              struct url *url_file = opt.trustservernames ? u : original_url;
               char *filename = (opt.output_document
                                 ? xstrdup (opt.output_document)
                                 : (con.target ? xstrdup (con.target)
-                                   : url_file_name (u, NULL)));
+                                   : url_file_name (url_file, NULL)));
               res = ftp_index (filename, u, f);
               if (res == FTPOK && opt.verbose)
                 {
@@ -2712,11 +2736,13 @@ ftp_loop (struct url *u, char **local_file, int *dt, struct url *proxy,
           /* ftp_retrieve_glob is a catch-all function that gets called
              if we need globbing, time-stamping, recursion or preserve
              permissions.  Its third argument is just what we really need.  */
-          res = ftp_retrieve_glob (u, &con,
+          res = ftp_retrieve_glob (u, original_url, &con,
                                    ispattern ? GLOB_GLOBALL : GLOB_GETONE);
         }
       else
-        res = ftp_loop_internal (u, NULL, &con, local_file, false);
+        {
+          res = ftp_loop_internal (u, original_url, NULL, &con, local_file, false);
+        }
     }
   if (res == FTPOK)
     res = RETROK;

src/ftp.h

@@ -169,7 +169,8 @@ enum wget_ftp_fstatus
 };
 
 struct fileinfo *ftp_parse_ls (const char *, const enum stype);
-uerr_t ftp_loop (struct url *, char **, int *, struct url *, bool, bool);
+uerr_t ftp_loop (struct url *, struct url *, char **, int *, struct url *,
+                 bool, bool);
 
 uerr_t ftp_index (const char *, struct url *, struct fileinfo *);
 

src/gnutls.c

@@ -36,7 +36,9 @@ as that of the covered work.  */
 #include <stdio.h>
 #include <dirent.h>
 #include <stdlib.h>
+#include <xalloc.h>
 
+#include <gnutls/abstract.h>
 #include <gnutls/gnutls.h>
 #include <gnutls/x509.h>
 #include <sys/ioctl.h>
@@ -518,6 +520,22 @@ _do_handshake (gnutls_session_t session, int fd, double timeout)
   return err;
 }
 
+static const char *
+_sni_hostname(const char *hostname)
+{
+  size_t len = strlen(hostname);
+
+  char *sni_hostname = xmemdup(hostname, len + 1);
+
+  /* Remove trailing dot(s) to fix #47408.
+   * Regarding RFC 6066 (SNI): The hostname is represented as a byte
+   * string using ASCII encoding without a trailing dot. */
+  while (len && sni_hostname[--len] == '.')
+    sni_hostname[len] = 0;
+
+  return sni_hostname;
+}
+
 bool
 ssl_connect_wget (int fd, const char *hostname, int *continue_session)
 {
@@ -530,8 +548,12 @@ ssl_connect_wget (int fd, const char *hostname, int *continue_session)
   /* We set the server name but only if it's not an IP address. */
   if (! is_valid_ip_address (hostname))
     {
-      gnutls_server_name_set (session, GNUTLS_NAME_DNS, hostname,
-                              strlen (hostname));
+      /* GnuTLS 3.4.x (x<=10) disrespects the length parameter, we have to construct a new string */
+      /* see https://gitlab.com/gnutls/gnutls/issues/78 */
+      const char *sni_hostname = _sni_hostname(hostname);
+
+      gnutls_server_name_set (session, GNUTLS_NAME_DNS, sni_hostname, strlen(sni_hostname));
+      xfree(sni_hostname);
     }
 
   gnutls_set_default_priority (session);
@@ -633,9 +655,13 @@ ssl_connect_wget (int fd, const char *hostname, int *continue_session)
         {
           if (!ctx || !ctx->session_data || gnutls_session_set_data (session, ctx->session_data->data, ctx->session_data->size))
             {
-              /* server does not want to continue the session */
-              gnutls_free (ctx->session_data->data);
-              gnutls_free (ctx->session_data);
+              if (ctx && ctx->session_data)
+                {
+                  /* server does not want to continue the session */
+                  if (ctx->session_data->data)
+                    gnutls_free (ctx->session_data->data);
+                  gnutls_free (ctx->session_data);
+                }
               gnutls_deinit (session);
               return false;
             }
@@ -667,6 +693,59 @@ ssl_connect_wget (int fd, const char *hostname, int *continue_session)
   return true;
 }
 
+static bool
+pkp_pin_peer_pubkey (gnutls_x509_crt_t cert, const char *pinnedpubkey)
+{
+  /* Scratch */
+  size_t len1 = 0, len2 = 0;
+  char *buff1 = NULL;
+
+  gnutls_pubkey_t key = NULL;
+
+  /* Result is returned to caller */
+  int ret = 0;
+  bool result = false;
+
+  /* if a path wasn't specified, don't pin */
+  if (NULL == pinnedpubkey)
+    return true;
+
+  if (NULL == cert)
+    return result;
+
+  /* Begin Gyrations to get the public key     */
+  gnutls_pubkey_init (&key);
+
+  ret = gnutls_pubkey_import_x509 (key, cert, 0);
+  if (ret < 0)
+    goto cleanup; /* failed */
+
+  ret = gnutls_pubkey_export (key, GNUTLS_X509_FMT_DER, NULL, &len1);
+  if (ret != GNUTLS_E_SHORT_MEMORY_BUFFER || len1 == 0)
+    goto cleanup; /* failed */
+
+  buff1 = xmalloc (len1);
+
+  len2 = len1;
+
+  ret = gnutls_pubkey_export (key, GNUTLS_X509_FMT_DER, buff1, &len2);
+  if (ret < 0 || len1 != len2)
+    goto cleanup; /* failed */
+
+  /* End Gyrations */
+
+  /* The one good exit point */
+  result = wg_pin_peer_pubkey (pinnedpubkey, buff1, len1);
+
+ cleanup:
+  if (NULL != key)
+    gnutls_pubkey_deinit (key);
+
+  xfree (buff1);
+
+  return result;
+}
+
 #define _CHECK_CERT(flag,msg) \
   if (status & (flag))\
     {\
@@ -687,6 +766,11 @@ ssl_check_certificate (int fd, const char *host)
      him about problems with the server's certificate.  */
   const char *severity = opt.check_cert ? _("ERROR") : _("WARNING");
   bool success = true;
+  bool pinsuccess = opt.pinnedpubkey == NULL;
+
+  /* The user explicitly said to not check for the certificate.  */
+  if (opt.check_cert == CHECK_CERT_QUIET && pinsuccess)
+    return success;
 
   err = gnutls_certificate_verify_peers2 (ctx->session, &status);
   if (err < 0)
@@ -711,6 +795,7 @@ ssl_check_certificate (int fd, const char *host)
       gnutls_x509_crt_t cert;
       const gnutls_datum_t *cert_list;
       unsigned int cert_list_size;
+      const char *sni_hostname;
 
       if ((err = gnutls_x509_crt_init (&cert)) < 0)
         {
@@ -745,13 +830,23 @@ ssl_check_certificate (int fd, const char *host)
           logprintf (LOG_NOTQUIET, _("The certificate has expired\n"));
           success = false;
         }
-      if (!gnutls_x509_crt_check_hostname (cert, host))
+      sni_hostname = _sni_hostname(host);
+      if (!gnutls_x509_crt_check_hostname (cert, sni_hostname))
         {
           logprintf (LOG_NOTQUIET,
                      _("The certificate's owner does not match hostname %s\n"),
-                     quote (host));
+                     quote (sni_hostname));
           success = false;
         }
+      xfree(sni_hostname);
+
+      pinsuccess = pkp_pin_peer_pubkey (cert, opt.pinnedpubkey);
+      if (!pinsuccess)
+        {
+          logprintf (LOG_ALWAYS, _("The public key does not match pinned public key!\n"));
+          success = false;
+        }
+
  crt_deinit:
       gnutls_x509_crt_deinit (cert);
     }
@@ -762,5 +857,6 @@ ssl_check_certificate (int fd, const char *host)
     }
 
  out:
-  return opt.check_cert ? success : true;
+  /* never return true if pinsuccess fails */
+  return !pinsuccess ? false : (opt.check_cert == CHECK_CERT_ON ? success : true);
 }

src/host.c

@@ -65,6 +65,7 @@ as that of the covered work.  */
 #include "host.h"
 #include "url.h"
 #include "hash.h"
+#include "ptimer.h"
 
 #ifndef NO_ADDRESS
 # define NO_ADDRESS NO_DATA
@@ -649,6 +650,112 @@ cache_remove (const char *host)
     }
 }
 
+#ifdef HAVE_LIBCARES
+#include <sys/select.h>
+#include <ares.h>
+extern ares_channel ares;
+
+static struct address_list *
+merge_address_lists (struct address_list *al1, struct address_list *al2)
+{
+  int count = al1->count + al2->count;
+
+  /* merge al2 into al1 */
+  al1->addresses = xrealloc (al1->addresses, sizeof (ip_address) * count);
+  memcpy (al1->addresses + al1->count, al2->addresses, sizeof (ip_address) * al2->count);
+  al1->count = count;
+
+  address_list_delete (al2);
+
+  return al1;
+}
+
+static struct address_list *
+address_list_from_hostent (struct hostent *host)
+{
+  int count, i;
+  struct address_list *al = xnew0 (struct address_list);
+
+  for (count = 0; host->h_addr_list[count]; count++)
+    ;
+
+  assert (count > 0);
+
+  al->addresses = xnew_array (ip_address, count);
+  al->count     = count;
+  al->refcount  = 1;
+
+  for (i = 0; i < count; i++)
+    {
+      ip_address *ip = &al->addresses[i];
+      ip->family = host->h_addrtype;
+      memcpy (IP_INADDR_DATA (ip), host->h_addr_list[i], ip->family == AF_INET ? 4 : 16);
+    }
+
+  return al;
+}
+
+/* Since GnuLib's select() (i.e. rpl_select()) cannot handle socket-numbers
+ * returned from C-ares, we must use the original select() from Winsock.
+ */
+#ifdef WINDOWS
+#undef select
+#endif
+
+static void
+wait_ares (ares_channel channel)
+{
+  struct ptimer *timer = NULL;
+
+  if (opt.dns_timeout)
+    timer = ptimer_new ();
+
+  for (;;)
+    {
+      struct timeval *tvp, tv;
+      fd_set read_fds, write_fds;
+      int nfds, rc;
+
+      FD_ZERO (&read_fds);
+      FD_ZERO (&write_fds);
+      nfds = ares_fds (channel, &read_fds, &write_fds);
+      if (nfds == 0)
+        break;
+
+      if (timer)
+        {
+          double max = opt.dns_timeout - ptimer_measure (timer);
+
+          tv.tv_sec = (long) max;
+          tv.tv_usec = 1000000 * (max - (long) max);
+          tvp = ares_timeout (channel, &tv, &tv);
+        }
+      else
+        tvp = ares_timeout (channel, NULL, &tv);
+
+      rc = select (nfds, &read_fds, &write_fds, NULL, tvp);
+      if (rc == 0 && timer && ptimer_measure (timer) >= opt.dns_timeout)
+        ares_cancel (channel);
+      else
+        ares_process (channel, &read_fds, &write_fds);
+    }
+}
+
+static void
+callback (void *arg, int status, int timeouts _GL_UNUSED, struct hostent *host)
+{
+  struct address_list **al = (struct address_list **) arg;
+
+  if (!host || status != ARES_SUCCESS)
+    {
+      *al = NULL;
+      return;
+    }
+
+  *al = address_list_from_hostent (host);
+}
+#endif
+
 /* Look up HOST in DNS and return a list of IP addresses.
 
    This function caches its result so that, if the same host is passed
@@ -755,80 +862,112 @@ lookup_host (const char *host, int flags)
     }
 
 #ifdef ENABLE_IPV6
-  {
-    int err;
-    struct addrinfo hints, *res;
+#ifdef HAVE_LIBCARES
+  if (ares)
+    {
+      struct address_list *al4;
+      struct address_list *al6;
 
-    xzero (hints);
-    hints.ai_socktype = SOCK_STREAM;
-    if (opt.ipv4_only)
-      hints.ai_family = AF_INET;
-    else if (opt.ipv6_only)
-      hints.ai_family = AF_INET6;
-    else
-      /* We tried using AI_ADDRCONFIG, but removed it because: it
-         misinterprets IPv6 loopbacks, it is broken on AIX 5.1, and
-         it's unneeded since we sort the addresses anyway.  */
+      if (opt.ipv4_only || !opt.ipv6_only)
+        ares_gethostbyname (ares, host, AF_INET, callback, &al4);
+      if (opt.ipv6_only || !opt.ipv4_only)
+        ares_gethostbyname (ares, host, AF_INET6, callback, &al6);
+
+      wait_ares (ares);
+
+      if (al4 && al6)
+        al = merge_address_lists (al4, al6);
+      else if (al4)
+        al = al4;
+      else
+        al = al6;
+    }
+  else
+#endif
+    {
+      int err;
+      struct addrinfo hints, *res;
+
+      xzero (hints);
+      hints.ai_socktype = SOCK_STREAM;
+      if (opt.ipv4_only)
+        hints.ai_family = AF_INET;
+      else if (opt.ipv6_only)
+        hints.ai_family = AF_INET6;
+      else
+        /* We tried using AI_ADDRCONFIG, but removed it because: it
+           misinterprets IPv6 loopbacks, it is broken on AIX 5.1, and
+           it's unneeded since we sort the addresses anyway.  */
         hints.ai_family = AF_UNSPEC;
 
-    if (flags & LH_BIND)
-      hints.ai_flags |= AI_PASSIVE;
+      if (flags & LH_BIND)
+        hints.ai_flags |= AI_PASSIVE;
 
 #ifdef AI_NUMERICHOST
-    if (numeric_address)
-      {
-        /* Where available, the AI_NUMERICHOST hint can prevent costly
-           access to DNS servers.  */
-        hints.ai_flags |= AI_NUMERICHOST;
-        timeout = 0;            /* no timeout needed when "resolving"
+      if (numeric_address)
+        {
+          /* Where available, the AI_NUMERICHOST hint can prevent costly
+             access to DNS servers.  */
+          hints.ai_flags |= AI_NUMERICHOST;
+          timeout = 0; /* no timeout needed when "resolving"
                                    numeric hosts -- avoid setting up
                                    signal handlers and such. */
-      }
+        }
 #endif
 
-    err = getaddrinfo_with_timeout (host, NULL, &hints, &res, timeout);
-    if (err != 0 || res == NULL)
-      {
-        if (!silent)
-          logprintf (LOG_VERBOSE, _("failed: %s.\n"),
-                     err != EAI_SYSTEM ? gai_strerror (err) : strerror (errno));
-        return NULL;
-      }
-    al = address_list_from_addrinfo (res);
-    freeaddrinfo (res);
-    if (!al)
-      {
-        logprintf (LOG_VERBOSE,
-                   _("failed: No IPv4/IPv6 addresses for host.\n"));
-        return NULL;
-      }
+      err = getaddrinfo_with_timeout (host, NULL, &hints, &res, timeout);
 
-    /* Reorder addresses so that IPv4 ones (or IPv6 ones, as per
-       --prefer-family) come first.  Sorting is stable so the order of
-       the addresses with the same family is undisturbed.  */
-    if (al->count > 1 && opt.prefer_family != prefer_none)
-      stable_sort (al->addresses, al->count, sizeof (ip_address),
-                   opt.prefer_family == prefer_ipv4
-                   ? cmp_prefer_ipv4 : cmp_prefer_ipv6);
-  }
+      if (err != 0 || res == NULL)
+        {
+          if (!silent)
+            logprintf (LOG_VERBOSE, _ ("failed: %s.\n"),
+                       err != EAI_SYSTEM ? gai_strerror (err) : strerror (errno));
+          return NULL;
+        }
+      al = address_list_from_addrinfo (res);
+      freeaddrinfo (res);
+    }
+
+  if (!al)
+    {
+      logprintf (LOG_VERBOSE,
+                 _ ("failed: No IPv4/IPv6 addresses for host.\n"));
+      return NULL;
+    }
+
+  /* Reorder addresses so that IPv4 ones (or IPv6 ones, as per
+     --prefer-family) come first.  Sorting is stable so the order of
+     the addresses with the same family is undisturbed.  */
+  if (al->count > 1 && opt.prefer_family != prefer_none)
+    stable_sort (al->addresses, al->count, sizeof (ip_address),
+                 opt.prefer_family == prefer_ipv4
+                 ? cmp_prefer_ipv4 : cmp_prefer_ipv6);
 #else  /* not ENABLE_IPV6 */
-  {
-    struct hostent *hptr = gethostbyname_with_timeout (host, timeout);
-    if (!hptr)
-      {
-        if (!silent)
-          {
-            if (errno != ETIMEDOUT)
-              logprintf (LOG_VERBOSE, _("failed: %s.\n"),
-                         host_errstr (h_errno));
-            else
-              logputs (LOG_VERBOSE, _("failed: timed out.\n"));
-          }
-        return NULL;
-      }
-    /* Do older systems have h_addr_list?  */
-    al = address_list_from_ipv4_addresses (hptr->h_addr_list);
-  }
+#ifdef HAVE_LIBCARES
+  if (ares)
+    {
+      ares_gethostbyname (ares, host, AF_INET, callback, &al);
+      wait_ares (ares);
+    }
+  else
+#endif
+    {
+      struct hostent *hptr = gethostbyname_with_timeout (host, timeout);
+      if (!hptr)
+        {
+          if (!silent)
+            {
+              if (errno != ETIMEDOUT)
+                logprintf (LOG_VERBOSE, _ ("failed: %s.\n"),
+                           host_errstr (h_errno));
+              else
+                logputs (LOG_VERBOSE, _ ("failed: timed out.\n"));
+            }
+          return NULL;
+        }
+      /* Do older systems have h_addr_list?  */
+      al = address_list_from_ipv4_addresses (hptr->h_addr_list);
+    }
 #endif /* not ENABLE_IPV6 */
 
   /* Print the addresses determined by DNS lookup, but no more than

src/hsts.c

@@ -53,6 +53,7 @@ as that of the covered work.  */
 struct hsts_store {
   struct hash_table *table;
   time_t last_mtime;
+  bool changed;
 };
 
 struct hsts_kh {
@@ -148,13 +149,14 @@ hsts_find_entry (hsts_store_t store,
 end:
   /* restore pointer or we'll get a SEGV */
   k->host = org_ptr;
-  xfree (k->host);
 
   /* copy parameters to previous frame */
   if (match_type)
     *match_type = match;
   if (kh)
     memcpy (kh, k, sizeof (struct hsts_kh));
+  else
+    xfree (k->host);
 
   xfree (k);
   return khi;
@@ -236,8 +238,7 @@ hsts_new_entry (hsts_store_t store,
 static void
 hsts_remove_entry (hsts_store_t store, struct hsts_kh *kh)
 {
-  if (hash_table_remove (store->table, kh))
-    xfree (kh->host);
+  hash_table_remove (store->table, kh);
 }
 
 static bool
@@ -316,7 +317,7 @@ hsts_store_dump (hsts_store_t store, FILE *fp)
   /* Print preliminary comments. We don't care if any of these fail. */
   fputs ("# HSTS 1.0 Known Hosts database for GNU Wget.\n", fp);
   fputs ("# Edit at your own risk.\n", fp);
-  fputs ("# <hostname>[:<port>]\t<incl. subdomains>\t<created>\t<max-age>\n", fp);
+  fputs ("# <hostname>\t<port>\t<incl. subdomains>\t<created>\t<max-age>\n", fp);
 
   /* Now cycle through the HSTS store in memory and dump the entries */
   for (hash_table_iterate (store->table, &it); hash_table_iter_next (&it);)
@@ -334,6 +335,22 @@ hsts_store_dump (hsts_store_t store, FILE *fp)
     }
 }
 
+/*
+ * Test:
+ *  - The file is a regular file (ie. not a symlink), and
+ *  - The file is not world-writable.
+ */
+static bool
+hsts_file_access_valid (const char *filename)
+{
+  struct_stat st;
+
+  if (stat (filename, &st) == -1)
+    return false;
+
+  return !(st.st_mode & S_IWOTH) && S_ISREG (st.st_mode);
+}
+
 /* HSTS API */
 
 /*
@@ -370,14 +387,19 @@ hsts_match (hsts_store_t store, struct url *u)
                   if (u->port == 80)
                     u->port = 443;
                   url_changed = true;
+                  store->changed = true;
                 }
             }
           else
-            hsts_remove_entry (store, kh);
+            {
+              hsts_remove_entry (store, kh);
+              store->changed = true;
+            }
         }
+      xfree (kh->host);
     }
 
-  xfree(kh);
+  xfree (kh);
 
   return url_changed;
 }
@@ -422,12 +444,14 @@ hsts_store_entry (hsts_store_t store,
       if (entry && match == CONGRUENT_MATCH)
         {
           if (max_age == 0)
-            hsts_remove_entry (store, kh);
+            {
+              hsts_remove_entry (store, kh);
+              store->changed = true;
+            }
           else if (max_age > 0)
             {
-              entry->include_subdomains = include_subdomains;
-
-              if (entry->max_age != max_age)
+              if (entry->max_age != max_age ||
+                  entry->include_subdomains != include_subdomains)
                 {
                   /* RFC 6797 states that 'max_age' is a TTL relative to the reception of the STS header
                      so we have to update the 'created' field too */
@@ -435,6 +459,9 @@ hsts_store_entry (hsts_store_t store,
                   if (t != -1)
                     entry->created = t;
                   entry->max_age = max_age;
+                  entry->include_subdomains = include_subdomains;
+
+                  store->changed = true;
                 }
             }
           /* we ignore negative max_ages */
@@ -449,11 +476,14 @@ hsts_store_entry (hsts_store_t store,
              happen we got a non-existent entry with max_age == 0.
           */
           result = hsts_add_entry (store, host, port, max_age, include_subdomains);
+          if (result)
+            store->changed = true;
         }
       /* we ignore new entries with max_age == 0 */
+      xfree (kh->host);
     }
 
-  xfree(kh);
+  xfree (kh);
 
   return result;
 }
@@ -462,36 +492,55 @@ hsts_store_t
 hsts_store_open (const char *filename)
 {
   hsts_store_t store = NULL;
-  struct stat st;
-  FILE *fp = NULL;
 
   store = xnew0 (struct hsts_store);
   store->table = hash_table_new (0, hsts_hash_func, hsts_cmp_func);
   store->last_mtime = 0;
+  store->changed = false;
 
   if (file_exists_p (filename))
     {
-      if (stat (filename, &st) == 0)
-        store->last_mtime = st.st_mtime;
-
-      fp = fopen (filename, "r");
-      if (!fp || !hsts_read_database (store, fp, false))
+      if (hsts_file_access_valid (filename))
         {
-          /* abort! */
+          struct_stat st;
+          FILE *fp = fopen (filename, "r");
+
+          if (!fp || !hsts_read_database (store, fp, false))
+            {
+              /* abort! */
+              hsts_store_close (store);
+              xfree (store);
+              fclose (fp);
+              goto out;
+            }
+
+          if (fstat (fileno (fp), &st) == 0)
+            store->last_mtime = st.st_mtime;
+
+          fclose (fp);
+        }
+      else
+        {
+          /*
+           * If we're not reading the HSTS database,
+           * then by all means act as if HSTS was disabled.
+           */
           hsts_store_close (store);
           xfree (store);
+
+          logprintf (LOG_NOTQUIET, "Will not apply HSTS. "
+                     "The HSTS database must be a regular and non-world-writable file.\n");
         }
-      if (fp)
-        fclose (fp);
     }
 
+out:
   return store;
 }
 
 void
 hsts_store_save (hsts_store_t store, const char *filename)
 {
-  struct stat st;
+  struct_stat st;
   FILE *fp = NULL;
   int fd = 0;
 
@@ -525,6 +574,12 @@ hsts_store_save (hsts_store_t store, const char *filename)
     }
 }
 
+bool
+hsts_store_has_changed (hsts_store_t store)
+{
+  return (store ? store->changed : false);
+}
+
 void
 hsts_store_close (hsts_store_t store)
 {
@@ -613,7 +668,7 @@ test_url_rewrite (hsts_store_t s, const char *url, int port, bool rewrite)
   if (rewrite)
     {
       if (port == 80)
-       mu_assert("URL: port should've been rewritten to 443", u.port == 443);
+        mu_assert("URL: port should've been rewritten to 443", u.port == 443);
       else
         mu_assert("URL: port should've been left intact", u.port == port);
       mu_assert("URL: scheme should've been rewritten to HTTPS", u.scheme == SCHEME_HTTPS);
@@ -686,7 +741,7 @@ test_hsts_url_rewrite_superdomain (void)
   s = open_hsts_test_store ();
   mu_assert("Could not open the HSTS store", s != NULL);
 
-  created = hsts_store_entry (s, SCHEME_HTTPS, "www.foo.com", 443, time(NULL) + 1234, true);
+  created = hsts_store_entry (s, SCHEME_HTTPS, "www.foo.com", 443, 1234, true);
   mu_assert("A new entry should've been created", created == true);
 
   TEST_URL_RW (s, "www.foo.com", 80);
@@ -707,7 +762,7 @@ test_hsts_url_rewrite_congruent (void)
   s = open_hsts_test_store ();
   mu_assert("Could not open the HSTS store", s != NULL);
 
-  created = hsts_store_entry (s, SCHEME_HTTPS, "foo.com", 443, time(NULL) + 1234, false);
+  created = hsts_store_entry (s, SCHEME_HTTPS, "foo.com", 443, 1234, false);
   mu_assert("A new entry should've been created", created == true);
 
   TEST_URL_RW (s, "foo.com", 80);
@@ -726,6 +781,7 @@ test_hsts_read_database (void)
   char *home = home_dir();
   char *file = NULL;
   FILE *fp = NULL;
+  time_t created = time(NULL) - 10;
 
   if (home)
     {
@@ -734,9 +790,9 @@ test_hsts_read_database (void)
       if (fp)
         {
           fputs ("# dummy comment\n", fp);
-          fputs ("foo.example.com\t0\t1\t1434224817\t123123123\n", fp);
-          fputs ("bar.example.com\t0\t0\t1434224817\t456456456\n", fp);
-          fputs ("test.example.com\t8080\t0\t1434224817\t789789789\n", fp);
+          fprintf (fp, "foo.example.com\t0\t1\t%ld\t123\n",(long) created);
+          fprintf (fp, "bar.example.com\t0\t0\t%ld\t456\n", (long) created);
+          fprintf (fp, "test.example.com\t8080\t0\t%ld\t789\n", (long) created);
           fclose (fp);
 
           table = hsts_store_open (file);

src/hsts.h

@@ -43,6 +43,7 @@ hsts_store_t hsts_store_open (const char *);
 
 void hsts_store_save (hsts_store_t, const char *);
 void hsts_store_close (hsts_store_t);
+bool hsts_store_has_changed (hsts_store_t);
 
 bool hsts_store_entry (hsts_store_t,
                        enum url_scheme, const char *, int,

src/html-url.c

@@ -56,6 +56,7 @@ typedef void (*tag_handler_t) (int, struct taginfo *, struct map_context *);
 DECLARE_TAG_HANDLER (tag_find_urls);
 DECLARE_TAG_HANDLER (tag_handle_base);
 DECLARE_TAG_HANDLER (tag_handle_form);
+DECLARE_TAG_HANDLER (tag_handle_img);
 DECLARE_TAG_HANDLER (tag_handle_link);
 DECLARE_TAG_HANDLER (tag_handle_meta);
 
@@ -105,7 +106,7 @@ static struct known_tag {
   { TAG_FORM,    "form",        tag_handle_form },
   { TAG_FRAME,   "frame",       tag_find_urls },
   { TAG_IFRAME,  "iframe",      tag_find_urls },
-  { TAG_IMG,     "img",         tag_find_urls },
+  { TAG_IMG,     "img",         tag_handle_img },
   { TAG_INPUT,   "input",       tag_find_urls },
   { TAG_LAYER,   "layer",       tag_find_urls },
   { TAG_LINK,    "link",        tag_handle_link },
@@ -183,7 +184,8 @@ static const char *additional_attributes[] = {
   "name",                       /* used by tag_handle_meta  */
   "content",                    /* used by tag_handle_meta  */
   "action",                     /* used by tag_handle_form  */
-  "style"                       /* used by check_style_attr */
+  "style",                      /* used by check_style_attr */
+  "srcset",                     /* used by tag_handle_img */
 };
 
 static struct hash_table *interesting_tags;
@@ -674,6 +676,88 @@ tag_handle_meta (int tagid _GL_UNUSED, struct taginfo *tag, struct map_context *
     }
 }
 
+/* Handle the IMG tag.  This requires special handling for the srcset attr,
+   while the traditional src/lowsrc/href attributes can be handled generically.
+*/
+
+static void
+tag_handle_img (int tagid, struct taginfo *tag, struct map_context *ctx) {
+  int attrind;
+  char *srcset;
+
+  /* Use the generic approach for the attributes without special syntax. */
+  tag_find_urls(tagid, tag, ctx);
+
+  srcset = find_attr (tag, "srcset", &attrind);
+  if (srcset)
+    {
+      /* These are relative to the input text. */
+      int base_ind = ATTR_POS (tag,attrind,ctx);
+      int size = strlen (srcset);
+
+      /* These are relative to srcset. */
+      int offset, url_start, url_end;
+
+      /* Make sure to line up base_ind with srcset[0], not outside quotes. */
+      if (ctx->text[base_ind] == '"' || ctx->text[base_ind] == '\'')
+        ++base_ind;
+
+      offset = 0;
+      while (offset < size)
+        {
+          bool has_descriptor = true;
+
+          /* Skip over initial whitespace and commas. Note there is no \v
+            in HTML5 whitespace. */
+          url_start = offset + strspn (srcset + offset, " \f\n\r\t,");
+
+          if (url_start == size)
+            return;
+
+          /* URL is any non-whitespace chars (including commas) - but with
+             trailing commas removed. */
+          url_end = url_start + strcspn (srcset + url_start, " \f\n\r\t");
+          while ((url_end - 1) > url_start && srcset[url_end - 1] == ',')
+            {
+              has_descriptor = false;
+              --url_end;
+            }
+
+          if (url_end > url_start)
+            {
+              char *url_text = strdupdelim (srcset + url_start,
+                                            srcset + url_end);
+              struct urlpos *up = append_url (url_text, base_ind + url_start,
+                                              url_end - url_start, ctx);
+              up->link_inline_p = 1;
+              up->link_noquote_html_p = 1;
+              xfree (url_text);
+            }
+
+          /* If the URL wasn't terminated by a , there may also be a descriptor
+             which we just skip. */
+          if (has_descriptor)
+            {
+              /* This is comma-terminated, except there may be one level of
+                 parentheses escaping that. */
+              bool in_paren = false;
+              for (offset = url_end; offset < size; ++offset)
+                {
+                  char c = srcset[offset];
+                  if (c == '(')
+                    in_paren = true;
+                  else if (c == ')' && in_paren)
+                    in_paren = false;
+                  else if (c == ',' && !in_paren)
+                    break;
+                }
+            }
+          else
+            offset = url_end;
+        }
+    }
+}
+
 /* Dispatch the tag handler appropriate for the tag we're mapping
    over.  See known_tags[] for definition of tag handlers.  */
 
@@ -752,10 +836,12 @@ get_urls_html (const char *file, const char *url, bool *meta_disallow_follow,
   map_html_tags (fm->content, fm->length, collect_tags_mapper, &ctx, flags,
                  NULL, interesting_attributes);
 
+#ifdef ENABLE_IRI
   /* Meta charset is only valid if there was no HTTP header Content-Type charset. */
   /* This is true for HTTP 1.0 and 1.1. */
   if (iri && !iri->content_encoding && meta_charset)
     set_content_encoding (iri, meta_charset);
+#endif
 
   DEBUGP (("no-follow in %s: %d\n", file, ctx.nofollow));
   if (meta_disallow_follow)

src/http.c

@@ -1872,7 +1872,7 @@ initialize_request (struct url *u, struct http_stat *hs, int *dt, struct url *pr
   /* Find the username and password for authentication. */
   *user = u->user;
   *passwd = u->passwd;
-  search_netrc (u->host, (const char **)&user, (const char **)&passwd, 0);
+  search_netrc (u->host, (const char **)user, (const char **)passwd, 0);
   *user = *user ? *user : (opt.http_user ? opt.http_user : opt.user);
   *passwd = *passwd ? *passwd : (opt.http_passwd ? opt.http_passwd : opt.passwd);
 
@@ -3424,13 +3424,16 @@ gethttp (struct url *u, struct http_stat *hs, int *dt, struct url *proxy,
       char *tmp = strchr (type, ';');
       if (tmp)
         {
+#ifdef ENABLE_IRI
           /* sXXXav: only needed if IRI support is enabled */
           char *tmp2 = tmp + 1;
+#endif
 
           while (tmp > type && c_isspace (tmp[-1]))
             --tmp;
           *tmp = '\0';
 
+#ifdef ENABLE_IRI
           /* Try to get remote encoding if needed */
           if (opt.enable_iri && !opt.encoding_remote)
             {
@@ -3439,6 +3442,7 @@ gethttp (struct url *u, struct http_stat *hs, int *dt, struct url *proxy,
                 set_content_encoding (iri, tmp);
               xfree(tmp);
             }
+#endif
         }
     }
   hs->newloc = resp_header_strdup (resp, "Location");
@@ -3794,7 +3798,6 @@ http_loop (struct url *u, struct url *original_url, char **newloc,
   struct http_stat hstat;        /* HTTP status */
   struct_stat st;
   bool send_head_first = true;
-  char *file_name;
   bool force_full_retrieve = false;
 
 
@@ -3864,11 +3867,6 @@ http_loop (struct url *u, struct url *original_url, char **newloc,
   if (opt.content_disposition && opt.always_rest)
     send_head_first = true;
 
-  if (!opt.output_document)
-      file_name = url_file_name (opt.trustservernames ? u : original_url, NULL);
-  else
-    file_name = xstrdup (opt.output_document);
-
 #ifdef HAVE_METALINK
   if (opt.metalink_over_http)
     {
@@ -3881,7 +3879,7 @@ http_loop (struct url *u, struct url *original_url, char **newloc,
     {
       /* Use conditional get request if requested
        * and if timestamp is known at this moment.  */
-      if (opt.if_modified_since && file_exists_p (file_name) && !send_head_first)
+      if (opt.if_modified_since && !send_head_first && got_name && file_exists_p (hstat.local_file))
         {
           *dt |= IF_MODIFIED_SINCE;
           {
@@ -3892,12 +3890,10 @@ http_loop (struct url *u, struct url *original_url, char **newloc,
         }
         /* Send preliminary HEAD request if -N is given and we have existing
          * destination file or content disposition is enabled.  */
-      else if (file_exists_p (file_name) || opt.content_disposition)
+      else if (opt.content_disposition || file_exists_p (hstat.local_file))
         send_head_first = true;
     }
 
-  xfree (file_name);
-
   /* THE loop */
   do
     {

src/init.c

@@ -115,6 +115,7 @@ CMD_DECLARE (cmd_spec_secure_protocol);
 CMD_DECLARE (cmd_spec_timeout);
 CMD_DECLARE (cmd_spec_useragent);
 CMD_DECLARE (cmd_spec_verbose);
+CMD_DECLARE (cmd_check_cert);
 
 /* List of recognized commands, each consisting of name, place and
    function.  When adding a new command, simply add it to the list,
@@ -142,6 +143,9 @@ static const struct {
   { "backups",          &opt.backups,           cmd_number },
   { "base",             &opt.base_href,         cmd_string },
   { "bindaddress",      &opt.bind_address,      cmd_string },
+#ifdef HAVE_LIBCARES
+  { "binddnsaddress",   &opt.bind_dns_address,  cmd_string },
+#endif
   { "bodydata",         &opt.body_data,         cmd_string },
   { "bodyfile",         &opt.body_file,         cmd_string },
 #ifdef HAVE_SSL
@@ -152,7 +156,7 @@ static const struct {
   { "cadirectory",      &opt.ca_directory,      cmd_directory },
   { "certificate",      &opt.cert_file,         cmd_file },
   { "certificatetype",  &opt.cert_type,         cmd_cert_type },
-  { "checkcertificate", &opt.check_cert,        cmd_boolean },
+  { "checkcertificate", &opt.check_cert,        cmd_check_cert },
 #endif
   { "chooseconfig",     &opt.choose_config,     cmd_file },
   { "connecttimeout",   &opt.connect_timeout,   cmd_time },
@@ -172,6 +176,9 @@ static const struct {
   { "dirprefix",        &opt.dir_prefix,        cmd_directory },
   { "dirstruct",        NULL,                   cmd_spec_dirstruct },
   { "dnscache",         &opt.dns_cache,         cmd_boolean },
+#ifdef HAVE_LIBCARES
+  { "dnsservers",       &opt.dns_servers,       cmd_string },
+#endif
   { "dnstimeout",       &opt.dns_timeout,       cmd_time },
   { "domains",          &opt.domains,           cmd_vector },
   { "dotbytes",         &opt.dot_bytes,         cmd_bytes },
@@ -253,6 +260,9 @@ static const struct {
   { "passiveftp",       &opt.ftp_pasv,          cmd_boolean },
   { "passwd",           &opt.ftp_passwd,        cmd_string },/* deprecated*/
   { "password",         &opt.passwd,            cmd_string },
+#ifdef HAVE_SSL
+  { "pinnedpubkey",     &opt.pinnedpubkey,      cmd_string },
+#endif
   { "postdata",         &opt.post_data,         cmd_string },
   { "postfile",         &opt.post_file_name,    cmd_file },
   { "preferfamily",     NULL,                   cmd_spec_prefer_family },
@@ -415,7 +425,7 @@ defaults (void)
   opt.retr_symlinks = true;
 
 #ifdef HAVE_SSL
-  opt.check_cert = true;
+  opt.check_cert = CHECK_CERT_ON;
   opt.ftps_resume_ssl = true;
   opt.ftps_fallback_to_ftp = false;
   opt.ftps_implicit = false;
@@ -955,6 +965,18 @@ static bool simple_atof (const char *, const char *, double *);
                      && (p)[3] == '\0')
 
 
+static int
+cmd_boolean_internal (const char *com _GL_UNUSED, const char *val, void *place _GL_UNUSED)
+{
+  if (CMP2 (val, 'o', 'n') || CMP3 (val, 'y', 'e', 's') || CMP1 (val, '1'))
+    /* "on", "yes" and "1" mean true. */
+    return 1;
+  else if (CMP3 (val, 'o', 'f', 'f') || CMP2 (val, 'n', 'o') || CMP1 (val, '0'))
+    /* "off", "no" and "0" mean false. */
+    return 0;
+  return -1;
+}
+
 /* Store the boolean value from VAL to PLACE.  COM is ignored,
    except for error messages.  */
 static bool
@@ -962,24 +984,62 @@ cmd_boolean (const char *com, const char *val, void *place)
 {
   bool value;
 
-  if (CMP2 (val, 'o', 'n') || CMP3 (val, 'y', 'e', 's') || CMP1 (val, '1'))
-    /* "on", "yes" and "1" mean true. */
-    value = true;
-  else if (CMP3 (val, 'o', 'f', 'f') || CMP2 (val, 'n', 'o') || CMP1 (val, '0'))
-    /* "off", "no" and "0" mean false. */
-    value = false;
-  else
+  switch (cmd_boolean_internal (com, val, place))
     {
-      fprintf (stderr,
-               _("%s: %s: Invalid boolean %s; use `on' or `off'.\n"),
-               exec_name, com, quote (val));
-      return false;
-    }
+    case 0:
+      value = false;
+      break;
 
+    case 1:
+      value = true;
+      break;
+
+    default:
+      {
+        fprintf (stderr,
+                 _("%s: %s: Invalid boolean %s; use `on' or `off'.\n"),
+                 exec_name, com, quote (val));
+        return false;
+      }
+    }
   *(bool *) place = value;
   return true;
 }
 
+/* Store the check_cert value from VAL to PLACE.  COM is ignored,
+   except for error messages.  */
+static bool
+cmd_check_cert (const char *com, const char *val, void *place)
+{
+  int value;
+
+  switch (cmd_boolean_internal (com, val, place))
+    {
+    case 0:
+      value = CHECK_CERT_OFF;
+      break;
+
+    case 1:
+      value = CHECK_CERT_ON;
+      break;
+
+    default:
+      {
+        if (!c_strcasecmp (val, "quiet"))
+          value = CHECK_CERT_QUIET;
+        else
+          {
+            fprintf (stderr,
+                     _("%s: %s: Invalid %s; use `on', `off' or `quiet'.\n"),
+                     exec_name, com, quote (val));
+            return false;
+          }
+      }
+    }
+  *(int *) place = value;
+  return true;
+}
+
 /* Set the non-negative integer value from VAL to PLACE.  With
    incorrect specification, the number remains unchanged.  */
 static bool
@@ -1871,6 +1931,18 @@ cleanup (void)
   xfree (opt.body_file);
   xfree (opt.rejected_log);
 
+#ifdef HAVE_LIBCARES
+#include <ares.h>
+  {
+    extern ares_channel ares;
+
+    xfree (opt.bind_dns_address);
+    xfree (opt.dns_servers);
+    ares_destroy (ares);
+    ares_library_cleanup ();
+  }
+#endif
+
 #endif /* DEBUG_MALLOC */
 }
 

src/iri.c

@@ -129,8 +129,8 @@ do_conversion (const char *tocode, const char *fromcode, char const *in_org, siz
   cd = iconv_open (tocode, fromcode);
   if (cd == (iconv_t)(-1))
     {
-      logprintf (LOG_VERBOSE, _("Conversion from %s to UTF-8 isn't supported\n"),
-                 quote (opt.locale));
+      logprintf (LOG_VERBOSE, _("Conversion from %s to %s isn't supported\n"),
+                 quote (fromcode), quote (tocode));
       *out = NULL;
       return false;
     }
@@ -146,7 +146,8 @@ do_conversion (const char *tocode, const char *fromcode, char const *in_org, siz
 
   for (;;)
     {
-      if (iconv (cd, &in, &inlen, out, &outlen) != (size_t)(-1))
+      if (iconv (cd, &in, &inlen, out, &outlen) != (size_t)(-1) &&
+          iconv (cd, NULL, NULL, out, &outlen) != (size_t)(-1))
         {
           *out = s;
           *(s + len - outlen - done) = '\0';
@@ -179,16 +180,10 @@ do_conversion (const char *tocode, const char *fromcode, char const *in_org, siz
         }
       else if (errno == E2BIG) /* Output buffer full */
         {
-          char *new;
-
           tooshort++;
           done = len;
-          outlen = done + inlen * 2;
-          new = xmalloc (outlen + 1);
-          memcpy (new, s, done);
-          xfree (s);
-          s = new;
-          len = outlen;
+          len = outlen = done + inlen * 2;
+          s = xrealloc (s, outlen + 1);
           *out = s + done;
         }
       else /* Weird, we got an unspecified error */

src/iri.h

@@ -60,7 +60,7 @@ void set_content_encoding (struct iri *i, const char *charset);
 
 extern struct iri dummy_iri;
 
-#define parse_charset(str)          (str, NULL)
+#define parse_charset(str)          NULL
 #define find_locale()               NULL
 #define check_encoding_name(str)    false
 #define locale_to_utf8(str)         (str)

src/log.c

@@ -351,6 +351,7 @@ logputs (enum log_options o, const char *s)
 {
   FILE *fp;
   FILE *warcfp;
+  int errno_save = errno;
 
   check_redirect_output ();
   if (o == LOG_PROGRESS)
@@ -358,10 +359,14 @@ logputs (enum log_options o, const char *s)
   else
     fp = get_log_fp ();
 
+  errno = errno_save;
+
   if (fp == NULL)
     return;
 
   warcfp = get_warc_log_fp ();
+  errno = errno_save;
+
   CHECK_VERBOSE (o);
 
   FPUTS (s, fp);
@@ -373,6 +378,8 @@ logputs (enum log_options o, const char *s)
     logflush ();
   else
     needs_flushing = true;
+
+  errno = errno_save;
 }
 
 struct logvprintf_state {
@@ -543,8 +550,10 @@ logprintf (enum log_options o, const char *fmt, ...)
   va_list args;
   struct logvprintf_state lpstate;
   bool done;
+  int errno_saved = errno;
 
   check_redirect_output ();
+  errno = errno_saved;
   if (inhibit_logging)
     return;
   CHECK_VERBOSE (o);
@@ -561,6 +570,8 @@ logprintf (enum log_options o, const char *fmt, ...)
         exit (WGET_EXIT_GENERIC_ERROR);
     }
   while (!done);
+
+  errno = errno_saved;
 }
 
 #ifdef ENABLE_DEBUG

src/main.c

@@ -86,6 +86,13 @@ as that of the covered work.  */
 struct iri dummy_iri;
 #endif
 
+#ifdef HAVE_LIBCARES
+#include <ares.h>
+ares_channel ares;
+#else
+void *ares;
+#endif
+
 struct options opt;
 
 /* defined in version.c */
@@ -197,10 +204,12 @@ save_hsts (void)
     {
       char *filename = get_hsts_database ();
 
-      if (filename)
-        DEBUGP (("Saving HSTS entries to %s\n", filename));
+      if (filename && hsts_store_has_changed (hsts_store))
+        {
+          DEBUGP (("Saving HSTS entries to %s\n", filename));
+          hsts_store_save (hsts_store, filename);
+        }
 
-      hsts_store_save (hsts_store, filename);
       hsts_store_close (hsts_store);
 
       xfree (filename);
@@ -252,6 +261,9 @@ static struct cmdline_option option_data[] =
     { "backups", 0, OPT_BOOLEAN, "backups", -1 },
     { "base", 'B', OPT_VALUE, "base", -1 },
     { "bind-address", 0, OPT_VALUE, "bindaddress", -1 },
+#ifdef HAVE_LIBCARES
+    { "bind-dns-address", 0, OPT_VALUE, "binddnsaddress", -1 },
+#endif
     { "body-data", 0, OPT_VALUE, "bodydata", -1 },
     { "body-file", 0, OPT_VALUE, "bodyfile", -1 },
     { IF_SSL ("ca-certificate"), 0, OPT_VALUE, "cacertificate", -1 },
@@ -277,6 +289,9 @@ static struct cmdline_option option_data[] =
     { "directories", 0, OPT_BOOLEAN, "dirstruct", -1 },
     { "directory-prefix", 'P', OPT_VALUE, "dirprefix", -1 },
     { "dns-cache", 0, OPT_BOOLEAN, "dnscache", -1 },
+#ifdef HAVE_LIBCARES
+    { "dns-servers", 0, OPT_VALUE, "dnsservers", -1 },
+#endif
     { "dns-timeout", 0, OPT_VALUE, "dnstimeout", -1 },
     { "domains", 'D', OPT_VALUE, "domains", -1 },
     { "dont-remove-listing", 0, OPT__DONT_REMOVE_LISTING, NULL, no_argument },
@@ -350,6 +365,7 @@ static struct cmdline_option option_data[] =
     { "parent", 0, OPT__PARENT, NULL, optional_argument },
     { "passive-ftp", 0, OPT_BOOLEAN, "passiveftp", -1 },
     { "password", 0, OPT_VALUE, "password", -1 },
+    { IF_SSL ("pinnedpubkey"), 0, OPT_VALUE, "pinnedpubkey", -1 },
     { "post-data", 0, OPT_VALUE, "postdata", -1 },
     { "post-file", 0, OPT_VALUE, "postfile", -1 },
     { "prefer-family", 0, OPT_VALUE, "preferfamily", -1 },
@@ -627,6 +643,12 @@ Download:\n"),
        --spider                    don't download anything\n"),
     N_("\
   -T,  --timeout=SECONDS           set all timeout values to SECONDS\n"),
+#ifdef HAVE_LIBCARES
+    N_("\
+       --dns-servers=ADDRESSES     list of DNS servers to query (comma separated)\n"),
+    N_("\
+       --bind-dns-address=ADDRESS  bind DNS resolver to ADDRESS (hostname or IP) on local host\n"),
+#endif
     N_("\
        --dns-timeout=SECS          set the DNS lookup timeout to SECS\n"),
     N_("\
@@ -784,6 +806,11 @@ HTTPS (SSL/TLS) options:\n"),
        --ca-directory=DIR          directory where hash list of CAs is stored\n"),
     N_("\
        --crl-file=FILE             file with bundle of CRLs\n"),
+    N_("\
+       --pinnedpubkey=FILE/HASHES  Public key (PEM/DER) file, or any number\n\
+                                   of base64 encoded sha256 hashes preceded by\n\
+                                   \'sha256//\' and seperated by \';\', to verify\n\
+                                   peer against\n"),
 #if defined(HAVE_LIBSSL) || defined(HAVE_LIBSSL32)
     N_("\
        --random-file=FILE          file with random data for seeding the SSL PRNG\n"),
@@ -1774,6 +1801,58 @@ only if outputting to a regular file.\n"));
         }
     }
 
+#ifdef HAVE_LIBCARES
+  if (opt.bind_dns_address || opt.dns_servers)
+    {
+      if (ares_library_init (ARES_LIB_INIT_ALL))
+        {
+          fprintf (stderr, _("Failed to init libcares\n"));
+          exit (WGET_EXIT_GENERIC_ERROR);
+        }
+
+      if (ares_init (&ares) != ARES_SUCCESS)
+        {
+          fprintf (stderr, _("Failed to init c-ares channel\n"));
+          exit (WGET_EXIT_GENERIC_ERROR);
+        }
+
+      if (opt.bind_dns_address)
+        {
+          struct in_addr a4;
+#ifdef ENABLE_IPV6
+          struct in6_addr a6;
+#endif
+
+          if (inet_pton (AF_INET, opt.bind_dns_address, &a4) == 1)
+            {
+              ares_set_local_ip4 (ares, ntohl (a4.s_addr));
+            }
+#ifdef ENABLE_IPV6
+          else if (inet_pton (AF_INET6, opt.bind_dns_address, &a6) == 1)
+            {
+              ares_set_local_ip6 (ares, (unsigned char *) &a6);
+            }
+#endif
+          else
+            {
+              fprintf (stderr, _("Failed to parse IP address '%s'\n"), opt.bind_dns_address);
+              exit (WGET_EXIT_GENERIC_ERROR);
+            }
+        }
+
+      if (opt.dns_servers)
+        {
+          int result;
+
+          if ((result = ares_set_servers_csv (ares, opt.dns_servers)) != ARES_SUCCESS)
+            {
+              fprintf (stderr, _("Failed to set DNS server(s) '%s' (%d)\n"), opt.dns_servers, result);
+              exit (WGET_EXIT_GENERIC_ERROR);
+            }
+        }
+    }
+#endif
+
 #ifdef __VMS
   /* Set global ODS5 flag according to the specified destination (if
      any), otherwise according to the current default device.
@@ -1842,13 +1921,21 @@ only if outputting to a regular file.\n"));
       else
         {
           if ((opt.recursive || opt.page_requisites)
-              && ((url_scheme (*t) != SCHEME_FTP && url_scheme (*t) != SCHEME_FTPS)
+              && ((url_scheme (*t) != SCHEME_FTP
+#ifdef HAVE_SSL
+              && url_scheme (*t) != SCHEME_FTPS
+#endif
+              )
                   || url_uses_proxy (url_parsed)))
             {
               int old_follow_ftp = opt.follow_ftp;
 
               /* Turn opt.follow_ftp on in case of recursive FTP retrieval */
-              if (url_scheme (*t) == SCHEME_FTP || url_scheme (*t) == SCHEME_FTPS)
+              if (url_scheme (*t) == SCHEME_FTP
+#ifdef HAVE_SSL
+                  || url_scheme (*t) == SCHEME_FTPS
+#endif
+                  )
                 opt.follow_ftp = 1;
 
               retrieve_tree (url_parsed, NULL);

src/metalink.c

@@ -36,7 +36,7 @@ as that of the covered work.  */
 #include "utils.h"
 #include "sha256.h"
 #include "xstrndup.h"
-#include <sys/errno.h>
+#include <errno.h>
 #include <unistd.h> /* For unlink.  */
 #include <metalink/metalink_parser.h>
 #ifdef HAVE_GPGME
@@ -168,7 +168,7 @@ retrieve_from_metalink (const metalink_t* metalink)
               FILE *local_file;
 
               /* Check the digest.  */
-              local_file = fopen (filename, "r");
+              local_file = fopen (filename, "rb");
               if (!local_file)
                 {
                   logprintf (LOG_NOTQUIET, _("Could not open downloaded file.\n"));
@@ -274,7 +274,7 @@ retrieve_from_metalink (const metalink_t* metalink)
                       goto gpg_skip_verification;
                     }
 
-                  DEBUGP (("Veryfying signature %s:\n%s\n",
+                  DEBUGP (("Verifying signature %s:\n%s\n",
                            quote (msig->mediatype),
                            msig->signature));
 
@@ -342,10 +342,7 @@ retrieve_from_metalink (const metalink_t* metalink)
                   /* The list is null-terminated.  */
                   for (gpgsig = gpgres->signatures; gpgsig; gpgsig = gpgsig->next)
                     {
-                      DEBUGP (("Checking signature 0x%p\n",
-                               (void *) gpgsig));
-                      DEBUGP (("Summary=0x%x Status=0x%x\n",
-                               gpgsig->summary, gpgsig->status & 0xFFFF));
+                      DEBUGP (("Checking signature %s\n", gpgsig->fpr));
 
                       if (gpgsig->summary
                           & (GPGME_SIGSUM_VALID | GPGME_SIGSUM_GREEN))

src/mswindows.c

@@ -62,7 +62,7 @@ void
 xsleep (double seconds)
 {
 #if defined(HAVE_USLEEP) && defined(HAVE_SLEEP)
-  if (seconds > 1000)
+  if (seconds >= 1)
     {
       /* Explained in utils.c. */
       sleep (seconds);

src/openssl.c

@@ -35,6 +35,7 @@ as that of the covered work.  */
 #include <errno.h>
 #include <unistd.h>
 #include <string.h>
+#include <xalloc.h>
 
 #include <openssl/ssl.h>
 #include <openssl/x509v3.h>
@@ -506,6 +507,22 @@ ssl_connect_with_timeout_callback(void *arg)
   ctx->result = SSL_connect(ctx->ssl);
 }
 
+static const char *
+_sni_hostname(const char *hostname)
+{
+  size_t len = strlen(hostname);
+
+  char *sni_hostname = xmemdup(hostname, len + 1);
+
+  /* Remove trailing dot(s) to fix #47408.
+   * Regarding RFC 6066 (SNI): The hostname is represented as a byte
+   * string using ASCII encoding without a trailing dot. */
+  while (len && sni_hostname[--len] == '.')
+    sni_hostname[len] = 0;
+
+  return sni_hostname;
+}
+
 /* Perform the SSL handshake on file descriptor FD, which is assumed
    to be connected to an SSL server.  The SSL handle provided by
    OpenSSL is registered with the file descriptor FD using
@@ -532,7 +549,12 @@ ssl_connect_wget (int fd, const char *hostname, int *continue_session)
      then use it whenever we have a hostname.  If not, don't, ever. */
   if (! is_valid_ip_address (hostname))
     {
-      if (! SSL_set_tlsext_host_name (conn, hostname))
+      const char *sni_hostname = _sni_hostname(hostname);
+
+      long rc = SSL_set_tlsext_host_name (conn, sni_hostname);
+      xfree(sni_hostname);
+
+      if (rc == 0)
         {
           DEBUGP (("Failed to set TLS server-name indication."));
           goto error;
@@ -650,6 +672,65 @@ static char *_get_rfc2253_formatted (X509_NAME *name)
   return out ? out : xstrdup("");
 }
 
+/*
+ * Heavily modified from:
+ * https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning#OpenSSL
+ */
+static bool
+pkp_pin_peer_pubkey (X509* cert, const char *pinnedpubkey)
+{
+  /* Scratch */
+  int len1 = 0, len2 = 0;
+  char *buff1 = NULL, *temp = NULL;
+
+  /* Result is returned to caller */
+  bool result = false;
+
+  /* if a path wasn't specified, don't pin */
+  if (!pinnedpubkey)
+    return true;
+
+  if (!cert)
+    return result;
+
+  /* Begin Gyrations to get the subjectPublicKeyInfo     */
+  /* Thanks to Viktor Dukhovni on the OpenSSL mailing list */
+
+  /* https://groups.google.com/group/mailing.openssl.users/browse_thread
+   /thread/d61858dae102c6c7 */
+  len1 = i2d_X509_PUBKEY (X509_get_X509_PUBKEY (cert), NULL);
+  if (len1 < 1)
+    goto cleanup; /* failed */
+
+  /* https://www.openssl.org/docs/crypto/buffer.html */
+  buff1 = temp = OPENSSL_malloc (len1);
+  if (!buff1)
+    goto cleanup; /* failed */
+
+  /* https://www.openssl.org/docs/crypto/d2i_X509.html */
+  len2 = i2d_X509_PUBKEY (X509_get_X509_PUBKEY (cert), (unsigned char **) &temp);
+
+  /*
+   * These checks are verifying we got back the same values as when we
+   * sized the buffer. It's pretty weak since they should always be the
+   * same. But it gives us something to test.
+   */
+  if ((len1 != len2) || !temp || ((temp - buff1) != len1))
+    goto cleanup; /* failed */
+
+  /* End Gyrations */
+
+  /* The one good exit point */
+  result = wg_pin_peer_pubkey (pinnedpubkey, buff1, len1);
+
+ cleanup:
+  /* https://www.openssl.org/docs/crypto/buffer.html */
+  if (NULL != buff1)
+    OPENSSL_free (buff1);
+
+  return result;
+}
+
 /* Verify the validity of the certificate presented by the server.
    Also check that the "common name" of the server, as presented by
    its certificate, corresponds to HOST.  (HOST typically comes from
@@ -673,6 +754,7 @@ ssl_check_certificate (int fd, const char *host)
   long vresult;
   bool success = true;
   bool alt_name_checked = false;
+  bool pinsuccess = opt.pinnedpubkey == NULL;
 
   /* If the user has specified --no-check-cert, we still want to warn
      him about problems with the server's certificate.  */
@@ -682,6 +764,10 @@ ssl_check_certificate (int fd, const char *host)
   SSL *conn = ctx->conn;
   assert (conn != NULL);
 
+  /* The user explicitly said to not check for the certificate.  */
+  if (opt.check_cert == CHECK_CERT_QUIET && pinsuccess)
+    return success;
+
   cert = SSL_get_peer_certificate (conn);
   if (!cert)
     {
@@ -758,9 +844,12 @@ ssl_check_certificate (int fd, const char *host)
     {
       /* Test subject alternative names */
 
+      /* SNI hostname must not have a trailing dot */
+      const char *sni_hostname = _sni_hostname(host);
+
       /* Do we want to check for dNSNAmes or ipAddresses (see RFC 2818)?
        * Signal it by host_in_octet_string. */
-      ASN1_OCTET_STRING *host_in_octet_string = a2i_IPADDRESS (host);
+      ASN1_OCTET_STRING *host_in_octet_string = a2i_IPADDRESS (sni_hostname);
 
       int numaltnames = sk_GENERAL_NAME_num (subjectAltNames);
       int i;
@@ -795,7 +884,7 @@ ssl_check_certificate (int fd, const char *host)
                   if (0 <= ASN1_STRING_to_UTF8 (&name_in_utf8, name->d.dNSName))
                     {
                       /* Compare and check for NULL attack in ASN1_STRING */
-                      if (pattern_match ((char *)name_in_utf8, host) &&
+                      if (pattern_match ((char *)name_in_utf8, sni_hostname) &&
                             (strlen ((char *)name_in_utf8) ==
                                 (size_t) ASN1_STRING_length (name->d.dNSName)))
                         {
@@ -816,9 +905,11 @@ ssl_check_certificate (int fd, const char *host)
           logprintf (LOG_NOTQUIET,
               _("%s: no certificate subject alternative name matches\n"
                 "\trequested host name %s.\n"),
-                     severity, quote_n (1, host));
+                     severity, quote_n (1, sni_hostname));
           success = false;
         }
+
+      xfree(sni_hostname);
     }
 
   if (alt_name_checked == false)
@@ -873,6 +964,13 @@ ssl_check_certificate (int fd, const char *host)
         }
     }
 
+    pinsuccess = pkp_pin_peer_pubkey (cert, opt.pinnedpubkey);
+    if (!pinsuccess)
+      {
+        logprintf (LOG_ALWAYS, _("The public key does not match pinned public key!\n"));
+        success = false;
+      }
+
 
   if (success)
     DEBUGP (("X509 certificate successfully verified and matches host %s\n",
@@ -880,13 +978,13 @@ ssl_check_certificate (int fd, const char *host)
   X509_free (cert);
 
  no_cert:
-  if (opt.check_cert && !success)
+  if (opt.check_cert == CHECK_CERT_ON && !success)
     logprintf (LOG_NOTQUIET, _("\
 To connect to %s insecurely, use `--no-check-certificate'.\n"),
                quotearg_style (escape_quoting_style, host));
 
-  /* Allow --no-check-cert to disable certificate checking. */
-  return opt.check_cert ? success : true;
+  /* never return true if pinsuccess fails */
+  return !pinsuccess ? false : (opt.check_cert == CHECK_CERT_ON ? success : true);
 }
 
 /*

src/options.h

@@ -29,6 +29,13 @@ Corresponding Source for a non-source form of such a combination
 shall include the source code for the parts of OpenSSL used as well
 as that of the covered work.  */
 
+enum CHECK_CERT_MODES
+{
+  CHECK_CERT_OFF,
+  CHECK_CERT_ON,
+  CHECK_CERT_QUIET
+};
+
 struct options
 {
   int verbose;                  /* Are we verbose?  (First set to -1,
@@ -92,6 +99,11 @@ struct options
   void *(*regex_compile_fun)(const char *);             /* Function to compile a regex. */
   bool (*regex_match_fun)(const void *, const char *);  /* Function to match a string to a regex. */
 
+#ifdef HAVE_LIBCARES
+  char *bind_dns_address;
+  char *dns_servers;
+#endif
+
   char **domains;               /* See host.c */
   char **exclude_domains;
   bool dns_cache;               /* whether we cache DNS lookups. */
@@ -215,7 +227,7 @@ struct options
     secure_protocol_tlsv1_2,
     secure_protocol_pfs
   } secure_protocol;            /* type of secure protocol to use. */
-  bool check_cert;              /* whether to validate the server's cert */
+  int check_cert;               /* whether to validate the server's cert */
   char *cert_file;              /* external client certificate to use. */
   char *private_key;            /* private key file (if not internal). */
   enum keyfile_type {
@@ -229,6 +241,11 @@ struct options
   char *ca_cert;                /* CA certificate file to use */
   char *crl_file;               /* file with CRLs */
 
+  char *pinnedpubkey;           /* Public key (PEM/DER) file, or any number
+                                   of base64 encoded sha256 hashes preceded by
+                                   \'sha256//\' and seperated by \';\', to verify
+                                   peer against */
+
   char *random_file;            /* file with random data to seed the PRNG */
   char *egd_file;               /* file name of the egd daemon socket */
   bool https_only;              /* whether to follow HTTPS only */

src/progress.c

@@ -594,7 +594,8 @@ bar_create (const char *f_download, wgint initial, wgint total)
   bp->width = screen_width - 1;
   /* + enough space for the terminating zero, and hopefully enough room
    * for multibyte characters. */
-  bp->buffer = xmalloc (bp->width + 100);
+#define BUF_LEN (bp->width + 100)
+  bp->buffer = xmalloc (BUF_LEN);
 
   logputs (LOG_VERBOSE, "\n");
 
@@ -854,7 +855,7 @@ get_eta (int *bcd)
 {
   /* TRANSLATORS: "ETA" is English-centric, but this must
      be short, ideally 3 chars.  Abbreviate if necessary.  */
-  static const char eta_str[] = N_("   eta %s");
+  static const char eta_str[] = N_("    eta %s");
   static const char *eta_trans;
   static int bytes_cols_diff;
   if (eta_trans == NULL)
@@ -903,11 +904,11 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
   char *p = bp->buffer;
   wgint size = bp->initial_length + bp->count;
 
-  int size_grouped_pad; /* Used to pad the field width for size_grouped. */
-
   struct bar_progress_hist *hist = &bp->hist;
   int orig_filename_cols = count_cols (bp->f_download);
 
+  int padding;
+
   /* The progress bar should look like this:
      file xx% [=======>             ] nnn.nnK 12.34KB/s  eta 36m 51s
 
@@ -928,12 +929,16 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
      "=====>..."       - progress bar             - the rest
   */
 
+  /* TODO: Ask the Turkish Translators to fix their translation for the "done"
+   * mode of progress bar. Use one less character. Once that is done, redice
+   * PROGRESS_ETA_LEN by 1.
+   */
 #define PROGRESS_FILENAME_LEN  MAX_FILENAME_COLS + 1
 #define PROGRESS_PERCENT_LEN   4
 #define PROGRESS_DECORAT_LEN   2
 #define PROGRESS_FILESIZE_LEN  7 + 1
-#define PROGRESS_DWNLOAD_RATE  8 + 1
-#define PROGRESS_ETA_LEN       14
+#define PROGRESS_DWNLOAD_RATE  8 + 2
+#define PROGRESS_ETA_LEN       15
 
   int progress_size = bp->width - (PROGRESS_FILENAME_LEN + PROGRESS_PERCENT_LEN +
                                    PROGRESS_DECORAT_LEN + PROGRESS_FILESIZE_LEN +
@@ -945,23 +950,23 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
   int cols_diff;
   const char *down_size;
 
+  memset (bp->buffer, '\0', BUF_LEN);
+
   if (progress_size < 5)
     progress_size = 0;
 
   if (orig_filename_cols <= MAX_FILENAME_COLS)
     {
-      int padding = MAX_FILENAME_COLS - orig_filename_cols;
-      sprintf (p, "%s ", bp->f_download);
-      p += orig_filename_cols + 1;
-      for (;padding;padding--)
-        *p++ = ' ';
+      padding = MAX_FILENAME_COLS - orig_filename_cols;
+      p += sprintf (p, "%s ", bp->f_download);
+      memset (p, ' ', padding);
+      p += padding;
     }
   else
     {
       int offset_cols;
       int bytes_in_filename, offset_bytes, col;
       int *cols_ret = &col;
-      int padding;
 
 #define MIN_SCROLL_TEXT 5
       if ((orig_filename_cols > MAX_FILENAME_COLS + MIN_SCROLL_TEXT) &&
@@ -992,9 +997,8 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
       memcpy (p, bp->f_download + offset_bytes, bytes_in_filename);
       p += bytes_in_filename;
       padding = MAX_FILENAME_COLS - (padding + *cols_ret);
-      for (;padding;padding--)
-          *p++ = ' ';
-      *p++ = ' ';
+      memset (p, ' ', padding + 1);
+      p += padding + 1;
     }
 
   /* "xx% " */
@@ -1002,15 +1006,13 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
     {
       int percentage = 100.0 * size / bp->total_length;
       assert (percentage <= 100);
-
-      if (percentage < 100)
-        sprintf (p, "%3d%%", percentage);
-      else
-        strcpy (p, "100%");
-      p += 4;
+      p += sprintf (p, "%3d%%", percentage);
     }
   else
-    APPEND_LITERAL ("    ");
+    {
+      memset (p, ' ', PROGRESS_PERCENT_LEN);
+      p += PROGRESS_PERCENT_LEN;
+    }
 
   /* The progress bar: "[====>      ]" or "[++==>      ]". */
   if (progress_size && bp->total_length > 0)
@@ -1022,7 +1024,6 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
       int dlsz = (double)size / bp->total_length * progress_size;
 
       char *begin;
-      int i;
 
       assert (dlsz <= progress_size);
       assert (insz <= dlsz);
@@ -1032,18 +1033,19 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
 
       /* Print the initial portion of the download with '+' chars, the
          rest with '=' and one '>'.  */
-      for (i = 0; i < insz; i++)
-        *p++ = '+';
+      memset (p, '+', insz);
+      p += insz;
+
       dlsz -= insz;
       if (dlsz > 0)
         {
-          for (i = 0; i < dlsz - 1; i++)
-            *p++ = '=';
+          memset (p, '=', dlsz-1);
+          p += dlsz - 1;
           *p++ = '>';
         }
 
-      while (p - begin < progress_size)
-        *p++ = ' ';
+      memset (p, ' ', (progress_size - (p - begin)));
+      p += (progress_size - (p - begin));
       *p++ = ']';
     }
   else if (progress_size)
@@ -1071,27 +1073,14 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
       *p++ = ']';
 
     }
- ++bp->tick;
+  ++bp->tick;
 
   /* " 234.56M" */
   down_size = human_readable (size, 1000, 2);
-  cols_diff = 7 - count_cols (down_size);
-  while (cols_diff > 0)
-  {
-    *p++=' ';
-    cols_diff--;
-  }
-  sprintf (p, " %s", down_size);
-  move_to_end (p);
-  /* Pad with spaces to 7 chars for the size_grouped field;
-   * couldn't use the field width specifier in sprintf, because
-   * it counts in bytes, not characters. */
-  for (size_grouped_pad = PROGRESS_FILESIZE_LEN - 7;
-       size_grouped_pad > 0;
-       --size_grouped_pad)
-    {
-      *p++ = ' ';
-    }
+  cols_diff = PROGRESS_FILESIZE_LEN - count_cols (down_size);
+  memset (p, ' ', cols_diff);
+  p += cols_diff;
+  p += sprintf (p, "%s", down_size);
 
   /* " 12.52Kb/s or 12.52KB/s" */
   if (hist->total_time > 0 && hist->total_bytes)
@@ -1104,12 +1093,11 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
       wgint dlquant = hist->total_bytes + bp->recent_bytes;
       double dltime = hist->total_time + (dl_total_time - bp->recent_start);
       double dlspeed = calc_rate (dlquant, dltime, &units);
-      sprintf (p, " %4.*f%s", dlspeed >= 99.95 ? 0 : dlspeed >= 9.995 ? 1 : 2,
+      p += sprintf (p, "  %4.*f%s", dlspeed >= 99.95 ? 0 : dlspeed >= 9.995 ? 1 : 2,
                dlspeed,  !opt.report_bps ? short_units[units] : short_units_bits[units]);
-      move_to_end (p);
     }
   else
-    APPEND_LITERAL (" --.-KB/s");
+    APPEND_LITERAL ("  --.-KB/s");
 
   if (!done)
     {
@@ -1144,14 +1132,14 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
               bp->last_eta_time = dl_total_time;
             }
 
-          sprintf (p, get_eta(&bytes_cols_diff),
+          p += sprintf (p, get_eta(&bytes_cols_diff),
                    eta_to_human_short (eta, false));
-          move_to_end (p);
         }
       else if (bp->total_length > 0)
         {
         skip_eta:
-          APPEND_LITERAL ("             ");
+          memset (p, ' ', PROGRESS_ETA_LEN);
+          p += PROGRESS_ETA_LEN;
         }
     }
   else
@@ -1161,21 +1149,25 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
       int ncols;
 
       /* Note to translators: this should not take up more room than
-         available here.  Abbreviate if necessary.  */
-      strcpy (p, _("   in "));
+         available here (6 columns).  Abbreviate if necessary.  */
+      strcpy (p, _("    in "));
       nbytes = strlen (p);
       ncols  = count_cols (p);
       bytes_cols_diff = nbytes - ncols;
-      p += nbytes;
       if (dl_total_time >= 10)
-        strcpy (p, eta_to_human_short ((int) (dl_total_time + 0.5), false));
+        ncols += sprintf (p + nbytes, "%s",  eta_to_human_short ((int) (dl_total_time + 0.5), false));
       else
-        sprintf (p, "%ss", print_decimal (dl_total_time));
-      move_to_end (p);
+        ncols += sprintf (p + nbytes, "%ss", print_decimal (dl_total_time));
+      p += ncols + bytes_cols_diff;
+      memset (p, ' ', PROGRESS_ETA_LEN - ncols);
+      p += PROGRESS_ETA_LEN - ncols;
     }
 
-  while (p - bp->buffer - bytes_cols_diff < bp->width)
-    *p++ = ' ';
+  padding = bp->width - count_cols (bp->buffer);
+  assert (padding >= 0 && "Padding length became non-positive!");
+  padding = padding > 0 ? padding : 0;
+  memset (p, ' ', padding);
+  p += padding;
   *p = '\0';
 
   /* 2014-11-14  Darshit Shah  <darnir@gmail.com>
@@ -1184,8 +1176,11 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
    * from the release code since we do not want Wget to crash and burn when the
    * assertion fails. Instead Wget should continue downloading and display a
    * horrible and irritating progress bar that spams the screen with newlines.
+   *
+   * By default, all assertions are disabled in a Wget build and are enabled
+   * only with the --enable-assert configure option.
    */
-  assert (count_cols (bp->buffer) <= bp->width + 1);
+  assert (count_cols (bp->buffer) == bp->width);
 }
 
 /* Print the contents of the buffer as a one-line ASCII "image" so
@@ -1203,8 +1198,6 @@ display_image (char *buf)
 static void
 bar_set_params (char *params)
 {
-  char *term = getenv ("TERM");
-
   if (params)
     {
       char *param = strtok (params, ":");
@@ -1224,12 +1217,6 @@ bar_set_params (char *params)
           dots.  */
        || !isatty (fileno (stderr))
 #endif
-       /* Normally we don't depend on terminal type because the
-          progress bar only uses ^M to move the cursor to the
-          beginning of line, which works even on dumb terminals.  But
-          Jamie Zawinski reports that ^M and ^H tricks don't work in
-          Emacs shell buffers, and only make a mess.  */
-       || (term && 0 == strcmp (term, "emacs"))
        )
       && !current_impl_locked)
     {

src/recur.c

@@ -234,17 +234,19 @@ retrieve_tree (struct url *start_url_parsed, struct iri *pi)
 
   FILE *rejectedlog = NULL; /* Don't write a rejected log. */
 
-#define COPYSTR(x)  (x) ? xstrdup(x) : NULL;
   /* Duplicate pi struct if not NULL */
   if (pi)
     {
+#define COPYSTR(x)  (x) ? xstrdup(x) : NULL;
       i->uri_encoding = COPYSTR (pi->uri_encoding);
       i->content_encoding = COPYSTR (pi->content_encoding);
       i->utf8_encode = pi->utf8_encode;
+#undef COPYSTR
     }
+#ifdef ENABLE_IRI
   else
     set_uri_encoding (i, opt.locale, true);
-#undef COPYSTR
+#endif
 
   queue = url_queue_new ();
   blacklist = make_string_hash_table (0);
@@ -610,7 +612,11 @@ download_child (const struct urlpos *upos, struct url *parent, int depth,
   u_scheme_like_http = schemes_are_similar_p (u->scheme, SCHEME_HTTP);
 
   /* 1. Schemes other than HTTP are normally not recursed into. */
-  if (!u_scheme_like_http && !((u->scheme == SCHEME_FTP || u->scheme == SCHEME_FTPS) && opt.follow_ftp))
+  if (!u_scheme_like_http && !((u->scheme == SCHEME_FTP
+#ifdef HAVE_SSL
+      || u->scheme == SCHEME_FTPS
+#endif
+      ) && opt.follow_ftp))
     {
       DEBUGP (("Not following non-HTTP schemes.\n"));
       reason = WG_RR_NONHTTP;

src/retr.c

@@ -830,14 +830,19 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file,
       if (redirection_count)
         oldrec = glob = false;
 
-      result = ftp_loop (u, &local_file, dt, proxy_url, recursive, glob);
+      result = ftp_loop (u, orig_parsed, &local_file, dt, proxy_url,
+                         recursive, glob);
       recursive = oldrec;
 
       /* There is a possibility of having HTTP being redirected to
          FTP.  In these cases we must decide whether the text is HTML
          according to the suffix.  The HTML suffixes are `.html',
          `.htm' and a few others, case-insensitive.  */
-      if (redirection_count && local_file && (u->scheme == SCHEME_FTP || u->scheme == SCHEME_FTPS))
+      if (redirection_count && local_file && (u->scheme == SCHEME_FTP
+#ifdef HAVE_SSL
+          || u->scheme == SCHEME_FTPS
+#endif
+          ))
         {
           if (has_html_suffix_p (local_file))
             *dt |= TEXTHTML;
@@ -868,11 +873,15 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file,
       xfree (mynewloc);
       mynewloc = construced_newloc;
 
-      /* Reset UTF-8 encoding state, keep the URI encoding and reset
+#ifdef ENABLE_IRI
+      /* Reset UTF-8 encoding state, set the URI encoding and reset
          the content encoding. */
       iri->utf8_encode = opt.enable_iri;
+      if (opt.encoding_remote)
+       set_uri_encoding (iri, opt.encoding_remote, true);
       set_content_encoding (iri, NULL);
       xfree (iri->orig_url);
+#endif
 
       /* Now, see if this new location makes sense. */
       newloc_parsed = url_parse (mynewloc, &up_error_code, iri, true);
@@ -1060,10 +1069,12 @@ retrieve_from_file (const char *file, bool html, int *count)
       if (dt & TEXTHTML)
         html = true;
 
+#ifdef ENABLE_IRI
       /* If we have a found a content encoding, use it.
        * ( == is okay, because we're checking for identical object) */
       if (iri->content_encoding != opt.locale)
           set_uri_encoding (iri, iri->content_encoding, false);
+#endif
 
       /* Reset UTF-8 encode status */
       iri->utf8_encode = opt.enable_iri;
@@ -1099,12 +1110,20 @@ retrieve_from_file (const char *file, bool html, int *count)
 
       proxy = getproxy (cur_url->url);
       if ((opt.recursive || opt.page_requisites)
-          && ((cur_url->url->scheme != SCHEME_FTP && cur_url->url->scheme != SCHEME_FTPS) || proxy))
+          && ((cur_url->url->scheme != SCHEME_FTP
+#ifdef HAVE_SSL
+          && cur_url->url->scheme != SCHEME_FTPS
+#endif
+          ) || proxy))
         {
           int old_follow_ftp = opt.follow_ftp;
 
           /* Turn opt.follow_ftp on in case of recursive FTP retrieval */
-          if (cur_url->url->scheme == SCHEME_FTP || cur_url->url->scheme == SCHEME_FTPS)
+          if (cur_url->url->scheme == SCHEME_FTP
+#ifdef HAVE_SSL
+              || cur_url->url->scheme == SCHEME_FTPS
+#endif
+              )
             opt.follow_ftp = 1;
 
           status = retrieve_tree (parsed_url ? parsed_url : cur_url->url,

src/sysdep.h

@@ -68,9 +68,6 @@ as that of the covered work.  */
 
 #ifdef NAMESPACE_TWEAKS
 
-/* Request the "Unix 98 compilation environment". */
-#define _XOPEN_SOURCE 500
-
 #endif /* NAMESPACE_TWEAKS */
 
 
@@ -146,16 +143,6 @@ int snprintf (char *str, size_t count, const char *fmt, ...);
 int vsnprintf (char *str, size_t count, const char *fmt, va_list arg);
 #endif
 
-/* Some systems (Linux libc5, "NCR MP-RAS 3.0", and others) don't
-   provide MAP_FAILED, a symbolic constant for the value returned by
-   mmap() when it doesn't work.  Usually, this constant should be -1.
-   This only makes sense for files that use mmap() and include
-   sys/mman.h *before* sysdep.h, but doesn't hurt others.  */
-
-#ifndef MAP_FAILED
-# define MAP_FAILED ((void *) -1)
-#endif
-
 /* Enable system fnmatch only on systems where fnmatch.h is usable.
    If the fnmatch on your system is buggy, undef this symbol and a
    replacement implementation will be used instead.  */

src/url.c

@@ -43,6 +43,11 @@ as that of the covered work.  */
 #include "host.h"  /* for is_valid_ipv6_address */
 #include "c-strcase.h"
 
+#if HAVE_ICONV
+#include <iconv.h>
+#include <langinfo.h>
+#endif
+
 #ifdef __VMS
 #include "vms.h"
 #endif /* def __VMS */
@@ -1399,8 +1404,8 @@ UVWC, VC, VC, VC,  VC, VC, VC, VC,   /* NUL SOH STX ETX  EOT ENQ ACK BEL */
    0,  0,  0,  0,   0,  0,  0,  0,   /* p   q   r   s    t   u   v   w   */
    0,  0,  0,  0,   W,  0,  0,  C,   /* x   y   z   {    |   }   ~   DEL */
 
-  C, C, C, C,  C, C, C, C,  C, C, C, C,  C, C, C, C, /* 128-143 */
-  C, C, C, C,  C, C, C, C,  C, C, C, C,  C, C, C, C, /* 144-159 */
+  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0, /* 128-143 */
+  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0, /* 144-159 */
   0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,
   0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,
 
@@ -1531,6 +1536,82 @@ append_uri_pathel (const char *b, const char *e, bool escaped,
   append_null (dest);
 }
 
+static char *
+convert_fname (char *fname)
+{
+  char *converted_fname = fname;
+#if HAVE_ICONV
+  const char *from_encoding = opt.encoding_remote;
+  const char *to_encoding = opt.locale;
+  iconv_t cd;
+  size_t len, done, inlen, outlen;
+  char *s;
+  const char *orig_fname = fname;
+
+  /* Defaults for remote and local encodings.  */
+  if (!from_encoding)
+    from_encoding = "UTF-8";
+  if (!to_encoding)
+    to_encoding = nl_langinfo (CODESET);
+
+  cd = iconv_open (to_encoding, from_encoding);
+  if (cd == (iconv_t)(-1))
+    logprintf (LOG_VERBOSE, _("Conversion from %s to %s isn't supported\n"),
+	       quote (from_encoding), quote (to_encoding));
+  else
+    {
+      inlen = strlen (fname);
+      len = outlen = inlen * 2;
+      converted_fname = s = xmalloc (outlen + 1);
+      done = 0;
+
+      for (;;)
+	{
+	  if (iconv (cd, &fname, &inlen, &s, &outlen) != (size_t)(-1)
+	      && iconv (cd, NULL, NULL, &s, &outlen) != (size_t)(-1))
+	    {
+	      *(converted_fname + len - outlen - done) = '\0';
+	      iconv_close(cd);
+	      DEBUGP (("Converted file name '%s' (%s) -> '%s' (%s)\n",
+		       orig_fname, from_encoding, converted_fname, to_encoding));
+	      xfree (orig_fname);
+	      return converted_fname;
+	    }
+
+	  /* Incomplete or invalid multibyte sequence */
+	  if (errno == EINVAL || errno == EILSEQ)
+	    {
+	      logprintf (LOG_VERBOSE,
+			 _("Incomplete or invalid multibyte sequence encountered\n"));
+	      xfree (converted_fname);
+	      converted_fname = (char *)orig_fname;
+	      break;
+	    }
+	  else if (errno == E2BIG) /* Output buffer full */
+	    {
+	      done = len;
+	      len = outlen = done + inlen * 2;
+	      converted_fname = xrealloc (converted_fname, outlen + 1);
+	      s = converted_fname + done;
+	    }
+	  else /* Weird, we got an unspecified error */
+	    {
+	      logprintf (LOG_VERBOSE, _("Unhandled errno %d\n"), errno);
+	      xfree (converted_fname);
+	      converted_fname = (char *)orig_fname;
+	      break;
+	    }
+	}
+      DEBUGP (("Failed to convert file name '%s' (%s) -> '?' (%s)\n",
+	       orig_fname, from_encoding, to_encoding));
+    }
+
+    iconv_close(cd);
+#endif
+
+  return converted_fname;
+}
+
 /* Append to DEST the directory structure that corresponds the
    directory part of URL's path.  For example, if the URL is
    http://server/dir1/dir2/file, this appends "/dir1/dir2".
@@ -1706,6 +1787,8 @@ url_file_name (const struct url *u, char *replaced_filename)
 
   xfree (temp_fnres.base);
 
+  fname = convert_fname (fname);
+
   /* Check the cases in which the unique extensions are not used:
      1) Clobbering is turned off (-nc).
      2) Retrieval with regetting.
@@ -1787,7 +1870,11 @@ path_simplify (enum url_scheme scheme, char *path)
               for (--t; t > beg && t[-1] != '/'; t--)
                 ;
             }
-          else if (scheme == SCHEME_FTP || scheme == SCHEME_FTPS)
+          else if (scheme == SCHEME_FTP
+#ifdef HAVE_SSL
+              || scheme == SCHEME_FTPS
+#endif
+              )
             {
               /* If we're at the beginning, copy the "../" literally
                  and move the beginning so a later ".." doesn't remove

src/utils.c

@@ -31,14 +31,12 @@ as that of the covered work.  */
 
 #include "wget.h"
 
+#include "sha256.h"
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <time.h>
 #include <unistd.h>
-#ifdef HAVE_MMAP
-# include <sys/mman.h>
-#endif
 #ifdef HAVE_PROCESS_H
 # include <process.h>  /* getpid() */
 #endif
@@ -89,6 +87,18 @@ as that of the covered work.  */
 # define USE_SIGNAL_TIMEOUT
 #endif
 
+/* Some systems (Linux libc5, "NCR MP-RAS 3.0", and others) don't
+   provide MAP_FAILED, a symbolic constant for the value returned by
+   mmap() when it doesn't work.  Usually, this constant should be -1.
+   This only makes sense for files that use mmap() and include
+   sys/mman.h *before* sysdep.h, but doesn't hurt others.  */
+#ifdef HAVE_MMAP
+# include <sys/mman.h>
+# ifndef MAP_FAILED
+#  define MAP_FAILED ((void *) -1)
+# endif
+#endif
+
 #include "utils.h"
 #include "hash.h"
 
@@ -2521,6 +2531,205 @@ wg_hex_to_string (char *str_buffer, const char *hex_buffer, size_t hex_len)
   str_buffer[2 * i] = '\0';
 }
 
+#ifdef HAVE_SSL
+
+/*
+ * Public key pem to der conversion
+ */
+
+static bool
+wg_pubkey_pem_to_der (const char *pem, unsigned char **der, size_t *der_len)
+{
+  char *stripped_pem, *begin_pos, *end_pos;
+  size_t pem_count, stripped_pem_count = 0, pem_len;
+  ssize_t size;
+  unsigned char *base64data;
+
+  *der = NULL;
+  *der_len = 0;
+
+  /* if no pem, exit. */
+  if (!pem)
+    return false;
+
+  begin_pos = strstr (pem, "-----BEGIN PUBLIC KEY-----");
+  if (!begin_pos)
+    return false;
+
+  pem_count = begin_pos - pem;
+  /* Invalid if not at beginning AND not directly following \n */
+  if (0 != pem_count && '\n' != pem[pem_count - 1])
+    return false;
+
+  /* 26 is length of "-----BEGIN PUBLIC KEY-----" */
+  pem_count += 26;
+
+  /* Invalid if not directly following \n */
+  end_pos = strstr (pem + pem_count, "\n-----END PUBLIC KEY-----");
+  if (!end_pos)
+    return false;
+
+  pem_len = end_pos - pem;
+
+  stripped_pem = xmalloc (pem_len - pem_count + 1);
+
+  /*
+   * Here we loop through the pem array one character at a time between the
+   * correct indices, and place each character that is not '\n' or '\r'
+   * into the stripped_pem array, which should represent the raw base64 string
+   */
+  while (pem_count < pem_len) {
+    if ('\n' != pem[pem_count] && '\r' != pem[pem_count])
+      stripped_pem[stripped_pem_count++] = pem[pem_count];
+    ++pem_count;
+  }
+  /* Place the null terminator in the correct place */
+  stripped_pem[stripped_pem_count] = '\0';
+
+  base64data = xmalloc (BASE64_LENGTH(stripped_pem_count));
+
+  size = base64_decode (stripped_pem, base64data);
+
+  if (size < 0) {
+    xfree (base64data);           /* malformed base64 from server */
+  } else {
+    *der = base64data;
+    *der_len = (size_t) size;
+  }
+
+  xfree (stripped_pem);
+
+  return *der_len > 0;
+}
+
+/*
+ * Generic pinned public key check.
+ */
+
+bool
+wg_pin_peer_pubkey (const char *pinnedpubkey, const char *pubkey, size_t pubkeylen)
+{
+  struct file_memory *fm;
+  unsigned char *buf = NULL, *pem_ptr = NULL;
+  size_t size, pem_len;
+  bool pem_read;
+  bool result = false;
+
+  size_t pinkeylen;
+  ssize_t decoded_hash_length;
+  char *pinkeycopy, *begin_pos, *end_pos;
+  unsigned char *sha256sumdigest = NULL, *expectedsha256sumdigest = NULL;
+
+  /* if a path wasn't specified, don't pin */
+  if (!pinnedpubkey)
+    return true;
+  if (!pubkey || !pubkeylen)
+    return result;
+
+  /* only do this if pinnedpubkey starts with "sha256//", length 8 */
+  if (strncmp (pinnedpubkey, "sha256//", 8) == 0) {
+    /* compute sha256sum of public key */
+    sha256sumdigest = xmalloc (SHA256_DIGEST_SIZE);
+    sha256_buffer (pubkey, pubkeylen, sha256sumdigest);
+    expectedsha256sumdigest = xmalloc (SHA256_DIGEST_SIZE + 1);
+
+    /* it starts with sha256//, copy so we can modify it */
+    pinkeylen = strlen (pinnedpubkey) + 1;
+    pinkeycopy = xmalloc (pinkeylen);
+    memcpy (pinkeycopy, pinnedpubkey, pinkeylen);
+
+    /* point begin_pos to the copy, and start extracting keys */
+    begin_pos = pinkeycopy;
+    do
+      {
+        end_pos = strstr (begin_pos, ";sha256//");
+        /*
+         * if there is an end_pos, null terminate,
+         * otherwise it'll go to the end of the original string
+         */
+        if (end_pos)
+          end_pos[0] = '\0';
+
+        /* decode base64 pinnedpubkey, 8 is length of "sha256//" */
+        decoded_hash_length = base64_decode (begin_pos + 8, expectedsha256sumdigest);
+        /* if valid base64, compare sha256 digests directly */
+        if (SHA256_DIGEST_SIZE == decoded_hash_length &&
+           !memcmp (sha256sumdigest, expectedsha256sumdigest, SHA256_DIGEST_SIZE)) {
+          result = true;
+          break;
+        }
+
+        /*
+         * change back the null-terminator we changed earlier,
+         * and look for next begin
+         */
+        if (end_pos) {
+          end_pos[0] = ';';
+          begin_pos = strstr (end_pos, "sha256//");
+        }
+      } while (end_pos && begin_pos);
+
+    xfree (sha256sumdigest);
+    xfree (expectedsha256sumdigest);
+    xfree (pinkeycopy);
+
+    return result;
+  }
+
+  /* fall back to assuming this is a file path */
+  fm = wget_read_file (pinnedpubkey);
+  if (!fm)
+    return result;
+
+  /* Check the file's size */
+  if (fm->length < 0 || fm->length > MAX_PINNED_PUBKEY_SIZE)
+    goto cleanup;
+
+  /*
+   * if the size of our certificate is bigger than the file
+   * size then it can't match
+   */
+  size = (size_t) fm->length;
+  if (pubkeylen > size)
+    goto cleanup;
+
+  /* If the sizes are the same, it can't be base64 encoded, must be der */
+  if (pubkeylen == size) {
+    if (!memcmp (pubkey, fm->content, pubkeylen))
+      result = true;
+    goto cleanup;
+  }
+
+  /*
+   * Otherwise we will assume it's PEM and try to decode it
+   * after placing null terminator
+   */
+  buf = xmalloc (size + 1);
+  memcpy (buf, fm->content, size);
+  buf[size] = '\0';
+
+  pem_read = wg_pubkey_pem_to_der ((const char *) buf, &pem_ptr, &pem_len);
+  /* if it wasn't read successfully, exit */
+  if (!pem_read)
+    goto cleanup;
+
+  /*
+   * if the size of our certificate doesn't match the size of
+   * the decoded file, they can't be the same, otherwise compare
+   */
+  if (pubkeylen == pem_len && !memcmp (pubkey, pem_ptr, pubkeylen))
+    result = true;
+
+ cleanup:
+  xfree (buf);
+  xfree (pem_ptr);
+  wget_read_file_free (fm);
+
+  return result;
+}
+
+#endif /* HAVE_SSL */
+
 #ifdef TESTING
 
 const char *

src/utils.h

@@ -37,6 +37,10 @@ as that of the covered work.  */
 /* Constant is using when we don`t know attempted size exactly */
 #define UNKNOWN_ATTEMPTED_SIZE -3
 
+#ifndef MAX_PINNED_PUBKEY_SIZE
+#define MAX_PINNED_PUBKEY_SIZE 1048576 /* 1MB */
+#endif
+
 /* Macros that interface to malloc, but know about type sizes, and
    cast the result to the appropriate type.  The casts are not
    necessary in standard C, but Wget performs them anyway for the sake
@@ -161,4 +165,9 @@ void wg_hex_to_string (char *str_buffer, const char *hex_buffer, size_t hex_len)
 
 extern unsigned char char_prop[];
 
+#ifdef HAVE_SSL
+/* Check pinned public key. */
+bool wg_pin_peer_pubkey (const char *pinnedpubkey, const char *pubkey, size_t pubkeylen);
+#endif
+
 #endif /* UTILS_H */

src/warc.c

@@ -246,6 +246,22 @@ warc_write_header (const char *name, const char *value)
   return warc_write_ok;
 }
 
+/* Writes a WARC header with a URI as value to the current WARC record.
+   This method may be run after warc_write_start_record and
+   before warc_write_block_from_file.  */
+static bool
+warc_write_header_uri (const char *name, const char *value)
+{
+  if (value)
+    {
+      warc_write_string (name);
+      warc_write_string (": <");
+      warc_write_string (value);
+      warc_write_string (">\r\n");
+    }
+  return warc_write_ok;
+}
+
 /* Copies the contents of DATA_IN to the WARC record.
    Adds a Content-Length header to the WARC record.
    Run this method after warc_write_header,
@@ -1292,7 +1308,7 @@ warc_write_request_record (const char *url, const char *timestamp_str,
 {
   warc_write_start_record ();
   warc_write_header ("WARC-Type", "request");
-  warc_write_header ("WARC-Target-URI", url);
+  warc_write_header_uri ("WARC-Target-URI", url);
   warc_write_header ("Content-Type", "application/http;msgtype=request");
   warc_write_date_header (timestamp_str);
   warc_write_header ("WARC-Record-ID", record_uuid);

src/wget.h

@@ -48,8 +48,10 @@ as that of the covered work.  */
 
 /* Disable assertions when debug support is not compiled in. */
 #ifndef ENABLE_DEBUG
+#ifndef NDEBUG
 # define NDEBUG
 #endif
+#endif
 
 /* Is OpenSSL or GNUTLS available? */
 #if defined HAVE_LIBSSL || defined HAVE_LIBSSL32 || defined HAVE_LIBGNUTLS

testenv/Makefile.am

@@ -37,7 +37,10 @@ AUTOMAKE_OPTIONS = parallel-tests
 AM_TESTS_ENVIRONMENT = export WGETRC=/dev/null; MAKE_CHECK=True; export MAKE_CHECK;\
  export PYTHONPATH=$$PYTHONPATH:$(srcdir); export VALGRIND_TESTS="@VALGRIND_TESTS@";
 
-SSL_TESTS = Test--https.py Test--https-crl.py
+if WITH_SSL
+  AM_TESTS_ENVIRONMENT += export SSL_TESTS=1;
+endif
+
 if HAVE_PYTHON3
   TESTS = Test-auth-basic-fail.py                   \
     Test-auth-basic.py                              \
@@ -58,6 +61,13 @@ if HAVE_PYTHON3
     Test-Head.py                                    \
     Test--https.py                                  \
     Test--https-crl.py                              \
+    Test-missing-scheme-retval.py                   \
+    Test-pinnedpubkey-der-https.py                  \
+    Test-pinnedpubkey-der-no-check-https.py         \
+    Test-pinnedpubkey-hash-https.py                 \
+    Test-pinnedpubkey-hash-no-check-fail-https.py   \
+    Test-pinnedpubkey-pem-fail-https.py             \
+    Test-pinnedpubkey-pem-https.py                  \
     Test-hsts.py                                    \
     Test-O.py                                       \
     Test-Post.py                                    \
@@ -67,14 +77,8 @@ if HAVE_PYTHON3
     Test-redirect-crash.py                          \
     Test-reserved-chars.py                          \
     Test-condget.py                                 \
-    $(SSL_TESTS)				    \
     $(METALINK_TESTS)
 
-  # added test cases expected to fail here and under TESTS
-if !WITH_SSL
-  XFAIL_TESTS = $(SSL_TESTS)
-endif
-
 endif
 
 EXTRA_DIST = certs conf exc misc server test README $(TESTS)

testenv/README

@@ -97,6 +97,7 @@ Environment Variables:
   the test suite will execute all the tests via this command.
   If it is set to "1", valgrind memcheck is enabled with hard coded options.
   This variable is set by ./configure --enable-valgrind-tests.
+* SSL_TESTS: This must be set to run any https tests.
 
 
 File Structure:

testenv/Test--convert-links--content-on-error.py

@@ -6,7 +6,6 @@ from misc.wget_file import WgetFile
 """
     This test ensures that Wget link conversion works also on HTTP error pages.
 """
-TEST_NAME = "Test--convert-links--content-on-error"
 ############# File Definitions ###############################################
 a_x_FileContent = """
 <!DOCTYPE html>
@@ -70,7 +69,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test--https-crl.py

@@ -8,7 +8,9 @@ import os
 """
     This test ensures that Wget can download files from HTTPS Servers
 """
-TEST_NAME = "HTTPS CRL"
+if os.getenv('SSL_TESTS') is None:
+    exit (77)
+
 ############# File Definitions ###############################################
 File1 = "Would you like some Tea?"
 File2 = "With lemon or cream?"
@@ -40,7 +42,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test,

testenv/Test--https.py

@@ -8,7 +8,9 @@ import os
 """
     This test ensures that Wget can download files from HTTPS Servers
 """
-TEST_NAME = "HTTPS Downloads"
+if os.getenv('SSL_TESTS') is None:
+    exit (77)
+
 ############# File Definitions ###############################################
 File1 = "Would you like some Tea?"
 File2 = "With lemon or cream?"
@@ -45,7 +47,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test,

testenv/Test--rejected-log.py

@@ -6,7 +6,6 @@ from misc.wget_file import WgetFile
 """
     This test executed Wget in recursive mode with a rejected log outputted.
 """
-TEST_NAME = "Rejected Log"
 ############# File Definitions ###############################################
 mainpage = """
 <html>
@@ -55,10 +54,10 @@ Disallow: /dummy.txt
 """
 
 log = """\
-REASON	U_URL	U_SCHEME	U_HOST	U_PORT	U_PATH	U_PARAMS	U_QUERY	U_FRAGMENT	P_URL	P_SCHEME	P_HOST	P_PORT	P_PATH	P_PARAMS	P_QUERY	P_FRAGMENT
-BLACKLIST	http%3A//127.0.0.1%3A{{port}}/index.html	SCHEME_HTTP	127.0.0.1	{{port}}	index.html				http%3A//127.0.0.1%3A{{port}}/secondpage.html	SCHEME_HTTP	127.0.0.1	{{port}}	secondpage.html
-ROBOTS	http%3A//127.0.0.1%3A{{port}}/dummy.txt	SCHEME_HTTP	127.0.0.1	{{port}}	dummy.txt				http%3A//127.0.0.1%3A{{port}}/thirdpage.html	SCHEME_HTTP	127.0.0.1	{{port}}	thirdpage.html
-SPANNEDHOST	http%3A//no.such.domain/	SCHEME_HTTP	no.such.domain	80					http%3A//127.0.0.1%3A{{port}}/thirdpage.html	SCHEME_HTTP	127.0.0.1	{{port}}	thirdpage.html
+REASON\tU_URL\tU_SCHEME\tU_HOST\tU_PORT\tU_PATH\tU_PARAMS\tU_QUERY\tU_FRAGMENT\tP_URL\tP_SCHEME\tP_HOST\tP_PORT\tP_PATH\tP_PARAMS\tP_QUERY\tP_FRAGMENT
+BLACKLIST\thttp%3A//127.0.0.1%3A{{port}}/index.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tindex.html\t\t\t\thttp%3A//127.0.0.1%3A{{port}}/secondpage.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tsecondpage.html\t\t\t
+ROBOTS\thttp%3A//127.0.0.1%3A{{port}}/dummy.txt\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tdummy.txt\t\t\t\thttp%3A//127.0.0.1%3A{{port}}/thirdpage.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tthirdpage.html\t\t\t
+SPANNEDHOST\thttp%3A//no.such.domain/\tSCHEME_HTTP\tno.such.domain\t80\t\t\t\t\thttp%3A//127.0.0.1%3A{{port}}/thirdpage.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tthirdpage.html\t\t\t
 """
 
 dummyfile = "Don't care."
@@ -93,7 +92,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test--spider-r.py

@@ -6,7 +6,6 @@ from misc.wget_file import WgetFile
 """
     This test executed Wget in Spider mode with recursive retrieval.
 """
-TEST_NAME = "Recursive Spider"
 ############# File Definitions ###############################################
 mainpage = """
 <html>
@@ -97,7 +96,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-504.py

@@ -18,7 +18,6 @@ from misc.wget_file import WgetFile
     considered 504 as a general Server Error, it would be a fatal failure and
     Wget would request File1 only once.
 """
-TEST_NAME = "504 Gateway Timeouts"
 ############# File Definitions ###############################################
 File1 = """All happy families are alike;
 Each unhappy family is unhappy in its own way"""
@@ -63,7 +62,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-Content-disposition-2.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     This test ensures that Wget parses the Content-Disposition header
     correctly and creates the appropriate file when the said filename exists.
 """
-TEST_NAME = "Content Disposition Clobber"
 ############# File Definitions ###############################################
 File1 = "Teapot"
 File2 = "The Teapot Protocol"
@@ -45,7 +44,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-Content-disposition.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     This test ensures that Wget parses the Content-Disposition header
     correctly and creates a local file accordingly.
 """
-TEST_NAME = "Content Disposition Header"
 ############# File Definitions ###############################################
 File1 = """All that is gold does not glitter,
         Not all those who wander are lost;
@@ -47,7 +46,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-Head.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     This test ensures that Wget correctly handles responses to HEAD requests
     and does not actually download any data
 """
-TEST_NAME = "HEAD Requests"
 ############# File Definitions ###############################################
 File1 = "You shall not pass!"
 
@@ -35,7 +34,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-O.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     This test ensures that Wget correctly handles the -O command for output
     filenames.
 """
-TEST_NAME = "Output Filename Command"
 ############# File Definitions ###############################################
 File1 = "Test Contents."
 
@@ -36,7 +35,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-Parallel-Proto.py

@@ -8,7 +8,6 @@ from misc.wget_file import WgetFile
     This is a Prototype Test File for multiple servers.
     Ideally this File should be copied and edited to write new tests.
 """
-TEST_NAME = "Parallel Prototype"
 ############# File Definitions ###############################################
 File1 = "Would you like some Tea?"
 File2 = "With lemon or cream?"
@@ -44,7 +43,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test,

testenv/Test-Post.py

@@ -6,7 +6,6 @@ from misc.wget_file import WgetFile
 """
     Simple test for HTTP POST Requests usiong the --method command
 """
-TEST_NAME = "HTTP POST Requests"
 ############# File Definitions ###############################################
 File1 = """A reader lives a thousand lives before he dies, said Jojen.
 The man who never reads lives only one"""
@@ -39,7 +38,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-Proto.py

@@ -8,7 +8,6 @@ from misc.wget_file import WgetFile
     This is a Prototype Test File.
     Ideally this File should be copied and edited to write new tests.
 """
-TEST_NAME = "Prototype"
 ############# File Definitions ###############################################
 File1 = "Would you like some Tea?"
 File2 = "With lemon or cream?"
@@ -65,7 +64,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test,

testenv/Test-auth-basic-fail.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     This test ensures that Wget returns the correct exit code when Basic
     authentcation failes due to a username/password error.
 """
-TEST_NAME = "Basic Authentication Failure"
 ############# File Definitions ###############################################
 File1 = "I am an invisble man."
 
@@ -42,7 +41,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-auth-basic.py

@@ -8,7 +8,6 @@ from misc.wget_file import WgetFile
     Also, we ensure that Wget saves the host after a successfull auth and
     doesn't wait for a challenge the second time.
 """
-TEST_NAME = "Basic Authorization"
 ############# File Definitions ###############################################
 File1 = "I am an invisble man."
 File2 = "I too am an invisible man."
@@ -50,7 +49,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-auth-both.py

@@ -8,7 +8,6 @@ from misc.wget_file import WgetFile
     Also, we ensure that Wget saves the host after a successfull auth and
     doesn't wait for a challenge the second time.
 """
-TEST_NAME = "Multiple authentication support"
 ############# File Definitions ###############################################
 File1 = "Would you like some Tea?"
 File2 = "With lemon or cream?"
@@ -78,7 +77,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-auth-digest.py

@@ -6,7 +6,6 @@ from misc.wget_file import WgetFile
 """
     This test ensures Wget's Digest Authorization Negotiation.
 """
-TEST_NAME = "Digest Authorization"
 ############# File Definitions ###############################################
 File1 = "Need a cookie?"
 File2 = "Want cookies with milk!"
@@ -57,7 +56,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-auth-no-challenge-url.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     This test ensures Wget's Basic Authorization Negotiation, when credentials
     are provided in-URL
 """
-TEST_NAME = "Auth no challenge in URL"
 ############# File Definitions ###############################################
 File1 = "Need a cookie?"
 
@@ -45,7 +44,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-auth-no-challenge.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     This test ensures Wget's Basic Authorization Negotiation, when the
     --auth-no-challenge command is used.
 """
-TEST_NAME = "Auth No Challenge Command"
 ############# File Definitions ###############################################
 File1 = "Need a cookie?"
 
@@ -45,7 +44,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-auth-retcode.py

@@ -8,7 +8,6 @@ from misc.wget_file import WgetFile
     a 403 Forbidden by the Server.
 """
 
-TEST_NAME = "Forbidden Retcode"
 
 ############# File Definitions ###############################################
 File1 = "Apples and Oranges? Really?"
@@ -41,7 +40,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-auth-with-content-disposition.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     This test ensures that Wget handles Content-Disposition correctly when
     coupled with Authentication
 """
-TEST_NAME = "Authentication with Content Disposition"
 ############# File Definitions ###############################################
 File1 = "Need a cookie?"
 
@@ -45,7 +44,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-c-full.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     Test Wget's response when the file requested already exists on disk with
     a filesize greater than or equal to the requested file.
 """
-TEST_NAME = "Test continue option"
 ############# File Definitions ###############################################
 File1 = "abababababababababababababababababababababababababababababababababab"
 File2 = "ababababababababababababababababababab"
@@ -44,7 +43,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-condget.py

@@ -6,7 +6,6 @@ from misc.wget_file import WgetFile
 """
     Simple test for HTTP Conditional-GET Requests using the -N command
 """
-TEST_NAME = "HTTP Conditional-GET Requests"
 ############# File Definitions ###############################################
 # Keep same length !
 Cont1 = """THIS IS 1 FILE"""
@@ -131,7 +130,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-cookie-401.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     This test ensures that Wget stores the cookie even in the event of a
     401 Unauthorized Response
 """
-TEST_NAME = "Basic Cookie 401 Response"
 ############# File Definitions ###############################################
 File1 = """All happy families are alike;
 Each unhappy family is unhappy in its own way"""
@@ -50,7 +49,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-cookie-domain-mismatch.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     This test ensures that Wget identifies bad servers trying to set cookies
     for a different domain and rejects them.
 """
-TEST_NAME = "Cookie Domain Mismatch"
 ############# File Definitions ###############################################
 File1 = "Would you care for a cup of coffee?"
 File2 = "Anyone for chocochip cookies?"
@@ -49,7 +48,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-cookie-expires.py

@@ -8,7 +8,6 @@ from misc.wget_file import WgetFile
     Simultaneuously, we also check if multiple cookies to the same domain
     are handled correctly
 """
-TEST_NAME = "Cookie Expires"
 ############# File Definitions ###############################################
 File1 = "Hello World!"
 File2 = "'Ello! This is Amazing!"
@@ -72,7 +71,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-cookie.py

@@ -6,7 +6,6 @@ from misc.wget_file import WgetFile
 """
     This test ensures that Wget's cookie jar support works correctly.
 """
-TEST_NAME = "Basic Cookie Functionality"
 ############# File Definitions ###############################################
 File1 = """All happy families are alike;
 Each unhappy family is unhappy in its own way"""
@@ -48,7 +47,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-hsts.py

@@ -9,6 +9,11 @@ import os
 """
 This test makes sure Wget can parse a given HSTS database and apply the indicated HSTS policy.
 """
+
+print (os.getenv('SSL_TESTS'))
+if os.getenv('SSL_TESTS') is None:
+    exit (77)
+
 def hsts_database_path():
     hsts_file = ".wget-hsts-testenv"
     return os.path.abspath(hsts_file)
@@ -25,8 +30,6 @@ def create_hsts_database(path, host, port):
     f.write(host + "\t" + str(port) + "\t0\t" + str(curtime) + "\t" + max_age + "\n")
     f.close()
 
-TEST_NAME = "HSTS basic test"
-
 File_Name = "hw"
 File_Content = "Hello, world!"
 File = WgetFile(File_Name, File_Content)
@@ -59,7 +62,6 @@ test_options = {
 }
 
 test = HTTPTest(
-        name = TEST_NAME,
         pre_hook = pre_test,
         post_hook = post_test,
         test_params = test_options,

testenv/Test-metalink-http.py

@@ -9,7 +9,6 @@ from base64 import b64encode
 """
     This is to test Metalink as HTTP file support in Wget.
 """
-TEST_NAME = "Metalink in HTTP"
 
 # Helper function for hostname, port and digest substitution
 def SubstituteServerInfo (text, host, port, digest):
@@ -109,7 +108,6 @@ post_test = {
 }
 
 http_test = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test,

testenv/Test-metalink-xml.py

@@ -8,7 +8,6 @@ import hashlib
 """
     This is to test Metalink as XML file support in Wget.
 """
-TEST_NAME = "Metalink in XML"
 ############# File Definitions ###############################################
 File1 = "Would you like some Tea?"
 File1_lowPref = "Do not take this"
@@ -69,7 +68,6 @@ post_test = {
 }
 
 http_test = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test,

testenv/Test-missing-scheme-retval.py

@@ -0,0 +1,42 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from test.base_test import HTTP
+from misc.wget_file import WgetFile
+import os
+
+"""
+    This test ensures that Wget complains about missing scheme
+"""
+############# File Definitions ###############################################
+A_File = WgetFile ("bar", 'Content')
+
+# put the URL into 'options' to avoid prepending scheme/localhost/port
+WGET_OPTIONS = "/foo/bar"
+WGET_URLS = [[]]
+
+Files = [[A_File]]
+
+ExpectedReturnCode = 1
+ExpectedDownloadedFiles = []
+
+################ Pre and Post Test Hooks #####################################
+pre_test = {
+    "ServerFiles"       : Files,
+}
+test_options = {
+    "WgetCommands"      : WGET_OPTIONS,
+    "Urls"              : WGET_URLS
+}
+post_test = {
+    "ExpectedFiles"     : ExpectedDownloadedFiles,
+    "ExpectedRetcode"   : ExpectedReturnCode
+}
+
+err = HTTPTest (
+                pre_hook=pre_test,
+                test_params=test_options,
+                post_hook=post_test,
+).begin ()
+
+exit (err)

testenv/Test-pinnedpubkey-der-https.py

@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from test.base_test import HTTP, HTTPS
+from misc.wget_file import WgetFile
+import os
+
+"""
+    This test ensures that Wget can download files from HTTPS Servers
+"""
+if os.getenv('SSL_TESTS') is None:
+    exit (77)
+
+############# File Definitions ###############################################
+File1 = "Would you like some Tea?"
+File2 = "With lemon or cream?"
+File3 = "Sure you're joking Mr. Feynman"
+
+A_File = WgetFile ("File1", File1)
+B_File = WgetFile ("File2", File2)
+C_File = WgetFile ("File3", File3)
+
+CAFILE = os.path.abspath(os.path.join(os.getenv('srcdir', '.'), 'certs', 'ca-cert.pem'))
+PINNEDPUBKEY = os.path.abspath(os.path.join(os.getenv('srcdir', '.'), 'certs', 'server-pubkey.der'))
+WGET_OPTIONS = "--pinnedpubkey=" + PINNEDPUBKEY + " --ca-certificate=" + CAFILE
+WGET_URLS = [["File1", "File2"]]
+
+Files = [[A_File, B_File]]
+Existing_Files = [C_File]
+
+Servers = [HTTPS]
+
+ExpectedReturnCode = 0
+ExpectedDownloadedFiles = [A_File, B_File, C_File]
+
+################ Pre and Post Test Hooks #####################################
+pre_test = {
+    "ServerFiles"       : Files,
+    "LocalFiles"        : Existing_Files
+}
+test_options = {
+    "WgetCommands"      : WGET_OPTIONS,
+    "Urls"              : WGET_URLS
+}
+post_test = {
+    "ExpectedFiles"     : ExpectedDownloadedFiles,
+    "ExpectedRetcode"   : ExpectedReturnCode
+}
+
+err = HTTPTest (
+                pre_hook=pre_test,
+                test_params=test_options,
+                post_hook=post_test,
+                protocols=Servers
+).begin ()
+
+exit (err)

testenv/Test-pinnedpubkey-der-no-check-https.py

@@ -0,0 +1,56 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from test.base_test import HTTP, HTTPS
+from misc.wget_file import WgetFile
+import os
+
+"""
+    This test ensures that Wget can download files from HTTPS Servers
+"""
+if os.getenv('SSL_TESTS') is None:
+    exit (77)
+
+############# File Definitions ###############################################
+File1 = "Would you like some Tea?"
+File2 = "With lemon or cream?"
+File3 = "Sure you're joking Mr. Feynman"
+
+A_File = WgetFile ("File1", File1)
+B_File = WgetFile ("File2", File2)
+C_File = WgetFile ("File3", File3)
+
+PINNEDPUBKEY = os.path.abspath(os.path.join(os.getenv('srcdir', '.'), 'certs', 'server-pubkey.der'))
+WGET_OPTIONS = "--no-check-certificate --pinnedpubkey=" + PINNEDPUBKEY
+WGET_URLS = [["File1", "File2"]]
+
+Files = [[A_File, B_File]]
+Existing_Files = [C_File]
+
+Servers = [HTTPS]
+
+ExpectedReturnCode = 0
+ExpectedDownloadedFiles = [A_File, B_File, C_File]
+
+################ Pre and Post Test Hooks #####################################
+pre_test = {
+    "ServerFiles"       : Files,
+    "LocalFiles"        : Existing_Files
+}
+test_options = {
+    "WgetCommands"      : WGET_OPTIONS,
+    "Urls"              : WGET_URLS
+}
+post_test = {
+    "ExpectedFiles"     : ExpectedDownloadedFiles,
+    "ExpectedRetcode"   : ExpectedReturnCode
+}
+
+err = HTTPTest (
+                pre_hook=pre_test,
+                test_params=test_options,
+                post_hook=post_test,
+                protocols=Servers
+).begin ()
+
+exit (err)

testenv/Test-pinnedpubkey-hash-https.py

@@ -0,0 +1,56 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from test.base_test import HTTP, HTTPS
+from misc.wget_file import WgetFile
+import os
+
+"""
+    This test ensures that Wget can download files from HTTPS Servers
+"""
+if os.getenv('SSL_TESTS') is None:
+    exit (77)
+
+############# File Definitions ###############################################
+File1 = "Would you like some Tea?"
+File2 = "With lemon or cream?"
+File3 = "Sure you're joking Mr. Feynman"
+
+A_File = WgetFile ("File1", File1)
+B_File = WgetFile ("File2", File2)
+C_File = WgetFile ("File3", File3)
+
+CAFILE = os.path.abspath(os.path.join(os.getenv('srcdir', '.'), 'certs', 'ca-cert.pem'))
+WGET_OPTIONS = "--pinnedpubkey=sha256//mHiEhWHvusnzP7COZk+SzSJ+Gl7nZT+ADx0PUnDD7mM= --ca-certificate=" + CAFILE
+WGET_URLS = [["File1", "File2"]]
+
+Files = [[A_File, B_File]]
+Existing_Files = [C_File]
+
+Servers = [HTTPS]
+
+ExpectedReturnCode = 0
+ExpectedDownloadedFiles = [A_File, B_File, C_File]
+
+################ Pre and Post Test Hooks #####################################
+pre_test = {
+    "ServerFiles"       : Files,
+    "LocalFiles"        : Existing_Files
+}
+test_options = {
+    "WgetCommands"      : WGET_OPTIONS,
+    "Urls"              : WGET_URLS
+}
+post_test = {
+    "ExpectedFiles"     : ExpectedDownloadedFiles,
+    "ExpectedRetcode"   : ExpectedReturnCode
+}
+
+err = HTTPTest (
+                pre_hook=pre_test,
+                test_params=test_options,
+                post_hook=post_test,
+                protocols=Servers
+).begin ()
+
+exit (err)

testenv/Test-pinnedpubkey-hash-no-check-fail-https.py

@@ -0,0 +1,51 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from test.base_test import HTTP, HTTPS
+from misc.wget_file import WgetFile
+import os
+
+"""
+    This test ensures that Wget can download files from HTTPS Servers
+"""
+if os.getenv('SSL_TESTS') is None:
+    exit (77)
+
+############# File Definitions ###############################################
+File1 = "Would you like some Tea?"
+File2 = "With lemon or cream?"
+
+A_File = WgetFile ("File1", File1)
+B_File = WgetFile ("File2", File2)
+
+WGET_OPTIONS = "--no-check-certificate --pinnedpubkey=sha256//mHiEhWHvusnzP7COZk+SzSJ+Gl7ZZT+ADx0PUnDD7mM="
+WGET_URLS = [["File1", "File2"]]
+
+Files = [[A_File, B_File]]
+
+Servers = [HTTPS]
+
+ExpectedReturnCode = 5
+ExpectedDownloadedFiles = []
+
+################ Pre and Post Test Hooks #####################################
+pre_test = {
+    "ServerFiles"       : Files
+}
+test_options = {
+    "WgetCommands"      : WGET_OPTIONS,
+    "Urls"              : WGET_URLS
+}
+post_test = {
+    "ExpectedFiles"     : ExpectedDownloadedFiles,
+    "ExpectedRetcode"   : ExpectedReturnCode
+}
+
+err = HTTPTest (
+                pre_hook=pre_test,
+                test_params=test_options,
+                post_hook=post_test,
+                protocols=Servers
+).begin ()
+
+exit (err)

testenv/Test-pinnedpubkey-pem-fail-https.py

@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from test.base_test import HTTP, HTTPS
+from misc.wget_file import WgetFile
+import os
+
+"""
+    This test ensures that Wget can download files from HTTPS Servers
+"""
+if os.getenv('SSL_TESTS') is None:
+    exit (77)
+
+############# File Definitions ###############################################
+File1 = "Would you like some Tea?"
+File2 = "With lemon or cream?"
+
+A_File = WgetFile ("File1", File1)
+B_File = WgetFile ("File2", File2)
+
+CAFILE = os.path.abspath(os.path.join(os.getenv('srcdir', '.'), 'certs', 'ca-cert.pem'))
+PINNEDPUBKEY = os.path.abspath(os.path.join(os.getenv('srcdir', '.'), 'certs', 'ca-key.pem'))
+WGET_OPTIONS = "--pinnedpubkey=" + PINNEDPUBKEY + " --ca-certificate=" + CAFILE
+WGET_URLS = [["File1", "File2"]]
+
+Files = [[A_File, B_File]]
+
+Servers = [HTTPS]
+
+ExpectedReturnCode = 5
+ExpectedDownloadedFiles = []
+
+################ Pre and Post Test Hooks #####################################
+pre_test = {
+    "ServerFiles"       : Files
+}
+test_options = {
+    "WgetCommands"      : WGET_OPTIONS,
+    "Urls"              : WGET_URLS
+}
+post_test = {
+    "ExpectedFiles"     : ExpectedDownloadedFiles,
+    "ExpectedRetcode"   : ExpectedReturnCode
+}
+
+err = HTTPTest (
+                pre_hook=pre_test,
+                test_params=test_options,
+                post_hook=post_test,
+                protocols=Servers
+).begin ()
+
+exit (err)

testenv/Test-pinnedpubkey-pem-https.py

@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from test.base_test import HTTP, HTTPS
+from misc.wget_file import WgetFile
+import os
+
+"""
+    This test ensures that Wget can download files from HTTPS Servers
+"""
+if os.getenv('SSL_TESTS') is None:
+    exit (77)
+
+############# File Definitions ###############################################
+File1 = "Would you like some Tea?"
+File2 = "With lemon or cream?"
+File3 = "Sure you're joking Mr. Feynman"
+
+A_File = WgetFile ("File1", File1)
+B_File = WgetFile ("File2", File2)
+C_File = WgetFile ("File3", File3)
+
+CAFILE = os.path.abspath(os.path.join(os.getenv('srcdir', '.'), 'certs', 'ca-cert.pem'))
+PINNEDPUBKEY = os.path.abspath(os.path.join(os.getenv('srcdir', '.'), 'certs', 'server-pubkey.pem'))
+WGET_OPTIONS = "--pinnedpubkey=" + PINNEDPUBKEY + " --ca-certificate=" + CAFILE
+WGET_URLS = [["File1", "File2"]]
+
+Files = [[A_File, B_File]]
+Existing_Files = [C_File]
+
+Servers = [HTTPS]
+
+ExpectedReturnCode = 0
+ExpectedDownloadedFiles = [A_File, B_File, C_File]
+
+################ Pre and Post Test Hooks #####################################
+pre_test = {
+    "ServerFiles"       : Files,
+    "LocalFiles"        : Existing_Files
+}
+test_options = {
+    "WgetCommands"      : WGET_OPTIONS,
+    "Urls"              : WGET_URLS
+}
+post_test = {
+    "ExpectedFiles"     : ExpectedDownloadedFiles,
+    "ExpectedRetcode"   : ExpectedReturnCode
+}
+
+err = HTTPTest (
+                pre_hook=pre_test,
+                test_params=test_options,
+                post_hook=post_test,
+                protocols=Servers
+).begin ()
+
+exit (err)

testenv/Test-redirect-crash.py

@@ -20,7 +20,6 @@ redirected = [
         "File%20formats/Images/SVG,%20Scalable%20Vector%20Graphics/html,%20W3C%20v1.2%20rec%20(tiny)/directory/"
 ]
 
-TEST_NAME = "Redirection crash"
 ############# File Definitions ###############################################
 Index = ""
 for i in urls:
@@ -65,7 +64,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-reserved-chars.py

@@ -11,7 +11,6 @@ This test ensures that Wget keeps reserved characters in URLs in non-UTF-8 chars
 # so we need to set LC_ALL="C" in order to reproduce it.
 environ["LC_ALL"] = "C"
 
-TEST_NAME = "URLs with reserved characters"
 ######### File Definitions #########
 RequestList = [
     [
@@ -50,7 +49,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/certs/README

@@ -75,3 +75,13 @@ Generating a signed CRL...
 Update times.
 The certificate will expire in (days): -1
 CRL Number (default: 6080006793650397145):
+
+To generate a public key in PEM format:
+$ openssl x509 -noout -pubkey < server-cert.pem > server-pubkey.pem
+
+To generate a public key in DER format:
+$ openssl x509 -noout -pubkey < server-cert.pem | openssl asn1parse -noout -inform pem -out server-pubkey.der
+
+To generate a sha256 hash of the public key:
+$ openssl x509 -noout -pubkey < server-cert.pem | openssl asn1parse -noout -inform pem -out /dev/stdout | openssl dgst -sha256 -binary | openssl base64
+mHiEhWHvusnzP7COZk+SzSJ+Gl7nZT+ADx0PUnDD7mM=

testenv/certs/server-pubkey.pem

@@ -0,0 +1,9 @@
+-----BEGIN PUBLIC KEY-----
+MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyMLca3nkR9K2XqYTfvX6
+kPf9ylHkwvGR1sGyzkyUg/ZMOGI84i0teaXyjGzgGNSbfB+fcZX2IkuZvNshYv7S
+RtGRDYsI8pR/4KWffPZkT6tfB1aVPyBV+/nU6l+SnaUsNVSot80pEZCCK+NIKYup
+jYup4HRJpU2+5oPcSmpnIgfQTlJmCOoEeBFG28aRzLSs6anlIjY0BIu6BSKhdr04
+taOlgPCh2x3cRGUvQMnVolbxMLxOqLHiLSixbNqv4tcEiKfRC9qv3+5Ec3SnWSre
+nReA0cqpamJNPnj5ZjHs96a/ipFfPXWzCInNQv4/DUO6tD2yZvMOACzPtXYUmdR4
+JwIDAQAB
+-----END PUBLIC KEY-----

testenv/server/http/http_server.py

@@ -369,7 +369,7 @@ class _Handler(BaseHTTPRequestHandler):
         rej_headers = header_obj.headers
         for header_line in rej_headers:
             header_recd = self.headers.get(header_line)
-            if not header_recd and header_recd == rej_headers[header_line]:
+            if header_recd and header_recd == rej_headers[header_line]:
                 self.send_error(400, 'Blacklisted Header %s received' %
                                 header_line)
                 self.finish_headers()

testenv/test/base_test.py

@@ -4,6 +4,7 @@ import shlex
 import traceback
 import re
 import time
+import sys
 from subprocess import call
 from misc.colour_terminal import print_red, print_blue
 from exc.test_failed import TestFailed
@@ -22,12 +23,12 @@ class BaseTest:
         * instantiate_server_by(protocol)
     """
 
-    def __init__(self, name, pre_hook, test_params, post_hook, protocols, req_protocols):
+    def __init__(self, pre_hook, test_params, post_hook, protocols, req_protocols):
         """
         Define the class-wide variables (or attributes).
         Attributes should not be defined outside __init__.
         """
-        self.name = name
+        self.name = os.path.basename(os.path.realpath(sys.argv[0]))
         # if pre_hook == None, then {} (an empty dict object) is passed to
         # self.pre_configs
         self.pre_configs = pre_hook or {}
@@ -162,6 +163,7 @@ class BaseTest:
                 shutil.rmtree(self.get_test_dir())
         except:
             print("Unknown Exception while trying to remove Test Environment.")
+            self.tests_passed = False
 
     def _exit_test(self):
         self.__test_cleanup()
@@ -263,4 +265,4 @@ class BaseTest:
                 traceback.print_tb(exc_tb)
         self.__test_cleanup()
 
-        return True
+        return self.tests_passed

testenv/test/http_test.py

@@ -13,14 +13,12 @@ class HTTPTest(BaseTest):
     # immediately after the call to Wget returns.
 
     def __init__(self,
-                 name="Unnamed Test",
                  pre_hook=None,
                  test_params=None,
                  post_hook=None,
                  protocols=(HTTP,),
                  req_protocols=None):
-        super(HTTPTest, self).__init__(name,
-                                       pre_hook,
+        super(HTTPTest, self).__init__(pre_hook,
                                        test_params,
                                        post_hook,
                                        protocols,

tests/Test-ftp-iri.px

@@ -26,12 +26,12 @@ my %urls = (
     },
 );
 
-my $cmdline = $WgetTest::WGETPATH . " --local-encoding=iso-8859-1 -S ftp://localhost:{{port}}/fran${ccedilla_l1}ais.txt";
+my $cmdline = $WgetTest::WGETPATH . " --local-encoding=iso-8859-1 --remote-encoding=utf-8 -S ftp://localhost:{{port}}/fran${ccedilla_l1}ais.txt";
 
 my $expected_error_code = 0;
 
 my %expected_downloaded_files = (
-    "fran${ccedilla_u8}ais.txt" => {
+    "fran${ccedilla_l1}ais.txt" => {
         content => $francais,
     },
 );

tests/Test-iri-forced-remote.px

@@ -27,8 +27,6 @@ my $eacute_l1 = "\xE9";
 my $eacute_u8 = "\xC3\xA9";
 my $eurosign_l15 = "\xA4";
 my $eurosign_u8 = "\xE2\x82\xAC";
-my $currency_l1 = "\xA4";
-my $currency_u8 = "\xC2\xA4";
 
 my $pageindex = <<EOF;
 <html>
@@ -120,8 +118,8 @@ my %urls = (
         code => "200",
         msg => "Ok",
         headers => {
-            # wrong charset here, overridden by --remote-encoding=iso-8859-1
-            "Content-type" => "text/html; charset=UTF-8",
+            # wrong charset specified by meta tag in $pagefrancais, overridden by HTTP Content-Type
+            "Content-type" => "text/html; charset=iso-8859-1",
         },
         content => $pagefrancais,
     },
@@ -151,7 +149,7 @@ my %urls = (
     },
 );
 
-my $cmdline = $WgetTest::WGETPATH . " --iri -e robots=on --trust-server-names --remote-encoding=iso-8859-1 -nH -r http://localhost:{{port}}/";
+my $cmdline = $WgetTest::WGETPATH . " --iri -e robots=on --trust-server-names --local-encoding=utf-8 -nH -r http://localhost:{{port}}/";
 
 my $expected_error_code = 0;
 
@@ -168,7 +166,7 @@ my %expected_downloaded_files = (
     "p2_${eacute_u8}${eacute_u8}n.html" => {
         content => $pageeen,
     },
-    "p3_${currency_u8}${currency_u8}${currency_u8}.html" => {
+    "p3_${eurosign_u8}${eurosign_u8}${eurosign_u8}.html" => {
         content => $pageeuro,
     },
 );

tests/Test-k.px

@@ -7,6 +7,8 @@ use HTTPTest;
 
 
 ###############################################################################
+my $osname = $^O;
+print "OS=$osname\n";
 
 my $index = <<EOF;
 <html>
@@ -61,7 +63,12 @@ my %urls = (
     },
 );
 
-my $cmdline = $WgetTest::WGETPATH . " -k -r -nH http://localhost:{{port}}/index.html";
+my $restrict = "unix";
+if ($osname eq "MSWin32") {
+  $restrict = "windows";
+}
+
+my $cmdline = $WgetTest::WGETPATH . " -k -r -nH --restrict-file-names=$restrict http://localhost:{{port}}/index.html";
 
 my $expected_error_code = 0;
 

tests/valgrind-suppressions

@@ -6,3 +6,175 @@
    fun:idn_encode
    fun:url_parse
 }
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:modify_param_name
+   fun:extract_param
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:modify_param_name
+   fun:extract_param
+   fun:parse_set_cookie
+   fun:cookie_handle_set_cookie
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:modify_param_name
+   fun:extract_param
+   fun:parse_set_cookie
+   fun:cookie_handle_set_cookie
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   <insert_a_suppression_name_here>
+   Memcheck:Cond
+   fun:memrchr
+   fun:extract_param
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   <insert_a_suppression_name_here>
+   Memcheck:Cond
+   fun:memrchr
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   <insert_a_suppression_name_here>
+   Memcheck:Cond
+   fun:memrchr
+   fun:extract_param
+   fun:parse_set_cookie.constprop.4
+   fun:cookie_handle_set_cookie
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}

tests/valgrind-suppressions-ssl

@@ -26,3 +26,141 @@
    ...
    obj:*/libcrypto.so.*
 }
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:modify_param_name
+   fun:extract_param
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   ...
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:modify_param_name
+   fun:extract_param
+   fun:parse_set_cookie
+   fun:cookie_handle_set_cookie
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   ...
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:modify_param_name
+   fun:extract_param
+   fun:parse_set_cookie
+   fun:cookie_handle_set_cookie
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   ...
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}
+
+{
+   Valgrind 3.4 bug suppression
+   Memcheck:Cond
+   fun:memrchr
+   fun:parse_content_disposition
+   fun:check_file_output
+   fun:gethttp
+   fun:http_loop
+   fun:retrieve_url
+   fun:main
+}