NEWS: update

If not --trust-server-names is used, FTP will also get the destination
file name from the original url specified by the user instead of the
redirected url.  Closes CVE-2016-4971.

* src/ftp.c (ftp_get_listing): Add argument original_url.
(getftp): Likewise.
(ftp_loop_internal): Likewise.  Use original_url to generate the
file name if --trust-server-names is not provided.
(ftp_retrieve_glob): Likewise.
(ftp_loop): Likewise.

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

.gitignore

@@ -2,6 +2,7 @@
 /.sc-start-sc_bindtextdomain
 /.sc-start-sc_prohibit_HAVE_MBRTOWC
 /.version
+/ABOUT-NLS
 /GNUmakefile
 /INSTALL
 /Makefile
@@ -25,6 +26,7 @@ testenv/*.log
 testenv/*.trs
 # build-aux/
 build-aux/.gitignore
+build-aux/ar-lib
 build-aux/compile
 build-aux/config.guess
 build-aux/config.sub

.travis.yml

@@ -0,0 +1,50 @@
+sudo: no
+
+language: c
+
+compiler:
+    - gcc
+
+notifications:
+    email:
+        recipients:
+            - darnir@gmail.com
+            - bug-wget@gnu.org
+        on_success: change
+        on_failure: always
+
+env:
+    global:
+        - PERLLIB=$PERLLIB:../../tests
+        - PERL5LIB=$PERL5LIB:../../tests
+    matrix:
+        - SSL=""
+        - SSL="openssl"
+        - SSL="gnutls"
+
+addons:
+    apt:
+        packages:
+            - automake
+            - autoconf
+            - autopoint
+            - flex
+            - texinfo
+            - pkg-config
+            - libgnutls-dev
+            - libssl-dev
+            - make
+            - libhttp-daemon-perl
+            - libio-socket-ssl-perl
+            - libidn11-dev
+            - gettext
+            - texlive
+            - python3
+            - valgrind
+            - language-pack-tr
+            - language-pack-ru
+
+script:
+    - make distclean || true
+    - ./bootstrap
+    - ./contrib/travis-ci $SSL

MAILING-LIST

@@ -23,15 +23,6 @@ Gmane archives conveniently include messages from both the current
 list, and the previous one. Messages also show up in the Gmane archives
 sooner than they do at `lists.gnu.org'.
 
-Bug Notices List
-----------------
-
-Additionally, there is the <wget-notify@addictivecode.org> mailing
-list. This is a non-discussion list that receives bug report
-notifications from the bug-tracker. To subscribe to this list, send an
-email to <wget-notify-join@addictivecode.org>, or visit
-`http://addictivecode.org/mailman/listinfo/wget-notify'.
-
 Obsolete Lists
 --------------
 

NEWS

@@ -1,12 +1,75 @@
 GNU Wget NEWS -- history of user-visible changes.
 
 Copyright (C) 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
-2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014 Free Software
-Foundation, Inc.
+2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Free
+Software Foundation, Inc.
 See the end for copying conditions.
 
 Please send GNU Wget bug reports to <bug-wget@gnu.org>.
 
+* Changes in Wget 1.18
+
+* By default, on server redirects to a FTP resource, use the original
+  URL to get the local file name. Close CVE-2016-4971.  This
+  introduces a backward-incompatibility for HTTP->FTP redirects and
+  any script that relies on the old  behaviour must use
+  --trust-server-names.
+
+* Check the HSTS file is not world-writable before using it.
+
+* Parse <img srcset> attributes on a recursive download.
+
+* Fix problem with SNI server names having trailing dot(s)
+
+* New options --bind-dns-address and --dns-servers.
+
+* When Wget is built with libiconv, it now converts non-ASCII URIs to
+  the locale's codeset when it creates files.  The encoding of the
+  remote files and URIs is taken from --remote-encoding, defaulting to
+  UTF-8.  The result is that non-ASCII URIs and files downloaded via
+  HTTP/HTTPS and FTP will have names on the local filesystem that
+  correspond to their remote names.
+
+* Changes in Wget 1.17.1
+
+* Fix compile error when IPv6 is disabled or SSL is not present.
+
+* Fix HSTS memory leak.
+
+* Fix progress output in non-C locales.
+
+* Fix SIGSEGV when -N and --content-disposition are used together.
+
+* Add --check-certificate=quiet to tell wget to not print any warning about
+  invalid certificates.
+
+* Changes in Wget 1.17
+
+** Remove FTP passive to active fallback due to privacy concerns.
+
+** Add support for --if-modified-since.
+
+** Add support for metalink through --input-metalink and --metalink-over-http.
+
+** Add support for HSTS through --hsts and --hsts-file.
+
+** Add option to restrict filenames under VMS.
+
+** Add support for --rejected-log which logs to a separate file the reasons why
+   URLs are being rejected and some context around it.
+
+** Add support for FTPS.
+
+** Do not download/save file on error when --spider enabled
+
+** Add --convert-file-only option. This option converts only the
+   filename part of the URLs, leaving the rest of the URLs untouched.
+
+* Changes in Wget 1.16.3
+
+** Fix a regression introduced by wget 1.16.2 that --quiet is not
+   really quiet anymore.
+
 * Changes in Wget 1.16.2
 
 ** Native uuid generation on Windows
@@ -114,7 +177,7 @@ Please send GNU Wget bug reports to <bug-wget@gnu.org>.
 
 ** Add support for TLS Server Name Indication.
 
-** Accept the arguments --accept-reject and --reject-regex.
+** Accept the arguments --accept-regex and --reject-regex.
 
 ** The GNU TLS backend honors correctly the timeout value.
 

README.checkout

@@ -5,7 +5,8 @@ Compiling From Repository Sources
    normally present in the distribution tarballs. Therefore, to build GNU
    Wget from the sources in the repository, you'll need to have one or
    more of the following (note that gettext, OpenSSL, GnuTLS, libidn,
-   libiconv, libpsl, libpcre and pkg-config are not absolutely required):
+   libiconv, libpsl, libpcre, pkg-config, libmetalink and GnuPG are not
+   absolutely required):
 
      * [20]autoconf (currently, GNU Wget requires version 2.61). This is
        needed to generate the configure script from configure.in. This is
@@ -23,13 +24,19 @@ Compiling From Repository Sources
      * [23]Perl, if you wish to generate the wget(1) manpage, or run the
        tests in the tests/ sub directory. Tarball distributions include an
        already-generated wget.1 manual. The command "make check" runs the
-       test suite written in perl. To execute all the tests you need
-       [24]libwww-perl and libio-socket-ssl-perl perl library. If
-       "perl -MCPAN -e 'install Bundle::LWP'" fails then you most likely don't
-       have cpan module installed. First download [25]CPAN and [26]install it.
-       Then execute "perl -MCPAN -e 'install Bundle::LWP'".
+       test suite written in perl and python (see below). To execute all the tests
+       you need libwww-perl and libio-socket-ssl-perl perl library.
+       GNU/Linux distros likely have these two libraries packaged.
+       If not, install them directly with "perl -MCPAN -e 'install Bundle::LWP'"[24].
+       If that fails then you most likely don't have the CPAN module installed.
+       First download [25]CPAN and [26]install it, then try again the above command.
        Now "make check" should pass most of the tests in the test suite.
 
+     * [45]Python, if you want to run the tests in the testenv/ subdirectory.
+       Keep in mind that "make check" will try to run all the perl and python
+       tests. More information about the test suite below in the section
+       "Testing and development".
+
      * [27]texinfo in order to generate Info, PostScript and/or HTML
        documentation. You don't need texinfo in order to generate the
        wget(1) manpage; however, note that the manpage does not include
@@ -89,6 +96,13 @@ Compiling From Repository Sources
        saved the .pc file. Example:
        $ PKG_CONFIG_PATH="." ./configure
 
+     * [46]libmetalink is needed to enable Metalink files support.
+
+     * [47]GnuPG with GPGME is used to verify GPG-signed Metalink resources.
+
+     * [48]libcares is needed to bind DNS resolving to a given IP address.
+       The command line options --dns-servers and --bind-dns-address are
+       only available when configured with --with-cares.
 
    For those who might be confused as to what to do once they check out
    the source code, considering configure and Makefile do not yet exist at
@@ -110,6 +124,8 @@ Compiling From Repository Sources
  $  src/wget --version
  GNU Wget 1.12-devel (9cb2563197bc)
 
+Testing and development
+
  All developers are requested to enable the assertions on their development
  builds to ensure a stable codebase. Assertions are added to state certain
  assumptions about the code and its data which all developers should be mindful
@@ -118,6 +134,29 @@ Compiling From Repository Sources
 
  $ ./configure --enable-assert [other configure options]
 
+ Both the Perl and Python test suites (test/ and testenv/) include support for GDB and Valgrind.
+ The environment variables GDB_TESTS and VALGRIND_TESTS are available to
+ enable such wrappers. If specified, Wget would be run through either of them
+ during the test. For example:
+
+ $ cd testenv
+ $ VALGRIND_TESTS=1 ./Test-O.py
+
+ That would execute Test-O.py test case, but running Wget through Valgrind.
+
+ GDB has preference over Valgrind. If both variables have been asserted,
+ Wget would be run through GDB.
+
+ If you run a test case through GDB, please bear in mind that it could give
+ a false negative. This is because some tests that expect Wget to fail
+ rely on Wget's return code. However, when run through GDB, its return code
+ will always be zero, causing the test to claim failure. This wrapper for
+ GDB is, however, very useful to tackle bugs, allowing one to write a test case
+ for some specific bug and then using GDB to fix it more easily. Otherwise,
+ a dedicated server would have to be set up and write a custom CGI just to reproduce
+ that bug, which might be tedious. Tests should only be run through GDB for that purpose.
+
+
  Copyright (C) 2008,2010,2014,2015 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later
  <http://www.gnu.org/licenses/gpl.html>.
@@ -169,3 +208,7 @@ References
   42. http://moinmo.in/GPL
   43. http://validator.w3.org/check?uri=referer
   44. http://wget.addictivecode.org/WikiLicense
+  45. https://www.python.org/
+  46. https://launchpad.net/libmetalink
+  47. https://www.gnupg.org
+  48. http://c-ares.haxx.se/

bootstrap.conf

@@ -38,6 +38,7 @@ close
 connect
 dirname
 fcntl
+flock
 fnmatch
 futimens
 ftello
@@ -64,6 +65,7 @@ mkstemp
 mkostemp
 crypto/md5
 crypto/sha1
+crypto/sha256
 quote
 quotearg
 recv

configure.ac

@@ -57,6 +57,8 @@ AC_CANONICAL_HOST
 AC_DEFINE_UNQUOTED([OS_TYPE], "$host_os",
                    [Define to be the name of the operating system.])
 
+dnl Non-verbose make
+m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])])
 
 dnl
 dnl Process External Libraries
@@ -70,11 +72,16 @@ dnl SSL: Configure SSL backend to use
 AC_ARG_WITH([ssl],
   [AS_HELP_STRING([--with-ssl={gnutls,openssl}], [specify SSL backend. GNU TLS is the default.])])
 
-
-dnl Zlib: Configure use iof zlib for compression
+dnl Zlib: Configure use of zlib for compression
 AC_ARG_WITH([zlib],
   [AS_HELP_STRING([--without-zlib], [disable zlib.])])
 
+dnl Metalink: Configure use of the Metalink library
+AC_ARG_WITH([metalink],
+  [AS_HELP_STRING([--with-metalink], [enable support for metalinks.])])
+
+dnl C-Ares: Configure use of the c-ares library for DNS lookup
+AC_ARG_WITH(cares, AS_HELP_STRING([--with-cares], [enable support for C-Ares DNS lookup.]), with_cares=$withval, with_cares=no)
 
 dnl
 dnl Process features
@@ -177,7 +184,7 @@ dnl
 dnl Gettext
 dnl
 AM_GNU_GETTEXT([external],[need-ngettext])
-AM_GNU_GETTEXT_VERSION([0.17])
+AM_GNU_GETTEXT_VERSION([0.18.1])
 
 AC_PROG_RANLIB
 
@@ -475,6 +482,34 @@ else
   fi
 fi
 
+dnl
+dnl Check for libmetalink
+dnl
+AS_IF([test x"$with_metalink" != xno], [
+  PKG_CHECK_MODULES([METALINK], libmetalink, [
+      LIBS="$METALINK_LIBS $LIBS"
+      CFLAGS="$METALINK_CFLAGS $CFLAGS"
+      AC_DEFINE([HAVE_METALINK], [1], [Define if using metalink.])
+      with_metalink=yes
+    ], [
+      with_metalink=no
+    ])
+
+  have_gpg=no
+  AS_IF([test x"$with_metalink" = xyes], [
+    dnl
+    dnl Check for GPGME
+    dnl
+    m4_ifdef([AM_PATH_GPGME], [
+      AM_PATH_GPGME([], [
+        LIBS="$GPGME_LIBS $LIBS"
+        CFLAGS="$GPGME_CFLAGS $CFLAGS"
+        AC_DEFINE([HAVE_GPGME], [1], [Define if GPGME is available.])
+        have_gpg=yes
+      ])
+    ])
+  ])
+])
 
 dnl **********************************************************************
 dnl Checks for IPv6
@@ -711,10 +746,36 @@ AS_IF([test "X$enable_pcre" != "Xno"],[
   ])
 ])
 
+dnl
+dnl Check for libcares (resolver library)
+dnl
+
+AS_IF([test "X$with_cares" == "Xyes"],[
+  PKG_CHECK_MODULES([CARES], libcares, [
+    CFLAGS="$CARES_CFLAGS $CFLAGS"
+    AC_CHECK_HEADER(ares.h, [
+      LIBS="$CARES_LIBS $LIBS"
+      AC_DEFINE([HAVE_LIBCARES], [1], [Define if libcares is available.])
+      RESOLVER_INFO="libcares, --bind-dns-address and --dns-servers available"
+    ])
+  ], [
+    AC_CHECK_HEADER(ares.h, [
+      AC_CHECK_LIB(cares, ares_set_local_ip4, [
+        LIBS="-lcares ${LIBS}"
+        AC_DEFINE([HAVE_LIBCARES], 1, [Define if libcares is available.])
+        RESOLVER_INFO="libcares, --bind-dns-address and --dns-servers available"
+      ])
+    ])
+  ])
+], [
+  RESOLVER_INFO="libc, --bind-dns-address and --dns-servers not available"
+])
+
 
 dnl Needed by src/Makefile.am
 AM_CONDITIONAL([IRI_IS_ENABLED], [test "X$iri" != "Xno"])
-
+AM_CONDITIONAL([WITH_SSL], [test "X$with_ssl" != "Xno"])
+AM_CONDITIONAL([METALINK_IS_ENABLED], [test "X$with_metalink" != "Xno"])
 
 dnl
 dnl Create output
@@ -743,4 +804,7 @@ AC_MSG_NOTICE([Summary of build options:
   Debugging:         $ENABLE_DEBUG
   Assertions:        $ENABLE_ASSERTION
   Valgrind:          $VALGRIND_INFO
+  Metalink:          $with_metalink
+  Resolver:          $RESOLVER_INFO
+  GPGME:             $have_gpg
 ])

contrib/check-hard

@@ -2,28 +2,30 @@
 #
 # Do some checking before 'git push'.
 
-CORES=`grep -c ^processor /proc/cpuinfo 2>/dev/null || echo 0`
+# Set a stricter bash mode
+set -e
+set -u
+
+CORES=$(grep -c ^processor /proc/cpuinfo 2> /dev/null || echo 0)
 if [ $CORES -lt 1 ]; then CORES=1; fi
 
-make -f Makefile distclean 2>/dev/null
+echo "Running: make distclean"
+make distclean > /dev/null || true
 
-# automatically exit on error
-set -e
-
-export CFLAGS="-std=c89 -pedantic -O2 -g -Wall -Wextra -Wstrict-prototypes -Wold-style-definition -Wwrite-strings -Wshadow -Wformat -Wformat-security -Wunreachable-code -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition"
+CFLAGS="-std=c89 -pedantic -O2 -g -Wall -Wextra -Wstrict-prototypes -Wold-style-definition -Wwrite-strings -Wshadow -Wformat -Wformat-security -Wunreachable-code -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition"
 
 CACHEFILE=$PWD/config_check.cache
 
 # measure time consumed and print it at the end of the script
 START=$(date +%s.%N)
 
-echo "./bootstrap"
-./bootstrap >/dev/null
+echo "Running: ./bootstrap"
+./bootstrap > /dev/null
 echo
 
 for CC in gcc clang; do
   export CC
-  echo "*** checking CC=$CC"
+  echo "*** Testing with CC=$CC"
 
   # the compiler changed, so we have to remove the cache file here
   rm -f $CACHEFILE
@@ -31,20 +33,20 @@ for CC in gcc clang; do
   for options in "" "--with-ssl=openssl"; do
     export DISTCHECK_CONFIGURE_FLAGS="-C --cache-file=$CACHEFILE $options"
     echo "  ./configure $DISTCHECK_CONFIGURE_FLAGS"
-    ./configure $DISTCHECK_CONFIGURE_FLAGS >/dev/null
+    ./configure $DISTCHECK_CONFIGURE_FLAGS CFLAGS="$CFLAGS" > /dev/null
 
     for xVALGRIND in 0 1; do
       for xLCALL in C tr_TR.utf8; do
         export TESTS_ENVIRONMENT="LC_ALL=$xLCALL VALGRIND_TESTS=$xVALGRIND"
         echo "    TESTS_ENVIRONMENT=\"$TESTS_ENVIRONMENT\"" make check -j$CORES
-        make check -j$CORES >/dev/null
+        make check -j$CORES > /dev/null
       done
     done
 
     unset TESTS_ENVIRONMENT
     export TESTS_ENVIRONMENT
-    echo "  make distcheck -j$CORES"
-    make distcheck -j$CORES
+    echo "  make distcheck CFLAGS=$CFLAGS -j$CORES"
+    make distcheck CFLAGS="$CFLAGS" -j$CORES > /dev/null
   done
 done
 

contrib/travis-ci

@@ -0,0 +1,60 @@
+#!/bin/bash
+
+# Set a stricter bash mode
+set -e
+set -u
+
+# Set up the subset of tests this Matrix instance will run.
+
+# One instance is forked for each of the SSL library possibilities
+SSL=${1:-""}
+case $SSL in
+    "") SSL_LIB="--without-ssl";;
+    "openssl") SSL_LIB="--with-ssl=openssl";;
+    "gnutls") SSL_LIB="--with-ssl=gnutls";;
+esac
+
+# For some reason. /proc/cpuinfo reports 16 cores on Travis, while the docs
+# claim that each instance has only 2 cores. We believe the docs and force a
+# value of n+1 here.
+CORES=3
+
+# Define a large number of Warning flags for the compiler. Hopefully, someone
+# will sit and analyze the output to clean the warnings from the codebase.
+CFLAGS="-std=c89 -pedantic -O2 -Wall -Wextra -Wstrict-prototypes -Wold-style-definition -Wwrite-strings -Wshadow -Wformat -Wformat-security -Wunreachable-code -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition"
+
+# A cachefile for ./configure. Saves significant time when reconfiguring
+CACHEFILE=$PWD/config_check.cache
+
+# measure time consumed and print it at the end of the script
+START=$(date +%s)
+
+# Test different Permutations of Wget. Always under Valgrind
+#
+# Options:
+#   - All Features Enabled
+#     - C       [Default Locale]
+#     - Russian [Multibyte / Multicolumn Locale]
+#     - Turkish [Special Test due to issues with language]
+#   - IPv6 Disabled
+#     - C
+for options in "" "--disable-ipv6"; do
+  export DISTCHECK_CONFIGURE_FLAGS="-C --cache-file=$CACHEFILE --enable-assert --enable-valgrind-tests $SSL_LIB $options"
+  echo "  ./configure $DISTCHECK_CONFIGURE_FLAGS CFLAGS=\"$CFLAGS\""
+  ./configure $DISTCHECK_CONFIGURE_FLAGS CFLAGS="$CFLAGS"
+
+  if [[ -z $options ]]; then
+      LANG_ARR=("ru_RU.UTF-8" "tr_TR.UTF-8" "C")
+  else
+      LANG_ARR=("C")
+  fi
+
+  for xLCALL in "${LANG_ARR[@]}"; do
+    export TESTS_ENVIRONMENT="LC_ALL=$xLCALL VALGRIND_TESTS=1"
+    echo "    TESTS_ENVIRONMENT=\"$TESTS_ENVIRONMENT\" make distcheck CFLAGS=$CFLAGS -j$CORES"
+    make distcheck CFLAGS="$CFLAGS" -j$CORES
+  done
+done
+
+END=$(date +%s)
+echo "Duration: $((END-START))"

doc/Makefile.am

@@ -1,6 +1,6 @@
 # Makefile for `wget' utility
-# Copyright (C) 1995, 1996, 1997, 2007, 2008, 2009, 2010, 2011 Free
-# Software Foundation, Inc.
+# Copyright (C) 1995, 1996, 1997, 2007, 2008, 2009, 2010, 2011, 2015
+# Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -57,7 +57,10 @@ wget.pod: $(srcdir)/wget.texi version.texi
 	$(TEXI2POD) -D VERSION="$(VERSION)" $(srcdir)/wget.texi $@
 
 $(MAN): wget.pod
-	$(POD2MAN) --center="GNU Wget" --release="GNU Wget @VERSION@" $? > $@
+	$(POD2MAN) --center="GNU Wget" --release="GNU Wget @VERSION@" --utf8 $? > $@; \
+	if [ $? != 0 ]; then \
+          $(POD2MAN) --center="GNU Wget" --release="GNU Wget @VERSION@" $? > $@; \
+	fi
 
 #wget.cat: $(MAN)
 #	nroff -man $? > $@

doc/fdl.texi

@@ -5,7 +5,8 @@
 @c hence no sectioning command or @node.
 
 @display
-Copyright @copyright{} 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc.
+Copyright @copyright{} 2000, 2001, 2002, 2007, 2008, 2015 Free Software
+Foundation, Inc.
 @uref{http://fsf.org/}
 
 Everyone is permitted to copy and distribute verbatim copies

doc/texi2pod.pl

@@ -1,7 +1,7 @@
 #! /usr/bin/env perl
 
-#   Copyright (C) 1999, 2000, 2001, 2003, 2007, 2009, 2010, 2011 Free
-#   Software Foundation, Inc.
+#   Copyright (C) 1999, 2000, 2001, 2003, 2007, 2009, 2010, 2011, 2015
+#   Free Software Foundation, Inc.
 
 # This file is part of GCC.
 
@@ -315,6 +315,8 @@ die "No filename or title\n" unless defined $fn && defined $tl;
 $sects{NAME} = "$fn \- $tl\n";
 $sects{FOOTNOTES} .= "=back\n" if exists $sects{FOOTNOTES};
 
+print "=encoding utf-8\n\n";
+
 for $sect (qw(NAME SYNOPSIS DESCRIPTION OPTIONS ENVIRONMENT EXITSTATUS
            FILES BUGS NOTES FOOTNOTES SEEALSO AUTHOR COPYRIGHT)) {
     if(exists $sects{$sect}) {

doc/wget.texi

@@ -2,6 +2,7 @@
 
 @c %**start of header
 @setfilename wget.info
+@documentencoding UTF-8
 @include version.texi
 @settitle GNU Wget @value{VERSION} Manual
 @c Disable the monstrous rectangles beside overfull hbox-es.
@@ -31,8 +32,8 @@ data.
 
 @c man begin COPYRIGHT
 Copyright @copyright{} 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
-2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-Inc.
+2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+Foundation, Inc.
 
 @iftex
 Permission is granted to make and distribute verbatim copies of
@@ -59,11 +60,11 @@ Texts.  A copy of the license is included in the section entitled
 @title GNU Wget @value{VERSION}
 @subtitle The non-interactive download utility
 @subtitle Updated for Wget @value{VERSION}, @value{UPDATED}
-@author by Hrvoje Nik@v{s}i@'{c} and others
+@author by Hrvoje Nikšić and others
 
 @ignore
 @c man begin AUTHOR
-Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
+Originally written by Hrvoje Nikšić <hniksic@xemacs.org>.
 @c man end
 @c man begin SEEALSO
 This is @strong{not} the complete manual for GNU Wget.
@@ -506,6 +507,23 @@ treated as @samp{html} if the Content-Type matches @samp{text/html}.
 Furthermore, the @var{file}'s location will be implicitly used as base
 href if none was specified.
 
+@cindex input-metalink
+@item --input-metalink=@var{file}
+Downloads files covered in local Metalink @var{file}. Metalink version 3
+and 4 are supported.
+
+@cindex metalink-over-http
+@item --metalink-over-http
+Issues HTTP HEAD request instead of GET and extracts Metalink metadata
+from response headers. Then it switches to Metalink download.
+If no valid Metalink metadata is found, it falls back to ordinary HTTP download.
+
+@cindex preferred-location
+@item --preferred-location
+Set preferred location for Metalink resources. This has effect if multiple
+resources with same priority are available.
+
+
 @cindex force html
 @item -F
 @itemx --force-html
@@ -533,6 +551,11 @@ would be resolved to @samp{http://foo/baz/b.html}.
 @cindex specify config 
 @item --config=@var{FILE}
 Specify the location of a startup file you wish to use.
+
+@item --rejected-log=@var{logfile}
+Logs all URL rejections to @var{logfile} as comma separated values.  The values
+include the reason of rejection, the URL and the parent URL it was found in.
+
 @end table
 
 @node Download Options, Directory Options, Logging and Input File Options, Invoking
@@ -548,6 +571,27 @@ the local machine.  @var{ADDRESS} may be specified as a hostname or IP
 address.  This option can be useful if your machine is bound to multiple
 IPs.
 
+@cindex bind DNS address
+@cindex client DNS address
+@cindex DNS IP address, client, DNS
+@item --bind-dns-address=@var{ADDRESS}
+[libcares only]
+This address overrides the route for DNS requests. If you ever need to
+circumvent the standard settings from /etc/resolv.conf, this option together
+with @samp{--dns-servers} is your friend.
+@var{ADDRESS} must be specified either as IPv4 or IPv6 address.
+Wget needs to be built with libcares for this option to be available.
+
+@cindex DNS server
+@cindex DNS IP address, client, DNS
+@item --dns-servers=@var{ADDRESSES}
+[libcares only]
+The given address(es) override the standard nameserver
+addresses,  e.g. as configured in /etc/resolv.conf.
+@var{ADDRESSES} may be specified either as IPv4 or IPv6 addresses,
+comma-separated.
+Wget needs to be built with libcares for this option to be available.
+
 @cindex retries
 @cindex tries
 @cindex number of tries
@@ -585,6 +629,9 @@ content will be placed in @var{file}. This was disabled in version
 1.11, but has been reinstated (with a warning) in 1.11.2, as there are
 some cases where this behavior can actually have some use.
 
+A combination with @samp{-nc} is only accepted if the given output
+file does not exist.
+
 Note that a combination with @samp{-k} is only permitted when
 downloading a single document, as in that case it will just convert
 all relative URIs to external ones; @samp{-k} makes no sense for
@@ -626,6 +673,9 @@ of a file depends on the local and remote timestamp and size of the
 file (@pxref{Time-Stamping}).  @samp{-nc} may not be specified at the
 same time as @samp{-N}.
 
+A combination with @samp{-O}/@samp{--output-document} is only accepted
+if the given output file does not exist.
+
 Note that when @samp{-nc} is specified, files with the suffixes
 @samp{.html} or @samp{.htm} will be loaded from the local disk and
 parsed as if they had been retrieved from the Web.
@@ -665,13 +715,11 @@ Without @samp{-c}, the previous example would just download the remote
 file to @file{ls-lR.Z.1}, leaving the truncated @file{ls-lR.Z} file
 alone.
 
-Beginning with Wget 1.7, if you use @samp{-c} on a non-empty file, and
-it turns out that the server does not support continued downloading,
-Wget will refuse to start the download from scratch, which would
-effectively ruin existing contents.  If you really want the download to
-start from scratch, remove the file.
+If you use @samp{-c} on a non-empty file, and the server does not support
+continued downloading, Wget will restart the download from scratch and overwrite
+the existing file entirely.
 
-Also beginning with Wget 1.7, if you use @samp{-c} on a file which is of
+Beginning with Wget 1.7, if you use @samp{-c} on a file which is of
 equal size as the one on the server, Wget will refuse to download the
 file and print an explanatory message.  The same happens when the file
 is smaller on the server than locally (presumably because it was changed
@@ -787,6 +835,10 @@ used alongside the @samp{--logfile} option.
 @itemx --timestamping
 Turn on time-stamping.  @xref{Time-Stamping}, for details.
 
+@item --no-if-modified-since
+Do not send If-Modified-Since header in @samp{-N} mode. Send preliminary HEAD
+request instead. This has only effect in @samp{-N} mode.
+
 @item --no-use-server-timestamps
 Don't set the local file's timestamp by the one on the server.
 
@@ -950,7 +1002,7 @@ specified in bytes (default), kilobytes (with @samp{k} suffix), or
 megabytes (with @samp{m} suffix).
 
 Note that quota will never affect downloading a single file.  So if you
-specify @samp{wget -Q10k ftp://wuarchive.wustl.edu/ls-lR.gz}, all of the
+specify @samp{wget -Q10k https://example.com/ls-lR.gz}, all of the
 @file{ls-lR.gz} will be downloaded.  The same goes even when several
 @sc{url}s are specified on the command-line.  However, quota is
 respected when retrieving either recursively, or from an input file.
@@ -1521,6 +1573,11 @@ Please note that wget does not require the content to be of the form
 simply transmit whatever data is provided to it. Most servers however expect
 the POST data to be in the above format when processing HTML Forms.
 
+When sending a POST request using the @samp{--post-file} option, Wget treats
+the file as a binary file and will send every character in the POST request
+without stripping trailing newline or formfeed characters. Any other control
+characters in the text will also be sent as-is in the POST request.
+
 Please be aware that Wget needs to know the size of the POST data in
 advance.  Therefore the argument to @code{--post-file} must be a regular
 file; specifying a FIFO or something like @file{/dev/stdin} won't work.
@@ -1548,11 +1605,11 @@ users:
 # @r{Log in to the server.  This can be done only once.}
 wget --save-cookies cookies.txt \
      --post-data 'user=foo&password=bar' \
-     http://server.com/auth.php
+     http://example.com/auth.php
 
 # @r{Now grab the page or pages we care about.}
 wget --load-cookies cookies.txt \
-     -p http://server.com/interesting/article.php
+     -p http://example.com/interesting/article.php
 @end group
 @end example
 
@@ -1637,8 +1694,9 @@ form-based authentication.
 
 @cindex SSL
 To support encrypted HTTP (HTTPS) downloads, Wget must be compiled
-with an external SSL library, currently OpenSSL.  If Wget is compiled
-without SSL support, none of these options are available.
+with an external SSL library. The current default is GnuTLS.
+In addition, Wget also supports HSTS (HTTP Strict Transport Security).
+If Wget is compiled without SSL support, none of these options are available.
 
 @table @samp
 @cindex SSL protocol, choose
@@ -1686,6 +1744,12 @@ this option to bypass the verification and proceed with the download.
 site's authenticity, or if you really don't care about the validity of
 its certificate.}  It is almost always a bad idea not to check the
 certificates when transmitting confidential or important data.
+For self-signed/internal certificates, you should download the certificate
+and verify against that instead of forcing this insecure mode.
+If you are really sure of not desiring any certificate verification, you
+can specify --check-certificate=quiet to tell wget to not print any
+warning about invalid certificates, albeit in most cases this is the
+wrong thing to do.
 
 @cindex SSL certificate
 @item --certificate=@var{file}
@@ -1733,6 +1797,18 @@ system-specified locations, chosen at OpenSSL installation time.
 Specifies a CRL file in @var{file}.  This is needed for certificates
 that have been revocated by the CAs.
 
+@cindex SSL Public Key Pin
+@item --pinnedpubkey=file/hashes
+Tells wget to use the specified public key file (or hashes) to verify the peer.
+This can be a path to a file which contains a single public key in PEM or DER
+format, or any number of base64 encoded sha256 hashes preceded by ``sha256//''
+and separated by ``;''
+
+When negotiating a TLS or SSL connection, the server sends a certificate
+indicating its identity. A public key is extracted from this certificate and if
+it does not exactly match the public key(s) provided to this option, wget will
+abort the connection before sending or receiving any data.
+
 @cindex entropy, specifying source of
 @cindex randomness, specifying source of
 @item --random-file=@var{file}
@@ -1768,6 +1844,71 @@ read random data from EGD socket specified using this option.
 If this option is not specified (and the equivalent startup command is
 not used), EGD is never contacted.  EGD is not needed on modern Unix
 systems that support @file{/dev/urandom}.
+
+@cindex HSTS
+@item --no-hsts
+Wget supports HSTS (HTTP Strict Transport Security, RFC 6797) by default.
+Use @samp{--no-hsts} to make Wget act as a non-HSTS-compliant UA. As a
+consequence, Wget would ignore all the @code{Strict-Transport-Security}
+headers, and would not enforce any existing HSTS policy.
+
+@item --hsts-file=@var{file}
+By default, Wget stores its HSTS database in @file{~/.wget-hsts}.
+You can use @samp{--hsts-file} to override this. Wget will use
+the supplied file as the HSTS database. Such file must conform to the
+correct HSTS database format used by Wget. If Wget cannot parse the provided
+file, the behaviour is unspecified.
+
+The Wget's HSTS database is a plain text file. Each line contains an HSTS entry
+(ie. a site that has issued a @code{Strict-Transport-Security} header and that
+therefore has specified a concrete HSTS policy to be applied). Lines starting with
+a dash (@code{#}) are ignored by Wget. Please note that in spite of this convenient
+human-readability hand-hacking the HSTS database is generally not a good idea.
+
+An HSTS entry line consists of several fields separated by one or more whitespace:
+
+@code{<hostname> SP [<port>] SP <include subdomains> SP <created> SP <max-age>}
+
+The @var{hostname} and @var{port} fields indicate the hostname and port to which
+the given HSTS policy applies. The @var{port} field may be zero, and it will, in
+most of the cases. That means that the port number will not be taken into account
+when deciding whether such HSTS policy should be applied on a given request (only
+the hostname will be evaluated). When @var{port} is different to zero, both the
+target hostname and the port will be evaluated and the HSTS policy will only be applied
+if both of them match. This feature has been included for testing/development purposes only.
+The Wget testsuite (in @file{testenv/}) creates HSTS databases with explicit ports
+with the purpose of ensuring Wget's correct behaviour. Applying HSTS policies to ports
+other than the default ones is discouraged by RFC 6797 (see Appendix B "Differences
+between HSTS Policy and Same-Origin Policy"). Thus, this functionality should not be used
+in production environments and @var{port} will typically be zero. The last three fields
+do what they are expected to. The field @var{include_subdomains} can either be @code{1}
+or @code{0} and it signals whether the subdomains of the target domain should be
+part of the given HSTS policy as well. The @var{created} and @var{max-age} fields
+hold the timestamp values of when such entry was created (first seen by Wget) and the
+HSTS-defined value 'max-age', which states how long should that HSTS policy remain active,
+measured in seconds elapsed since the timestamp stored in @var{created}. Once that time
+has passed, that HSTS policy will no longer be valid and will eventually be removed
+from the database.
+
+If you supply your own HSTS database via @samp{--hsts-file}, be aware that Wget
+may modify the provided file if any change occurs between the HSTS policies
+requested by the remote servers and those in the file. When Wget exists,
+it effectively updates the HSTS database by rewriting the database file with the new entries.
+
+If the supplied file does not exist, Wget will create one. This file will contain the new HSTS
+entries. If no HSTS entries were generated (no @code{Strict-Transport-Security} headers
+were sent by any of the servers) then no file will be created, not even an empty one. This
+behaviour applies to the default database file (@file{~/.wget-hsts}) as well: it will not be
+created until some server enforces an HSTS policy.
+
+Care is taken not to override possible changes made by other Wget processes at
+the same time over the HSTS database. Before dumping the updated HSTS entries
+on the file, Wget will re-read it and merge the changes.
+
+Using a custom HSTS database and/or modifying an existing one is discouraged.
+For more information about the potential security threats arised from such practice,
+see section 14 "Security Considerations" of RFC 6797, specially section 14.9
+"Creative Manipulation of HSTS Policy Store".
 @end table
 
 @cindex WARC
@@ -1910,6 +2051,43 @@ this option has no effect.  Symbolic links are always traversed in this
 case.
 @end table
 
+@section FTPS Options
+
+@table @samp
+@item --ftps-implicit
+This option tells Wget to use FTPS implicitly. Implicit FTPS consists of initializing
+SSL/TLS from the very beginning of the control connection. This option does not send
+an @code{AUTH TLS} command: it assumes the server speaks FTPS and directly starts an
+SSL/TLS connection. If the attempt is successful, the session continues just like
+regular FTPS (@code{PBSZ} and @code{PROT} are sent, etc.).
+Implicit FTPS is no longer a requirement for FTPS implementations, and thus
+many servers may not support it. If @samp{--ftps-implicit} is passed and no explicit
+port number specified, the default port for implicit FTPS, 990, will be used, instead
+of the default port for the "normal" (explicit) FTPS which is the same as that of FTP,
+21.
+
+@item --no-ftps-resume-ssl
+Do not resume the SSL/TLS session in the data channel. When starting a data connection,
+Wget tries to resume the SSL/TLS session previously started in the control connection.
+SSL/TLS session resumption avoids performing an entirely new handshake by reusing
+the SSL/TLS parameters of a previous session. Typically, the FTPS servers want it that way,
+so Wget does this by default. Under rare circumstances however, one might want to
+start an entirely new SSL/TLS session in every data connection.
+This is what @samp{--no-ftps-resume-ssl} is for.
+
+@item --ftps-clear-data-connection
+All the data connections will be in plain text. Only the control connection will be
+under SSL/TLS. Wget will send a @code{PROT C} command to achieve this, which must be
+approved by the server.
+
+@item --ftps-fallback-to-ftp
+Fall back to FTP if FTPS is not supported by the target server. For security reasons,
+this option is not asserted by default. The default behaviour is to exit with an error.
+If a server does not successfully reply to the initial @code{AUTH TLS} command, or in the
+case of implicit FTPS, if the initial SSL/TLS connection attempt is rejected, it is
+considered that such server does not support FTPS.
+@end table
+
 @node Recursive Retrieval Options, Recursive Accept/Reject Options, FTP Options, Invoking
 @section Recursive Retrieval Options
 
@@ -1988,6 +2166,23 @@ Note that only at the end of the download can Wget know which links have
 been downloaded.  Because of that, the work done by @samp{-k} will be
 performed at the end of all the downloads.
 
+@item --convert-file-only
+This option converts only the filename part of the URLs, leaving the rest
+of the URLs untouched. This filename part is sometimes referred to as the
+"basename", although we avoid that term here in order not to cause confusion.
+
+It works particularly well in conjunction with @samp{--adjust-extension}, although
+this coupling is not enforced. It proves useful to populate Internet caches
+with files downloaded from different hosts.
+
+Example: if some link points to @file{//foo.com/bar.cgi?xyz} with
+@samp{--adjust-extension} asserted and its local destination is intended to be
+@file{./foo.com/bar.cgi?xyz.css}, then the link would be converted to
+@file{//foo.com/bar.cgi?xyz.css}. Note that only the filename part has been
+modified. The rest of the URL has been left untouched, including the net path
+(@code{//}) which would otherwise be processed by Wget and converted to the
+effective scheme (ie. @code{http://}).
+
 @cindex backing up converted files
 @item -K
 @itemx --backup-converted
@@ -2397,11 +2592,11 @@ The @samp{-D} option allows you to specify the domains that will be
 followed, thus limiting the recursion only to the hosts that belong to
 these domains.  Obviously, this makes sense only in conjunction with
 @samp{-H}.  A typical example would be downloading the contents of
-@samp{www.server.com}, but allowing downloads from
-@samp{images.server.com}, etc.:
+@samp{www.example.com}, but allowing downloads from
+@samp{images.example.com}, etc.:
 
 @example
-wget -rH -Dserver.com http://www.server.com/
+wget -rH -Dexample.com http://www.example.com/
 @end example
 
 You can specify more than one address by separating them with a comma,
@@ -2641,7 +2836,7 @@ These links are not relative:
 @example
 <a href="/foo.gif">
 <a href="/foo/bar.gif">
-<a href="http://www.server.com/foo/bar.gif">
+<a href="http://www.example.com/foo/bar.gif">
 @end example
 
 Using this option guarantees that recursive retrieval will not span
@@ -3511,7 +3706,7 @@ same directory structure the original has, with only one try per
 document, saving the log of the activities to @file{gnulog}:
 
 @example
-wget -r http://www.gnu.org/ -o gnulog
+wget -r https://www.gnu.org/ -o gnulog
 @end example
 
 @item
@@ -3519,7 +3714,7 @@ The same as the above, but convert the links in the downloaded files to
 point to local files, so you can view the documents off-line:
 
 @example
-wget --convert-links -r http://www.gnu.org/ -o gnulog
+wget --convert-links -r https://www.gnu.org/ -o gnulog
 @end example
 
 @item
@@ -3529,22 +3724,22 @@ sheets, are also downloaded.  Also make sure the downloaded page
 references the downloaded links.
 
 @example
-wget -p --convert-links http://www.server.com/dir/page.html
+wget -p --convert-links http://www.example.com/dir/page.html
 @end example
 
-The @sc{html} page will be saved to @file{www.server.com/dir/page.html}, and
-the images, stylesheets, etc., somewhere under @file{www.server.com/},
+The @sc{html} page will be saved to @file{www.example.com/dir/page.html}, and
+the images, stylesheets, etc., somewhere under @file{www.example.com/},
 depending on where they were on the remote server.
 
 @item
-The same as the above, but without the @file{www.server.com/} directory.
+The same as the above, but without the @file{www.example.com/} directory.
 In fact, I don't want to have all those random server directories
 anyway---just save @emph{all} those files under a @file{download/}
 subdirectory of the current directory.
 
 @example
 wget -p --convert-links -nH -nd -Pdownload \
-     http://www.server.com/dir/page.html
+     http://www.example.com/dir/page.html
 @end example
 
 @item
@@ -3573,12 +3768,12 @@ wget -r -l2 -P/tmp ftp://wuarchive.wustl.edu/
 
 @item
 You want to download all the @sc{gif}s from a directory on an @sc{http}
-server.  You tried @samp{wget http://www.server.com/dir/*.gif}, but that
+server.  You tried @samp{wget http://www.example.com/dir/*.gif}, but that
 didn't work because @sc{http} retrieval does not support globbing.  In
 that case, use:
 
 @example
-wget -r -l1 --no-parent -A.gif http://www.server.com/dir/
+wget -r -l1 --no-parent -A.gif http://www.example.com/dir/
 @end example
 
 More verbose, but the effect is the same.  @samp{-r -l1} means to
@@ -3594,7 +3789,7 @@ interrupted.  Now you do not want to clobber the files already present.
 It would be:
 
 @example
-wget -nc -r http://www.gnu.org/
+wget -nc -r https://www.gnu.org/
 @end example
 
 @item
@@ -3602,7 +3797,7 @@ If you want to encode your own username and password to @sc{http} or
 @sc{ftp}, use the appropriate @sc{url} syntax (@pxref{URL Format}).
 
 @example
-wget ftp://hniksic:mypassword@@unix.server.com/.emacs
+wget ftp://hniksic:mypassword@@unix.example.com/.emacs
 @end example
 
 Note, however, that this usage is not advisable on multi-user systems
@@ -3639,7 +3834,7 @@ to recheck a site each Sunday:
 
 @example
 crontab
-0 0 * * 0 wget --mirror http://www.gnu.org/ -o /home/me/weeklog
+0 0 * * 0 wget --mirror https://www.gnu.org/ -o /home/me/weeklog
 @end example
 
 @item
@@ -3651,7 +3846,7 @@ would look like this:
 
 @example
 wget --mirror --convert-links --backup-converted  \
-     http://www.gnu.org/ -o /home/me/weeklog
+     https://www.gnu.org/ -o /home/me/weeklog
 @end example
 
 @item
@@ -3664,13 +3859,13 @@ or @samp{application/xhtml+xml} to @file{@var{name}.html}.
 @example
 wget --mirror --convert-links --backup-converted \
      --html-extension -o /home/me/weeklog        \
-     http://www.gnu.org/
+     https://www.gnu.org/
 @end example
 
 Or, with less typing:
 
 @example
-wget -m -k -K -E http://www.gnu.org/ -o /home/me/weeklog
+wget -m -k -K -E https://www.gnu.org/ -o /home/me/weeklog
 @end example
 @end itemize
 @c man end
@@ -3777,14 +3972,14 @@ username and password.
 Like all GNU utilities, the latest version of Wget can be found at the
 master GNU archive site ftp.gnu.org, and its mirrors.  For example,
 Wget @value{VERSION} can be found at
-@url{ftp://ftp.gnu.org/pub/gnu/wget/wget-@value{VERSION}.tar.gz}
+@url{https://ftp.gnu.org/pub/gnu/wget/wget-@value{VERSION}.tar.gz}
 
 @node Web Site, Mailing Lists, Distribution, Various
 @section Web Site
 @cindex web site
 
 The official web site for GNU Wget is at
-@url{http://www.gnu.org/software/wget/}. However, most useful
+@url{https//www.gnu.org/software/wget/}. However, most useful
 information resides at ``The Wget Wgiki'',
 @url{http://wget.addictivecode.org/}.
 
@@ -3798,14 +3993,14 @@ information resides at ``The Wget Wgiki'',
 The primary mailinglist for discussion, bug-reports, or questions
 about GNU Wget is at @email{bug-wget@@gnu.org}. To subscribe, send an
 email to @email{bug-wget-join@@gnu.org}, or visit
-@url{http://lists.gnu.org/mailman/listinfo/bug-wget}.
+@url{https://lists.gnu.org/mailman/listinfo/bug-wget}.
 
 You do not need to subscribe to send a message to the list; however,
 please note that unsubscribed messages are moderated, and may take a
 while before they hit the list---@strong{usually around a day}.  If
 you want your message to show up immediately, please subscribe to the
 list before posting. Archives for the list may be found at
-@url{http://lists.gnu.org/pipermail/bug-wget/}.
+@url{https://lists.gnu.org/archive/html/bug-wget/}.
 
 An NNTP/Usenettish gateway is also available via
 @uref{http://gmane.org/about.php,Gmane}. You can see the Gmane
@@ -3813,15 +4008,7 @@ archives at
 @url{http://news.gmane.org/gmane.comp.web.wget.general}. Note that the
 Gmane archives conveniently include messages from both the current
 list, and the previous one. Messages also show up in the Gmane
-archives sooner than they do at @url{lists.gnu.org}.
-
-@unnumberedsubsec Bug Notices List
-
-Additionally, there is the @email{wget-notify@@addictivecode.org} mailing
-list. This is a non-discussion list that receives bug report
-notifications from the bug-tracker. To subscribe to this list,
-send an email to @email{wget-notify-join@@addictivecode.org},
-or visit @url{http://addictivecode.org/mailman/listinfo/wget-notify}.
+archives sooner than they do at @url{https://lists.gnu.org}.
 
 @unnumberedsubsec Obsolete Lists
 
@@ -3833,7 +4020,7 @@ discussing patches to GNU Wget.
 Messages from @email{wget@@sunsite.dk} are archived at
 @itemize @tie{}
 @item
-@url{http://www.mail-archive.com/wget%40sunsite.dk/} and at
+@url{https://www.mail-archive.com/wget%40sunsite.dk/} and at
 @item
 @url{http://news.gmane.org/gmane.comp.web.wget.general} (which also
 continues to archive the current list, @email{bug-wget@@gnu.org}).
@@ -3862,7 +4049,7 @@ via IRC at @code{irc.freenode.org}, @code{#wget}. Come check it out!
 
 @c man begin BUGS
 You are welcome to submit bug reports via the GNU Wget bug tracker (see
-@url{http://wget.addictivecode.org/BugTracker}).
+@url{https://savannah.gnu.org/bugs/?func=additem&group=wget}).
 
 Before actually submitting a bug report, please try to follow a few
 simple guidelines.
@@ -3879,7 +4066,7 @@ Lists}).
 @item
 Try to repeat the bug in as simple circumstances as possible.  E.g. if
 Wget crashes while downloading @samp{wget -rl0 -kKE -t5 --no-proxy
-http://yoyodyne.com -o /tmp/log}, you should try to see if the crash is
+http://example.com -o /tmp/log}, you should try to see if the crash is
 repeatable, and if will occur with a simpler set of options.  You might
 even try to start the download at the page where the crash occurred to
 see if that page somehow triggered the crash.
@@ -3944,7 +4131,7 @@ Windows-related features might look at them.
 
 Support for building on MS-DOS via DJGPP has been contributed by Gisle
 Vanem; a port to VMS is maintained by Steven Schweda, and is available
-at @url{http://antinode.org/}.
+at @url{https://antinode.info/dec/sw/wget.html}.
 
 @node Signals,  , Portability, Various
 @section Signals
@@ -4022,23 +4209,23 @@ download an individual page.  Because of that, Wget honors RES when
 downloading recursively.  For instance, when you issue:
 
 @example
-wget -r http://www.server.com/
+wget -r http://www.example.com/
 @end example
 
-First the index of @samp{www.server.com} will be downloaded.  If Wget
+First the index of @samp{www.example.com} will be downloaded.  If Wget
 finds that it wants to download more documents from that server, it will
-request @samp{http://www.server.com/robots.txt} and, if found, use it
+request @samp{http://www.example.com/robots.txt} and, if found, use it
 for further downloads.  @file{robots.txt} is loaded only once per each
 server.
 
 Until version 1.8, Wget supported the first version of the standard,
 written by Martijn Koster in 1994 and available at
-@url{http://www.robotstxt.org/wc/norobots.html}.  As of version 1.8,
+@url{http://www.robotstxt.org/robotstxt.html}.  As of version 1.8,
 Wget has supported the additional directives specified in the internet
 draft @samp{<draft-koster-robots-00.txt>} titled ``A Method for Web
 Robots Control''.  The draft, which has as far as I know never made to
 an @sc{rfc}, is available at
-@url{http://www.robotstxt.org/wc/norobots-rfc.txt}.
+@url{http://www.robotstxt.org/norobots-rfc.txt}.
 
 This manual no longer includes the text of the Robot Exclusion Standard.
 
@@ -4096,12 +4283,7 @@ me).
 @section Contributors
 @cindex contributors
 
-@iftex
-GNU Wget was written by Hrvoje Nik@v{s}i@'{c} @email{hniksic@@xemacs.org},
-@end iftex
-@ifnottex
-GNU Wget was written by Hrvoje Niksic @email{hniksic@@xemacs.org}.
-@end ifnottex
+GNU Wget was written by Hrvoje Nikšić @email{hniksic@@xemacs.org},
 
 However, the development of Wget could never have gone as far as it has, were
 it not for the help of many people, either with bug reports, feature proposals,
@@ -4141,14 +4323,8 @@ portability fixes.
 Gordon Matzigkeit---@file{.netrc} support.
 
 @item
-@iftex
-Zlatko @v{C}alu@v{s}i@'{c}, Tomislav Vujec and Dra@v{z}en
-Ka@v{c}ar---feature suggestions and ``philosophical'' discussions.
-@end iftex
-@ifnottex
-Zlatko Calusic, Tomislav Vujec and Drazen Kacar---feature suggestions
-and ``philosophical'' discussions.
-@end ifnottex
+Zlatko Čalušić, Tomislav Vujec and Dražen
+Kačar---feature suggestions and ``philosophical'' discussions.
 
 @item
 Darko Budor---initial port to Windows.
@@ -4158,21 +4334,11 @@ Antonio Rosella---help and suggestions, plus the initial Italian
 translation.
 
 @item
-@iftex
-Tomislav Petrovi@'{c}, Mario Miko@v{c}evi@'{c}---many bug reports and
+Tomislav Petrović, Mario Mikočević---many bug reports and
 suggestions.
-@end iftex
-@ifnottex
-Tomislav Petrovic, Mario Mikocevic---many bug reports and suggestions.
-@end ifnottex
 
 @item
-@iftex
-Fran@,{c}ois Pinard---many thorough bug reports and discussions.
-@end iftex
-@ifnottex
-Francois Pinard---many thorough bug reports and discussions.
-@end ifnottex
+François Pinard---many thorough bug reports and discussions.
 
 @item
 Karl Eichwalder---lots of help with internationalization, Makefile
@@ -4235,12 +4401,7 @@ Wanderlei Cavassin,
 Gilles Cedoc,
 Tim Charron,
 Noel Cragg,
-@iftex
-Kristijan @v{C}onka@v{s},
-@end iftex
-@ifnottex
-Kristijan Conkas,
-@end ifnottex
+Kristijan Čonkaš,
 John Daily,
 Andreas Damm,
 Ahmon Dancy,
@@ -4250,27 +4411,12 @@ Alexander Dergachev,
 Andrew Deryabin,
 Ulrich Drepper,
 Marc Duponcheel,
-@iftex
-Damir D@v{z}eko,
-@end iftex
-@ifnottex
-Damir Dzeko,
-@end ifnottex
+Damir Džeko,
 Alan Eldridge,
 Hans-Andreas Engel,
-@iftex
-Aleksandar Erkalovi@'{c},
-@end iftex
-@ifnottex
-Aleksandar Erkalovic,
-@end ifnottex
+Aleksandar Erkalović,
 Andy Eskilsson,
-@iftex
-Jo@~{a}o Ferreira,
-@end iftex
-@ifnottex
-Joao Ferreira,
-@end ifnottex
+João Ferreira,
 Christian Fraenkel,
 David Fritz,
 Mike Frysinger,
@@ -4295,25 +4441,10 @@ Richard Huveneers,
 Jonas Jensen,
 Larry Jones,
 Simon Josefsson,
-@iftex
-Mario Juri@'{c},
-@end iftex
-@ifnottex
-Mario Juric,
-@end ifnottex
-@iftex
-Hack Kampbj@o rn,
-@end iftex
-@ifnottex
-Hack Kampbjorn,
-@end ifnottex
+Mario Jurić,
+Hack Kampbjørn,
 Const Kaplinsky,
-@iftex
-Goran Kezunovi@'{c},
-@end iftex
-@ifnottex
-Goran Kezunovic,
-@end ifnottex
+Goran Kezunović,
 Igor Khristophorov,
 Robert Kleine,
 KOJIMA Haime,
@@ -4322,31 +4453,14 @@ Alexander Kourakos,
 Martin Kraemer,
 Sami Krank,
 Jay Krell,
-@tex
-$\Sigma\acute{\iota}\mu o\varsigma\;
-\Xi\varepsilon\nu\iota\tau\acute{\epsilon}\lambda\lambda\eta\varsigma$
-(Simos KSenitellis),
-@end tex
-@ifnottex
-Simos KSenitellis,
-@end ifnottex
+Σίμος Ξενιτέλλης (Simos KSenitellis),
 Christian Lackas,
 Hrvoje Lacko,
 Daniel S.@: Lewart,
-@iftex
-Nicol@'{a}s Lichtmeier,
-@end iftex
-@ifnottex
-Nicolas Lichtmeier,
-@end ifnottex
+Nicolás Lichtmeier,
 Dave Love,
 Alexander V.@: Lukyanov,
-@iftex
-Thomas Lu@ss{}nig,
-@end iftex
-@ifnottex
-Thomas Lussnig,
-@end ifnottex
+Thomas Lußnig,
 Andre Majorel,
 Aurelien Marchand,
 Matthew J.@: Mellon,
@@ -4367,31 +4481,15 @@ Leonid Petrov,
 Simone Piunno,
 Andrew Pollock,
 Steve Pothier,
-@iftex
-Jan P@v{r}ikryl,
-@end iftex
-@ifnottex
-Jan Prikryl,
-@end ifnottex
+Jan Přikryl,
 Marin Purgar,
-@iftex
-Csaba R@'{a}duly,
-@end iftex
-@ifnottex
-Csaba Raduly,
-@end ifnottex
+Csaba Ráduly,
 Keith Refson,
 Bill Richardson,
 Tyler Riddle,
 Tobias Ringstrom,
 Jochen Roderburg,
-@c Texinfo doesn't grok @'{@i}, so we have to use TeX itself.
-@tex
-Juan Jos\'{e} Rodr\'{\i}guez,
-@end tex
-@ifnottex
-Juan Jose Rodriguez,
-@end ifnottex
+Juan José Rodríguez,
 Maciej W.@: Rozycki,
 Edward J.@: Sabol,
 Heinz Salzmann,
@@ -4418,12 +4516,7 @@ Dave Turner,
 Gisle Vanem,
 Rabin Vincent,
 Russell Vincent,
-@iftex
-@v{Z}eljko Vrba,
-@end iftex
-@ifnottex
-Zeljko Vrba,
-@end ifnottex
+Željko Vrba,
 Charles G Waldman,
 Douglas E.@: Wegscheid,
 Ralf Wildenhues,
@@ -4432,12 +4525,7 @@ Benjamin Wolsey,
 Saint Xavier,
 YAMAZAKI Makoto,
 Jasmin Zainul,
-@iftex
-Bojan @v{Z}drnja,
-@end iftex
-@ifnottex
-Bojan Zdrnja,
-@end ifnottex
+Bojan Ždrnja,
 Kristijan Zimmer,
 Xin Zou.
 

msdos/config.h

@@ -1,6 +1,6 @@
 /* Configuration header file for MS-DOS/Watt-32
-   Copyright (C) 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   Copyright (C) 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
    This file is part of GNU Wget.
 

msdos/msdos.c

@@ -1,5 +1,5 @@
 /* MSDOS utility functions.
-   Copyright (C) 2009, 2010, 2011 Free Software Foundation, Inc.
+   Copyright (C) 2009, 2010, 2011, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

po/POTFILES.in

@@ -26,6 +26,7 @@ src/init.c
 src/iri.c
 src/log.c
 src/main.c
+src/metalink.c
 src/mswindows.c
 src/netrc.c
 src/openssl.c

src/Makefile.am

@@ -1,6 +1,6 @@
 # Makefile for `wget' utility
 # Copyright (C) 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
-# 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software
+# 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
 # Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
@@ -35,6 +35,10 @@ if IRI_IS_ENABLED
 IRI_OBJ = iri.c
 endif
 
+if METALINK_IS_ENABLED
+METALINK_OBJ = metalink.c
+endif
+
 # The following line is losing on some versions of make!
 DEFS     = @DEFS@ -DSYSTEM_WGETRC=\"$(sysconfdir)/wgetrc\" -DLOCALEDIR=\"$(localedir)\"
 LIBS     = @LIBICONV@ @LIBINTL@ @LIBS@ $(LIB_CLOCK_GETTIME)
@@ -44,16 +48,16 @@ EXTRA_DIST = css.l css.c css_.c build_info.c.in
 bin_PROGRAMS = wget
 wget_SOURCES = connect.c convert.c cookies.c ftp.c	\
 		css_.c css-url.c	\
-		ftp-basic.c ftp-ls.c hash.c host.c html-parse.c html-url.c	\
+		ftp-basic.c ftp-ls.c hash.c host.c hsts.c html-parse.c html-url.c	\
 		http.c init.c log.c main.c netrc.c progress.c ptimer.c	\
 		recur.c res.c retr.c spider.c url.c warc.c	\
-		utils.c exits.c build_info.c $(IRI_OBJ)	\
+		utils.c exits.c build_info.c $(IRI_OBJ) $(METALINK_OBJ)	\
 		css-url.h css-tokens.h connect.h convert.h cookies.h	\
-		ftp.h hash.h host.h html-parse.h html-url.h	\
+		ftp.h hash.h host.h hsts.h  html-parse.h html-url.h	\
 		http.h http-ntlm.h init.h log.h mswindows.h netrc.h	\
 		options.h progress.h ptimer.h recur.h res.h retr.h	\
 		spider.h ssl.h sysdep.h url.h warc.h utils.h wget.h iri.h	\
-		exits.h version.h
+		exits.h version.h metalink.h
 nodist_wget_SOURCES = version.c
 EXTRA_wget_SOURCES = iri.c
 LDADD = $(LIBOBJS) ../lib/libgnu.a

src/build_info.c.in

@@ -8,6 +8,10 @@ nls             defined ENABLE_NLS
 ntlm            defined ENABLE_NTLM
 opie            defined ENABLE_OPIE
 psl             defined HAVE_LIBPSL
+cares            defined HAVE_LIBCARES
+
+metalink        defined HAVE_METALINK
+gpgme           defined HAVE_GPGME
 
 ssl choice:
     openssl     defined HAVE_LIBSSL || defined HAVE_LIBSSL32

src/connect.c

@@ -1,6 +1,6 @@
 /* Establishing and handling network connections.
    Copyright (C) 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
-   2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software
+   2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
    Foundation, Inc.
 
 This file is part of GNU Wget.
@@ -369,7 +369,14 @@ connect_to_ip (const ip_address *ip, int port, const char *print)
        logprintf.  */
     int save_errno = errno;
     if (sock >= 0)
-      fd_close (sock);
+      {
+#ifdef WIN32
+	/* If the connection timed out, fd_close will hang in Gnulib's
+	   close_fd_maybe_socket, inside the call to WSAEnumNetworkEvents.  */
+	if (errno != ETIMEDOUT)
+#endif
+	  fd_close (sock);
+      }
     if (print)
       logprintf (LOG_NOTQUIET, _("failed: %s.\n"), strerror (errno));
     errno = save_errno;

src/connect.h

@@ -1,7 +1,7 @@
 /* Declarations for connect.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/convert.c

@@ -1,6 +1,6 @@
 /* Conversion of links to local files.
-   Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014
-   Free Software Foundation, Inc.
+   Copyright (C) 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011,
+   2014, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -46,6 +46,7 @@ as that of the covered work.  */
 #include "html-url.h"
 #include "css-url.h"
 #include "iri.h"
+#include "xstrndup.h"
 
 static struct hash_table *dl_file_url_map;
 struct hash_table *dl_url_file_map;
@@ -136,8 +137,9 @@ convert_links_in_hashtable (struct hash_table *downloaded_set,
                  form.  We do this even if the URL already is in
                  relative form, because our directory structure may
                  not be identical to that on the server (think `-nd',
-                 `--cut-dirs', etc.)  */
-              cur_url->convert = CO_CONVERT_TO_RELATIVE;
+                 `--cut-dirs', etc.). If --convert-file-only was passed,
+                 we only convert the basename portion of the URL.  */
+              cur_url->convert = (opt.convert_file_only ? CO_CONVERT_BASENAME_ONLY : CO_CONVERT_TO_RELATIVE);
               cur_url->local_name = xstrdup (local_name);
               DEBUGP (("will convert url %s to local %s\n", u->url, local_name));
             }
@@ -193,7 +195,7 @@ convert_all_links (void)
   convert_links_in_hashtable (downloaded_css_set, 1, &file_count);
 
   secs = ptimer_measure (timer);
-  logprintf (LOG_VERBOSE, _("Converted %d files in %s seconds.\n"),
+  logprintf (LOG_VERBOSE, _("Converted links in %d files in %s seconds.\n"),
              file_count, print_decimal (secs));
 
   ptimer_destroy (timer);
@@ -206,6 +208,7 @@ static const char *replace_attr_refresh_hack (const char *, int, FILE *,
                                               const char *, int);
 static char *local_quote_string (const char *, bool);
 static char *construct_relative (const char *, const char *);
+static char *convert_basename (const char *, const struct urlpos *);
 
 /* Change the links in one file.  LINKS is a list of links in the
    document, along with their positions and the desired direction of
@@ -221,7 +224,7 @@ convert_links (const char *file, struct urlpos *links)
   struct urlpos *link;
   int to_url_count = 0, to_file_count = 0;
 
-  logprintf (LOG_VERBOSE, _("Converting %s... "), file);
+  logprintf (LOG_VERBOSE, _("Converting links in %s... "), file);
 
   {
     /* First we do a "dry run": go through the list L and see whether
@@ -305,7 +308,7 @@ convert_links (const char *file, struct urlpos *links)
             char *quoted_newname = local_quote_string (newname,
                                                        link->link_css_p);
 
-            if (link->link_css_p)
+            if (link->link_css_p || link->link_noquote_html_p)
               p = replace_plain (p, link->size, fp, quoted_newname);
             else if (!link->link_refresh_p)
               p = replace_attr (p, link->size, fp, quoted_newname);
@@ -315,9 +318,32 @@ convert_links (const char *file, struct urlpos *links)
 
             DEBUGP (("TO_RELATIVE: %s to %s at position %d in %s.\n",
                      link->url->url, newname, link->pos, file));
+
             xfree (newname);
             xfree (quoted_newname);
             ++to_file_count;
+            break;
+          }
+        case CO_CONVERT_BASENAME_ONLY:
+          {
+            char *newname = convert_basename (p, link);
+            char *quoted_newname = local_quote_string (newname, link->link_css_p);
+
+            if (link->link_css_p || link->link_noquote_html_p)
+              p = replace_plain (p, link->size, fp, quoted_newname);
+            else if (!link->link_refresh_p)
+              p = replace_attr (p, link->size, fp, quoted_newname);
+            else
+              p = replace_attr_refresh_hack (p, link->size, fp, quoted_newname,
+                                             link->refresh_timeout);
+
+            DEBUGP (("Converted file part only: %s to %s at position %d in %s.\n",
+                     link->url->url, newname, link->pos, file));
+
+            xfree (newname);
+            xfree (quoted_newname);
+            ++to_file_count;
+
             break;
           }
         case CO_CONVERT_TO_COMPLETE:
@@ -326,7 +352,7 @@ convert_links (const char *file, struct urlpos *links)
             char *newlink = link->url->url;
             char *quoted_newlink = html_quote_string (newlink);
 
-            if (link->link_css_p)
+            if (link->link_css_p || link->link_noquote_html_p)
               p = replace_plain (p, link->size, fp, newlink);
             else if (!link->link_refresh_p)
               p = replace_attr (p, link->size, fp, quoted_newlink);
@@ -336,6 +362,7 @@ convert_links (const char *file, struct urlpos *links)
 
             DEBUGP (("TO_COMPLETE: <something> to %s at position %d in %s.\n",
                      newlink, link->pos, file));
+
             xfree (quoted_newlink);
             ++to_url_count;
             break;
@@ -414,14 +441,89 @@ construct_relative (const char *basefile, const char *linkfile)
         ++basedirs;
     }
 
-  /* Construct LINK as explained above. */
-  link = xmalloc (3 * basedirs + strlen (linkfile) + 1);
-  for (i = 0; i < basedirs; i++)
-    memcpy (link + 3 * i, "../", 3);
-  strcpy (link + 3 * i, linkfile);
+  if (!basedirs && (b = strpbrk (linkfile, "/:")) && *b == ':')
+    {
+      link = xmalloc (2 + strlen (linkfile) + 1);
+      memcpy (link, "./", 2);
+      strcpy (link + 2, linkfile);
+    }
+  else
+    {
+      /* Construct LINK as explained above. */
+      link = xmalloc (3 * basedirs + strlen (linkfile) + 1);
+      for (i = 0; i < basedirs; i++)
+        memcpy (link + 3 * i, "../", 3);
+      strcpy (link + 3 * i, linkfile);
+    }
+
   return link;
 }
 
+/* Construct and return a "transparent proxy" URL
+   reflecting changes made by --adjust-extension to the file component
+   (i.e., "basename") of the original URL, but leaving the "dirname"
+   of the URL (protocol://hostname... portion) untouched.
+
+   Think: populating a squid cache via a recursive wget scrape, where
+   changing URLs to work locally with "file://..." is NOT desirable.
+
+   Example:
+
+   if
+                     p = "//foo.com/bar.cgi?xyz"
+   and
+      link->local_name = "docroot/foo.com/bar.cgi?xyz.css"
+   then
+
+      new_construct_func(p, link);
+   will return
+      "//foo.com/bar.cgi?xyz.css"
+
+   Essentially, we do s/$(basename orig_url)/$(basename link->local_name)/
+*/
+static char *
+convert_basename (const char *p, const struct urlpos *link)
+{
+  int len = link->size;
+  char *url = NULL;
+  char *org_basename = NULL, *local_basename = NULL;
+  char *result = NULL;
+
+  if (*p == '"' || *p == '\'')
+    {
+      len -= 2;
+      p++;
+    }
+
+  url = xstrndup (p, len);
+
+  org_basename = strrchr (url, '/');
+  if (org_basename)
+    org_basename++;
+  else
+    org_basename = url;
+
+  local_basename = strrchr (link->local_name, '/');
+  if (local_basename)
+    local_basename++;
+  else
+    local_basename = url;
+
+  /*
+   * If the basenames differ, graft the adjusted basename (local_basename)
+   * onto the original URL.
+   */
+  if (strcmp (org_basename, local_basename) == 0)
+    result = url;
+  else
+    {
+      result = uri_merge (url, local_basename);
+      xfree (url);
+    }
+
+  return result;
+}
+
 /* Used by write_backup_file to remember which files have been
    written. */
 static struct hash_table *converted_files;

src/convert.h

@@ -1,6 +1,6 @@
 /* Declarations for convert.c
-   Copyright (C) 2003, 2004, 2005, 2006, 2009, 2010, 2011 Free Software
-   Foundation, Inc.
+   Copyright (C) 2003, 2004, 2005, 2006, 2009, 2010, 2011, 2015 Free
+   Software Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -40,6 +40,8 @@ enum convert_options {
   CO_NOCONVERT = 0,             /* don't convert this URL */
   CO_CONVERT_TO_RELATIVE,       /* convert to relative, e.g. to
                                    "../../otherdir/foo.gif" */
+  CO_CONVERT_BASENAME_ONLY,     /* convert the file portion only (basename)
+                                   leaving the rest of the URL unchanged */
   CO_CONVERT_TO_COMPLETE,       /* convert to absolute, e.g. to
                                    "http://orighost/somedir/bar.jpg". */
   CO_NULLIFY_BASE               /* change to empty string. */
@@ -67,6 +69,7 @@ struct urlpos {
   unsigned int link_base_p  :1;     /* the url came from <base href=...> */
   unsigned int link_inline_p    :1; /* needed to render the page */
   unsigned int link_css_p   :1;     /* the url came from CSS */
+  unsigned int link_noquote_html_p :1; /* from HTML, but doesn't need " */
   unsigned int link_expect_html :1; /* expected to contain HTML */
   unsigned int link_expect_css  :1; /* expected to contain CSS */
 

src/cookies.c

@@ -1,6 +1,6 @@
 /* Support for cookies.
    Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009,
-   2010, 2011 Free Software Foundation, Inc.
+   2010, 2011, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/cookies.h

@@ -1,6 +1,6 @@
 /* Support for cookies.
    Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009,
-   2010, 2011 Free Software Foundation, Inc.
+   2010, 2011, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/css-tokens.h

@@ -1,5 +1,6 @@
 /* Declarations for css.lex
-   Copyright (C) 2006, 2009, 2010, 2011 Free Software Foundation, Inc.
+   Copyright (C) 2006, 2009, 2010, 2011, 2015 Free Software Foundation,
+   Inc.
 
 This file is part of GNU Wget.
 

src/css-url.c

@@ -1,6 +1,6 @@
 /* Collect URLs from CSS source.
-   Copyright (C) 1998, 2000, 2001, 2002, 2003, 2009, 2010, 2011, 2014 Free
-   Software Foundation, Inc.
+   Copyright (C) 1998, 2000, 2001, 2002, 2003, 2009, 2010, 2011, 2014,
+   2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/css-url.h

@@ -1,5 +1,6 @@
 /* Declarations for css-url.c.
-   Copyright (C) 2006, 2009, 2010, 2011 Free Software Foundation, Inc.
+   Copyright (C) 2006, 2009, 2010, 2011, 2015 Free Software Foundation,
+   Inc.
 
 This file is part of GNU Wget.
 

src/css.l

@@ -6,7 +6,8 @@
 %{
 /* Lex source for CSS tokenizing.
    Taken from http://www.w3.org/TR/CSS21/grammar.html#q2
-   Copyright (C) 2006, 2009, 2010, 2011 Free Software Foundation, Inc.
+   Copyright (C) 2006, 2009, 2010, 2011, 2015 Free Software Foundation,
+   Inc.
 
 This file is part of GNU Wget.
 

src/exits.c

@@ -1,5 +1,6 @@
 /* Exit status handling.
-   Copyright (C) 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+   Copyright (C) 2009, 2010, 2011, 2012, 2015 Free Software Foundation,
+   Inc.
 
    This file is part of GNU Wget.
 

src/exits.h

@@ -1,5 +1,6 @@
 /* Exit status related declarations.
-   Copyright (C) 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+   Copyright (C) 2009, 2010, 2011, 2012, 2015 Free Software Foundation,
+   Inc.
 
 This file is part of GNU Wget.
 

src/ftp-basic.c

@@ -1,7 +1,7 @@
 /* Basic FTP routines.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -135,6 +135,23 @@ ftp_request (const char *command, const char *value)
   return res;
 }
 
+uerr_t
+ftp_greeting (int csock)
+{
+  uerr_t err = FTPOK;
+  char *response = NULL;
+
+  err = ftp_response (csock, &response);
+  if (err != FTPOK)
+    goto bail;
+  if (*response != '2')
+    err = FTPSRVERR;
+
+bail:
+  if (response)
+    xfree (response);
+  return err;
+}
 /* Sends the USER and PASS commands to the server, to control
    connection socket csock.  */
 uerr_t
@@ -144,16 +161,6 @@ ftp_login (int csock, const char *acc, const char *pass)
   char *request, *respline;
   int nwritten;
 
-  /* Get greeting.  */
-  err = ftp_response (csock, &respline);
-  if (err != FTPOK)
-    return err;
-  if (*respline != '2')
-    {
-      xfree (respline);
-      return FTPSRVERR;
-    }
-  xfree (respline);
   /* Send USER username.  */
   request = ftp_request ("USER", acc);
   nwritten = fd_write (csock, request, strlen (request), -1);
@@ -481,6 +488,119 @@ ftp_eprt (int csock, int *local_sock)
 }
 #endif
 
+#ifdef HAVE_SSL
+/*
+ * The following three functions defined into this #ifdef block
+ * wrap the extended FTP commands defined in RFC 2228 (FTP Security Extensions).
+ * Currently, only FTPS is supported, so these functions are only compiled when SSL
+ * support is available, because there's no point in using FTPS when there's no SSL.
+ * Shall someone add new secure FTP protocols in the future, feel free to remove this
+ * #ifdef, or add new constants to it.
+ */
+
+/*
+ * Sends an AUTH command as defined by RFC 2228,
+ * deriving its argument from the scheme. For example, if the provided scheme
+ * is SCHEME_FTPS, the command sent will be "AUTH TLS". Currently, this is the only
+ * scheme supported, so this function will return FTPNOAUTH when supplied a different
+ * one. It will also return FTPNOAUTH if the target server does not support FTPS.
+ */
+uerr_t
+ftp_auth (int csock, enum url_scheme scheme)
+{
+  uerr_t err = 0;
+  int written = 0;
+  char *request = NULL, *response = NULL;
+
+  if (scheme == SCHEME_FTPS)
+    {
+      request = ftp_request ("AUTH", "TLS");
+      written = fd_write (csock, request, strlen (request), -1);
+      if (written < 0)
+        {
+          err = WRITEFAILED;
+          goto bail;
+        }
+      err = ftp_response (csock, &response);
+      if (err != FTPOK)
+        goto bail;
+      if (*response != '2')
+        err = FTPNOAUTH;
+    }
+  else
+    err = FTPNOAUTH;
+
+bail:
+  xfree (request);
+  xfree (response);
+
+  return err;
+}
+
+uerr_t
+ftp_pbsz (int csock, int pbsz)
+{
+  uerr_t err = 0;
+  int written = 0;
+  char spbsz[5];
+  char *request = NULL, *response = NULL;
+
+  snprintf (spbsz, 5, "%d", pbsz);
+  request = ftp_request ("PBSZ", spbsz);
+  written = fd_write (csock, request, strlen (request), -1);
+  if (written < 0)
+    {
+      err = WRITEFAILED;
+      goto bail;
+    }
+
+  err = ftp_response (csock, &response);
+  if (err != FTPOK)
+    goto bail;
+  if (*response != '2')
+    err = FTPNOPBSZ;
+
+bail:
+  xfree (request);
+  xfree (response);
+
+  return err;
+}
+
+uerr_t
+ftp_prot (int csock, enum prot_level prot)
+{
+  uerr_t err = 0;
+  int written = 0;
+  char *request = NULL, *response = NULL;
+  /* value must be a single character value */
+  char value[2];
+
+  value[0] = prot;
+  value[1] = '\0';
+
+  request = ftp_request ("PROT", value);
+  written = fd_write (csock, request, strlen (request), -1);
+  if (written < 0)
+    {
+      err = WRITEFAILED;
+      goto bail;
+    }
+
+  err = ftp_response (csock, &response);
+  if (err != FTPOK)
+    goto bail;
+  if (*response != '2')
+    err = FTPNOPROT;
+
+bail:
+  xfree (request);
+  xfree (response);
+
+  return err;
+}
+#endif /* HAVE_SSL */
+
 /* Similar to ftp_port, but uses `PASV' to initiate the passive FTP
    transfer.  Reads the response from server and parses it.  Reads the
    host and port addresses and returns them.  */

src/ftp-ls.c

@@ -1,7 +1,7 @@
 /* Parsing FTP `ls' output.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/ftp-opie.c

@@ -1,6 +1,6 @@
 /* Opie (s/key) support for FTP.
    Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006,
-   2007, 2008, 2009, 2010, 2011 Free Software Foundation, Inc.
+   2007, 2008, 2009, 2010, 2011, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/ftp.c

@@ -1,7 +1,7 @@
 /* File Transfer Protocol support.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -44,6 +44,7 @@ as that of the covered work.  */
 #include "url.h"
 #include "retr.h"
 #include "ftp.h"
+#include "ssl.h"
 #include "connect.h"
 #include "host.h"
 #include "netrc.h"
@@ -235,24 +236,96 @@ print_length (wgint size, wgint start, bool authoritative)
   logputs (LOG_VERBOSE, !authoritative ? _(" (unauthoritative)\n") : "\n");
 }
 
-static uerr_t ftp_get_listing (struct url *, ccon *, struct fileinfo **);
+static uerr_t ftp_get_listing (struct url *, struct url *, ccon *, struct fileinfo **);
+
+static uerr_t
+get_ftp_greeting(int csock, ccon *con)
+{
+  uerr_t err = 0;
+
+  /* Get the server's greeting */
+  err = ftp_greeting (csock);
+  if (err != FTPOK)
+    {
+      logputs (LOG_NOTQUIET, "Error in server response. Closing.\n");
+      fd_close (csock);
+      con->csock = -1;
+    }
+
+  return err;
+}
+
+#ifdef HAVE_SSL
+static uerr_t
+init_control_ssl_connection (int csock, struct url *u, bool *using_control_security)
+{
+  bool using_security = false;
+
+  /* If '--ftps-implicit' was passed, perform the SSL handshake directly,
+   * and do not send an AUTH command.
+   * Otherwise send an AUTH sequence before login,
+   * and perform the SSL handshake if accepted by server.
+   */
+  if (!opt.ftps_implicit && !opt.server_response)
+    logputs (LOG_VERBOSE, "==> AUTH TLS ... ");
+  if (opt.ftps_implicit || ftp_auth (csock, SCHEME_FTPS) == FTPOK)
+    {
+      if (!ssl_connect_wget (csock, u->host, NULL))
+        {
+          fd_close (csock);
+          return CONSSLERR;
+        }
+      else if (!ssl_check_certificate (csock, u->host))
+        {
+          fd_close (csock);
+          return VERIFCERTERR;
+        }
+
+      if (!opt.ftps_implicit && !opt.server_response)
+        logputs (LOG_VERBOSE, " done.\n");
+
+      /* If implicit FTPS was requested, we act as "normal" FTP, but over SSL.
+       * We're not using RFC 2228 commands.
+       */
+      using_security = true;
+    }
+  else
+    {
+      /* The server does not support 'AUTH TLS'.
+       * Check if --ftps-fallback-to-ftp was passed. */
+      if (opt.ftps_fallback_to_ftp)
+        {
+          logputs (LOG_NOTQUIET, "Server does not support AUTH TLS. Falling back to FTP.\n");
+          using_security = false;
+        }
+      else
+        {
+          fd_close (csock);
+          return FTPNOAUTH;
+        }
+    }
+
+  *using_control_security = using_security;
+  return NOCONERROR;
+}
+#endif
 
 /* Retrieves a file with denoted parameters through opening an FTP
    connection to the server.  It always closes the data connection,
    and closes the control connection in case of error.  If warc_tmp
    is non-NULL, the downloaded data will be written there as well.  */
 static uerr_t
-getftp (struct url *u, wgint passed_expected_bytes, wgint *qtyread,
+getftp (struct url *u, struct url *original_url,
+        wgint passed_expected_bytes, wgint *qtyread,
         wgint restval, ccon *con, int count, wgint *last_expected_bytes,
         FILE *warc_tmp)
 {
   int csock, dtsock, local_sock, res;
   uerr_t err = RETROK;          /* appease the compiler */
-  FILE *fp;
+  FILE *fp = NULL;
   char *respline, *tms;
   const char *user, *passwd, *tmrate;
   int cmd = con->cmd;
-  bool pasv_mode_open = false;
   wgint expected_bytes = 0;
   bool got_expected_bytes = false;
   bool rest_failed = false;
@@ -261,6 +334,15 @@ getftp (struct url *u, wgint passed_expected_bytes, wgint *qtyread,
   char type_char;
   bool try_again;
   bool list_a_used = false;
+#ifdef HAVE_SSL
+  enum prot_level prot = (opt.ftps_clear_data_connection ? PROT_CLEAR : PROT_PRIVATE);
+  /* these variables tell whether the target server
+   * accepts the security extensions (RFC 2228) or not,
+   * and whether we're actually using any of them
+   * (encryption at the control connection only,
+   * or both at control and data connections) */
+  bool using_control_security = false, using_data_security = false;
+#endif
 
   assert (con != NULL);
   assert (con->target != NULL);
@@ -286,8 +368,36 @@ getftp (struct url *u, wgint passed_expected_bytes, wgint *qtyread,
   local_sock = -1;
   con->dltime = 0;
 
+#ifdef HAVE_SSL
+  if (u->scheme == SCHEME_FTPS)
+    {
+      /* Initialize SSL layer first */
+      if (!ssl_init ())
+        {
+          scheme_disable (SCHEME_FTPS);
+          logprintf (LOG_NOTQUIET, _("Could not initialize SSL. It will be disabled."));
+          err = SSLINITFAILED;
+          return err;
+        }
+
+      /* If we're using the default FTP port and implicit FTPS was requested,
+       * rewrite the port to the default *implicit* FTPS port.
+       */
+      if (opt.ftps_implicit && u->port == DEFAULT_FTP_PORT)
+        {
+          DEBUGP (("Implicit FTPS was specified. Rewriting default port to %d.\n", DEFAULT_FTPS_IMPLICIT_PORT));
+          u->port = DEFAULT_FTPS_IMPLICIT_PORT;
+        }
+    }
+#endif
+
   if (!(cmd & DO_LOGIN))
-    csock = con->csock;
+    {
+      csock = con->csock;
+#ifdef HAVE_SSL
+      using_data_security = con->st & DATA_CHANNEL_SECURITY;
+#endif
+    }
   else                          /* cmd & DO_LOGIN */
     {
       char    *host = con->proxy ? con->proxy->host : u->host;
@@ -309,6 +419,43 @@ getftp (struct url *u, wgint passed_expected_bytes, wgint *qtyread,
       else
         con->csock = -1;
 
+#ifdef HAVE_SSL
+      if (u->scheme == SCHEME_FTPS)
+        {
+          /* If we're in implicit FTPS mode, we have to set up SSL/TLS before everything else.
+           * Otherwise we first read the server's greeting, and then send an "AUTH TLS".
+           */
+          if (opt.ftps_implicit)
+            {
+              err = init_control_ssl_connection (csock, u, &using_control_security);
+              if (err != NOCONERROR)
+                return err;
+              err = get_ftp_greeting (csock, con);
+              if (err != FTPOK)
+                return err;
+            }
+          else
+            {
+              err = get_ftp_greeting (csock, con);
+              if (err != FTPOK)
+                return err;
+              err = init_control_ssl_connection (csock, u, &using_control_security);
+              if (err != NOCONERROR)
+                return err;
+            }
+        }
+      else
+        {
+          err = get_ftp_greeting (csock, con);
+          if (err != FTPOK)
+            return err;
+        }
+#else
+      err = get_ftp_greeting (csock, con);
+      if (err != FTPOK)
+        return err;
+#endif
+
       /* Second: Login with proper USER/PASS sequence.  */
       logprintf (LOG_VERBOSE, _("Logging in as %s ... "),
                  quotearg_style (escape_quoting_style, user));
@@ -366,6 +513,46 @@ Error in server response, closing control connection.\n"));
         default:
           abort ();
         }
+
+#ifdef HAVE_SSL
+      if (using_control_security)
+        {
+          /* Send the PBSZ and PROT commands, in that order.
+           * If we are here it means that the server has already accepted
+           * some form of FTPS. Thus, these commands must work.
+           * If they don't work, that's an error. There's no sense in honoring
+           * --ftps-fallback-to-ftp or similar options. */
+          if (u->scheme == SCHEME_FTPS)
+            {
+              if (!opt.server_response)
+                logputs (LOG_VERBOSE, "==> PBSZ 0 ... ");
+              if ((err = ftp_pbsz (csock, 0)) == FTPNOPBSZ)
+                {
+                  logputs (LOG_NOTQUIET, _("Server did not accept the 'PBSZ 0' command.\n"));
+                  return err;
+                }
+              if (!opt.server_response)
+                logputs (LOG_VERBOSE, "done.");
+
+              if (!opt.server_response)
+                logprintf (LOG_VERBOSE, "  ==> PROT %c ... ", prot);
+              if ((err = ftp_prot (csock, prot)) == FTPNOPROT)
+                {
+                  logprintf (LOG_NOTQUIET, _("Server did not accept the 'PROT %c' command.\n"), prot);
+                  return err;
+                }
+              if (!opt.server_response)
+                logputs (LOG_VERBOSE, "done.\n");
+
+              if (prot != PROT_CLEAR)
+                {
+                  using_data_security = true;
+                  con->st |= DATA_CHANNEL_SECURITY;
+                }
+            }
+        }
+#endif
+
       /* Third: Get the system type */
       if (!opt.server_response)
         logprintf (LOG_VERBOSE, "==> SYST ... ");
@@ -883,13 +1070,19 @@ Error in server response, closing control connection.\n"));
                           ? CONERROR : CONIMPOSSIBLE);
                 }
 
-              pasv_mode_open = true;  /* Flag to avoid accept port */
               if (!opt.server_response)
                 logputs (LOG_VERBOSE, _("done.    "));
-            } /* err==FTP_OK */
-        }
+            }
+          else
+            return err;
 
-      if (!pasv_mode_open)   /* Try to use a port command if PASV failed */
+          /*
+           * We do not want to fall back from PASSIVE mode to ACTIVE mode !
+           * The reason is the PORT command exposes the client's real IP address
+           * to the server. Bad for someone who relies on privacy via a ftp proxy.
+           */
+        }
+      else
         {
           err = ftp_do_port (csock, &local_sock);
           /* FTPRERR, WRITEFAILED, bindport (FTPSYSERR), HOSTERR,
@@ -996,7 +1189,7 @@ Error in server response, closing control connection.\n"));
         {
           bool exists = false;
           struct fileinfo *f;
-          uerr_t _res = ftp_get_listing (u, con, &f);
+          uerr_t _res = ftp_get_listing (u, original_url, con, &f);
           /* Set the DO_RETR command flag again, because it gets unset when
              calling ftp_get_listing() and would otherwise cause an assertion
              failure earlier on when this function gets repeatedly called
@@ -1148,8 +1341,8 @@ Error in server response, closing control connection.\n"));
     }
 
   /* If no transmission was required, then everything is OK.  */
-  if (!pasv_mode_open)  /* we are not using pasive mode so we need
-                              to accept */
+  if (!opt.ftp_pasv)  /* we are not using passive mode so we need
+                         to accept */
     {
       /* Wait for the server to connect to the address we're waiting
          at.  */
@@ -1308,6 +1501,38 @@ Error in server response, closing control connection.\n"));
   else if (expected_bytes)
     print_length (expected_bytes, restval, false);
 
+#ifdef HAVE_SSL
+  if (u->scheme == SCHEME_FTPS && using_data_security)
+    {
+      /* We should try to restore the existing SSL session in the data connection
+       * and fall back to establishing a new session if the server doesn't want to restore it.
+       */
+      if (!opt.ftps_resume_ssl || !ssl_connect_wget (dtsock, u->host, &csock))
+        {
+          if (opt.ftps_resume_ssl)
+            logputs (LOG_NOTQUIET, "Server does not want to resume the SSL session. Trying with a new one.\n");
+          if (!ssl_connect_wget (dtsock, u->host, NULL))
+            {
+              fd_close (csock);
+              fd_close (dtsock);
+              err = CONERROR;
+              logputs (LOG_NOTQUIET, "Could not perform SSL handshake.\n");
+              goto exit_error;
+            }
+        }
+      else
+        logputs (LOG_NOTQUIET, "Resuming SSL session in data connection.\n");
+
+      if (!ssl_check_certificate (dtsock, u->host))
+        {
+          fd_close (csock);
+          fd_close (dtsock);
+          err = CONERROR;
+          goto exit_error;
+        }
+    }
+#endif
+
   /* Get the contents of the document.  */
   flags = 0;
   if (restval && rest_failed)
@@ -1372,10 +1597,18 @@ Error in server response, closing control connection.\n"));
      become apparent later.  */
   if (*respline != '2')
     {
-      xfree (respline);
       if (res != -1)
         logprintf (LOG_NOTQUIET, "%s (%s) - ", tms, tmrate);
       logputs (LOG_NOTQUIET, _("Data transfer aborted.\n"));
+#ifdef HAVE_SSL
+      if (!c_strncasecmp (respline, "425", 3) && u->scheme == SCHEME_FTPS)
+        {
+          logputs (LOG_NOTQUIET, "FTPS server rejects new SSL sessions in the data connection.\n");
+          xfree (respline);
+          return FTPRESTFAIL;
+        }
+#endif
+      xfree (respline);
       return FTPRETRINT;
     }
   xfree (respline);
@@ -1532,6 +1765,13 @@ Error in server response, closing control connection.\n"));
     }
   } while (try_again);
   return RETRFINISHED;
+
+exit_error:
+
+  /* If fp is a regular file, close and try to remove it */
+  if (fp && !output_stream)
+    fclose (fp);
+  return err;
 }
 
 /* A one-file FTP loop.  This is the part where FTP retrieval is
@@ -1540,7 +1780,8 @@ Error in server response, closing control connection.\n"));
    This loop either gets commands from con, or (if ON_YOUR_OWN is
    set), makes them up to retrieve the file given by the URL.  */
 static uerr_t
-ftp_loop_internal (struct url *u, struct fileinfo *f, ccon *con, char **local_file)
+ftp_loop_internal (struct url *u, struct url *original_url, struct fileinfo *f,
+                   ccon *con, char **local_file, bool force_full_retrieve)
 {
   int count, orig_lp;
   wgint restval, len = 0, qtyread = 0;
@@ -1565,7 +1806,7 @@ ftp_loop_internal (struct url *u, struct fileinfo *f, ccon *con, char **local_fi
     {
       /* URL-derived file.  Consider "-O file" name. */
       xfree (con->target);
-      con->target = url_file_name (u, NULL);
+      con->target = url_file_name (opt.trustservernames || !original_url ? u : original_url, NULL);
       if (!opt.output_document)
         locf = con->target;
       else
@@ -1642,6 +1883,8 @@ ftp_loop_internal (struct url *u, struct fileinfo *f, ccon *con, char **local_fi
       /* Decide whether or not to restart.  */
       if (con->cmd & DO_LIST)
         restval = 0;
+      else if (force_full_retrieve)
+        restval = 0;
       else if (opt.start_pos >= 0)
         restval = opt.start_pos;
       else if (opt.always_rest
@@ -1681,8 +1924,8 @@ ftp_loop_internal (struct url *u, struct fileinfo *f, ccon *con, char **local_fi
 
       /* If we are working on a WARC record, getftp should also write
          to the warc_tmp file. */
-      err = getftp (u, len, &qtyread, restval, con, count, &last_expected_bytes,
-                    warc_tmp);
+      err = getftp (u, original_url, len, &qtyread, restval, con, count,
+                    &last_expected_bytes, warc_tmp);
 
       if (con->csock == -1)
         con->st &= ~DONE_CWD;
@@ -1692,11 +1935,17 @@ ftp_loop_internal (struct url *u, struct fileinfo *f, ccon *con, char **local_fi
       switch (err)
         {
         case HOSTERR: case CONIMPOSSIBLE: case FWRITEERR: case FOPENERR:
-        case FTPNSFOD: case FTPLOGINC: case FTPNOPASV: case CONTNOTSUPPORTED:
-        case UNLINKERR: case WARC_TMP_FWRITEERR:
+        case FTPNSFOD: case FTPLOGINC: case FTPNOPASV: case FTPNOAUTH: case FTPNOPBSZ: case FTPNOPROT:
+        case UNLINKERR: case WARC_TMP_FWRITEERR: case CONSSLERR: case CONTNOTSUPPORTED:
+#ifdef HAVE_SSL
+          if (err == FTPNOAUTH)
+            logputs (LOG_NOTQUIET, "Server does not support AUTH TLS.\n");
+          if (opt.ftps_implicit)
+            logputs (LOG_NOTQUIET, "Server does not like implicit FTPS connections.\n");
+#endif
           /* Fatal errors, give up.  */
           if (warc_tmp != NULL)
-            fclose (warc_tmp);
+              fclose (warc_tmp);
           return err;
         case CONSOCKERR: case CONERROR: case FTPSRVERR: case FTPRERR:
         case WRITEFAILED: case FTPUNKNOWNTYPE: case FTPSYSERR:
@@ -1771,10 +2020,12 @@ ftp_loop_internal (struct url *u, struct fileinfo *f, ccon *con, char **local_fi
 
           warc_res = warc_write_resource_record (NULL, u->url, NULL, NULL,
                                                   warc_ip, NULL, warc_tmp, -1);
+
           if (! warc_res)
             return WARC_ERR;
 
           /* warc_write_resource_record has also closed warc_tmp. */
+          warc_tmp = NULL;
         }
 
       if (con->cmd & DO_LIST)
@@ -1821,6 +2072,9 @@ Removing file due to --delete-after in ftp_loop_internal():\n"));
       if (local_file)
         *local_file = xstrdup (locf);
 
+      if (warc_tmp != NULL)
+        fclose (warc_tmp);
+
       return RETROK;
     } while (!opt.ntry || (count < opt.ntry));
 
@@ -1829,13 +2083,18 @@ Removing file due to --delete-after in ftp_loop_internal():\n"));
       fd_close (con->csock);
       con->csock = -1;
     }
+
+  if (warc_tmp != NULL)
+    fclose (warc_tmp);
+
   return TRYLIMEXC;
 }
 
 /* Return the directory listing in a reusable format.  The directory
    is specifed in u->dir.  */
 static uerr_t
-ftp_get_listing (struct url *u, ccon *con, struct fileinfo **f)
+ftp_get_listing (struct url *u, struct url *original_url, ccon *con,
+                 struct fileinfo **f)
 {
   uerr_t err;
   char *uf;                     /* url file name */
@@ -1856,7 +2115,7 @@ ftp_get_listing (struct url *u, ccon *con, struct fileinfo **f)
 
   con->target = xstrdup (lf);
   xfree (lf);
-  err = ftp_loop_internal (u, NULL, con, NULL);
+  err = ftp_loop_internal (u, original_url, NULL, con, NULL, false);
   lf = xstrdup (con->target);
   xfree (con->target);
   con->target = old_target;
@@ -1879,8 +2138,9 @@ ftp_get_listing (struct url *u, ccon *con, struct fileinfo **f)
   return err;
 }
 
-static uerr_t ftp_retrieve_dirs (struct url *, struct fileinfo *, ccon *);
-static uerr_t ftp_retrieve_glob (struct url *, ccon *, int);
+static uerr_t ftp_retrieve_dirs (struct url *, struct url *,
+                                 struct fileinfo *, ccon *);
+static uerr_t ftp_retrieve_glob (struct url *, struct url *, ccon *, int);
 static struct fileinfo *delelement (struct fileinfo *, struct fileinfo **);
 static void freefileinfo (struct fileinfo *f);
 
@@ -1892,7 +2152,8 @@ static void freefileinfo (struct fileinfo *f);
    If opt.recursive is set, after all files have been retrieved,
    ftp_retrieve_dirs will be called to retrieve the directories.  */
 static uerr_t
-ftp_retrieve_list (struct url *u, struct fileinfo *f, ccon *con)
+ftp_retrieve_list (struct url *u, struct url *original_url,
+                   struct fileinfo *f, ccon *con)
 {
   static int depth = 0;
   uerr_t err;
@@ -1901,6 +2162,7 @@ ftp_retrieve_list (struct url *u, struct fileinfo *f, ccon *con)
   time_t tml;
   bool dlthis; /* Download this (file). */
   const char *actual_target = NULL;
+  bool force_full_retrieve = false;
 
   /* Increase the depth.  */
   ++depth;
@@ -1980,9 +2242,10 @@ ftp_retrieve_list (struct url *u, struct fileinfo *f, ccon *con)
 Remote file no newer than local file %s -- not retrieving.\n"), quote (con->target));
                   dlthis = false;
                 }
-              else if (eq_size)
+              else if (f->tstamp > tml)
                 {
-                  /* Remote file is newer or sizes cannot be matched */
+                  /* Remote file is newer */
+                  force_full_retrieve = true;
                   logprintf (LOG_VERBOSE, _("\
 Remote file is newer than local file %s -- retrieving.\n\n"),
                              quote (con->target));
@@ -2051,7 +2314,10 @@ Already have correct symlink %s -> %s\n\n"),
           else                /* opt.retr_symlinks */
             {
               if (dlthis)
-                err = ftp_loop_internal (u, f, con, NULL);
+                {
+                  err = ftp_loop_internal (u, original_url, f, con, NULL,
+                                           force_full_retrieve);
+                }
             } /* opt.retr_symlinks */
           break;
         case FT_DIRECTORY:
@@ -2062,7 +2328,10 @@ Already have correct symlink %s -> %s\n\n"),
         case FT_PLAINFILE:
           /* Call the retrieve loop.  */
           if (dlthis)
-            err = ftp_loop_internal (u, f, con, NULL);
+            {
+              err = ftp_loop_internal (u, original_url, f, con, NULL,
+                                       force_full_retrieve);
+            }
           break;
         case FT_UNKNOWN:
           logprintf (LOG_NOTQUIET, _("%s: unknown/unsupported file type.\n"),
@@ -2127,7 +2396,7 @@ Already have correct symlink %s -> %s\n\n"),
   /* We do not want to call ftp_retrieve_dirs here */
   if (opt.recursive &&
       !(opt.reclevel != INFINITE_RECURSION && depth >= opt.reclevel))
-    err = ftp_retrieve_dirs (u, orig, con);
+    err = ftp_retrieve_dirs (u, original_url, orig, con);
   else if (opt.recursive)
     DEBUGP ((_("Will not retrieve dirs since depth is %d (max %d).\n"),
              depth, opt.reclevel));
@@ -2140,7 +2409,8 @@ Already have correct symlink %s -> %s\n\n"),
    ftp_retrieve_glob on each directory entry.  The function knows
    about excluded directories.  */
 static uerr_t
-ftp_retrieve_dirs (struct url *u, struct fileinfo *f, ccon *con)
+ftp_retrieve_dirs (struct url *u, struct url *original_url,
+                   struct fileinfo *f, ccon *con)
 {
   char *container = NULL;
   int container_size = 0;
@@ -2190,7 +2460,7 @@ Not descending to %s as it is excluded/not-included.\n"),
       odir = xstrdup (u->dir);  /* because url_set_dir will free
                                    u->dir. */
       url_set_dir (u, newdir);
-      ftp_retrieve_glob (u, con, GLOB_GETALL);
+      ftp_retrieve_glob (u, original_url, con, GLOB_GETALL);
       url_set_dir (u, odir);
       xfree (odir);
 
@@ -2249,14 +2519,15 @@ is_invalid_entry (struct fileinfo *f)
    GLOB_GLOBALL, use globbing; if it's GLOB_GETALL, download the whole
    directory.  */
 static uerr_t
-ftp_retrieve_glob (struct url *u, ccon *con, int action)
+ftp_retrieve_glob (struct url *u, struct url *original_url,
+                   ccon *con, int action)
 {
   struct fileinfo *f, *start;
   uerr_t res;
 
   con->cmd |= LEAVE_PENDING;
 
-  res = ftp_get_listing (u, con, &start);
+  res = ftp_get_listing (u, original_url, con, &start);
   if (res != RETROK)
     return res;
   /* First: weed out that do not conform the global rules given in
@@ -2352,7 +2623,7 @@ ftp_retrieve_glob (struct url *u, ccon *con, int action)
   if (start)
     {
       /* Just get everything.  */
-      res = ftp_retrieve_list (u, start, con);
+      res = ftp_retrieve_list (u, original_url, start, con);
     }
   else
     {
@@ -2368,7 +2639,7 @@ ftp_retrieve_glob (struct url *u, ccon *con, int action)
         {
           /* Let's try retrieving it anyway.  */
           con->st |= ON_YOUR_OWN;
-          res = ftp_loop_internal (u, NULL, con, NULL);
+          res = ftp_loop_internal (u, original_url, NULL, con, NULL, false);
           return res;
         }
 
@@ -2388,8 +2659,8 @@ ftp_retrieve_glob (struct url *u, ccon *con, int action)
    of URL.  Inherently, its capabilities are limited on what can be
    encoded into a URL.  */
 uerr_t
-ftp_loop (struct url *u, char **local_file, int *dt, struct url *proxy,
-          bool recursive, bool glob)
+ftp_loop (struct url *u, struct url *original_url, char **local_file, int *dt,
+          struct url *proxy, bool recursive, bool glob)
 {
   ccon con;                     /* FTP connection */
   uerr_t res;
@@ -2410,16 +2681,17 @@ ftp_loop (struct url *u, char **local_file, int *dt, struct url *proxy,
   if (!*u->file && !recursive)
     {
       struct fileinfo *f;
-      res = ftp_get_listing (u, &con, &f);
+      res = ftp_get_listing (u, original_url, &con, &f);
 
       if (res == RETROK)
         {
           if (opt.htmlify && !opt.spider)
             {
+              struct url *url_file = opt.trustservernames ? u : original_url;
               char *filename = (opt.output_document
                                 ? xstrdup (opt.output_document)
                                 : (con.target ? xstrdup (con.target)
-                                   : url_file_name (u, NULL)));
+                                   : url_file_name (url_file, NULL)));
               res = ftp_index (filename, u, f);
               if (res == FTPOK && opt.verbose)
                 {
@@ -2464,11 +2736,13 @@ ftp_loop (struct url *u, char **local_file, int *dt, struct url *proxy,
           /* ftp_retrieve_glob is a catch-all function that gets called
              if we need globbing, time-stamping, recursion or preserve
              permissions.  Its third argument is just what we really need.  */
-          res = ftp_retrieve_glob (u, &con,
+          res = ftp_retrieve_glob (u, original_url, &con,
                                    ispattern ? GLOB_GLOBALL : GLOB_GETONE);
         }
       else
-        res = ftp_loop_internal (u, NULL, &con, local_file);
+        {
+          res = ftp_loop_internal (u, original_url, NULL, &con, local_file, false);
+        }
     }
   if (res == FTPOK)
     res = RETROK;

src/ftp.h

@@ -1,7 +1,7 @@
 /* Declarations for FTP support.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -33,6 +33,7 @@ as that of the covered work.  */
 #define FTP_H
 
 #include "host.h"
+#include "url.h"
 
 /* System types. */
 enum stype
@@ -53,10 +54,27 @@ enum ustype
   UST_OTHER
 };
 
+#ifdef HAVE_SSL
+/* Data channel protection levels (to be used with PBSZ) */
+enum prot_level
+{
+  PROT_CLEAR = 'C',
+  PROT_SAFE = 'S',
+  PROT_CONFIDENTIAL = 'E',
+  PROT_PRIVATE = 'P'
+};
+#endif
+
 uerr_t ftp_response (int, char **);
+uerr_t ftp_greeting (int);
 uerr_t ftp_login (int, const char *, const char *);
 uerr_t ftp_port (int, int *);
 uerr_t ftp_pasv (int, ip_address *, int *);
+#ifdef HAVE_SSL
+uerr_t ftp_auth (int, enum url_scheme);
+uerr_t ftp_pbsz (int, int);
+uerr_t ftp_prot (int, enum prot_level);
+#endif
 #ifdef ENABLE_IPV6
 uerr_t ftp_lprt (int, int *);
 uerr_t ftp_lpsv (int, ip_address *, int *);
@@ -142,15 +160,17 @@ enum wget_ftp_fstatus
   AVOID_LIST    = 0x0008,   /* It tells us if during this
                                session we have to avoid to use
                                "LIST". */
-  LIST_AFTER_LIST_A_CHECK_DONE  = 0x0010
+  LIST_AFTER_LIST_A_CHECK_DONE  = 0x0010,
                             /* It tells us if we have already
                                checked "LIST" after the first
                                "LIST -a" to handle the case of
                                file/folders named "-a". */
+  DATA_CHANNEL_SECURITY = 0x0020 /* Establish a secure data channel */
 };
 
 struct fileinfo *ftp_parse_ls (const char *, const enum stype);
-uerr_t ftp_loop (struct url *, char **, int *, struct url *, bool, bool);
+uerr_t ftp_loop (struct url *, struct url *, char **, int *, struct url *,
+                 bool, bool);
 
 uerr_t ftp_index (const char *, struct url *, struct fileinfo *);
 

src/gnutls.c

@@ -1,6 +1,6 @@
 /* SSL support via GnuTLS library.
-   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 Free Software
-   Foundation, Inc.
+   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2015
+   Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -36,7 +36,9 @@ as that of the covered work.  */
 #include <stdio.h>
 #include <dirent.h>
 #include <stdlib.h>
+#include <xalloc.h>
 
+#include <gnutls/abstract.h>
 #include <gnutls/gnutls.h>
 #include <gnutls/x509.h>
 #include <sys/ioctl.h>
@@ -56,6 +58,9 @@ as that of the covered work.  */
 
 #include "host.h"
 
+static int
+_do_handshake (gnutls_session_t session, int fd, double timeout);
+
 static int
 key_type_to_gnutls_type (enum keyfile_type type)
 {
@@ -184,13 +189,13 @@ ssl_init (void)
   /* Use the private key from the cert file unless otherwise specified. */
   if (opt.cert_file && !opt.private_key)
     {
-      opt.private_key = opt.cert_file;
+      opt.private_key = xstrdup (opt.cert_file);
       opt.private_key_type = opt.cert_type;
     }
   /* Use the cert from the private key file unless otherwise specified. */
   if (!opt.cert_file && opt.private_key)
     {
-      opt.cert_file = opt.private_key;
+      opt.cert_file = xstrdup (opt.private_key);
       opt.cert_type = opt.private_key_type;
     }
 
@@ -219,6 +224,7 @@ cert to be of the same type.\n"));
 struct wgnutls_transport_context
 {
   gnutls_session_t session;       /* GnuTLS session handle */
+  gnutls_datum_t *session_data;
   int last_error;               /* last error returned by read/write/... */
 
   /* Since GnuTLS doesn't support the equivalent to recv(...,
@@ -276,6 +282,12 @@ wgnutls_read_timeout (int fd, char *buf, int bufsize, void *arg, double timeout)
         {
           ret = gnutls_record_recv (ctx->session, buf, bufsize);
           timed_out = timeout && ptimer_measure (timer) >= timeout;
+          if (!timed_out && ret == GNUTLS_E_REHANDSHAKE)
+            {
+              DEBUGP (("GnuTLS: *** REHANDSHAKE while reading\n"));
+              if ((ret = _do_handshake (ctx->session, fd, timeout)) == 0)
+                ret = GNUTLS_E_AGAIN; /* restart reading */
+            }
         }
     }
   while (ret == GNUTLS_E_INTERRUPTED || (ret == GNUTLS_E_AGAIN && !timed_out));
@@ -405,6 +417,11 @@ wgnutls_close (int fd, void *arg)
 {
   struct wgnutls_transport_context *ctx = arg;
   /*gnutls_bye (ctx->session, GNUTLS_SHUT_RDWR);*/
+  if (ctx->session_data)
+    {
+      gnutls_free (ctx->session_data->data);
+      gnutls_free (ctx->session_data);
+    }
   gnutls_deinit (ctx->session);
   xfree (ctx);
   close (fd);
@@ -419,24 +436,124 @@ static struct transport_implementation wgnutls_transport =
   wgnutls_peek, wgnutls_errstr, wgnutls_close
 };
 
-bool
-ssl_connect_wget (int fd, const char *hostname)
+static int
+_do_handshake (gnutls_session_t session, int fd, double timeout)
 {
 #ifdef F_GETFL
   int flags = 0;
 #endif
+  int err;
+
+  if (timeout)
+    {
+#ifdef F_GETFL
+      flags = fcntl (fd, F_GETFL, 0);
+      if (flags < 0)
+        return flags;
+      if (fcntl (fd, F_SETFL, flags | O_NONBLOCK))
+        return -1;
+#else
+      /* XXX: Assume it was blocking before.  */
+      const int one = 1;
+      if (ioctl (fd, FIONBIO, &one) < 0)
+        return -1;
+#endif
+    }
+
+  /* We don't stop the handshake process for non-fatal errors */
+  do
+    {
+      err = gnutls_handshake (session);
+
+      if (timeout && err == GNUTLS_E_AGAIN)
+        {
+          if (gnutls_record_get_direction (session))
+            {
+              /* wait for writeability */
+              err = select_fd (fd, timeout, WAIT_FOR_WRITE);
+            }
+          else
+            {
+              /* wait for readability */
+              err = select_fd (fd, timeout, WAIT_FOR_READ);
+            }
+
+          if (err <= 0)
+            {
+              if (err == 0)
+                {
+                  errno = ETIMEDOUT;
+                  err = -1;
+                }
+              break;
+            }
+
+           err = GNUTLS_E_AGAIN;
+        }
+      else if (err < 0)
+        {
+          logprintf (LOG_NOTQUIET, "GnuTLS: %s\n", gnutls_strerror (err));
+          if (err == GNUTLS_E_WARNING_ALERT_RECEIVED ||
+              err == GNUTLS_E_FATAL_ALERT_RECEIVED)
+            {
+              gnutls_alert_description_t alert = gnutls_alert_get (session);
+              const char *str = gnutls_alert_get_name (alert);
+              logprintf (LOG_NOTQUIET, "GnuTLS: received alert [%d]: %s\n",
+                         alert, str ? str : "(unknown)");
+            }
+        }
+    }
+  while (err && gnutls_error_is_fatal (err) == 0);
+
+  if (timeout)
+    {
+#ifdef F_GETFL
+      if (fcntl (fd, F_SETFL, flags) < 0)
+        return -1;
+#else
+      const int zero = 0;
+      if (ioctl (fd, FIONBIO, &zero) < 0)
+        return -1;
+#endif
+    }
+
+  return err;
+}
+
+static const char *
+_sni_hostname(const char *hostname)
+{
+  size_t len = strlen(hostname);
+
+  char *sni_hostname = xmemdup(hostname, len + 1);
+
+  /* Remove trailing dot(s) to fix #47408.
+   * Regarding RFC 6066 (SNI): The hostname is represented as a byte
+   * string using ASCII encoding without a trailing dot. */
+  while (len && sni_hostname[--len] == '.')
+    sni_hostname[len] = 0;
+
+  return sni_hostname;
+}
+
+bool
+ssl_connect_wget (int fd, const char *hostname, int *continue_session)
+{
   struct wgnutls_transport_context *ctx;
   gnutls_session_t session;
   int err;
-  const char *str;
 
   gnutls_init (&session, GNUTLS_CLIENT);
 
   /* We set the server name but only if it's not an IP address. */
   if (! is_valid_ip_address (hostname))
     {
-      gnutls_server_name_set (session, GNUTLS_NAME_DNS, hostname,
-                              strlen (hostname));
+      /* GnuTLS 3.4.x (x<=10) disrespects the length parameter, we have to construct a new string */
+      /* see https://gitlab.com/gnutls/gnutls/issues/78 */
+      const char *sni_hostname = _sni_hostname(hostname);
+
+      gnutls_server_name_set (session, GNUTLS_NAME_DNS, sni_hostname, strlen(sni_hostname));
+      xfree(sni_hostname);
     }
 
   gnutls_set_default_priority (session);
@@ -531,79 +648,32 @@ ssl_connect_wget (int fd, const char *hostname)
       return false;
     }
 
-  if (opt.connect_timeout)
+  if (continue_session)
     {
-#ifdef F_GETFL
-      flags = fcntl (fd, F_GETFL, 0);
-      if (flags < 0)
-        return flags;
-      if (fcntl (fd, F_SETFL, flags | O_NONBLOCK))
-        return -1;
-#else
-      /* XXX: Assume it was blocking before.  */
-      const int one = 1;
-      if (ioctl (fd, FIONBIO, &one) < 0)
-        return -1;
-#endif
-    }
-
-  /* We don't stop the handshake process for non-fatal errors */
-  do
-    {
-      err = gnutls_handshake (session);
-
-      if (opt.connect_timeout && err == GNUTLS_E_AGAIN)
+      ctx = (struct wgnutls_transport_context *) fd_transport_context (*continue_session);
+      if (!gnutls_session_is_resumed (session))
         {
-          if (gnutls_record_get_direction (session))
+          if (!ctx || !ctx->session_data || gnutls_session_set_data (session, ctx->session_data->data, ctx->session_data->size))
             {
-              /* wait for writeability */
-              err = select_fd (fd, opt.connect_timeout, WAIT_FOR_WRITE);
-            }
-          else
-            {
-              /* wait for readability */
-              err = select_fd (fd, opt.connect_timeout, WAIT_FOR_READ);
-            }
-
-          if (err <= 0)
-            {
-              if (err == 0)
+              if (ctx && ctx->session_data)
                 {
-                  errno = ETIMEDOUT;
-                  err = -1;
+                  /* server does not want to continue the session */
+                  if (ctx->session_data->data)
+                    gnutls_free (ctx->session_data->data);
+                  gnutls_free (ctx->session_data);
                 }
-              break;
+              gnutls_deinit (session);
+              return false;
             }
-
-           err = GNUTLS_E_AGAIN;
         }
-      else if (err < 0)
+      else
         {
-          logprintf (LOG_NOTQUIET, "GnuTLS: %s\n", gnutls_strerror (err));
-          if (err == GNUTLS_E_WARNING_ALERT_RECEIVED ||
-              err == GNUTLS_E_FATAL_ALERT_RECEIVED)
-            {
-              gnutls_alert_description_t alert = gnutls_alert_get (session);
-              str = gnutls_alert_get_name (alert);
-              if (str == NULL)
-                str = "(unknown)";
-              logprintf (LOG_NOTQUIET, "GnuTLS: received alert [%d]: %s\n", alert, str);
-            }
+          logputs (LOG_ALWAYS, "SSL session has already been resumed. Continuing.\n");
+          continue_session = NULL;
         }
     }
-  while (err && gnutls_error_is_fatal (err) == 0);
 
-  if (opt.connect_timeout)
-    {
-#ifdef F_GETFL
-      if (fcntl (fd, F_SETFL, flags) < 0)
-        return -1;
-#else
-      const int zero = 0;
-      if (ioctl (fd, FIONBIO, &zero) < 0)
-        return -1;
-#endif
-    }
+  err = _do_handshake (session, fd, opt.connect_timeout);
 
   if (err < 0)
     {
@@ -612,11 +682,70 @@ ssl_connect_wget (int fd, const char *hostname)
     }
 
   ctx = xnew0 (struct wgnutls_transport_context);
+  ctx->session_data = xnew0 (gnutls_datum_t);
   ctx->session = session;
+  if (gnutls_session_get_data2 (session, ctx->session_data))
+    {
+      xfree (ctx->session_data);
+      logprintf (LOG_NOTQUIET, "WARNING: Could not save SSL session data for socket %d\n", fd);
+    }
   fd_register_transport (fd, &wgnutls_transport, ctx);
   return true;
 }
 
+static bool
+pkp_pin_peer_pubkey (gnutls_x509_crt_t cert, const char *pinnedpubkey)
+{
+  /* Scratch */
+  size_t len1 = 0, len2 = 0;
+  char *buff1 = NULL;
+
+  gnutls_pubkey_t key = NULL;
+
+  /* Result is returned to caller */
+  int ret = 0;
+  bool result = false;
+
+  /* if a path wasn't specified, don't pin */
+  if (NULL == pinnedpubkey)
+    return true;
+
+  if (NULL == cert)
+    return result;
+
+  /* Begin Gyrations to get the public key     */
+  gnutls_pubkey_init (&key);
+
+  ret = gnutls_pubkey_import_x509 (key, cert, 0);
+  if (ret < 0)
+    goto cleanup; /* failed */
+
+  ret = gnutls_pubkey_export (key, GNUTLS_X509_FMT_DER, NULL, &len1);
+  if (ret != GNUTLS_E_SHORT_MEMORY_BUFFER || len1 == 0)
+    goto cleanup; /* failed */
+
+  buff1 = xmalloc (len1);
+
+  len2 = len1;
+
+  ret = gnutls_pubkey_export (key, GNUTLS_X509_FMT_DER, buff1, &len2);
+  if (ret < 0 || len1 != len2)
+    goto cleanup; /* failed */
+
+  /* End Gyrations */
+
+  /* The one good exit point */
+  result = wg_pin_peer_pubkey (pinnedpubkey, buff1, len1);
+
+ cleanup:
+  if (NULL != key)
+    gnutls_pubkey_deinit (key);
+
+  xfree (buff1);
+
+  return result;
+}
+
 #define _CHECK_CERT(flag,msg) \
   if (status & (flag))\
     {\
@@ -637,6 +766,11 @@ ssl_check_certificate (int fd, const char *host)
      him about problems with the server's certificate.  */
   const char *severity = opt.check_cert ? _("ERROR") : _("WARNING");
   bool success = true;
+  bool pinsuccess = opt.pinnedpubkey == NULL;
+
+  /* The user explicitly said to not check for the certificate.  */
+  if (opt.check_cert == CHECK_CERT_QUIET && pinsuccess)
+    return success;
 
   err = gnutls_certificate_verify_peers2 (ctx->session, &status);
   if (err < 0)
@@ -661,6 +795,7 @@ ssl_check_certificate (int fd, const char *host)
       gnutls_x509_crt_t cert;
       const gnutls_datum_t *cert_list;
       unsigned int cert_list_size;
+      const char *sni_hostname;
 
       if ((err = gnutls_x509_crt_init (&cert)) < 0)
         {
@@ -695,13 +830,23 @@ ssl_check_certificate (int fd, const char *host)
           logprintf (LOG_NOTQUIET, _("The certificate has expired\n"));
           success = false;
         }
-      if (!gnutls_x509_crt_check_hostname (cert, host))
+      sni_hostname = _sni_hostname(host);
+      if (!gnutls_x509_crt_check_hostname (cert, sni_hostname))
         {
           logprintf (LOG_NOTQUIET,
                      _("The certificate's owner does not match hostname %s\n"),
-                     quote (host));
+                     quote (sni_hostname));
           success = false;
         }
+      xfree(sni_hostname);
+
+      pinsuccess = pkp_pin_peer_pubkey (cert, opt.pinnedpubkey);
+      if (!pinsuccess)
+        {
+          logprintf (LOG_ALWAYS, _("The public key does not match pinned public key!\n"));
+          success = false;
+        }
+
  crt_deinit:
       gnutls_x509_crt_deinit (cert);
     }
@@ -712,5 +857,6 @@ ssl_check_certificate (int fd, const char *host)
     }
 
  out:
-  return opt.check_cert ? success : true;
+  /* never return true if pinsuccess fails */
+  return !pinsuccess ? false : (opt.check_cert == CHECK_CERT_ON ? success : true);
 }

src/hash.c

@@ -1,6 +1,6 @@
 /* Hash tables.
    Copyright (C) 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008,
-   2009, 2010, 2011 Free Software Foundation, Inc.
+   2009, 2010, 2011, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/hash.h

@@ -1,5 +1,5 @@
 /* Hash table declarations.
-   Copyright (C) 2000, 2007, 2008, 2009, 2010, 2011 Free Software
+   Copyright (C) 2000, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
    Foundation, Inc.
 
 This file is part of GNU Wget.

src/host.c

@@ -1,7 +1,7 @@
 /* Host name resolution and matching.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -65,6 +65,7 @@ as that of the covered work.  */
 #include "host.h"
 #include "url.h"
 #include "hash.h"
+#include "ptimer.h"
 
 #ifndef NO_ADDRESS
 # define NO_ADDRESS NO_DATA
@@ -649,6 +650,112 @@ cache_remove (const char *host)
     }
 }
 
+#ifdef HAVE_LIBCARES
+#include <sys/select.h>
+#include <ares.h>
+extern ares_channel ares;
+
+static struct address_list *
+merge_address_lists (struct address_list *al1, struct address_list *al2)
+{
+  int count = al1->count + al2->count;
+
+  /* merge al2 into al1 */
+  al1->addresses = xrealloc (al1->addresses, sizeof (ip_address) * count);
+  memcpy (al1->addresses + al1->count, al2->addresses, sizeof (ip_address) * al2->count);
+  al1->count = count;
+
+  address_list_delete (al2);
+
+  return al1;
+}
+
+static struct address_list *
+address_list_from_hostent (struct hostent *host)
+{
+  int count, i;
+  struct address_list *al = xnew0 (struct address_list);
+
+  for (count = 0; host->h_addr_list[count]; count++)
+    ;
+
+  assert (count > 0);
+
+  al->addresses = xnew_array (ip_address, count);
+  al->count     = count;
+  al->refcount  = 1;
+
+  for (i = 0; i < count; i++)
+    {
+      ip_address *ip = &al->addresses[i];
+      ip->family = host->h_addrtype;
+      memcpy (IP_INADDR_DATA (ip), host->h_addr_list[i], ip->family == AF_INET ? 4 : 16);
+    }
+
+  return al;
+}
+
+/* Since GnuLib's select() (i.e. rpl_select()) cannot handle socket-numbers
+ * returned from C-ares, we must use the original select() from Winsock.
+ */
+#ifdef WINDOWS
+#undef select
+#endif
+
+static void
+wait_ares (ares_channel channel)
+{
+  struct ptimer *timer = NULL;
+
+  if (opt.dns_timeout)
+    timer = ptimer_new ();
+
+  for (;;)
+    {
+      struct timeval *tvp, tv;
+      fd_set read_fds, write_fds;
+      int nfds, rc;
+
+      FD_ZERO (&read_fds);
+      FD_ZERO (&write_fds);
+      nfds = ares_fds (channel, &read_fds, &write_fds);
+      if (nfds == 0)
+        break;
+
+      if (timer)
+        {
+          double max = opt.dns_timeout - ptimer_measure (timer);
+
+          tv.tv_sec = (long) max;
+          tv.tv_usec = 1000000 * (max - (long) max);
+          tvp = ares_timeout (channel, &tv, &tv);
+        }
+      else
+        tvp = ares_timeout (channel, NULL, &tv);
+
+      rc = select (nfds, &read_fds, &write_fds, NULL, tvp);
+      if (rc == 0 && timer && ptimer_measure (timer) >= opt.dns_timeout)
+        ares_cancel (channel);
+      else
+        ares_process (channel, &read_fds, &write_fds);
+    }
+}
+
+static void
+callback (void *arg, int status, int timeouts _GL_UNUSED, struct hostent *host)
+{
+  struct address_list **al = (struct address_list **) arg;
+
+  if (!host || status != ARES_SUCCESS)
+    {
+      *al = NULL;
+      return;
+    }
+
+  *al = address_list_from_hostent (host);
+}
+#endif
+
 /* Look up HOST in DNS and return a list of IP addresses.
 
    This function caches its result so that, if the same host is passed
@@ -755,80 +862,112 @@ lookup_host (const char *host, int flags)
     }
 
 #ifdef ENABLE_IPV6
-  {
-    int err;
-    struct addrinfo hints, *res;
+#ifdef HAVE_LIBCARES
+  if (ares)
+    {
+      struct address_list *al4;
+      struct address_list *al6;
 
-    xzero (hints);
-    hints.ai_socktype = SOCK_STREAM;
-    if (opt.ipv4_only)
-      hints.ai_family = AF_INET;
-    else if (opt.ipv6_only)
-      hints.ai_family = AF_INET6;
-    else
-      /* We tried using AI_ADDRCONFIG, but removed it because: it
-         misinterprets IPv6 loopbacks, it is broken on AIX 5.1, and
-         it's unneeded since we sort the addresses anyway.  */
+      if (opt.ipv4_only || !opt.ipv6_only)
+        ares_gethostbyname (ares, host, AF_INET, callback, &al4);
+      if (opt.ipv6_only || !opt.ipv4_only)
+        ares_gethostbyname (ares, host, AF_INET6, callback, &al6);
+
+      wait_ares (ares);
+
+      if (al4 && al6)
+        al = merge_address_lists (al4, al6);
+      else if (al4)
+        al = al4;
+      else
+        al = al6;
+    }
+  else
+#endif
+    {
+      int err;
+      struct addrinfo hints, *res;
+
+      xzero (hints);
+      hints.ai_socktype = SOCK_STREAM;
+      if (opt.ipv4_only)
+        hints.ai_family = AF_INET;
+      else if (opt.ipv6_only)
+        hints.ai_family = AF_INET6;
+      else
+        /* We tried using AI_ADDRCONFIG, but removed it because: it
+           misinterprets IPv6 loopbacks, it is broken on AIX 5.1, and
+           it's unneeded since we sort the addresses anyway.  */
         hints.ai_family = AF_UNSPEC;
 
-    if (flags & LH_BIND)
-      hints.ai_flags |= AI_PASSIVE;
+      if (flags & LH_BIND)
+        hints.ai_flags |= AI_PASSIVE;
 
 #ifdef AI_NUMERICHOST
-    if (numeric_address)
-      {
-        /* Where available, the AI_NUMERICHOST hint can prevent costly
-           access to DNS servers.  */
-        hints.ai_flags |= AI_NUMERICHOST;
-        timeout = 0;            /* no timeout needed when "resolving"
+      if (numeric_address)
+        {
+          /* Where available, the AI_NUMERICHOST hint can prevent costly
+             access to DNS servers.  */
+          hints.ai_flags |= AI_NUMERICHOST;
+          timeout = 0; /* no timeout needed when "resolving"
                                    numeric hosts -- avoid setting up
                                    signal handlers and such. */
-      }
+        }
 #endif
 
-    err = getaddrinfo_with_timeout (host, NULL, &hints, &res, timeout);
-    if (err != 0 || res == NULL)
-      {
-        if (!silent)
-          logprintf (LOG_VERBOSE, _("failed: %s.\n"),
-                     err != EAI_SYSTEM ? gai_strerror (err) : strerror (errno));
-        return NULL;
-      }
-    al = address_list_from_addrinfo (res);
-    freeaddrinfo (res);
-    if (!al)
-      {
-        logprintf (LOG_VERBOSE,
-                   _("failed: No IPv4/IPv6 addresses for host.\n"));
-        return NULL;
-      }
+      err = getaddrinfo_with_timeout (host, NULL, &hints, &res, timeout);
 
-    /* Reorder addresses so that IPv4 ones (or IPv6 ones, as per
-       --prefer-family) come first.  Sorting is stable so the order of
-       the addresses with the same family is undisturbed.  */
-    if (al->count > 1 && opt.prefer_family != prefer_none)
-      stable_sort (al->addresses, al->count, sizeof (ip_address),
-                   opt.prefer_family == prefer_ipv4
-                   ? cmp_prefer_ipv4 : cmp_prefer_ipv6);
-  }
+      if (err != 0 || res == NULL)
+        {
+          if (!silent)
+            logprintf (LOG_VERBOSE, _ ("failed: %s.\n"),
+                       err != EAI_SYSTEM ? gai_strerror (err) : strerror (errno));
+          return NULL;
+        }
+      al = address_list_from_addrinfo (res);
+      freeaddrinfo (res);
+    }
+
+  if (!al)
+    {
+      logprintf (LOG_VERBOSE,
+                 _ ("failed: No IPv4/IPv6 addresses for host.\n"));
+      return NULL;
+    }
+
+  /* Reorder addresses so that IPv4 ones (or IPv6 ones, as per
+     --prefer-family) come first.  Sorting is stable so the order of
+     the addresses with the same family is undisturbed.  */
+  if (al->count > 1 && opt.prefer_family != prefer_none)
+    stable_sort (al->addresses, al->count, sizeof (ip_address),
+                 opt.prefer_family == prefer_ipv4
+                 ? cmp_prefer_ipv4 : cmp_prefer_ipv6);
 #else  /* not ENABLE_IPV6 */
-  {
-    struct hostent *hptr = gethostbyname_with_timeout (host, timeout);
-    if (!hptr)
-      {
-        if (!silent)
-          {
-            if (errno != ETIMEDOUT)
-              logprintf (LOG_VERBOSE, _("failed: %s.\n"),
-                         host_errstr (h_errno));
-            else
-              logputs (LOG_VERBOSE, _("failed: timed out.\n"));
-          }
-        return NULL;
-      }
-    /* Do older systems have h_addr_list?  */
-    al = address_list_from_ipv4_addresses (hptr->h_addr_list);
-  }
+#ifdef HAVE_LIBCARES
+  if (ares)
+    {
+      ares_gethostbyname (ares, host, AF_INET, callback, &al);
+      wait_ares (ares);
+    }
+  else
+#endif
+    {
+      struct hostent *hptr = gethostbyname_with_timeout (host, timeout);
+      if (!hptr)
+        {
+          if (!silent)
+            {
+              if (errno != ETIMEDOUT)
+                logprintf (LOG_VERBOSE, _ ("failed: %s.\n"),
+                           host_errstr (h_errno));
+              else
+                logputs (LOG_VERBOSE, _ ("failed: timed out.\n"));
+            }
+          return NULL;
+        }
+      /* Do older systems have h_addr_list?  */
+      al = address_list_from_ipv4_addresses (hptr->h_addr_list);
+    }
 #endif /* not ENABLE_IPV6 */
 
   /* Print the addresses determined by DNS lookup, but no more than

src/host.h

@@ -1,7 +1,7 @@
 /* Declarations for host.c
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/hsts.c

@@ -0,0 +1,819 @@
+/* HTTP Strict Transport Security (HSTS) support.
+   Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2015 Free Software
+   Foundation, Inc.
+
+This file is part of GNU Wget.
+
+GNU Wget is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 3 of the License, or
+ (at your option) any later version.
+
+GNU Wget is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Wget.  If not, see <http://www.gnu.org/licenses/>.
+
+Additional permission under GNU GPL version 3 section 7
+
+If you modify this program, or any covered work, by linking or
+combining it with the OpenSSL project's OpenSSL library (or a
+modified version of that library), containing parts covered by the
+terms of the OpenSSL or SSLeay licenses, the Free Software Foundation
+grants you additional permission to convey the resulting work.
+Corresponding Source for a non-source form of such a combination
+shall include the source code for the parts of OpenSSL used as well
+as that of the covered work.  */
+#include "wget.h"
+
+#ifdef HAVE_HSTS
+#include "hsts.h"
+#include "host.h" /* for is_valid_ip_address() */
+#include "init.h" /* for home_dir() */
+#include "utils.h"
+#include "hash.h"
+#include "c-ctype.h"
+#ifdef TESTING
+#include "test.h"
+#endif
+
+#include <unistd.h>
+#include <sys/types.h>
+#include <stdlib.h>
+#include <time.h>
+#include <sys/stat.h>
+#include <string.h>
+#include <stdio.h>
+#include <sys/file.h>
+
+struct hsts_store {
+  struct hash_table *table;
+  time_t last_mtime;
+  bool changed;
+};
+
+struct hsts_kh {
+  char *host;
+  int explicit_port;
+};
+
+struct hsts_kh_info {
+  time_t created;
+  time_t max_age;
+  bool include_subdomains;
+};
+
+enum hsts_kh_match {
+  NO_MATCH,
+  SUPERDOMAIN_MATCH,
+  CONGRUENT_MATCH
+};
+
+#define hsts_is_host_name_valid(host) (!is_valid_ip_address (host))
+#define hsts_is_scheme_valid(scheme) (scheme == SCHEME_HTTPS)
+#define hsts_is_host_eligible(scheme, host) \
+    (hsts_is_scheme_valid (scheme) && hsts_is_host_name_valid (host))
+
+#define DEFAULT_HTTP_PORT 80
+#define DEFAULT_SSL_PORT  443
+#define CHECK_EXPLICIT_PORT(p1, p2) (p1 == 0 || p1 == p2)
+#define MAKE_EXPLICIT_PORT(s, p) (s == SCHEME_HTTPS ? (p == DEFAULT_SSL_PORT ? 0 : p) \
+    : (p == DEFAULT_HTTP_PORT ? 0 : p))
+
+/* Hashing and comparison functions for the hash table */
+
+static unsigned long
+hsts_hash_func (const void *key)
+{
+  struct hsts_kh *k = (struct hsts_kh *) key;
+  const char *h = NULL;
+  unsigned int hash = k->explicit_port;
+
+  for (h = k->host; *h; h++)
+    hash = hash * 31 + *h;
+
+  return hash;
+}
+
+static int
+hsts_cmp_func (const void *h1, const void *h2)
+{
+  struct hsts_kh *kh1 = (struct hsts_kh *) h1,
+      *kh2 = (struct hsts_kh *) h2;
+
+  return (!strcmp (kh1->host, kh2->host)) && (kh1->explicit_port == kh2->explicit_port);
+}
+
+/* Private functions. Feel free to make some of these public when needed. */
+
+static struct hsts_kh_info *
+hsts_find_entry (hsts_store_t store,
+                 const char *host, int explicit_port,
+                 enum hsts_kh_match *match_type,
+                 struct hsts_kh *kh)
+{
+  struct hsts_kh *k = NULL;
+  struct hsts_kh_info *khi = NULL;
+  enum hsts_kh_match match = NO_MATCH;
+  char *pos = NULL;
+  char *org_ptr = NULL;
+
+  k = (struct hsts_kh *) xnew (struct hsts_kh);
+  k->host = xstrdup_lower (host);
+  k->explicit_port = explicit_port;
+
+  /* save pointer so that we don't get into trouble later when freeing */
+  org_ptr = k->host;
+
+  khi = (struct hsts_kh_info *) hash_table_get (store->table, k);
+  if (khi)
+    {
+      match = CONGRUENT_MATCH;
+      goto end;
+    }
+
+  while (match == NO_MATCH &&
+      (pos = strchr (k->host, '.')) && pos - k->host > 0 &&
+      strchr (pos + 1, '.'))
+    {
+      k->host += (pos - k->host + 1);
+      khi = (struct hsts_kh_info *) hash_table_get (store->table, k);
+      if (khi)
+        match = SUPERDOMAIN_MATCH;
+    }
+
+end:
+  /* restore pointer or we'll get a SEGV */
+  k->host = org_ptr;
+
+  /* copy parameters to previous frame */
+  if (match_type)
+    *match_type = match;
+  if (kh)
+    memcpy (kh, k, sizeof (struct hsts_kh));
+  else
+    xfree (k->host);
+
+  xfree (k);
+  return khi;
+}
+
+static bool
+hsts_new_entry_internal (hsts_store_t store,
+                         const char *host, int port,
+                         time_t created, time_t max_age,
+                         bool include_subdomains,
+                         bool check_validity,
+                         bool check_expired,
+                         bool check_duplicates)
+{
+  struct hsts_kh *kh = xnew (struct hsts_kh);
+  struct hsts_kh_info *khi = xnew0 (struct hsts_kh_info);
+  bool success = false;
+
+  kh->host = xstrdup_lower (host);
+  kh->explicit_port = MAKE_EXPLICIT_PORT (SCHEME_HTTPS, port);
+
+  khi->created = created;
+  khi->max_age = max_age;
+  khi->include_subdomains = include_subdomains;
+
+  /* Check validity */
+  if (check_validity && !hsts_is_host_name_valid (host))
+    goto bail;
+
+  if (check_expired && ((khi->created + khi->max_age) < khi->created))
+    goto bail;
+
+  if (check_duplicates && hash_table_contains (store->table, kh))
+    goto bail;
+
+  /* Now store the new entry */
+  hash_table_put (store->table, kh, khi);
+  success = true;
+
+bail:
+  if (!success)
+    {
+      /* abort! */
+      xfree (kh->host);
+      xfree (kh);
+      xfree (khi);
+    }
+
+  return success;
+}
+
+/*
+   Creates a new entry, but does not check whether that entry already exists.
+   This function assumes that check has already been done by the caller.
+ */
+static bool
+hsts_add_entry (hsts_store_t store,
+                const char *host, int port,
+                time_t max_age, bool include_subdomains)
+{
+  time_t t = time (NULL);
+
+  /* It might happen time() returned -1 */
+  return (t < 0 ?
+      false :
+      hsts_new_entry_internal (store, host, port, t, max_age, include_subdomains, false, true, false));
+}
+
+/* Creates a new entry, unless an identical one already exists. */
+static bool
+hsts_new_entry (hsts_store_t store,
+                const char *host, int port,
+                time_t created, time_t max_age,
+                bool include_subdomains)
+{
+  return hsts_new_entry_internal (store, host, port, created, max_age, include_subdomains, true, true, true);
+}
+
+static void
+hsts_remove_entry (hsts_store_t store, struct hsts_kh *kh)
+{
+  hash_table_remove (store->table, kh);
+}
+
+static bool
+hsts_store_merge (hsts_store_t store,
+                  const char *host, int port,
+                  time_t created, time_t max_age,
+                  bool include_subdomains)
+{
+  enum hsts_kh_match match_type = NO_MATCH;
+  struct hsts_kh_info *khi = NULL;
+  bool success = false;
+
+  port = MAKE_EXPLICIT_PORT (SCHEME_HTTPS, port);
+  khi = hsts_find_entry (store, host, port, &match_type, NULL);
+  if (khi && match_type == CONGRUENT_MATCH && created > khi->created)
+    {
+      /* update the entry with the new info */
+      khi->created = created;
+      khi->max_age = max_age;
+      khi->include_subdomains = include_subdomains;
+
+      success = true;
+    }
+  else if (!khi)
+    success = hsts_new_entry (store, host, port, created, max_age, include_subdomains);
+
+  return success;
+}
+
+static bool
+hsts_read_database (hsts_store_t store, FILE *fp, bool merge_with_existing_entries)
+{
+  char *line = NULL, *p;
+  size_t len = 0;
+  int items_read;
+  bool result = false;
+  bool (*func)(hsts_store_t, const char *, int, time_t, time_t, bool);
+
+  char host[256];
+  int port;
+  time_t created, max_age;
+  int include_subdomains;
+
+  func = (merge_with_existing_entries ? hsts_store_merge : hsts_new_entry);
+
+  while (getline (&line, &len, fp) > 0)
+    {
+      for (p = line; c_isspace (*p); p++)
+        ;
+
+      if (*p == '#')
+        continue;
+
+      items_read = sscanf (p, "%255s %d %d %lu %lu",
+                           host,
+                           &port,
+                           &include_subdomains,
+                           (unsigned long *) &created,
+                           (unsigned long *) &max_age);
+
+      if (items_read == 5)
+        func (store, host, port, created, max_age, !!include_subdomains);
+    }
+
+  xfree (line);
+  result = true;
+
+  return result;
+}
+
+static void
+hsts_store_dump (hsts_store_t store, FILE *fp)
+{
+  hash_table_iterator it;
+
+  /* Print preliminary comments. We don't care if any of these fail. */
+  fputs ("# HSTS 1.0 Known Hosts database for GNU Wget.\n", fp);
+  fputs ("# Edit at your own risk.\n", fp);
+  fputs ("# <hostname>\t<port>\t<incl. subdomains>\t<created>\t<max-age>\n", fp);
+
+  /* Now cycle through the HSTS store in memory and dump the entries */
+  for (hash_table_iterate (store->table, &it); hash_table_iter_next (&it);)
+    {
+      struct hsts_kh *kh = (struct hsts_kh *) it.key;
+      struct hsts_kh_info *khi = (struct hsts_kh_info *) it.value;
+
+      if (fprintf (fp, "%s\t%d\t%d\t%lu\t%lu\n",
+                   kh->host, kh->explicit_port, khi->include_subdomains,
+                   khi->created, khi->max_age) < 0)
+        {
+          logprintf (LOG_ALWAYS, "Could not write the HSTS database correctly.\n");
+          break;
+        }
+    }
+}
+
+/*
+ * Test:
+ *  - The file is a regular file (ie. not a symlink), and
+ *  - The file is not world-writable.
+ */
+static bool
+hsts_file_access_valid (const char *filename)
+{
+  struct_stat st;
+
+  if (stat (filename, &st) == -1)
+    return false;
+
+  return !(st.st_mode & S_IWOTH) && S_ISREG (st.st_mode);
+}
+
+/* HSTS API */
+
+/*
+   Changes the given URLs according to the HSTS policy.
+
+   If there's no host in the store that either congruently
+   or not, matches the given URL, no changes are made.
+   Returns true if the URL was changed, or false
+   if it was left intact.
+ */
+bool
+hsts_match (hsts_store_t store, struct url *u)
+{
+  bool url_changed = false;
+  struct hsts_kh_info *entry = NULL;
+  struct hsts_kh *kh = xnew(struct hsts_kh);
+  enum hsts_kh_match match = NO_MATCH;
+  int port = MAKE_EXPLICIT_PORT (u->scheme, u->port);
+
+  /* avoid doing any computation if we're already in HTTPS */
+  if (!hsts_is_scheme_valid (u->scheme))
+    {
+      entry = hsts_find_entry (store, u->host, port, &match, kh);
+      if (entry)
+        {
+          if ((entry->created + entry->max_age) >= time(NULL))
+            {
+              if ((match == CONGRUENT_MATCH) ||
+                  (match == SUPERDOMAIN_MATCH && entry->include_subdomains))
+                {
+                  /* we found a matching Known HSTS Host
+                     rewrite the URL */
+                  u->scheme = SCHEME_HTTPS;
+                  if (u->port == 80)
+                    u->port = 443;
+                  url_changed = true;
+                  store->changed = true;
+                }
+            }
+          else
+            {
+              hsts_remove_entry (store, kh);
+              store->changed = true;
+            }
+        }
+      xfree (kh->host);
+    }
+
+  xfree (kh);
+
+  return url_changed;
+}
+
+/*
+   Add a new HSTS Known Host to the HSTS store.
+
+   If the host already exists, its information is updated,
+   or it'll be removed from the store if max_age is zero.
+
+   Bear in mind that the store is kept in memory, and will not
+   be written to disk until hsts_store_save is called.
+   This function regrows the in-memory HSTS store if necessary.
+
+   Currently, for a host to be taken into consideration,
+   two conditions have to be met:
+     - Connection must be through a secure channel (HTTPS).
+     - The host must not be an IPv4 or IPv6 address.
+
+   The RFC 6797 states that hosts that match IPv4 or IPv6 format
+   should be discarded at URI rewrite time. But we short-circuit
+   that check here, since there's no point in storing a host that
+   will never be matched.
+
+   Returns true if a new entry was actually created, or false
+   if an existing entry was updated/deleted. */
+bool
+hsts_store_entry (hsts_store_t store,
+                  enum url_scheme scheme, const char *host, int port,
+                  time_t max_age, bool include_subdomains)
+{
+  bool result = false;
+  enum hsts_kh_match match = NO_MATCH;
+  struct hsts_kh *kh = xnew(struct hsts_kh);
+  struct hsts_kh_info *entry = NULL;
+  time_t t = 0;
+
+  if (hsts_is_host_eligible (scheme, host))
+    {
+      port = MAKE_EXPLICIT_PORT (scheme, port);
+      entry = hsts_find_entry (store, host, port, &match, kh);
+      if (entry && match == CONGRUENT_MATCH)
+        {
+          if (max_age == 0)
+            {
+              hsts_remove_entry (store, kh);
+              store->changed = true;
+            }
+          else if (max_age > 0)
+            {
+              if (entry->max_age != max_age ||
+                  entry->include_subdomains != include_subdomains)
+                {
+                  /* RFC 6797 states that 'max_age' is a TTL relative to the reception of the STS header
+                     so we have to update the 'created' field too */
+                  t = time (NULL);
+                  if (t != -1)
+                    entry->created = t;
+                  entry->max_age = max_age;
+                  entry->include_subdomains = include_subdomains;
+
+                  store->changed = true;
+                }
+            }
+          /* we ignore negative max_ages */
+        }
+      else if (entry == NULL || match == SUPERDOMAIN_MATCH)
+        {
+          /* Either we didn't find a matching host,
+             or we got a superdomain match.
+             In either case, we create a new entry.
+
+             We have to perform an explicit check because it might
+             happen we got a non-existent entry with max_age == 0.
+          */
+          result = hsts_add_entry (store, host, port, max_age, include_subdomains);
+          if (result)
+            store->changed = true;
+        }
+      /* we ignore new entries with max_age == 0 */
+      xfree (kh->host);
+    }
+
+  xfree (kh);
+
+  return result;
+}
+
+hsts_store_t
+hsts_store_open (const char *filename)
+{
+  hsts_store_t store = NULL;
+
+  store = xnew0 (struct hsts_store);
+  store->table = hash_table_new (0, hsts_hash_func, hsts_cmp_func);
+  store->last_mtime = 0;
+  store->changed = false;
+
+  if (file_exists_p (filename))
+    {
+      if (hsts_file_access_valid (filename))
+        {
+          struct_stat st;
+          FILE *fp = fopen (filename, "r");
+
+          if (!fp || !hsts_read_database (store, fp, false))
+            {
+              /* abort! */
+              hsts_store_close (store);
+              xfree (store);
+              fclose (fp);
+              goto out;
+            }
+
+          if (fstat (fileno (fp), &st) == 0)
+            store->last_mtime = st.st_mtime;
+
+          fclose (fp);
+        }
+      else
+        {
+          /*
+           * If we're not reading the HSTS database,
+           * then by all means act as if HSTS was disabled.
+           */
+          hsts_store_close (store);
+          xfree (store);
+
+          logprintf (LOG_NOTQUIET, "Will not apply HSTS. "
+                     "The HSTS database must be a regular and non-world-writable file.\n");
+        }
+    }
+
+out:
+  return store;
+}
+
+void
+hsts_store_save (hsts_store_t store, const char *filename)
+{
+  struct_stat st;
+  FILE *fp = NULL;
+  int fd = 0;
+
+  if (filename && hash_table_count (store->table) > 0)
+    {
+      fp = fopen (filename, "a+");
+      if (fp)
+        {
+          /* Lock the file to avoid potential race conditions */
+          fd = fileno (fp);
+          flock (fd, LOCK_EX);
+
+          /* If the file has changed, merge the changes with our in-memory data
+             before dumping them to the file.
+             Otherwise we could potentially overwrite the data stored by other Wget processes.
+           */
+          if (store->last_mtime && stat (filename, &st) == 0 && st.st_mtime > store->last_mtime)
+            hsts_read_database (store, fp, true);
+
+          /* We've merged the latest changes so we can now truncate the file
+             and dump everything. */
+          fseek (fp, 0, SEEK_SET);
+          ftruncate (fd, 0);
+
+          /* now dump to the file */
+          hsts_store_dump (store, fp);
+
+          /* fclose is expected to unlock the file for us */
+          fclose (fp);
+        }
+    }
+}
+
+bool
+hsts_store_has_changed (hsts_store_t store)
+{
+  return (store ? store->changed : false);
+}
+
+void
+hsts_store_close (hsts_store_t store)
+{
+  hash_table_iterator it;
+
+  /* free all the host fields */
+  for (hash_table_iterate (store->table, &it); hash_table_iter_next (&it);)
+    {
+      xfree (((struct hsts_kh *) it.key)->host);
+      xfree (it.key);
+      xfree (it.value);
+    }
+
+  hash_table_destroy (store->table);
+}
+
+#ifdef TESTING
+/* I know I'm really evil because I'm writing macros
+   that change control flow. But we're testing, who will tell? :D
+ */
+#define TEST_URL_RW(s, u, p) do { \
+    if (test_url_rewrite (s, u, p, true)) \
+      return test_url_rewrite (s, u, p, true); \
+  } while (0)
+
+#define TEST_URL_NORW(s, u, p) do { \
+    if (test_url_rewrite (s, u, p, false)) \
+      return test_url_rewrite (s, u, p, false); \
+  } while (0)
+
+static char *
+get_hsts_store_filename (void)
+{
+  char *home = NULL, *filename = NULL;
+  FILE *fp = NULL;
+
+  home = home_dir ();
+  if (home)
+    {
+      filename = aprintf ("%s/.wget-hsts-test", home);
+      fp = fopen (filename, "w");
+      if (fp)
+        fclose (fp);
+    }
+
+  xfree (home);
+  return filename;
+}
+
+static hsts_store_t
+open_hsts_test_store (void)
+{
+  char *filename = NULL;
+  hsts_store_t table = NULL;
+
+  filename = get_hsts_store_filename ();
+  table = hsts_store_open (filename);
+  xfree (filename);
+
+  return table;
+}
+
+static void
+close_hsts_test_store (hsts_store_t store)
+{
+  char *filename = NULL;
+
+  filename = get_hsts_store_filename ();
+  unlink (filename);
+  xfree (filename);
+  xfree (store);
+}
+
+static const char*
+test_url_rewrite (hsts_store_t s, const char *url, int port, bool rewrite)
+{
+  bool result;
+  struct url u;
+
+  u.host = xstrdup (url);
+  u.port = port;
+  u.scheme = SCHEME_HTTP;
+
+  result = hsts_match (s, &u);
+
+  if (rewrite)
+    {
+      if (port == 80)
+        mu_assert("URL: port should've been rewritten to 443", u.port == 443);
+      else
+        mu_assert("URL: port should've been left intact", u.port == port);
+      mu_assert("URL: scheme should've been rewritten to HTTPS", u.scheme == SCHEME_HTTPS);
+      mu_assert("result should've been true", result == true);
+    }
+  else
+    {
+      mu_assert("URL: port should've been left intact", u.port == port);
+      mu_assert("URL: scheme should've been left intact", u.scheme == SCHEME_HTTP);
+      mu_assert("result should've been false", result == false);
+    }
+
+  xfree (u.host);
+  return NULL;
+}
+
+const char *
+test_hsts_new_entry (void)
+{
+  enum hsts_kh_match match = NO_MATCH;
+  struct hsts_kh_info *khi;
+  hsts_store_t s;
+  bool created;
+
+  s = open_hsts_test_store ();
+  mu_assert("Could not open the HSTS store. This could be due to lack of memory.", s != NULL);
+
+  created = hsts_store_entry (s, SCHEME_HTTP, "www.foo.com", 80, 1234, true);
+  mu_assert("No entry should have been created.", created == false);
+
+  created = hsts_store_entry (s, SCHEME_HTTPS, "www.foo.com", 443, 1234, true);
+  mu_assert("A new entry should have been created", created == true);
+
+  khi = hsts_find_entry (s, "www.foo.com", MAKE_EXPLICIT_PORT (SCHEME_HTTPS, 443), &match, NULL);
+  mu_assert("Should've been a congruent match", match == CONGRUENT_MATCH);
+  mu_assert("No valid HSTS info was returned", khi != NULL);
+  mu_assert("Variable 'max_age' should be 1234", khi->max_age == 1234);
+  mu_assert("Variable 'include_subdomains' should be asserted", khi->include_subdomains == true);
+
+  khi = hsts_find_entry (s, "b.www.foo.com", MAKE_EXPLICIT_PORT (SCHEME_HTTPS, 443), &match, NULL);
+  mu_assert("Should've been a superdomain match", match == SUPERDOMAIN_MATCH);
+  mu_assert("No valid HSTS info was returned", khi != NULL);
+  mu_assert("Variable 'max_age' should be 1234", khi->max_age == 1234);
+  mu_assert("Variable 'include_subdomains' should be asserted", khi->include_subdomains == true);
+
+  khi = hsts_find_entry (s, "ww.foo.com", MAKE_EXPLICIT_PORT (SCHEME_HTTPS, 443), &match, NULL);
+  mu_assert("Should've been no match", match == NO_MATCH);
+
+  khi = hsts_find_entry (s, "foo.com", MAKE_EXPLICIT_PORT (SCHEME_HTTPS, 443), &match, NULL);
+  mu_assert("Should've been no match", match == NO_MATCH);
+
+  khi = hsts_find_entry (s, ".foo.com", MAKE_EXPLICIT_PORT (SCHEME_HTTPS, 443), &match, NULL);
+  mu_assert("Should've been no match", match == NO_MATCH);
+
+  khi = hsts_find_entry (s, ".www.foo.com", MAKE_EXPLICIT_PORT (SCHEME_HTTPS, 443), &match, NULL);
+  mu_assert("Should've been no match", match == NO_MATCH);
+
+  hsts_store_close (s);
+  close_hsts_test_store (s);
+
+  return NULL;
+}
+
+const char*
+test_hsts_url_rewrite_superdomain (void)
+{
+  hsts_store_t s;
+  bool created;
+
+  s = open_hsts_test_store ();
+  mu_assert("Could not open the HSTS store", s != NULL);
+
+  created = hsts_store_entry (s, SCHEME_HTTPS, "www.foo.com", 443, 1234, true);
+  mu_assert("A new entry should've been created", created == true);
+
+  TEST_URL_RW (s, "www.foo.com", 80);
+  TEST_URL_RW (s, "bar.www.foo.com", 80);
+
+  hsts_store_close (s);
+  close_hsts_test_store (s);
+
+  return NULL;
+}
+
+const char*
+test_hsts_url_rewrite_congruent (void)
+{
+  hsts_store_t s;
+  bool created;
+
+  s = open_hsts_test_store ();
+  mu_assert("Could not open the HSTS store", s != NULL);
+
+  created = hsts_store_entry (s, SCHEME_HTTPS, "foo.com", 443, 1234, false);
+  mu_assert("A new entry should've been created", created == true);
+
+  TEST_URL_RW (s, "foo.com", 80);
+  TEST_URL_NORW (s, "www.foo.com", 80);
+
+  hsts_store_close (s);
+  close_hsts_test_store (s);
+
+  return NULL;
+}
+
+const char*
+test_hsts_read_database (void)
+{
+  hsts_store_t table;
+  char *home = home_dir();
+  char *file = NULL;
+  FILE *fp = NULL;
+  time_t created = time(NULL) - 10;
+
+  if (home)
+    {
+      file = aprintf ("%s/.wget-hsts-testing", home);
+      fp = fopen (file, "w");
+      if (fp)
+        {
+          fputs ("# dummy comment\n", fp);
+          fprintf (fp, "foo.example.com\t0\t1\t%ld\t123\n",(long) created);
+          fprintf (fp, "bar.example.com\t0\t0\t%ld\t456\n", (long) created);
+          fprintf (fp, "test.example.com\t8080\t0\t%ld\t789\n", (long) created);
+          fclose (fp);
+
+          table = hsts_store_open (file);
+
+          TEST_URL_RW (table, "foo.example.com", 80);
+          TEST_URL_RW (table, "www.foo.example.com", 80);
+          TEST_URL_RW (table, "bar.example.com", 80);
+
+          TEST_URL_NORW(table, "www.bar.example.com", 80);
+
+          TEST_URL_RW (table, "test.example.com", 8080);
+
+          hsts_store_close (table);
+          close_hsts_test_store (table);
+          unlink (file);
+        }
+      xfree (file);
+      xfree (home);
+    }
+
+  return NULL;
+}
+#endif /* TESTING */
+#endif /* HAVE_HSTS */

src/hsts.h

@@ -0,0 +1,54 @@
+/* Declarations for hsts.c
+   Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2015 Free Software
+   Foundation, Inc.
+
+This file is part of GNU Wget.
+
+GNU Wget is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 3 of the License, or
+ (at your option) any later version.
+
+GNU Wget is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Wget.  If not, see <http://www.gnu.org/licenses/>.
+
+Additional permission under GNU GPL version 3 section 7
+
+If you modify this program, or any covered work, by linking or
+combining it with the OpenSSL project's OpenSSL library (or a
+modified version of that library), containing parts covered by the
+terms of the OpenSSL or SSLeay licenses, the Free Software Foundation
+grants you additional permission to convey the resulting work.
+Corresponding Source for a non-source form of such a combination
+shall include the source code for the parts of OpenSSL used as well
+as that of the covered work.  */
+#include "wget.h"
+
+#ifdef HAVE_HSTS
+
+#ifndef HSTS_H
+#define HSTS_H
+
+#include "url.h"
+
+typedef struct hsts_store *hsts_store_t;
+
+hsts_store_t hsts_store_open (const char *);
+
+void hsts_store_save (hsts_store_t, const char *);
+void hsts_store_close (hsts_store_t);
+bool hsts_store_has_changed (hsts_store_t);
+
+bool hsts_store_entry (hsts_store_t,
+                       enum url_scheme, const char *, int,
+                       time_t, bool);
+bool hsts_match (hsts_store_t, struct url *);
+
+#endif /* HSTS_H */
+#endif /* HAVE_HSTS */

src/html-parse.c

@@ -1,6 +1,6 @@
 /* HTML parser for Wget.
    Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006,
-   2007, 2008, 2009, 2010, 2011 Free Software Foundation, Inc.
+   2007, 2008, 2009, 2010, 2011, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/html-parse.h

@@ -1,6 +1,6 @@
 /* Declarations for html-parse.c.
    Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006,
-   2007, 2008, 2009, 2010, 2011 Free Software Foundation, Inc.
+   2007, 2008, 2009, 2010, 2011, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/html-url.c

@@ -1,6 +1,7 @@
 /* Collect URLs from HTML source.
    Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006,
-   2007, 2008, 2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+   2007, 2008, 2009, 2010, 2011, 2012, 2015 Free Software Foundation,
+   Inc.
 
 This file is part of GNU Wget.
 
@@ -55,6 +56,7 @@ typedef void (*tag_handler_t) (int, struct taginfo *, struct map_context *);
 DECLARE_TAG_HANDLER (tag_find_urls);
 DECLARE_TAG_HANDLER (tag_handle_base);
 DECLARE_TAG_HANDLER (tag_handle_form);
+DECLARE_TAG_HANDLER (tag_handle_img);
 DECLARE_TAG_HANDLER (tag_handle_link);
 DECLARE_TAG_HANDLER (tag_handle_meta);
 
@@ -104,7 +106,7 @@ static struct known_tag {
   { TAG_FORM,    "form",        tag_handle_form },
   { TAG_FRAME,   "frame",       tag_find_urls },
   { TAG_IFRAME,  "iframe",      tag_find_urls },
-  { TAG_IMG,     "img",         tag_find_urls },
+  { TAG_IMG,     "img",         tag_handle_img },
   { TAG_INPUT,   "input",       tag_find_urls },
   { TAG_LAYER,   "layer",       tag_find_urls },
   { TAG_LINK,    "link",        tag_handle_link },
@@ -182,7 +184,8 @@ static const char *additional_attributes[] = {
   "name",                       /* used by tag_handle_meta  */
   "content",                    /* used by tag_handle_meta  */
   "action",                     /* used by tag_handle_form  */
-  "style"                       /* used by check_style_attr */
+  "style",                      /* used by check_style_attr */
+  "srcset",                     /* used by tag_handle_img */
 };
 
 static struct hash_table *interesting_tags;
@@ -303,6 +306,7 @@ append_url (const char *link_uri, int position, int size,
           logprintf (LOG_NOTQUIET,
                      _("%s: Cannot resolve incomplete link %s.\n"),
                      ctx->document_file, link_uri);
+          iri_free (iri);
           return NULL;
         }
 
@@ -311,6 +315,7 @@ append_url (const char *link_uri, int position, int size,
         {
           DEBUGP (("%s: link \"%s\" doesn't parse.\n",
                    ctx->document_file, link_uri));
+          iri_free (iri);
           return NULL;
         }
     }
@@ -334,6 +339,7 @@ append_url (const char *link_uri, int position, int size,
           DEBUGP (("%s: merged link \"%s\" doesn't parse.\n",
                    ctx->document_file, complete_uri));
           xfree (complete_uri);
+          iri_free (iri);
           return NULL;
         }
       xfree (complete_uri);
@@ -670,6 +676,88 @@ tag_handle_meta (int tagid _GL_UNUSED, struct taginfo *tag, struct map_context *
     }
 }
 
+/* Handle the IMG tag.  This requires special handling for the srcset attr,
+   while the traditional src/lowsrc/href attributes can be handled generically.
+*/
+
+static void
+tag_handle_img (int tagid, struct taginfo *tag, struct map_context *ctx) {
+  int attrind;
+  char *srcset;
+
+  /* Use the generic approach for the attributes without special syntax. */
+  tag_find_urls(tagid, tag, ctx);
+
+  srcset = find_attr (tag, "srcset", &attrind);
+  if (srcset)
+    {
+      /* These are relative to the input text. */
+      int base_ind = ATTR_POS (tag,attrind,ctx);
+      int size = strlen (srcset);
+
+      /* These are relative to srcset. */
+      int offset, url_start, url_end;
+
+      /* Make sure to line up base_ind with srcset[0], not outside quotes. */
+      if (ctx->text[base_ind] == '"' || ctx->text[base_ind] == '\'')
+        ++base_ind;
+
+      offset = 0;
+      while (offset < size)
+        {
+          bool has_descriptor = true;
+
+          /* Skip over initial whitespace and commas. Note there is no \v
+            in HTML5 whitespace. */
+          url_start = offset + strspn (srcset + offset, " \f\n\r\t,");
+
+          if (url_start == size)
+            return;
+
+          /* URL is any non-whitespace chars (including commas) - but with
+             trailing commas removed. */
+          url_end = url_start + strcspn (srcset + url_start, " \f\n\r\t");
+          while ((url_end - 1) > url_start && srcset[url_end - 1] == ',')
+            {
+              has_descriptor = false;
+              --url_end;
+            }
+
+          if (url_end > url_start)
+            {
+              char *url_text = strdupdelim (srcset + url_start,
+                                            srcset + url_end);
+              struct urlpos *up = append_url (url_text, base_ind + url_start,
+                                              url_end - url_start, ctx);
+              up->link_inline_p = 1;
+              up->link_noquote_html_p = 1;
+              xfree (url_text);
+            }
+
+          /* If the URL wasn't terminated by a , there may also be a descriptor
+             which we just skip. */
+          if (has_descriptor)
+            {
+              /* This is comma-terminated, except there may be one level of
+                 parentheses escaping that. */
+              bool in_paren = false;
+              for (offset = url_end; offset < size; ++offset)
+                {
+                  char c = srcset[offset];
+                  if (c == '(')
+                    in_paren = true;
+                  else if (c == ')' && in_paren)
+                    in_paren = false;
+                  else if (c == ',' && !in_paren)
+                    break;
+                }
+            }
+          else
+            offset = url_end;
+        }
+    }
+}
+
 /* Dispatch the tag handler appropriate for the tag we're mapping
    over.  See known_tags[] for definition of tag handlers.  */
 
@@ -748,10 +836,12 @@ get_urls_html (const char *file, const char *url, bool *meta_disallow_follow,
   map_html_tags (fm->content, fm->length, collect_tags_mapper, &ctx, flags,
                  NULL, interesting_attributes);
 
+#ifdef ENABLE_IRI
   /* Meta charset is only valid if there was no HTTP header Content-Type charset. */
   /* This is true for HTTP 1.0 and 1.1. */
   if (iri && !iri->content_encoding && meta_charset)
     set_content_encoding (iri, meta_charset);
+#endif
 
   DEBUGP (("no-follow in %s: %d\n", file, ctx.nofollow));
   if (meta_disallow_follow)

src/html-url.h

@@ -1,5 +1,5 @@
 /* Declarations for html-url.c.
-   Copyright (C) 1995, 1996, 1997, 2009, 2010, 2011 Free Software
+   Copyright (C) 1995, 1996, 1997, 2009, 2010, 2011, 2015 Free Software
    Foundation, Inc.
 
 This file is part of GNU Wget.

src/http-ntlm.c

@@ -1,6 +1,6 @@
 /* NTLM code.
-   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software
-   Foundation, Inc.
+   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free
+   Software Foundation, Inc.
    Contributed by Daniel Stenberg.
 
 This file is part of GNU Wget.
@@ -523,13 +523,13 @@ ntlm_output (struct ntlmdata *ntlm, const char *user, const char *passwd,
               0x0, 0x0,
 
 #ifdef USE_NTRESPONSES
-                SHORTPAIR (0x18), /* NT-response length, twice */
+              SHORTPAIR (0x18), /* NT-response length, twice */
               SHORTPAIR (0x18),
 #else
-                0x0, 0x0,
+              0x0, 0x0,
               0x0, 0x0,
 #endif
-                SHORTPAIR (ntrespoff),
+              SHORTPAIR (ntrespoff),
               0x0, 0x0,
 
               SHORTPAIR (domlen),

src/http-ntlm.h

@@ -1,8 +1,8 @@
 #ifndef __HTTP_NTLM_H
 #define __HTTP_NTLM_H
 /* Declarations for http_ntlm.c
-   Copyright (C) 1995, 1996, 1997, 2000, 2007, 2008, 2009, 2010, 2011
-   Free Software Foundation, Inc.
+   Copyright (C) 1995, 1996, 1997, 2000, 2007, 2008, 2009, 2010, 2011,
+   2015 Free Software Foundation, Inc.
    Contributed by Daniel Stenberg.
 
 This file is part of GNU Wget.

src/http.h

@@ -1,6 +1,6 @@
 /* Declarations for HTTP.
-   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software
-   Foundation, Inc.
+   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free
+   Software Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -31,6 +31,8 @@ as that of the covered work.  */
 #ifndef HTTP_H
 #define HTTP_H
 
+#include "hsts.h"
+
 struct url;
 
 uerr_t http_loop (struct url *, struct url *, char **, char **, const char *,

src/init.c

@@ -1,7 +1,7 @@
 /* Reading/parsing the initialization file.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2014 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2014, 2015 Free
+   Software Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -115,6 +115,7 @@ CMD_DECLARE (cmd_spec_secure_protocol);
 CMD_DECLARE (cmd_spec_timeout);
 CMD_DECLARE (cmd_spec_useragent);
 CMD_DECLARE (cmd_spec_verbose);
+CMD_DECLARE (cmd_check_cert);
 
 /* List of recognized commands, each consisting of name, place and
    function.  When adding a new command, simply add it to the list,
@@ -142,6 +143,9 @@ static const struct {
   { "backups",          &opt.backups,           cmd_number },
   { "base",             &opt.base_href,         cmd_string },
   { "bindaddress",      &opt.bind_address,      cmd_string },
+#ifdef HAVE_LIBCARES
+  { "binddnsaddress",   &opt.bind_dns_address,  cmd_string },
+#endif
   { "bodydata",         &opt.body_data,         cmd_string },
   { "bodyfile",         &opt.body_file,         cmd_string },
 #ifdef HAVE_SSL
@@ -152,13 +156,14 @@ static const struct {
   { "cadirectory",      &opt.ca_directory,      cmd_directory },
   { "certificate",      &opt.cert_file,         cmd_file },
   { "certificatetype",  &opt.cert_type,         cmd_cert_type },
-  { "checkcertificate", &opt.check_cert,        cmd_boolean },
+  { "checkcertificate", &opt.check_cert,        cmd_check_cert },
 #endif
   { "chooseconfig",     &opt.choose_config,     cmd_file },
   { "connecttimeout",   &opt.connect_timeout,   cmd_time },
   { "contentdisposition", &opt.content_disposition, cmd_boolean },
   { "contentonerror",   &opt.content_on_error,  cmd_boolean },
   { "continue",         &opt.always_rest,       cmd_boolean },
+  { "convertfileonly",  &opt.convert_file_only, cmd_boolean },
   { "convertlinks",     &opt.convert_links,     cmd_boolean },
   { "cookies",          &opt.cookies,           cmd_boolean },
 #ifdef HAVE_SSL
@@ -171,6 +176,9 @@ static const struct {
   { "dirprefix",        &opt.dir_prefix,        cmd_directory },
   { "dirstruct",        NULL,                   cmd_spec_dirstruct },
   { "dnscache",         &opt.dns_cache,         cmd_boolean },
+#ifdef HAVE_LIBCARES
+  { "dnsservers",       &opt.dns_servers,       cmd_string },
+#endif
   { "dnstimeout",       &opt.dns_timeout,       cmd_time },
   { "domains",          &opt.domains,           cmd_vector },
   { "dotbytes",         &opt.dot_bytes,         cmd_bytes },
@@ -188,12 +196,22 @@ static const struct {
   { "ftppasswd",        &opt.ftp_passwd,        cmd_string }, /* deprecated */
   { "ftppassword",      &opt.ftp_passwd,        cmd_string },
   { "ftpproxy",         &opt.ftp_proxy,         cmd_string },
+#ifdef HAVE_SSL
+  { "ftpscleardataconnection", &opt.ftps_clear_data_connection, cmd_boolean },
+  { "ftpsfallbacktoftp", &opt.ftps_fallback_to_ftp, cmd_boolean },
+  { "ftpsimplicit",     &opt.ftps_implicit,     cmd_boolean },
+  { "ftpsresumessl",    &opt.ftps_resume_ssl,   cmd_boolean },
+#endif
 #ifdef __VMS
   { "ftpstmlf",         &opt.ftp_stmlf,         cmd_boolean },
 #endif /* def __VMS */
   { "ftpuser",          &opt.ftp_user,          cmd_string },
   { "glob",             &opt.ftp_glob,          cmd_boolean },
   { "header",           NULL,                   cmd_spec_header },
+#ifdef HAVE_HSTS
+  { "hsts",             &opt.hsts,              cmd_boolean },
+  { "hsts-file",        &opt.hsts_file,         cmd_file },
+#endif
   { "htmlextension",    &opt.adjust_extension,  cmd_boolean }, /* deprecated */
   { "htmlify",          NULL,                   cmd_spec_htmlify },
   { "httpkeepalive",    &opt.http_keep_alive,   cmd_boolean },
@@ -205,6 +223,7 @@ static const struct {
 #endif
   { "httpsproxy",       &opt.https_proxy,       cmd_string },
   { "httpuser",         &opt.http_user,         cmd_string },
+  { "if-modified-since", &opt.if_modified_since, cmd_boolean },
   { "ignorecase",       &opt.ignore_case,       cmd_boolean },
   { "ignorelength",     &opt.ignore_length,     cmd_boolean },
   { "ignoretags",       &opt.ignore_tags,       cmd_vector },
@@ -214,6 +233,9 @@ static const struct {
   { "inet6only",        &opt.ipv6_only,         cmd_boolean },
 #endif
   { "input",            &opt.input_filename,    cmd_file },
+#ifdef HAVE_METALINK
+  { "input-metalink",   &opt.input_metalink,    cmd_file },
+#endif
   { "iri",              &opt.enable_iri,        cmd_boolean },
   { "keepsessioncookies", &opt.keep_session_cookies, cmd_boolean },
   { "limitrate",        &opt.limit_rate,        cmd_bytes },
@@ -222,6 +244,9 @@ static const struct {
   { "logfile",          &opt.lfilename,         cmd_file },
   { "login",            &opt.ftp_user,          cmd_string },/* deprecated*/
   { "maxredirect",      &opt.max_redirect,      cmd_number },
+#ifdef HAVE_METALINK
+  { "metalink-over-http", &opt.metalink_over_http, cmd_boolean },
+#endif
   { "method",           &opt.method,            cmd_string_uppercase },
   { "mirror",           NULL,                   cmd_spec_mirror },
   { "netrc",            &opt.netrc,             cmd_boolean },
@@ -235,9 +260,15 @@ static const struct {
   { "passiveftp",       &opt.ftp_pasv,          cmd_boolean },
   { "passwd",           &opt.ftp_passwd,        cmd_string },/* deprecated*/
   { "password",         &opt.passwd,            cmd_string },
+#ifdef HAVE_SSL
+  { "pinnedpubkey",     &opt.pinnedpubkey,      cmd_string },
+#endif
   { "postdata",         &opt.post_data,         cmd_string },
   { "postfile",         &opt.post_file_name,    cmd_file },
   { "preferfamily",     NULL,                   cmd_spec_prefer_family },
+#ifdef HAVE_METALINK
+  { "preferred-location", &opt.preferred_location, cmd_string },
+#endif
   { "preservepermissions", &opt.preserve_perm,  cmd_boolean },
 #ifdef HAVE_SSL
   { "privatekey",       &opt.private_key,       cmd_file },
@@ -260,6 +291,7 @@ static const struct {
   { "referer",          &opt.referer,           cmd_string },
   { "regextype",        &opt.regex_type,        cmd_spec_regex_type },
   { "reject",           &opt.rejects,           cmd_vector },
+  { "rejectedlog",      &opt.rejected_log,      cmd_file },
   { "rejectregex",      &opt.rejectregex_s,     cmd_string },
   { "relativeonly",     &opt.relative_only,     cmd_boolean },
   { "remoteencoding",   &opt.encoding_remote,   cmd_string },
@@ -356,11 +388,13 @@ defaults (void)
   opt.htmlify = true;
   opt.http_keep_alive = true;
   opt.use_proxy = true;
+  opt.convert_file_only = false;
   tmp = getenv ("no_proxy");
   if (tmp)
     opt.no_proxy = sepstring (tmp);
   opt.prefer_family = prefer_none;
   opt.allow_cache = true;
+  opt.if_modified_since = true;
 
   opt.read_timeout = 900;
   opt.use_robots = true;
@@ -391,12 +425,18 @@ defaults (void)
   opt.retr_symlinks = true;
 
 #ifdef HAVE_SSL
-  opt.check_cert = true;
+  opt.check_cert = CHECK_CERT_ON;
+  opt.ftps_resume_ssl = true;
+  opt.ftps_fallback_to_ftp = false;
+  opt.ftps_implicit = false;
+  opt.ftps_clear_data_connection = false;
 #endif
 
   /* The default for file name restriction defaults to the OS type. */
 #if defined(WINDOWS) || defined(MSDOS) || defined(__CYGWIN__)
   opt.restrict_files_os = restrict_windows;
+#elif defined(__VMS)
+  opt.restrict_files_os = restrict_vms;
 #else
   opt.restrict_files_os = restrict_unix;
 #endif
@@ -437,6 +477,11 @@ defaults (void)
   opt.start_pos = -1;
   opt.show_progress = -1;
   opt.noscroll = false;
+
+#ifdef HAVE_HSTS
+  /* HSTS is enabled by default */
+  opt.hsts = true;
+#endif
 }
 
 /* Return the user's home directory (strdup-ed), or NULL if none is
@@ -562,9 +607,7 @@ wgetrc_file_name (void)
      SYSTEM_WGETRC should not be defined under WINDOWS.  */
   if (!file)
     {
-      char *home = home_dir ();
-      xfree (file);
-      home = ws_mypath ();
+      char *home = ws_mypath ();
       if (home)
         {
           file = aprintf ("%s/wget.ini", home);
@@ -922,6 +965,18 @@ static bool simple_atof (const char *, const char *, double *);
                      && (p)[3] == '\0')
 
 
+static int
+cmd_boolean_internal (const char *com _GL_UNUSED, const char *val, void *place _GL_UNUSED)
+{
+  if (CMP2 (val, 'o', 'n') || CMP3 (val, 'y', 'e', 's') || CMP1 (val, '1'))
+    /* "on", "yes" and "1" mean true. */
+    return 1;
+  else if (CMP3 (val, 'o', 'f', 'f') || CMP2 (val, 'n', 'o') || CMP1 (val, '0'))
+    /* "off", "no" and "0" mean false. */
+    return 0;
+  return -1;
+}
+
 /* Store the boolean value from VAL to PLACE.  COM is ignored,
    except for error messages.  */
 static bool
@@ -929,24 +984,62 @@ cmd_boolean (const char *com, const char *val, void *place)
 {
   bool value;
 
-  if (CMP2 (val, 'o', 'n') || CMP3 (val, 'y', 'e', 's') || CMP1 (val, '1'))
-    /* "on", "yes" and "1" mean true. */
-    value = true;
-  else if (CMP3 (val, 'o', 'f', 'f') || CMP2 (val, 'n', 'o') || CMP1 (val, '0'))
-    /* "off", "no" and "0" mean false. */
-    value = false;
-  else
+  switch (cmd_boolean_internal (com, val, place))
     {
-      fprintf (stderr,
-               _("%s: %s: Invalid boolean %s; use `on' or `off'.\n"),
-               exec_name, com, quote (val));
-      return false;
-    }
+    case 0:
+      value = false;
+      break;
 
+    case 1:
+      value = true;
+      break;
+
+    default:
+      {
+        fprintf (stderr,
+                 _("%s: %s: Invalid boolean %s; use `on' or `off'.\n"),
+                 exec_name, com, quote (val));
+        return false;
+      }
+    }
   *(bool *) place = value;
   return true;
 }
 
+/* Store the check_cert value from VAL to PLACE.  COM is ignored,
+   except for error messages.  */
+static bool
+cmd_check_cert (const char *com, const char *val, void *place)
+{
+  int value;
+
+  switch (cmd_boolean_internal (com, val, place))
+    {
+    case 0:
+      value = CHECK_CERT_OFF;
+      break;
+
+    case 1:
+      value = CHECK_CERT_ON;
+      break;
+
+    default:
+      {
+        if (!c_strcasecmp (val, "quiet"))
+          value = CHECK_CERT_QUIET;
+        else
+          {
+            fprintf (stderr,
+                     _("%s: %s: Invalid %s; use `on', `off' or `quiet'.\n"),
+                     exec_name, com, quote (val));
+            return false;
+          }
+      }
+    }
+  *(int *) place = value;
+  return true;
+}
+
 /* Set the non-negative integer value from VAL to PLACE.  With
    incorrect specification, the number remains unchanged.  */
 static bool
@@ -1481,6 +1574,8 @@ cmd_spec_restrict_file_names (const char *com, const char *val, void *place_igno
 
       if (VAL_IS ("unix"))
         restrict_os = restrict_unix;
+      else if (VAL_IS ("vms"))
+        restrict_os = restrict_vms;
       else if (VAL_IS ("windows"))
         restrict_os = restrict_windows;
       else if (VAL_IS ("lowercase"))
@@ -1495,7 +1590,7 @@ cmd_spec_restrict_file_names (const char *com, const char *val, void *place_igno
         {
           fprintf (stderr, _("\
 %s: %s: Invalid restriction %s,\n\
-    use [unix|windows],[lowercase|uppercase],[nocontrol],[ascii].\n"),
+    use [unix|vms|windows],[lowercase|uppercase],[nocontrol],[ascii].\n"),
                    exec_name, com, quote (val));
           return false;
         }
@@ -1789,6 +1884,10 @@ cleanup (void)
   xfree (opt.lfilename);
   xfree (opt.dir_prefix);
   xfree (opt.input_filename);
+#ifdef HAVE_METALINK
+  xfree (opt.input_metalink);
+  xfree (opt.preferred_location);
+#endif
   xfree (opt.output_document);
   free_vec (opt.accepts);
   free_vec (opt.rejects);
@@ -1830,6 +1929,19 @@ cleanup (void)
   xfree (opt.post_data);
   xfree (opt.body_data);
   xfree (opt.body_file);
+  xfree (opt.rejected_log);
+
+#ifdef HAVE_LIBCARES
+#include <ares.h>
+  {
+    extern ares_channel ares;
+
+    xfree (opt.bind_dns_address);
+    xfree (opt.dns_servers);
+    ares_destroy (ares);
+    ares_library_cleanup ();
+  }
+#endif
 
 #endif /* DEBUG_MALLOC */
 }

src/init.h

@@ -1,7 +1,7 @@
 /* Declarations for init.c.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/iri.c

@@ -1,5 +1,6 @@
 /* IRI related functions.
-   Copyright (C) 2008, 2009, 2010, 2011 Free Software Foundation, Inc.
+   Copyright (C) 2008, 2009, 2010, 2011, 2015 Free Software Foundation,
+   Inc.
 
 This file is part of GNU Wget.
 
@@ -52,8 +53,9 @@ as that of the covered work.  */
 /* Given a string containing "charset=XXX", return the encoding if found,
    or NULL otherwise */
 char *
-parse_charset (char *str)
+parse_charset (const char *str)
 {
+  const char *end;
   char *charset;
 
   if (!str || !*str)
@@ -64,14 +66,14 @@ parse_charset (char *str)
     return NULL;
 
   str += 8;
-  charset = str;
+  end = str;
 
   /* sXXXav: which chars should be banned ??? */
-  while (*charset && !c_isspace (*charset))
-    charset++;
+  while (*end && !c_isspace (*end))
+    end++;
 
   /* sXXXav: could strdupdelim return NULL ? */
-  charset = strdupdelim (str, charset);
+  charset = strdupdelim (str, end);
 
   /* Do a minimum check on the charset value */
   if (!check_encoding_name (charset))
@@ -94,9 +96,9 @@ find_locale (void)
 
 /* Basic check of an encoding name. */
 bool
-check_encoding_name (char *encoding)
+check_encoding_name (const char *encoding)
 {
-  char *s = encoding;
+  const char *s = encoding;
 
   while (*s)
     {
@@ -116,27 +118,26 @@ check_encoding_name (char *encoding)
    will contain the transcoded string on success. *out content is
    unspecified otherwise. */
 static bool
-do_conversion (const char *tocode, const char *fromcode, char *in, size_t inlen, char **out)
+do_conversion (const char *tocode, const char *fromcode, char const *in_org, size_t inlen, char **out)
 {
   iconv_t cd;
   /* sXXXav : hummm hard to guess... */
   size_t len, done, outlen;
   int invalid = 0, tooshort = 0;
-  char *s, *in_org, *in_save;
+  char *s, *in, *in_save;
 
   cd = iconv_open (tocode, fromcode);
   if (cd == (iconv_t)(-1))
     {
-      logprintf (LOG_VERBOSE, _("Conversion from %s to UTF-8 isn't supported\n"),
-                 quote (opt.locale));
+      logprintf (LOG_VERBOSE, _("Conversion from %s to %s isn't supported\n"),
+                 quote (fromcode), quote (tocode));
       *out = NULL;
       return false;
     }
 
   /* iconv() has to work on an unescaped string */
-  in_org = in;
-  in_save = in = xstrndup(in, inlen);
-  url_unescape(in);
+  in_save = in = xstrndup (in_org, inlen);
+  url_unescape_except_reserved (in);
   inlen = strlen(in);
 
   len = outlen = inlen * 2;
@@ -145,13 +146,21 @@ do_conversion (const char *tocode, const char *fromcode, char *in, size_t inlen,
 
   for (;;)
     {
-      if (iconv (cd, &in, &inlen, out, &outlen) != (size_t)(-1))
+      if (iconv (cd, &in, &inlen, out, &outlen) != (size_t)(-1) &&
+          iconv (cd, NULL, NULL, out, &outlen) != (size_t)(-1))
         {
           *out = s;
           *(s + len - outlen - done) = '\0';
           xfree(in_save);
           iconv_close(cd);
-          DEBUGP (("converted '%s' (%s) -> '%s' (%s)\n", in_org, fromcode, *out, tocode));
+          IF_DEBUG
+          {
+            /* not not print out embedded passwords, in_org might be an URL */
+            if (!strchr(in_org, '@') && !strchr(*out, '@'))
+              debug_logprintf ("converted '%s' (%s) -> '%s' (%s)\n", in_org, fromcode, *out, tocode);
+            else
+              debug_logprintf ("%s: logging suppressed, strings may contain password\n", __func__);
+          }
           return true;
         }
 
@@ -171,16 +180,10 @@ do_conversion (const char *tocode, const char *fromcode, char *in, size_t inlen,
         }
       else if (errno == E2BIG) /* Output buffer full */
         {
-          char *new;
-
           tooshort++;
           done = len;
-          outlen = done + inlen * 2;
-          new = xmalloc (outlen + 1);
-          memcpy (new, s, done);
-          xfree (s);
-          s = new;
-          len = outlen;
+          len = outlen = done + inlen * 2;
+          s = xrealloc (s, outlen + 1);
           *out = s + done;
         }
       else /* Weird, we got an unspecified error */
@@ -192,7 +195,14 @@ do_conversion (const char *tocode, const char *fromcode, char *in, size_t inlen,
 
     xfree(in_save);
     iconv_close(cd);
-    DEBUGP (("converted '%s' (%s) -> '%s' (%s)\n", in_org, fromcode, *out, tocode));
+    IF_DEBUG
+    {
+      /* not not print out embedded passwords, in_org might be an URL */
+      if (!strchr(in_org, '@') && !strchr(*out, '@'))
+        debug_logprintf ("converted '%s' (%s) -> '%s' (%s)\n", in_org, fromcode, *out, tocode);
+      else
+        debug_logprintf ("%s: logging suppressed, strings may contain password\n", __func__);
+    }
     return false;
 }
 
@@ -216,42 +226,96 @@ locale_to_utf8 (const char *str)
   if (do_conversion ("UTF-8", opt.locale, (char *) str, strlen ((char *) str), &new))
     return (const char *) new;
 
+  xfree (new);
   return str;
 }
 
+/*
+ * Work around a libidn <= 1.30 vulnerability.
+ *
+ * The function checks for a valid UTF-8 character sequence before
+ * passing it to idna_to_ascii_8z().
+ *
+ * [1] http://lists.gnu.org/archive/html/help-libidn/2015-05/msg00002.html
+ * [2] https://lists.gnu.org/archive/html/bug-wget/2015-06/msg00002.html
+ * [3] http://curl.haxx.se/mail/lib-2015-06/0143.html
+ */
+static bool
+_utf8_is_valid(const char *utf8)
+{
+  const unsigned char *s = (const unsigned char *) utf8;
+
+  while (*s)
+    {
+      if ((*s & 0x80) == 0) /* 0xxxxxxx ASCII char */
+        s++;
+      else if ((*s & 0xE0) == 0xC0) /* 110xxxxx 10xxxxxx */
+        {
+          if ((s[1] & 0xC0) != 0x80)
+            return false;
+          s+=2;
+        }
+      else if ((*s & 0xF0) == 0xE0) /* 1110xxxx 10xxxxxx 10xxxxxx */
+        {
+          if ((s[1] & 0xC0) != 0x80 || (s[2] & 0xC0) != 0x80)
+            return false;
+          s+=3;
+        }
+      else if ((*s & 0xF8) == 0xF0) /* 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx */
+        {
+          if ((s[1] & 0xC0) != 0x80 || (s[2] & 0xC0) != 0x80 || (s[3] & 0xC0) != 0x80)
+            return false;
+          s+=4;
+        }
+      else
+        return false;
+    }
+
+  return true;
+}
+
 /* Try to "ASCII encode" UTF-8 host. Return the new domain on success or NULL
    on error. */
 char *
-idn_encode (struct iri *i, char *host)
+idn_encode (const struct iri *i, const char *host)
 {
-  char *new;
   int ret;
+  char *ascii_encoded;
+  char *utf8_encoded = NULL;
 
   /* Encode to UTF-8 if not done */
   if (!i->utf8_encode)
     {
-      if (!remote_to_utf8 (i, (const char *) host, (const char **) &new))
+      if (!remote_to_utf8 (i, host, &utf8_encoded))
           return NULL;  /* Nothing to encode or an error occured */
-      host = new;
     }
 
-  /* toASCII UTF-8 NULL terminated string */
-  ret = idna_to_ascii_8z (host, &new, IDNA_FLAGS);
+  if (!_utf8_is_valid(utf8_encoded ? utf8_encoded : host))
+    {
+      logprintf (LOG_VERBOSE, _("Invalid UTF-8 sequence: %s\n"),
+                 quote(utf8_encoded ? utf8_encoded : host));
+      xfree (utf8_encoded);
+      return NULL;
+    }
+
+  /* Store in ascii_encoded the ASCII UTF-8 NULL terminated string */
+  ret = idna_to_ascii_8z (utf8_encoded ? utf8_encoded : host, &ascii_encoded, IDNA_FLAGS);
+  xfree (utf8_encoded);
+
   if (ret != IDNA_SUCCESS)
     {
-      /* sXXXav : free new when needed ! */
       logprintf (LOG_VERBOSE, _("idn_encode failed (%d): %s\n"), ret,
                  quote (idna_strerror (ret)));
       return NULL;
     }
 
-  return new;
+  return ascii_encoded;
 }
 
 /* Try to decode an "ASCII encoded" host. Return the new domain in the locale
    on success or NULL on error. */
 char *
-idn_decode (char *host)
+idn_decode (const char *host)
 {
   char *new;
   int ret;
@@ -270,7 +334,7 @@ idn_decode (char *host)
 /* Try to transcode string str from remote encoding to UTF-8. On success, *new
    contains the transcoded string. *new content is unspecified otherwise. */
 bool
-remote_to_utf8 (struct iri *iri, const char *str, const char **new)
+remote_to_utf8 (const struct iri *iri, const char *str, char **new)
 {
   bool ret = false;
 
@@ -292,7 +356,7 @@ remote_to_utf8 (struct iri *iri, const char *str, const char **new)
       return false;
     }
 
-  if (do_conversion ("UTF-8", iri->uri_encoding, (char *) str, strlen (str), (char **) new))
+  if (do_conversion ("UTF-8", iri->uri_encoding, str, strlen (str), new))
     ret = true;
 
   /* Test if something was converted */
@@ -344,7 +408,7 @@ iri_free (struct iri *i)
 /* Set uri_encoding of struct iri i. If a remote encoding was specified, use
    it unless force is true. */
 void
-set_uri_encoding (struct iri *i, char *charset, bool force)
+set_uri_encoding (struct iri *i, const char *charset, bool force)
 {
   DEBUGP (("URI encoding = %s\n", charset ? quote (charset) : "None"));
   if (!force && opt.encoding_remote)
@@ -361,7 +425,7 @@ set_uri_encoding (struct iri *i, char *charset, bool force)
 
 /* Set content_encoding of struct iri i. */
 void
-set_content_encoding (struct iri *i, char *charset)
+set_content_encoding (struct iri *i, const char *charset)
 {
   DEBUGP (("URI content encoding = %s\n", charset ? quote (charset) : "None"));
   if (opt.encoding_remote)

src/iri.h

@@ -1,5 +1,6 @@
 /* Internationalization related declarations.
-   Copyright (C) 2008, 2009, 2010, 2011 Free Software Foundation, Inc.
+   Copyright (C) 2008, 2009, 2010, 2011, 2015 Free Software Foundation,
+   Inc.
 
 This file is part of GNU Wget.
 
@@ -42,24 +43,24 @@ struct iri {
 # include <idna.h>
 # include <idn-free.h>
 
-char *parse_charset (char *str);
+char *parse_charset (const char *str);
 char *find_locale (void);
-bool check_encoding_name (char *encoding);
+bool check_encoding_name (const char *encoding);
 const char *locale_to_utf8 (const char *str);
-char *idn_encode (struct iri *i, char *host);
-char *idn_decode (char *host);
-bool remote_to_utf8 (struct iri *i, const char *str, const char **new);
+char *idn_encode (const struct iri *i, const char *host);
+char *idn_decode (const char *host);
+bool remote_to_utf8 (const struct iri *i, const char *str, char **new);
 struct iri *iri_new (void);
 struct iri *iri_dup (const struct iri *);
 void iri_free (struct iri *i);
-void set_uri_encoding (struct iri *i, char *charset, bool force);
-void set_content_encoding (struct iri *i, char *charset);
+void set_uri_encoding (struct iri *i, const char *charset, bool force);
+void set_content_encoding (struct iri *i, const char *charset);
 
 #else /* ENABLE_IRI */
 
 extern struct iri dummy_iri;
 
-#define parse_charset(str)          (str, NULL)
+#define parse_charset(str)          NULL
 #define find_locale()               NULL
 #define check_encoding_name(str)    false
 #define locale_to_utf8(str)         (str)

src/log.c

@@ -1,6 +1,6 @@
 /* Messages logging.
    Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006,
-   2007, 2008, 2009, 2010, 2011 Free Software Foundation, Inc.
+   2007, 2008, 2009, 2010, 2011, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -351,6 +351,7 @@ logputs (enum log_options o, const char *s)
 {
   FILE *fp;
   FILE *warcfp;
+  int errno_save = errno;
 
   check_redirect_output ();
   if (o == LOG_PROGRESS)
@@ -358,10 +359,14 @@ logputs (enum log_options o, const char *s)
   else
     fp = get_log_fp ();
 
+  errno = errno_save;
+
   if (fp == NULL)
     return;
 
   warcfp = get_warc_log_fp ();
+  errno = errno_save;
+
   CHECK_VERBOSE (o);
 
   FPUTS (s, fp);
@@ -373,6 +378,8 @@ logputs (enum log_options o, const char *s)
     logflush ();
   else
     needs_flushing = true;
+
+  errno = errno_save;
 }
 
 struct logvprintf_state {
@@ -543,8 +550,10 @@ logprintf (enum log_options o, const char *fmt, ...)
   va_list args;
   struct logvprintf_state lpstate;
   bool done;
+  int errno_saved = errno;
 
   check_redirect_output ();
+  errno = errno_saved;
   if (inhibit_logging)
     return;
   CHECK_VERBOSE (o);
@@ -561,6 +570,8 @@ logprintf (enum log_options o, const char *fmt, ...)
         exit (WGET_EXIT_GENERIC_ERROR);
     }
   while (!done);
+
+  errno = errno_saved;
 }
 
 #ifdef ENABLE_DEBUG
@@ -598,11 +609,18 @@ log_init (const char *file, bool appendp)
 {
   if (file)
     {
-      logfp = fopen (file, appendp ? "a" : "w");
-      if (!logfp)
+      if (HYPHENP (file))
         {
-          fprintf (stderr, "%s: %s: %s\n", exec_name, file, strerror (errno));
-          exit (WGET_EXIT_GENERIC_ERROR);
+          logfp = stdout;
+        }
+      else
+        {
+          logfp = fopen (file, appendp ? "a" : "w");
+          if (!logfp)
+            {
+              fprintf (stderr, "%s: %s: %s\n", exec_name, file, strerror (errno));
+              exit (WGET_EXIT_GENERIC_ERROR);
+            }
         }
     }
   else

src/log.h

@@ -1,6 +1,6 @@
 /* Declarations for log.c.
    Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006,
-   2007, 2008, 2009, 2010, 2011 Free Software Foundation, Inc.
+   2007, 2008, 2009, 2010, 2011, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/main.c

@@ -1,5 +1,7 @@
 /* Command line parsing.
-   Copyright (C) 1996-2014 Free Software Foundation, Inc.
+   Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Free
+   Software Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -52,6 +54,7 @@ as that of the covered work.  */
 #include "convert.h"
 #include "spider.h"
 #include "http.h"               /* for save_cookies */
+#include "hsts.h"               /* for initializing hsts_store to NULL */
 #include "ptimer.h"
 #include "warc.h"
 #include "version.h"
@@ -61,6 +64,11 @@ as that of the covered work.  */
 #include <getpass.h>
 #include <quote.h>
 
+#ifdef HAVE_METALINK
+# include <metalink/metalink_parser.h>
+# include "metalink.h"
+#endif
+
 #ifdef WINDOWS
 # include <io.h>
 # include <fcntl.h>
@@ -78,6 +86,13 @@ as that of the covered work.  */
 struct iri dummy_iri;
 #endif
 
+#ifdef HAVE_LIBCARES
+#include <ares.h>
+ares_channel ares;
+#else
+void *ares;
+#endif
+
 struct options opt;
 
 /* defined in version.c */
@@ -134,6 +149,74 @@ i18n_initialize (void)
 #endif /* ENABLE_NLS */
 }
 
+#ifdef HAVE_HSTS
+/* make the HSTS store global */
+hsts_store_t hsts_store;
+
+static char*
+get_hsts_database (void)
+{
+  char *home;
+
+  if (opt.hsts_file)
+    return xstrdup (opt.hsts_file);
+
+  home = home_dir ();
+  if (home)
+    {
+      char *dir = aprintf ("%s/.wget-hsts", home);
+      xfree(home);
+      return dir;
+    }
+
+  return NULL;
+}
+
+static void
+load_hsts (void)
+{
+  if (!hsts_store)
+    {
+      char *filename = get_hsts_database ();
+
+      if (filename)
+        {
+          DEBUGP (("Reading HSTS entries from %s\n", filename));
+
+          hsts_store = hsts_store_open (filename);
+
+          if (!hsts_store)
+            logprintf (LOG_NOTQUIET, "ERROR: could not open HSTS store at '%s'. "
+                       "HSTS will be disabled.\n",
+                       filename);
+        }
+      else
+        logprintf (LOG_NOTQUIET, "ERROR: could not open HSTS store. HSTS will be disabled.\n");
+
+      xfree (filename);
+    }
+}
+
+static void
+save_hsts (void)
+{
+  if (hsts_store)
+    {
+      char *filename = get_hsts_database ();
+
+      if (filename && hsts_store_has_changed (hsts_store))
+        {
+          DEBUGP (("Saving HSTS entries to %s\n", filename));
+          hsts_store_save (hsts_store, filename);
+        }
+
+      hsts_store_close (hsts_store);
+
+      xfree (filename);
+    }
+}
+#endif
+
 /* Definition of command-line options. */
 
 static void _Noreturn print_help (void);
@@ -178,6 +261,9 @@ static struct cmdline_option option_data[] =
     { "backups", 0, OPT_BOOLEAN, "backups", -1 },
     { "base", 'B', OPT_VALUE, "base", -1 },
     { "bind-address", 0, OPT_VALUE, "bindaddress", -1 },
+#ifdef HAVE_LIBCARES
+    { "bind-dns-address", 0, OPT_VALUE, "binddnsaddress", -1 },
+#endif
     { "body-data", 0, OPT_VALUE, "bodydata", -1 },
     { "body-file", 0, OPT_VALUE, "bodyfile", -1 },
     { IF_SSL ("ca-certificate"), 0, OPT_VALUE, "cacertificate", -1 },
@@ -190,6 +276,7 @@ static struct cmdline_option option_data[] =
     { "config", 0, OPT_VALUE, "chooseconfig", -1 },
     { "connect-timeout", 0, OPT_VALUE, "connecttimeout", -1 },
     { "continue", 'c', OPT_BOOLEAN, "continue", -1 },
+    { "convert-file-only", 0, OPT_BOOLEAN, "convertfileonly", -1 },
     { "convert-links", 'k', OPT_BOOLEAN, "convertlinks", -1 },
     { "content-disposition", 0, OPT_BOOLEAN, "contentdisposition", -1 },
     { "content-on-error", 0, OPT_BOOLEAN, "contentonerror", -1 },
@@ -202,6 +289,9 @@ static struct cmdline_option option_data[] =
     { "directories", 0, OPT_BOOLEAN, "dirstruct", -1 },
     { "directory-prefix", 'P', OPT_VALUE, "dirprefix", -1 },
     { "dns-cache", 0, OPT_BOOLEAN, "dnscache", -1 },
+#ifdef HAVE_LIBCARES
+    { "dns-servers", 0, OPT_VALUE, "dnsservers", -1 },
+#endif
     { "dns-timeout", 0, OPT_VALUE, "dnstimeout", -1 },
     { "domains", 'D', OPT_VALUE, "domains", -1 },
     { "dont-remove-listing", 0, OPT__DONT_REMOVE_LISTING, NULL, no_argument },
@@ -219,10 +309,20 @@ static struct cmdline_option option_data[] =
     { "ftp-stmlf", 0, OPT_BOOLEAN, "ftpstmlf", -1 },
 #endif /* def __VMS */
     { "ftp-user", 0, OPT_VALUE, "ftpuser", -1 },
+#ifdef HAVE_SSL
+    { "ftps-clear-data-connection", 0, OPT_BOOLEAN, "ftpscleardataconnection", -1 },
+    { "ftps-fallback-to-ftp", 0, OPT_BOOLEAN, "ftpsfallbacktoftp", -1 },
+    { "ftps-implicit", 0, OPT_BOOLEAN, "ftpsimplicit", -1 },
+    { "ftps-resume-ssl", 0, OPT_BOOLEAN, "ftpsresumessl", -1 },
+#endif
     { "glob", 0, OPT_BOOLEAN, "glob", -1 },
     { "header", 0, OPT_VALUE, "header", -1 },
     { "help", 'h', OPT_FUNCALL, (void *)print_help, no_argument },
     { "host-directories", 0, OPT_BOOLEAN, "addhostdir", -1 },
+#ifdef HAVE_HSTS
+    { "hsts", 0, OPT_BOOLEAN, "hsts", -1},
+    { "hsts-file", 0, OPT_VALUE, "hsts-file", -1 },
+#endif
     { "html-extension", 'E', OPT_BOOLEAN, "adjustextension", -1 }, /* deprecated */
     { "htmlify", 0, OPT_BOOLEAN, "htmlify", -1 },
     { "http-keep-alive", 0, OPT_BOOLEAN, "httpkeepalive", -1 },
@@ -239,13 +339,20 @@ static struct cmdline_option option_data[] =
     { "inet6-only", '6', OPT_BOOLEAN, "inet6only", -1 },
 #endif
     { "input-file", 'i', OPT_VALUE, "input", -1 },
+#ifdef HAVE_METALINK
+    { "input-metalink", 0, OPT_VALUE, "input-metalink", -1 },
+#endif
     { "iri", 0, OPT_BOOLEAN, "iri", -1 },
     { "keep-session-cookies", 0, OPT_BOOLEAN, "keepsessioncookies", -1 },
     { "level", 'l', OPT_VALUE, "reclevel", -1 },
     { "limit-rate", 0, OPT_VALUE, "limitrate", -1 },
     { "load-cookies", 0, OPT_VALUE, "loadcookies", -1 },
     { "local-encoding", 0, OPT_VALUE, "localencoding", -1 },
+    { "rejected-log", 0, OPT_VALUE, "rejectedlog", -1 },
     { "max-redirect", 0, OPT_VALUE, "maxredirect", -1 },
+#ifdef HAVE_METALINK
+    { "metalink-over-http", 0, OPT_BOOLEAN, "metalink-over-http", -1 },
+#endif
     { "method", 0, OPT_VALUE, "method", -1 },
     { "mirror", 'm', OPT_BOOLEAN, "mirror", -1 },
     { "no", 'n', OPT__NO, NULL, required_argument },
@@ -258,9 +365,13 @@ static struct cmdline_option option_data[] =
     { "parent", 0, OPT__PARENT, NULL, optional_argument },
     { "passive-ftp", 0, OPT_BOOLEAN, "passiveftp", -1 },
     { "password", 0, OPT_VALUE, "password", -1 },
+    { IF_SSL ("pinnedpubkey"), 0, OPT_VALUE, "pinnedpubkey", -1 },
     { "post-data", 0, OPT_VALUE, "postdata", -1 },
     { "post-file", 0, OPT_VALUE, "postfile", -1 },
     { "prefer-family", 0, OPT_VALUE, "preferfamily", -1 },
+#ifdef HAVE_METALINK
+    { "preferred-location", 0, OPT_VALUE, "preferred-location", -1 },
+#endif
     { "preserve-permissions", 0, OPT_BOOLEAN, "preservepermissions", -1 },
     { IF_SSL ("private-key"), 0, OPT_VALUE, "privatekey", -1 },
     { IF_SSL ("private-key-type"), 0, OPT_VALUE, "privatekeytype", -1 },
@@ -299,6 +410,7 @@ static struct cmdline_option option_data[] =
     { "strict-comments", 0, OPT_BOOLEAN, "strictcomments", -1 },
     { "timeout", 'T', OPT_VALUE, "timeout", -1 },
     { "timestamping", 'N', OPT_BOOLEAN, "timestamping", -1 },
+    { "if-modified-since", 0, OPT_BOOLEAN, "if-modified-since", -1 },
     { "tries", 't', OPT_VALUE, "tries", -1 },
     { "unlink", 0, OPT_BOOLEAN, "unlink", -1 },
     { "trust-server-names", 0, OPT_BOOLEAN, "trustservernames", -1 },
@@ -340,8 +452,7 @@ no_prefix (const char *s)
 
   char *cp = p;
   int size = 3 + strlen (s) + 1;  /* "no-STRING\0" */
-  if (p + size >= buffer + sizeof (buffer))
-    abort ();
+  assert(p + size <= buffer + sizeof (buffer));
 
   cp[0] = 'n', cp[1] = 'o', cp[2] = '-';
   strcpy (cp + 3, s);
@@ -481,6 +592,10 @@ Logging and input file:\n"),
        --report-speed=TYPE         output bandwidth as TYPE.  TYPE can be bits\n"),
     N_("\
   -i,  --input-file=FILE           download URLs found in local or external FILE\n"),
+#ifdef HAVE_METALINK
+    N_("\
+       --input-metalink=FILE       download files covered in local Metalink FILE\n"),
+#endif
     N_("\
   -F,  --force-html                treat input file as HTML\n"),
     N_("\
@@ -490,6 +605,8 @@ Logging and input file:\n"),
        --config=FILE               specify config file to use\n"),
     N_("\
        --no-config                 do not read any config file\n"),
+    N_("\
+       --rejected-log=FILE         log reasons for URL rejection to FILE\n"),
     "\n",
 
     N_("\
@@ -515,7 +632,10 @@ Download:\n"),
   -N,  --timestamping              don't re-retrieve files unless newer than\n\
                                      local\n"),
     N_("\
-  --no-use-server-timestamps       don't set the local file's timestamp by\n\
+       --no-if-modified-since      don't use conditional if-modified-since get\n\
+                                     requests in timestamping mode\n"),
+    N_("\
+       --no-use-server-timestamps  don't set the local file's timestamp by\n\
                                      the one on the server\n"),
     N_("\
   -S,  --server-response           print server response\n"),
@@ -523,6 +643,12 @@ Download:\n"),
        --spider                    don't download anything\n"),
     N_("\
   -T,  --timeout=SECONDS           set all timeout values to SECONDS\n"),
+#ifdef HAVE_LIBCARES
+    N_("\
+       --dns-servers=ADDRESSES     list of DNS servers to query (comma separated)\n"),
+    N_("\
+       --bind-dns-address=ADDRESS  bind DNS resolver to ADDRESS (hostname or IP) on local host\n"),
+#endif
     N_("\
        --dns-timeout=SECS          set the DNS lookup timeout to SECS\n"),
     N_("\
@@ -572,6 +698,12 @@ Download:\n"),
        --remote-encoding=ENC       use ENC as the default remote encoding\n"),
     N_("\
        --unlink                    remove file before clobber\n"),
+#ifdef HAVE_METALINK
+    N_("\
+       --metalink-over-http        use Metalink metadata from HTTP response headers\n"),
+    N_("\
+       --preferred-location        preferred location for Metalink resources\n"),
+#endif
     "\n",
 
     N_("\
@@ -674,6 +806,11 @@ HTTPS (SSL/TLS) options:\n"),
        --ca-directory=DIR          directory where hash list of CAs is stored\n"),
     N_("\
        --crl-file=FILE             file with bundle of CRLs\n"),
+    N_("\
+       --pinnedpubkey=FILE/HASHES  Public key (PEM/DER) file, or any number\n\
+                                   of base64 encoded sha256 hashes preceded by\n\
+                                   \'sha256//\' and seperated by \';\', to verify\n\
+                                   peer against\n"),
 #if defined(HAVE_LIBSSL) || defined(HAVE_LIBSSL32)
     N_("\
        --random-file=FILE          file with random data for seeding the SSL PRNG\n"),
@@ -685,6 +822,16 @@ HTTPS (SSL/TLS) options:\n"),
     "\n",
 #endif /* HAVE_SSL */
 
+#ifdef HAVE_HSTS
+    N_("\
+HSTS options:\n"),
+    N_("\
+       --no-hsts                   disable HSTS\n"),
+    N_("\
+       --hsts-file                 path of HSTS database (will override default)\n"),
+    "\n",
+#endif
+
     N_("\
 FTP options:\n"),
 #ifdef __VMS
@@ -707,6 +854,20 @@ FTP options:\n"),
        --retr-symlinks             when recursing, get linked-to files (not dir)\n"),
     "\n",
 
+#ifdef HAVE_SSL
+    N_("\
+FTPS options:\n"),
+    N_("\
+       --ftps-implicit                 use implicit FTPS (default port is 990)\n"),
+    N_("\
+       --ftps-resume-ssl               resume the SSL/TLS session started in the control connection when\n"
+        "                                         opening a data connection\n"),
+    N_("\
+       --ftps-clear-data-connection    cipher the control channel only; all the data will be in plaintext\n"),
+    N_("\
+       --ftps-fallback-to-ftp          fall back to FTP if FTPS is not supported in the target server\n"),
+#endif
+
     N_("\
 WARC options:\n"),
     N_("\
@@ -743,6 +904,8 @@ Recursive download:\n"),
     N_("\
   -k,  --convert-links             make links in downloaded HTML or CSS point to\n\
                                      local files\n"),
+    N_("\
+       --convert-file-only         convert the file part of the URLs only (usually known as the basename)\n"),
     N_("\
        --backups=N                 before writing file X, rotate up to N backup files\n"),
 
@@ -874,7 +1037,10 @@ format_and_print_line (const char *prefix, const char *line,
   line_dup = xstrdup (line);
 
   if (printf ("%s", prefix) < 0)
-    return -1;
+    {
+      xfree (line_dup);
+      return -1;
+    }
 
   /* Wrap to new line after prefix. */
   remaining_chars = 0;
@@ -889,7 +1055,10 @@ format_and_print_line (const char *prefix, const char *line,
       if (remaining_chars <= (int) strlen (token))
         {
           if (printf ("\n%*c", TABULATION, ' ') < 0)
-            return -1;
+            {
+              xfree (line_dup);
+              return -1;
+            }
           remaining_chars = line_length - TABULATION;
         }
       if (printf ("%s ", token) < 0)
@@ -902,7 +1071,10 @@ format_and_print_line (const char *prefix, const char *line,
     }
 
   if (printf ("\n") < 0)
-    return -1;
+    {
+      xfree (line_dup);
+      return -1;
+    }
 
   xfree (line_dup);
   return 0;
@@ -986,7 +1158,7 @@ print_version (void)
   /* TRANSLATORS: When available, an actual copyright character
      (circle-c) should be used in preference to "(C)". */
   if (printf (_("\
-Copyright (C) %s Free Software Foundation, Inc.\n"), "2014") < 0)
+Copyright (C) %s Free Software Foundation, Inc.\n"), "2015") < 0)
     exit (WGET_EXIT_IO_FAIL);
   if (fputs (_("\
 License GPLv3+: GNU GPL version 3 or later\n\
@@ -1245,11 +1417,14 @@ main (int argc, char **argv)
   /* All user options have now been processed, so it's now safe to do
      interoption dependency checks. */
 
-  if (opt.noclobber && opt.convert_links)
+  if (opt.noclobber && (opt.convert_links || opt.convert_file_only))
     {
       fprintf (stderr,
-               _("Both --no-clobber and --convert-links were specified,"
-                 " only --convert-links will be used.\n"));
+               opt.convert_links ?
+                   _("Both --no-clobber and --convert-links were specified,"
+                     " only --convert-links will be used.\n") :
+                    _("Both --no-clobber and --convert-file-only were specified,"
+                      " only --convert-file-only will be used.\n"));
       opt.noclobber = false;
     }
 
@@ -1272,6 +1447,12 @@ main (int argc, char **argv)
   if (opt.verbose == -1)
     opt.verbose = !opt.quiet;
 
+  if (!opt.verbose && opt.show_progress == -1)
+    opt.show_progress = false;
+
+  if (opt.quiet && opt.show_progress == -1)
+    opt.show_progress = false;
+
   /* Sanity checks.  */
   if (opt.verbose && opt.quiet)
     {
@@ -1297,11 +1478,11 @@ Can't timestamp and not clobber old files at the same time.\n"));
 #endif
   if (opt.output_document)
     {
-      if (opt.convert_links
+      if ((opt.convert_links || opt.convert_file_only)
           && (nurl > 1 || opt.page_requisites || opt.recursive))
         {
           fputs (_("\
-Cannot specify both -k and -O if multiple URLs are given, or in combination\n\
+Cannot specify both -k or --convert-file-only and -O if multiple URLs are given, or in combination\n\
 with -p or -r. See the manual for details.\n\n"), stderr);
           print_usage (1);
           exit (WGET_EXIT_GENERIC_ERROR);
@@ -1388,7 +1569,11 @@ for details.\n\n"));
       opt.always_rest = false;
     }
 
-  if (!nurl && !opt.input_filename)
+  if (!nurl && !opt.input_filename
+#ifdef HAVE_METALINK
+      && !opt.input_metalink
+#endif
+      )
     {
       /* No URL specified.  */
       fprintf (stderr, _("%s: missing URL\n"), exec_name);
@@ -1602,15 +1787,72 @@ for details.\n\n"));
           if (fstat (fileno (output_stream), &st) == 0 && S_ISREG (st.st_mode))
             output_stream_regular = true;
         }
-      if (!output_stream_regular && opt.convert_links)
+      if (!output_stream_regular && (opt.convert_links || opt.recursive))
         {
-          fprintf (stderr, _("-k can be used together with -O only if \
+          fprintf (stderr, _("-k or -r can be used together with -O only if \
 outputting to a regular file.\n"));
-          print_usage (1);
+          exit (WGET_EXIT_GENERIC_ERROR);
+        }
+      if (!output_stream_regular && (opt.convert_links || opt.convert_file_only))
+        {
+          fprintf (stderr, _("--convert-links or --convert-file-only can be used together \
+only if outputting to a regular file.\n"));
           exit (WGET_EXIT_GENERIC_ERROR);
         }
     }
 
+#ifdef HAVE_LIBCARES
+  if (opt.bind_dns_address || opt.dns_servers)
+    {
+      if (ares_library_init (ARES_LIB_INIT_ALL))
+        {
+          fprintf (stderr, _("Failed to init libcares\n"));
+          exit (WGET_EXIT_GENERIC_ERROR);
+        }
+
+      if (ares_init (&ares) != ARES_SUCCESS)
+        {
+          fprintf (stderr, _("Failed to init c-ares channel\n"));
+          exit (WGET_EXIT_GENERIC_ERROR);
+        }
+
+      if (opt.bind_dns_address)
+        {
+          struct in_addr a4;
+#ifdef ENABLE_IPV6
+          struct in6_addr a6;
+#endif
+
+          if (inet_pton (AF_INET, opt.bind_dns_address, &a4) == 1)
+            {
+              ares_set_local_ip4 (ares, ntohl (a4.s_addr));
+            }
+#ifdef ENABLE_IPV6
+          else if (inet_pton (AF_INET6, opt.bind_dns_address, &a6) == 1)
+            {
+              ares_set_local_ip6 (ares, (unsigned char *) &a6);
+            }
+#endif
+          else
+            {
+              fprintf (stderr, _("Failed to parse IP address '%s'\n"), opt.bind_dns_address);
+              exit (WGET_EXIT_GENERIC_ERROR);
+            }
+        }
+
+      if (opt.dns_servers)
+        {
+          int result;
+
+          if ((result = ares_set_servers_csv (ares, opt.dns_servers)) != ARES_SUCCESS)
+            {
+              fprintf (stderr, _("Failed to set DNS server(s) '%s' (%d)\n"), opt.dns_servers, result);
+              exit (WGET_EXIT_GENERIC_ERROR);
+            }
+        }
+    }
+#endif
+
 #ifdef __VMS
   /* Set global ODS5 flag according to the specified destination (if
      any), otherwise according to the current default device.
@@ -1645,6 +1887,16 @@ outputting to a regular file.\n"));
   signal (SIGWINCH, progress_handle_sigwinch);
 #endif
 
+#ifdef HAVE_HSTS
+  /* Load the HSTS database.
+     Maybe all the URLs are FTP(S), in which case HSTS would not be needed,
+     but this is the best place to do it, and it shouldn't be a critical
+     performance hit.
+   */
+  if (opt.hsts)
+    load_hsts ();
+#endif
+
   /* Retrieve the URLs from argument list.  */
   for (t = url; *t; t++)
     {
@@ -1669,12 +1921,21 @@ outputting to a regular file.\n"));
       else
         {
           if ((opt.recursive || opt.page_requisites)
-              && (url_scheme (*t) != SCHEME_FTP || url_uses_proxy (url_parsed)))
+              && ((url_scheme (*t) != SCHEME_FTP
+#ifdef HAVE_SSL
+              && url_scheme (*t) != SCHEME_FTPS
+#endif
+              )
+                  || url_uses_proxy (url_parsed)))
             {
               int old_follow_ftp = opt.follow_ftp;
 
               /* Turn opt.follow_ftp on in case of recursive FTP retrieval */
-              if (url_scheme (*t) == SCHEME_FTP)
+              if (url_scheme (*t) == SCHEME_FTP
+#ifdef HAVE_SSL
+                  || url_scheme (*t) == SCHEME_FTPS
+#endif
+                  )
                 opt.follow_ftp = 1;
 
               retrieve_tree (url_parsed, NULL);
@@ -1682,10 +1943,10 @@ outputting to a regular file.\n"));
               opt.follow_ftp = old_follow_ftp;
             }
           else
-          {
-            retrieve_url (url_parsed, *t, &filename, &redirected_URL, NULL,
-                          &dt, opt.recursive, iri, true);
-          }
+            {
+              retrieve_url (url_parsed, *t, &filename, &redirected_URL, NULL,
+                            &dt, opt.recursive, iri, true);
+            }
 
           if (opt.delete_after && filename != NULL && file_exists_p (filename))
             {
@@ -1713,6 +1974,57 @@ outputting to a regular file.\n"));
                    opt.input_filename);
     }
 
+#ifdef HAVE_METALINK
+  /* Finally, from metlink file, if any.  */
+  if (opt.input_metalink)
+    {
+      metalink_error_t meta_err;
+      uerr_t retr_err;
+      metalink_t *metalink;
+
+      meta_err = metalink_parse_file (opt.input_metalink, &metalink);
+
+      if (meta_err)
+        {
+          logprintf (LOG_NOTQUIET, _("Unable to parse metalink file %s.\n"),
+                     opt.input_metalink);
+          retr_err = METALINK_PARSE_ERROR;
+        }
+      else
+        {
+          /* We need to sort the resources if preferred location
+             was specified by the user.  */
+          if (opt.preferred_location && opt.preferred_location[0])
+            {
+              metalink_file_t **mfile_ptr;
+              for (mfile_ptr = metalink->files; *mfile_ptr; mfile_ptr++)
+                {
+                  metalink_resource_t **mres_ptr;
+                  metalink_file_t *mfile = *mfile_ptr;
+                  size_t mres_count = 0;
+
+                  for (mres_ptr = mfile->resources; *mres_ptr; mres_ptr++)
+                    mres_count++;
+
+                  stable_sort (mfile->resources,
+                               mres_count,
+                               sizeof (metalink_resource_t *),
+                               metalink_res_cmp);
+                }
+            }
+          retr_err = retrieve_from_metalink (metalink);
+          if (retr_err != RETROK)
+            {
+              logprintf (LOG_NOTQUIET,
+                         _("Could not download all resources from %s.\n"),
+                         quote (opt.input_metalink));
+            }
+          metalink_delete (metalink);
+        }
+      inform_exit_status (retr_err);
+    }
+#endif /* HAVE_METALINK */
+
   /* Print broken links. */
   if (opt.recursive && opt.spider)
     print_broken_links ();
@@ -1752,7 +2064,12 @@ outputting to a regular file.\n"));
   if (opt.cookies_output)
     save_cookies ();
 
-  if (opt.convert_links && !opt.delete_after)
+#ifdef HAVE_HSTS
+  if (opt.hsts && hsts_store)
+    save_hsts ();
+#endif
+
+  if ((opt.convert_links || opt.convert_file_only) && !opt.delete_after)
     convert_all_links ();
 
   cleanup ();

src/metalink.c

@@ -0,0 +1,782 @@
+/* Metalink module.
+   Copyright (C) 2015 Free Software Foundation, Inc.
+
+This file is part of GNU Wget.
+
+GNU Wget is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 3 of the License, or (at
+your option) any later version.
+
+GNU Wget is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Wget.  If not, see <http://www.gnu.org/licenses/>.
+
+Additional permission under GNU GPL version 3 section 7
+
+If you modify this program, or any covered work, by linking or
+combining it with the OpenSSL project's OpenSSL library (or a
+modified version of that library), containing parts covered by the
+terms of the OpenSSL or SSLeay licenses, the Free Software Foundation
+grants you additional permission to convey the resulting work.
+Corresponding Source for a non-source form of such a combination
+shall include the source code for the parts of OpenSSL used as well
+as that of the covered work.  */
+
+#include "wget.h"
+#ifdef HAVE_METALINK
+
+#include "metalink.h"
+#include "retr.h"
+#include "exits.h"
+#include "utils.h"
+#include "sha256.h"
+#include "xstrndup.h"
+#include <errno.h>
+#include <unistd.h> /* For unlink.  */
+#include <metalink/metalink_parser.h>
+#ifdef HAVE_GPGME
+#include <gpgme.h>
+#include <fcntl.h> /* For open and close.  */
+#endif
+
+#ifdef TESTING
+#include "test.h"
+#endif
+
+/* Loop through all files in metalink structure and retrieve them.
+   Returns RETROK if all files were downloaded.
+   Returns last retrieval error (from retrieve_url) if some files
+   could not be downloaded.  */
+uerr_t
+retrieve_from_metalink (const metalink_t* metalink)
+{
+  metalink_file_t **mfile_ptr;
+  uerr_t last_retr_err = RETROK; /* Store last encountered retrieve error.  */
+
+  FILE *_output_stream = output_stream;
+  bool _output_stream_regular = output_stream_regular;
+  char *_output_document = opt.output_document;
+
+  DEBUGP (("Retrieving from Metalink\n"));
+
+  /* No files to download.  */
+  if (!metalink->files)
+    return RETROK;
+
+  if (opt.output_document)
+    {
+      /* We cannot support output_document as we need to compute checksum
+         of downloaded file, and to remove it if the checksum is bad.  */
+      logputs (LOG_NOTQUIET,
+               _("-O not supported for metalink download. Ignoring.\n"));
+    }
+
+  for (mfile_ptr = metalink->files; *mfile_ptr; mfile_ptr++)
+    {
+      metalink_file_t *mfile = *mfile_ptr;
+      metalink_resource_t **mres_ptr;
+      char *filename = NULL;
+      bool hash_ok = false;
+
+      uerr_t retr_err = METALINK_MISSING_RESOURCE;
+
+      /* -1 -> file should be rejected
+         0 -> could not verify
+         1 -> verified successfully  */
+      char sig_status = 0;
+
+      output_stream = NULL;
+
+      DEBUGP (("Processing metalink file %s...\n", quote (mfile->name)));
+
+      /* Resources are sorted by priority.  */
+      for (mres_ptr = mfile->resources; *mres_ptr; mres_ptr++)
+        {
+          metalink_resource_t *mres = *mres_ptr;
+          metalink_checksum_t **mchksum_ptr, *mchksum;
+          struct iri *iri;
+          struct url *url;
+          int url_err;
+
+          if (!RES_TYPE_SUPPORTED (mres->type))
+            {
+              logprintf (LOG_VERBOSE,
+                         _("Resource type %s not supported, ignoring...\n"),
+                         quote (mres->type));
+              continue;
+            }
+
+          retr_err = METALINK_RETR_ERROR;
+
+          /* If output_stream is not NULL, then we have failed on
+             previous resource and are retrying. Thus, remove the file.  */
+          if (output_stream)
+            {
+              fclose (output_stream);
+              output_stream = NULL;
+              if (unlink (filename))
+                logprintf (LOG_NOTQUIET, "unlink: %s\n", strerror (errno));
+              xfree (filename);
+            }
+
+          /* Parse our resource URL.  */
+          iri = iri_new ();
+          set_uri_encoding (iri, opt.locale, true);
+          url = url_parse (mres->url, &url_err, iri, false);
+
+          if (!url)
+            {
+              char *error = url_error (mres->url, url_err);
+              logprintf (LOG_NOTQUIET, "%s: %s.\n", mres->url, error);
+              xfree (error);
+              inform_exit_status (URLERROR);
+              iri_free (iri);
+              continue;
+            }
+          else
+            {
+              /* Avoid recursive Metalink from HTTP headers.  */
+              bool _metalink_http = opt.metalink_over_http;
+
+              /* Assure proper local file name regardless of the URL
+                 of particular Metalink resource.
+                 To do that we create the local file here and put
+                 it as output_stream. We restore the original configuration
+                 after we are finished with the file.  */
+              output_stream = unique_create (mfile->name, true, &filename);
+              output_stream_regular = true;
+
+              /* Store the real file name for displaying in messages.  */
+              opt.output_document = filename;
+
+              opt.metalink_over_http = false;
+              DEBUGP (("Storing to %s\n", filename));
+              retr_err = retrieve_url (url, mres->url, NULL, NULL,
+                                       NULL, NULL, opt.recursive, iri, false);
+              opt.metalink_over_http = _metalink_http;
+            }
+          url_free (url);
+          iri_free (iri);
+
+          if (retr_err == RETROK)
+            {
+              FILE *local_file;
+
+              /* Check the digest.  */
+              local_file = fopen (filename, "rb");
+              if (!local_file)
+                {
+                  logprintf (LOG_NOTQUIET, _("Could not open downloaded file.\n"));
+                  continue;
+                }
+
+              for (mchksum_ptr = mfile->checksums; *mchksum_ptr; mchksum_ptr++)
+                {
+                  char sha256[SHA256_DIGEST_SIZE];
+                  char sha256_txt[2 * SHA256_DIGEST_SIZE + 1];
+
+                  mchksum = *mchksum_ptr;
+
+                  /* I have seen both variants...  */
+                  if (strcasecmp (mchksum->type, "sha256")
+                      && strcasecmp (mchksum->type, "sha-256"))
+                    {
+                      DEBUGP (("Ignoring unsupported checksum type %s.\n",
+                               quote (mchksum->type)));
+                      continue;
+                    }
+
+                  logprintf (LOG_VERBOSE, _("Computing checksum for %s\n"),
+                             quote (mfile->name));
+
+                  sha256_stream (local_file, sha256);
+                  wg_hex_to_string (sha256_txt, sha256, SHA256_DIGEST_SIZE);
+                  DEBUGP (("Declared hash: %s\n", mchksum->hash));
+                  DEBUGP (("Computed hash: %s\n", sha256_txt));
+                  if (!strcmp (sha256_txt, mchksum->hash))
+                    {
+                      logputs (LOG_VERBOSE,
+                               _("Checksum matches.\n"));
+                      hash_ok = true;
+                    }
+                  else
+                    {
+                      logprintf (LOG_NOTQUIET,
+                                 _("Checksum mismatch for file %s.\n"),
+                                 quote (mfile->name));
+                      hash_ok = false;
+                    }
+
+                  /* Stop as soon as we checked the supported checksum.  */
+                  break;
+                } /* Iterate over available checksums.  */
+              fclose (local_file);
+              local_file = NULL;
+
+              if (!hash_ok)
+                continue;
+
+              sig_status = 0; /* Not verified.  */
+
+#ifdef HAVE_GPGME
+              /* Check the crypto signature.
+
+                 Note that the signtures from Metalink in XML will not be
+                 parsed when using libmetalink version older than 0.1.3.
+                 Metalink-over-HTTP is not affected by this problem.  */
+              if (mfile->signature)
+                {
+                  metalink_signature_t *msig = mfile->signature;
+                  gpgme_error_t gpgerr;
+                  gpgme_ctx_t gpgctx;
+                  gpgme_data_t gpgsigdata, gpgdata;
+                  gpgme_verify_result_t gpgres;
+                  gpgme_signature_t gpgsig;
+                  gpgme_protocol_t gpgprot = GPGME_PROTOCOL_UNKNOWN;
+                  int fd = -1;
+
+                  /* Initialize the library - as name suggests.  */
+                  gpgme_check_version (NULL);
+
+                  /* Open data file.  */
+                  fd = open (filename, O_RDONLY);
+                  if (fd == -1)
+                    {
+                      logputs (LOG_NOTQUIET,
+                               _("Could not open downloaded file for signature "
+                                 "verification.\n"));
+                      goto gpg_skip_verification;
+                    }
+
+                  /* Assign file descriptor to GPG data structure.  */
+                  gpgerr = gpgme_data_new_from_fd (&gpgdata, fd);
+                  if (gpgerr != GPG_ERR_NO_ERROR)
+                    {
+                      logprintf (LOG_NOTQUIET,
+                                 "GPGME data_new_from_fd: %s\n",
+                                 gpgme_strerror (gpgerr));
+                      goto gpg_skip_verification;
+                    }
+
+                  /* Prepare new GPGME context.  */
+                  gpgerr = gpgme_new (&gpgctx);
+                  if (gpgerr != GPG_ERR_NO_ERROR)
+                    {
+                      logprintf (LOG_NOTQUIET,
+                                 "GPGME new: %s\n",
+                                 gpgme_strerror (gpgerr));
+                      gpgme_data_release (gpgdata);
+                      goto gpg_skip_verification;
+                    }
+
+                  DEBUGP (("Verifying signature %s:\n%s\n",
+                           quote (msig->mediatype),
+                           msig->signature));
+
+                  /* Check signature type.  */
+                  if (!strcmp (msig->mediatype, "application/pgp-signature"))
+                    gpgprot = GPGME_PROTOCOL_OpenPGP;
+                  else /* Unsupported signature type.  */
+                    {
+                      gpgme_release (gpgctx);
+                      gpgme_data_release (gpgdata);
+                      goto gpg_skip_verification;
+                    }
+
+                  gpgerr = gpgme_set_protocol (gpgctx, gpgprot);
+                  if (gpgerr != GPG_ERR_NO_ERROR)
+                    {
+                      logprintf (LOG_NOTQUIET,
+                                 "GPGME set_protocol: %s\n",
+                                 gpgme_strerror (gpgerr));
+                      gpgme_release (gpgctx);
+                      gpgme_data_release (gpgdata);
+                      goto gpg_skip_verification;
+                    }
+
+                  /* Load the signature.  */
+                  gpgerr = gpgme_data_new_from_mem (&gpgsigdata,
+                                                    msig->signature,
+                                                    strlen (msig->signature),
+                                                    0);
+                  if (gpgerr != GPG_ERR_NO_ERROR)
+                    {
+                      logprintf (LOG_NOTQUIET,
+                                 _("GPGME data_new_from_mem: %s\n"),
+                                 gpgme_strerror (gpgerr));
+                      gpgme_release (gpgctx);
+                      gpgme_data_release (gpgdata);
+                      goto gpg_skip_verification;
+                    }
+
+                  /* Verify the signature.  */
+                  gpgerr = gpgme_op_verify (gpgctx, gpgsigdata, gpgdata, NULL);
+                  if (gpgerr != GPG_ERR_NO_ERROR)
+                    {
+                      logprintf (LOG_NOTQUIET,
+                                 _("GPGME op_verify: %s\n"),
+                                 gpgme_strerror (gpgerr));
+                      gpgme_data_release (gpgsigdata);
+                      gpgme_release (gpgctx);
+                      gpgme_data_release (gpgdata);
+                      goto gpg_skip_verification;
+                    }
+
+                  /* Check the results.  */
+                  gpgres = gpgme_op_verify_result (gpgctx);
+                  if (!gpgres)
+                    {
+                      logputs (LOG_NOTQUIET,
+                               _("GPGME op_verify_result: NULL\n"));
+                      gpgme_data_release (gpgsigdata);
+                      gpgme_release (gpgctx);
+                      gpgme_data_release (gpgdata);
+                      goto gpg_skip_verification;
+                    }
+
+                  /* The list is null-terminated.  */
+                  for (gpgsig = gpgres->signatures; gpgsig; gpgsig = gpgsig->next)
+                    {
+                      DEBUGP (("Checking signature %s\n", gpgsig->fpr));
+
+                      if (gpgsig->summary
+                          & (GPGME_SIGSUM_VALID | GPGME_SIGSUM_GREEN))
+                        {
+                          logputs (LOG_VERBOSE,
+                                   _("Signature validation suceeded.\n"));
+                          sig_status = 1;
+                          break;
+                        }
+
+                      if (gpgsig->summary & GPGME_SIGSUM_RED)
+                        {
+                          logputs (LOG_NOTQUIET,
+                                   _("Invalid signature. Rejecting resource.\n"));
+                          sig_status = -1;
+                          break;
+                        }
+
+                      if (gpgsig->summary == 0
+                          && (gpgsig->status & 0xFFFF) == GPG_ERR_NO_ERROR)
+                        {
+                          logputs (LOG_VERBOSE,
+                                   _("Data matches signature, but signature "
+                                     "is not trusted.\n"));
+                        }
+
+                      if ((gpgsig->status & 0xFFFF) != GPG_ERR_NO_ERROR)
+                        {
+                          logprintf (LOG_NOTQUIET,
+                                     "GPGME: %s\n",
+                                     gpgme_strerror (gpgsig->status & 0xFFFF));
+                        }
+                    }
+                  gpgme_data_release (gpgsigdata);
+                  gpgme_release (gpgctx);
+                  gpgme_data_release (gpgdata);
+gpg_skip_verification:
+                  if (fd != -1)
+                    close (fd);
+                } /* endif (mfile->signature) */
+#endif
+              /* Stop if file was downloaded with success.  */
+              if (sig_status >= 0)
+                break;
+            } /* endif RETR_OK.  */
+        } /* Iterate over resources.  */
+
+      if (retr_err != RETROK)
+        {
+          logprintf (LOG_VERBOSE, _("Failed to download %s. Skipping resource.\n"),
+                     quote (mfile->name));
+        }
+      else if (!hash_ok)
+        {
+          retr_err = METALINK_CHKSUM_ERROR;
+          logprintf (LOG_NOTQUIET,
+                     _("File %s retrieved but checksum does not match. "
+                       "\n"), quote (mfile->name));
+        }
+#ifdef HAVE_GPGME
+        /* Signature will be only validated if hash check was successful.  */
+      else if (sig_status < 0)
+        {
+          retr_err = METALINK_SIG_ERROR;
+          logprintf (LOG_NOTQUIET,
+                     _("File %s retrieved but signature does not match. "
+                       "\n"), quote (mfile->name));
+        }
+#endif
+      last_retr_err = retr_err == RETROK ? last_retr_err : retr_err;
+
+      /* Remove the file if error encountered or if option specified.
+         Note: the file has been downloaded using *_loop. Therefore, it
+         is not necessary to keep the file for continuated download.  */
+      if ((retr_err != RETROK || opt.delete_after)
+           && filename != NULL && file_exists_p (filename))
+        {
+          logprintf (LOG_VERBOSE, _("Removing %s.\n"), quote (filename));
+          if (unlink (filename))
+            logprintf (LOG_NOTQUIET, "unlink: %s\n", strerror (errno));
+        }
+      if (output_stream)
+        {
+          fclose (output_stream);
+          output_stream = NULL;
+        }
+      xfree (filename);
+    } /* Iterate over files.  */
+
+  /* Restore original values.  */
+  opt.output_document = _output_document;
+  output_stream_regular = _output_stream_regular;
+  output_stream = _output_stream;
+
+  return last_retr_err;
+}
+
+int metalink_res_cmp (const void* v1, const void* v2)
+{
+  const metalink_resource_t *res1 = *(metalink_resource_t **) v1,
+                            *res2 = *(metalink_resource_t **) v2;
+  if (res1->preference != res2->preference)
+    return res2->preference - res1->preference;
+  if (res1->priority != res2->priority)
+    return res1->priority - res2->priority;
+  if (opt.preferred_location)
+    {
+      int cmp = 0;
+      if (res1->location &&
+          !strcasecmp (opt.preferred_location, res1->location))
+        cmp -= 1;
+      if (res2->location &&
+          !strcasecmp (opt.preferred_location, res2->location))
+        cmp += 1;
+      return cmp;
+    }
+  return 0;
+}
+
+/*
+  Find value of given key. This is intended for Link header, but will
+  work with any header that uses ';' as field separator and '=' as key-value
+  separator.
+
+  Link           = "Link" ":" #link-value
+  link-value     = "<" URI-Reference ">" *( ";" link-param )
+  link-param     = ( ( "rel" "=" relation-types )
+                 | ( "anchor" "=" <"> URI-Reference <"> )
+                 | ( "rev" "=" relation-types )
+                 | ( "hreflang" "=" Language-Tag )
+                 | ( "media" "=" ( MediaDesc | ( <"> MediaDesc <"> ) ) )
+                 | ( "title" "=" quoted-string )
+                 | ( "title*" "=" ext-value )
+                 | ( "type" "=" ( media-type | quoted-mt ) )
+                 | ( link-extension ) )
+  link-extension = ( parmname [ "=" ( ptoken | quoted-string ) ] )
+                 | ( ext-name-star "=" ext-value )
+  ext-name-star  = parmname "*" ; reserved for RFC2231-profiled
+                                ; extensions.  Whitespace NOT
+                                ; allowed in between.
+  ptoken         = 1*ptokenchar
+  ptokenchar     = "!" | "#" | "$" | "%" | "&" | "'" | "("
+                 | ")" | "*" | "+" | "-" | "." | "/" | DIGIT
+                 | ":" | "<" | "=" | ">" | "?" | "@" | ALPHA
+                 | "[" | "]" | "^" | "_" | "`" | "{" | "|"
+                 | "}" | "~"
+  media-type     = type-name "/" subtype-name
+  quoted-mt      = <"> media-type <">
+  relation-types = relation-type
+                 | <"> relation-type *( 1*SP relation-type ) <">
+  relation-type  = reg-rel-type | ext-rel-type
+  reg-rel-type   = LOALPHA *( LOALPHA | DIGIT | "." | "-" )
+  ext-rel-type   = URI
+
+ See more: rfc5988
+*/
+bool
+find_key_value (const char *start, const char *end, const char *key, char **value)
+{
+  const char *eq;
+  size_t key_len = strlen (key);
+  const char *val_beg, *val_end;
+  const char *key_beg;
+
+  key_beg = start;
+
+  while (key_beg + key_len + 1 < end)
+    {
+      /* Skip whitespaces.  */
+      while (key_beg + key_len + 1 < end && c_isspace (*key_beg))
+        key_beg++;
+      if (strncmp (key_beg, key, key_len))
+        {
+          /* Find next token.  */
+          while (key_beg + key_len + 1 < end && *key_beg != ';')
+            key_beg++;
+          key_beg++;
+          continue;
+        }
+      else
+        {
+          /* Find equals sign.  */
+          eq = key_beg + key_len;
+          while (eq < end && c_isspace (*eq))
+            eq++;
+          if (eq == end)
+            return false;
+          if (*eq != '=')
+            {
+              key_beg++;
+              continue;
+            }
+
+          val_beg = eq + 1;
+          while (val_beg < end && c_isspace (*val_beg))
+            val_beg++;
+          if (val_beg == end)
+            return false;
+          val_end = val_beg + 1;
+          while (val_end < end && *val_end != ';' && !c_isspace (*val_end))
+            val_end++;
+          *value = xstrndup (val_beg, val_end - val_beg);
+          return true;
+        }
+    }
+  *value = NULL;
+  return false;
+}
+
+/* This is to check if given token exists in HTTP header. Tokens are
+   separated by ';'. */
+bool
+has_key (const char *start, const char *end, const char *key)
+{
+  const char *pos; /* Here would the token start.  */
+  size_t key_len = strlen (key);
+
+  pos = start;
+  while (pos + key_len <= end)
+    {
+      /* Skip whitespaces at beginning.  */
+      while (pos + key_len <= end && c_isspace (*pos))
+        pos++;
+
+      /* Does the prefix of pos match our key?  */
+      if (strncmp (key, pos, key_len))
+        {
+          /* This was not a match.
+             Skip all characters until beginning of next token.  */
+          while (pos + key_len <= end && *pos != ';')
+            pos++;
+          pos++;
+          continue;
+        }
+
+      /* key is prefix of pos. Is it the exact token or just a prefix?  */
+      pos += key_len;
+      while (pos < end && c_isspace (*pos))
+        pos++;
+      if (pos == end || *pos == ';')
+        return true;
+
+      /* This was not a match (just a prefix).
+         Skip all characters until beginning of next token.  */
+      while (pos + key_len <= end && *pos != ';')
+        pos++;
+      pos++;
+    }
+  return false;
+}
+
+/* Find all key=value pairs delimited with ';' or ','. This is intended for
+   Digest header parsing.
+   The usage is:
+
+   const char *pos;
+   for (pos = header_beg; pos = find_key_values (pos, header_end, &key, &val); pos++)
+   {
+     ...
+   }
+
+ */
+const char *
+find_key_values (const char *start, const char *end, char **key, char **value)
+{
+  const char *key_start, *key_end;
+  const char *eq;
+  const char *val_start, *val_end;
+
+  eq = start;
+  while (eq < end && *eq != '=')
+    {
+      /* Skip tokens without =value part.  */
+      if (*eq == ';' || *eq == ',')
+        start = eq + 1;
+      eq++;
+    }
+
+  if (eq >= end)
+    return NULL;
+
+  key_start = start;
+  while (key_start < eq && c_isspace (*key_start))
+    key_start++;
+
+  key_end = eq - 1;
+  while (key_end > key_start && c_isspace (*key_end))
+    key_end--;
+  key_end++;
+
+  val_start = eq + 1;
+  while (val_start < end && c_isspace (*val_start))
+    val_start++;
+
+  val_end = val_start;
+
+  while (val_end < end && *val_end != ';' &&
+         *val_end != ',' && !c_isspace (*val_end))
+    val_end++;
+
+  *key = xstrndup (key_start, key_end - key_start);
+  *value = xstrndup (val_start, val_end - val_start);
+
+  /* Skip trailing whitespaces.  */
+  while (val_end < end && c_isspace (*val_end))
+    val_end++;
+
+  return val_end;
+}
+
+#ifdef TESTING
+const char *
+test_find_key_values (void)
+{
+  static const char *header_data = "key1=val1;key2=val2 ;key3=val3; key4=val4"\
+                                   " ; key5=val5;key6 =val6;key7= val7; "\
+                                   "key8 = val8 ;    key9    =   val9       "\
+                                   "    ,key10= val10,key11,key12=val12";
+  static const struct
+  {
+    const char *key;
+    const char *val;
+  } test_array[] =
+  {
+    { "key1", "val1" },
+    { "key2", "val2" },
+    { "key3", "val3" },
+    { "key4", "val4" },
+    { "key5", "val5" },
+    { "key6", "val6" },
+    { "key7", "val7" },
+    { "key8", "val8" },
+    { "key9", "val9" },
+    { "key10", "val10" },
+    { "key12", "val12" }
+  };
+  const char *pos;
+  char *key, *value;
+  size_t i = 0;
+
+  for (pos = header_data; (pos = find_key_values (pos,
+                                                 header_data + strlen (header_data),
+                                                 &key, &value)); pos++)
+    {
+      mu_assert ("test_find_key_values: wrong result",
+                 !strcmp (test_array[i].val, value) &&
+                 !strcmp (test_array[i].key, key));
+      xfree (key);
+      xfree (value);
+      i++;
+    }
+
+  return NULL;
+}
+
+const char *
+test_find_key_value (void)
+{
+  static const char *header_data = "key1=val1;key2=val2 ;key3=val3; key4=val4"\
+                                   " ; key5=val5;key6 =val6;key7= val7; "\
+                                   "key8 = val8 ;    key9    =   val9       ";
+  static const struct
+  {
+    const char *key;
+    const char *val;
+    bool result;
+  } test_array[] =
+  {
+    { "key1",  "val1", true },
+    { "key2",  "val2", true },
+    { "key3",  "val3", true },
+    { "key4",  "val4", true },
+    { "key5",  "val5", true },
+    { "key6",  "val6", true },
+    { "key7",  "val7", true },
+    { "key8",  "val8", true },
+    { "key9",  "val9", true },
+    { "key10", NULL,   false },
+    {  "ey1",  NULL,   false },
+    { "dey1",  NULL,   false }
+  };
+  size_t i;
+
+  for (i=0; i < countof (test_array); ++i)
+    {
+      bool result;
+      char *value;
+
+      result = find_key_value (header_data,
+                               header_data + strlen(header_data),
+                               test_array[i].key, &value);
+
+      mu_assert ("test_find_key_value: wrong result",
+                 result == test_array[i].result &&
+                 ((!test_array[i].result && !value) ||
+                  !strcmp (value, test_array[i].val)));
+
+      xfree (value);
+    }
+
+  return NULL;
+}
+
+const char *
+test_has_key (void)
+{
+  static const char *header_data = "key1=val2;token1;xyz; token2;xyz;token3 ;"\
+                                   "xyz; token4 ;xyz;   token5  ";
+  struct
+  {
+    const char *token;
+    bool result;
+  } test_array[] =
+  {
+    { "key1=val2", true },
+    { "token1", true },
+    { "token2", true },
+    { "token3", true },
+    { "token4", true },
+    { "token5", true },
+    { "token6", false },
+    { "oken1", false },
+    { "poken1", false },
+    { "key1=val2", true }
+  };
+  size_t i;
+
+  for (i = 0; i < countof (test_array); ++i)
+    mu_assert ("test_has_key: wrong result",
+               has_key (header_data, header_data + strlen (header_data),
+                        test_array[i].token) == test_array[i].result);
+
+  return NULL;
+}
+#endif
+
+#endif /* HAVE_METALINK */

src/metalink.h

@@ -0,0 +1,60 @@
+/* Declarations for metalink.c.
+   Copyright (C) 2015 Free Software Foundation, Inc.
+
+This file is part of GNU Wget.
+
+GNU Wget is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 3 of the License, or
+(at your option) any later version.
+
+GNU Wget is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Wget.  If not, see <http://www.gnu.org/licenses/>.
+
+Additional permission under GNU GPL version 3 section 7
+
+If you modify this program, or any covered work, by linking or
+combining it with the OpenSSL project's OpenSSL library (or a
+modified version of that library), containing parts covered by the
+terms of the OpenSSL or SSLeay licenses, the Free Software Foundation
+grants you additional permission to convey the resulting work.
+Corresponding Source for a non-source form of such a combination
+shall include the source code for the parts of OpenSSL used as well
+as that of the covered work.  */
+#if ! defined METALINK_H && defined HAVE_METALINK
+#define	METALINK_H
+
+#include <metalink/metalink_types.h>
+#include "wget.h"
+
+#ifdef HAVE_SSL
+# define RES_TYPE_SUPPORTED(x)\
+    ((!x) || !strcmp (x, "http") || !strcmp (x, "https") || !strcmp (x, "ftp") || !strcmp (x, "ftps"))
+#else
+# define RES_TYPE_SUPPORTED(x)\
+    ((!x) || !strcmp (x, "ftp") || !strcmp (x, "http"))
+#endif
+
+#define DEFAULT_PRI 999999
+#define VALID_PRI_RANGE(x) ((x) > 0 && (x) < 1000000)
+
+uerr_t retrieve_from_metalink (const metalink_t *metalink);
+
+int metalink_res_cmp (const void *res1, const void *res2);
+
+bool find_key_value (const char *start,
+                     const char *end,
+                     const char *key,
+                     char **value);
+bool has_key (const char *start, const char *end, const char *key);
+const char *find_key_values (const char *start,
+                             const char *end,
+                             char **key,
+                             char **value);
+
+#endif	/* METALINK_H */

src/mswindows.c

@@ -1,7 +1,7 @@
 /* mswindows.c -- Windows-specific support
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -62,7 +62,7 @@ void
 xsleep (double seconds)
 {
 #if defined(HAVE_USLEEP) && defined(HAVE_SLEEP)
-  if (seconds > 1000)
+  if (seconds >= 1)
     {
       /* Explained in utils.c. */
       sleep (seconds);

src/mswindows.h

@@ -1,7 +1,7 @@
 /* Declarations for windows
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/netrc.c

@@ -1,5 +1,5 @@
 /* Read and parse the .netrc file to get hosts, accounts, and passwords.
-   Copyright (C) 1996, 2007, 2008, 2009, 2010, 2011 Free Software
+   Copyright (C) 1996, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
    Foundation, Inc.
 
 This file is part of GNU Wget.

src/netrc.h

@@ -1,6 +1,6 @@
 /* Declarations for netrc.c
-   Copyright (C) 1996, 1996, 1997, 2007, 2008, 2009, 2010, 2011 Free
-   Software Foundation, Inc.
+   Copyright (C) 1996, 1996, 1997, 2007, 2008, 2009, 2010, 2011, 2015
+   Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/openssl.c

@@ -1,6 +1,6 @@
 /* SSL support via OpenSSL library.
    Copyright (C) 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008,
-   2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+   2009, 2010, 2011, 2012, 2015 Free Software Foundation, Inc.
    Originally contributed by Christian Fraenkel.
 
 This file is part of GNU Wget.
@@ -35,6 +35,7 @@ as that of the covered work.  */
 #include <errno.h>
 #include <unistd.h>
 #include <string.h>
+#include <xalloc.h>
 
 #include <openssl/ssl.h>
 #include <openssl/x509v3.h>
@@ -292,10 +293,17 @@ ssl_init (void)
   /* Use the private key from the cert file unless otherwise specified. */
   if (opt.cert_file && !opt.private_key)
     {
-      opt.private_key = opt.cert_file;
+      opt.private_key = xstrdup (opt.cert_file);
       opt.private_key_type = opt.cert_type;
     }
 
+  /* Use cert from private key file unless otherwise specified. */
+  if (opt.private_key && !opt.cert_file)
+    {
+      opt.cert_file = xstrdup (opt.private_key);
+      opt.cert_type = opt.private_key_type;
+    }
+
   if (opt.cert_file)
     if (SSL_CTX_use_certificate_file (ssl_ctx, opt.cert_file,
                                       key_type_to_ssl_type (opt.cert_type))
@@ -327,6 +335,7 @@ ssl_init (void)
 struct openssl_transport_context
 {
   SSL *conn;                    /* SSL connection handle */
+  SSL_SESSION *sess;            /* SSL session info */
   char *last_error;             /* last error printed with openssl_errstr */
 };
 
@@ -498,6 +507,22 @@ ssl_connect_with_timeout_callback(void *arg)
   ctx->result = SSL_connect(ctx->ssl);
 }
 
+static const char *
+_sni_hostname(const char *hostname)
+{
+  size_t len = strlen(hostname);
+
+  char *sni_hostname = xmemdup(hostname, len + 1);
+
+  /* Remove trailing dot(s) to fix #47408.
+   * Regarding RFC 6066 (SNI): The hostname is represented as a byte
+   * string using ASCII encoding without a trailing dot. */
+  while (len && sni_hostname[--len] == '.')
+    sni_hostname[len] = 0;
+
+  return sni_hostname;
+}
+
 /* Perform the SSL handshake on file descriptor FD, which is assumed
    to be connected to an SSL server.  The SSL handle provided by
    OpenSSL is registered with the file descriptor FD using
@@ -507,7 +532,7 @@ ssl_connect_with_timeout_callback(void *arg)
    Returns true on success, false on failure.  */
 
 bool
-ssl_connect_wget (int fd, const char *hostname)
+ssl_connect_wget (int fd, const char *hostname, int *continue_session)
 {
   SSL *conn;
   struct scwt_context scwt_ctx;
@@ -520,11 +545,16 @@ ssl_connect_wget (int fd, const char *hostname)
   if (!conn)
     goto error;
 #if OPENSSL_VERSION_NUMBER >= 0x0090806fL && !defined(OPENSSL_NO_TLSEXT)
-  /* If the SSL library was build with support for ServerNameIndication
+  /* If the SSL library was built with support for ServerNameIndication
      then use it whenever we have a hostname.  If not, don't, ever. */
   if (! is_valid_ip_address (hostname))
     {
-      if (! SSL_set_tlsext_host_name (conn, hostname))
+      const char *sni_hostname = _sni_hostname(hostname);
+
+      long rc = SSL_set_tlsext_host_name (conn, sni_hostname);
+      xfree(sni_hostname);
+
+      if (rc == 0)
         {
           DEBUGP (("Failed to set TLS server-name indication."));
           goto error;
@@ -532,6 +562,14 @@ ssl_connect_wget (int fd, const char *hostname)
     }
 #endif
 
+  if (continue_session)
+    {
+      /* attempt to resume a previous SSL session */
+      ctx = (struct openssl_transport_context *) fd_transport_context (*continue_session);
+      if (!ctx || !ctx->sess || !SSL_set_session (conn, ctx->sess))
+        goto error;
+    }
+
 #ifndef FD_TO_SOCKET
 # define FD_TO_SOCKET(X) (X)
 #endif
@@ -550,6 +588,9 @@ ssl_connect_wget (int fd, const char *hostname)
 
   ctx = xnew0 (struct openssl_transport_context);
   ctx->conn = conn;
+  ctx->sess = SSL_get0_session (conn);
+  if (!ctx->sess)
+    logprintf (LOG_NOTQUIET, "WARNING: Could not save SSL session data for socket %d\n", fd);
 
   /* Register FD with Wget's transport layer, i.e. arrange that our
      functions are used for reading, writing, and polling.  */
@@ -631,6 +672,65 @@ static char *_get_rfc2253_formatted (X509_NAME *name)
   return out ? out : xstrdup("");
 }
 
+/*
+ * Heavily modified from:
+ * https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning#OpenSSL
+ */
+static bool
+pkp_pin_peer_pubkey (X509* cert, const char *pinnedpubkey)
+{
+  /* Scratch */
+  int len1 = 0, len2 = 0;
+  char *buff1 = NULL, *temp = NULL;
+
+  /* Result is returned to caller */
+  bool result = false;
+
+  /* if a path wasn't specified, don't pin */
+  if (!pinnedpubkey)
+    return true;
+
+  if (!cert)
+    return result;
+
+  /* Begin Gyrations to get the subjectPublicKeyInfo     */
+  /* Thanks to Viktor Dukhovni on the OpenSSL mailing list */
+
+  /* https://groups.google.com/group/mailing.openssl.users/browse_thread
+   /thread/d61858dae102c6c7 */
+  len1 = i2d_X509_PUBKEY (X509_get_X509_PUBKEY (cert), NULL);
+  if (len1 < 1)
+    goto cleanup; /* failed */
+
+  /* https://www.openssl.org/docs/crypto/buffer.html */
+  buff1 = temp = OPENSSL_malloc (len1);
+  if (!buff1)
+    goto cleanup; /* failed */
+
+  /* https://www.openssl.org/docs/crypto/d2i_X509.html */
+  len2 = i2d_X509_PUBKEY (X509_get_X509_PUBKEY (cert), (unsigned char **) &temp);
+
+  /*
+   * These checks are verifying we got back the same values as when we
+   * sized the buffer. It's pretty weak since they should always be the
+   * same. But it gives us something to test.
+   */
+  if ((len1 != len2) || !temp || ((temp - buff1) != len1))
+    goto cleanup; /* failed */
+
+  /* End Gyrations */
+
+  /* The one good exit point */
+  result = wg_pin_peer_pubkey (pinnedpubkey, buff1, len1);
+
+ cleanup:
+  /* https://www.openssl.org/docs/crypto/buffer.html */
+  if (NULL != buff1)
+    OPENSSL_free (buff1);
+
+  return result;
+}
+
 /* Verify the validity of the certificate presented by the server.
    Also check that the "common name" of the server, as presented by
    its certificate, corresponds to HOST.  (HOST typically comes from
@@ -654,6 +754,7 @@ ssl_check_certificate (int fd, const char *host)
   long vresult;
   bool success = true;
   bool alt_name_checked = false;
+  bool pinsuccess = opt.pinnedpubkey == NULL;
 
   /* If the user has specified --no-check-cert, we still want to warn
      him about problems with the server's certificate.  */
@@ -663,6 +764,10 @@ ssl_check_certificate (int fd, const char *host)
   SSL *conn = ctx->conn;
   assert (conn != NULL);
 
+  /* The user explicitly said to not check for the certificate.  */
+  if (opt.check_cert == CHECK_CERT_QUIET && pinsuccess)
+    return success;
+
   cert = SSL_get_peer_certificate (conn);
   if (!cert)
     {
@@ -739,9 +844,12 @@ ssl_check_certificate (int fd, const char *host)
     {
       /* Test subject alternative names */
 
+      /* SNI hostname must not have a trailing dot */
+      const char *sni_hostname = _sni_hostname(host);
+
       /* Do we want to check for dNSNAmes or ipAddresses (see RFC 2818)?
        * Signal it by host_in_octet_string. */
-      ASN1_OCTET_STRING *host_in_octet_string = a2i_IPADDRESS (host);
+      ASN1_OCTET_STRING *host_in_octet_string = a2i_IPADDRESS (sni_hostname);
 
       int numaltnames = sk_GENERAL_NAME_num (subjectAltNames);
       int i;
@@ -776,7 +884,7 @@ ssl_check_certificate (int fd, const char *host)
                   if (0 <= ASN1_STRING_to_UTF8 (&name_in_utf8, name->d.dNSName))
                     {
                       /* Compare and check for NULL attack in ASN1_STRING */
-                      if (pattern_match ((char *)name_in_utf8, host) &&
+                      if (pattern_match ((char *)name_in_utf8, sni_hostname) &&
                             (strlen ((char *)name_in_utf8) ==
                                 (size_t) ASN1_STRING_length (name->d.dNSName)))
                         {
@@ -797,9 +905,11 @@ ssl_check_certificate (int fd, const char *host)
           logprintf (LOG_NOTQUIET,
               _("%s: no certificate subject alternative name matches\n"
                 "\trequested host name %s.\n"),
-                     severity, quote_n (1, host));
+                     severity, quote_n (1, sni_hostname));
           success = false;
         }
+
+      xfree(sni_hostname);
     }
 
   if (alt_name_checked == false)
@@ -854,6 +964,13 @@ ssl_check_certificate (int fd, const char *host)
         }
     }
 
+    pinsuccess = pkp_pin_peer_pubkey (cert, opt.pinnedpubkey);
+    if (!pinsuccess)
+      {
+        logprintf (LOG_ALWAYS, _("The public key does not match pinned public key!\n"));
+        success = false;
+      }
+
 
   if (success)
     DEBUGP (("X509 certificate successfully verified and matches host %s\n",
@@ -861,13 +978,13 @@ ssl_check_certificate (int fd, const char *host)
   X509_free (cert);
 
  no_cert:
-  if (opt.check_cert && !success)
+  if (opt.check_cert == CHECK_CERT_ON && !success)
     logprintf (LOG_NOTQUIET, _("\
 To connect to %s insecurely, use `--no-check-certificate'.\n"),
                quotearg_style (escape_quoting_style, host));
 
-  /* Allow --no-check-cert to disable certificate checking. */
-  return opt.check_cert ? success : true;
+  /* never return true if pinsuccess fails */
+  return !pinsuccess ? false : (opt.check_cert == CHECK_CERT_ON ? success : true);
 }
 
 /*

src/options.h

@@ -1,7 +1,7 @@
 /* struct options.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -29,6 +29,13 @@ Corresponding Source for a non-source form of such a combination
 shall include the source code for the parts of OpenSSL used as well
 as that of the covered work.  */
 
+enum CHECK_CERT_MODES
+{
+  CHECK_CERT_OFF,
+  CHECK_CERT_ON,
+  CHECK_CERT_QUIET
+};
+
 struct options
 {
   int verbose;                  /* Are we verbose?  (First set to -1,
@@ -58,6 +65,11 @@ struct options
   char *dir_prefix;             /* The top of directory tree */
   char *lfilename;              /* Log filename */
   char *input_filename;         /* Input filename */
+#ifdef HAVE_METALINK
+  char *input_metalink;         /* Input metalink file */
+  bool metalink_over_http;      /* Use Metalink if present in HTTP response */
+  char *preferred_location;     /* Preferred location for Metalink resources */
+#endif
   char *choose_config;          /* Specified config file */
   bool noconfig;                /* Ignore all config files? */
   bool force_html;              /* Is the input file an HTML file? */
@@ -87,6 +99,11 @@ struct options
   void *(*regex_compile_fun)(const char *);             /* Function to compile a regex. */
   bool (*regex_match_fun)(const void *, const char *);  /* Function to match a string to a regex. */
 
+#ifdef HAVE_LIBCARES
+  char *bind_dns_address;
+  char *dns_servers;
+#endif
+
   char **domains;               /* See host.c */
   char **exclude_domains;
   bool dns_cache;               /* whether we cache DNS lookups. */
@@ -165,6 +182,7 @@ struct options
 #endif
 
   bool timestamping;            /* Whether to use time-stamping. */
+  bool if_modified_since;       /* Whether to use conditional get requests.  */
 
   bool backup_converted;        /* Do we save pre-converted files as *.orig? */
   int backups;                  /* Are numeric backups made? */
@@ -176,6 +194,9 @@ struct options
                                    NULL. */
   bool convert_links;           /* Will the links be converted
                                    locally? */
+  bool convert_file_only;       /* Convert only the file portion of the URI (i.e. basename).
+                                   Leave everything else untouched. */
+
   bool remove_listing;          /* Do we remove .listing files
                                    generated by FTP? */
   bool htmlify;                 /* Do we HTML-ify the OS-dependent
@@ -206,7 +227,7 @@ struct options
     secure_protocol_tlsv1_2,
     secure_protocol_pfs
   } secure_protocol;            /* type of secure protocol to use. */
-  bool check_cert;              /* whether to validate the server's cert */
+  int check_cert;               /* whether to validate the server's cert */
   char *cert_file;              /* external client certificate to use. */
   char *private_key;            /* private key file (if not internal). */
   enum keyfile_type {
@@ -220,9 +241,18 @@ struct options
   char *ca_cert;                /* CA certificate file to use */
   char *crl_file;               /* file with CRLs */
 
+  char *pinnedpubkey;           /* Public key (PEM/DER) file, or any number
+                                   of base64 encoded sha256 hashes preceded by
+                                   \'sha256//\' and seperated by \';\', to verify
+                                   peer against */
+
   char *random_file;            /* file with random data to seed the PRNG */
   char *egd_file;               /* file name of the egd daemon socket */
   bool https_only;              /* whether to follow HTTPS only */
+  bool ftps_resume_ssl;
+  bool ftps_fallback_to_ftp;
+  bool ftps_implicit;
+  bool ftps_clear_data_connection;
 #endif /* HAVE_SSL */
 
   bool cookies;                 /* whether cookies are used. */
@@ -239,6 +269,7 @@ struct options
 
   enum {
     restrict_unix,
+    restrict_vms,
     restrict_windows
   } restrict_files_os;          /* file name restriction ruleset. */
   bool restrict_files_ctrl;     /* non-zero if control chars in URLs
@@ -288,6 +319,13 @@ struct options
   bool show_all_dns_entries;    /* Show all the DNS entries when resolving a
                                    name. */
   bool report_bps;              /*Output bandwidth in bits format*/
+
+  char *rejected_log;           /* The file to log rejected URLS to. */
+
+#ifdef HAVE_HSTS
+  bool hsts;
+  char *hsts_file;
+#endif
 };
 
 extern struct options opt;

src/progress.c

@@ -1,6 +1,6 @@
 /* Download progress.
    Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009,
-   2010, 2011 Free Software Foundation, Inc.
+   2010, 2011, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -594,7 +594,8 @@ bar_create (const char *f_download, wgint initial, wgint total)
   bp->width = screen_width - 1;
   /* + enough space for the terminating zero, and hopefully enough room
    * for multibyte characters. */
-  bp->buffer = xmalloc (bp->width + 100);
+#define BUF_LEN (bp->width + 100)
+  bp->buffer = xmalloc (BUF_LEN);
 
   logputs (LOG_VERBOSE, "\n");
 
@@ -693,7 +694,7 @@ bar_finish (void *progress, double dltime)
    The idea is that for fast downloads, we get the speed over exactly
    the last three seconds.  For slow downloads (where a network read
    takes more than 150ms to complete), we get the speed over a larger
-   time period, as large as it takes to complete thirty reads.  This
+   time period, as large as it takes to complete twenty reads.  This
    is good because slow downloads tend to fluctuate more and a
    3-second average would be too erratic.  */
 
@@ -854,7 +855,7 @@ get_eta (int *bcd)
 {
   /* TRANSLATORS: "ETA" is English-centric, but this must
      be short, ideally 3 chars.  Abbreviate if necessary.  */
-  static const char eta_str[] = N_("   eta %s");
+  static const char eta_str[] = N_("    eta %s");
   static const char *eta_trans;
   static int bytes_cols_diff;
   if (eta_trans == NULL)
@@ -903,11 +904,11 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
   char *p = bp->buffer;
   wgint size = bp->initial_length + bp->count;
 
-  int size_grouped_pad; /* Used to pad the field width for size_grouped. */
-
   struct bar_progress_hist *hist = &bp->hist;
   int orig_filename_cols = count_cols (bp->f_download);
 
+  int padding;
+
   /* The progress bar should look like this:
      file xx% [=======>             ] nnn.nnK 12.34KB/s  eta 36m 51s
 
@@ -928,12 +929,16 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
      "=====>..."       - progress bar             - the rest
   */
 
+  /* TODO: Ask the Turkish Translators to fix their translation for the "done"
+   * mode of progress bar. Use one less character. Once that is done, redice
+   * PROGRESS_ETA_LEN by 1.
+   */
 #define PROGRESS_FILENAME_LEN  MAX_FILENAME_COLS + 1
 #define PROGRESS_PERCENT_LEN   4
 #define PROGRESS_DECORAT_LEN   2
 #define PROGRESS_FILESIZE_LEN  7 + 1
-#define PROGRESS_DWNLOAD_RATE  8 + 1
-#define PROGRESS_ETA_LEN       14
+#define PROGRESS_DWNLOAD_RATE  8 + 2
+#define PROGRESS_ETA_LEN       15
 
   int progress_size = bp->width - (PROGRESS_FILENAME_LEN + PROGRESS_PERCENT_LEN +
                                    PROGRESS_DECORAT_LEN + PROGRESS_FILESIZE_LEN +
@@ -945,41 +950,55 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
   int cols_diff;
   const char *down_size;
 
+  memset (bp->buffer, '\0', BUF_LEN);
+
   if (progress_size < 5)
     progress_size = 0;
 
   if (orig_filename_cols <= MAX_FILENAME_COLS)
     {
-      int padding = MAX_FILENAME_COLS - orig_filename_cols;
-      sprintf (p, "%s ", bp->f_download);
-      p += orig_filename_cols + 1;
-      for (;padding;padding--)
-        *p++ = ' ';
+      padding = MAX_FILENAME_COLS - orig_filename_cols;
+      p += sprintf (p, "%s ", bp->f_download);
+      memset (p, ' ', padding);
+      p += padding;
     }
   else
     {
       int offset_cols;
       int bytes_in_filename, offset_bytes, col;
       int *cols_ret = &col;
-      int padding;
 
 #define MIN_SCROLL_TEXT 5
       if ((orig_filename_cols > MAX_FILENAME_COLS + MIN_SCROLL_TEXT) &&
           !opt.noscroll &&
           !done)
-        offset_cols = ((int) bp->tick) % (orig_filename_cols - MAX_FILENAME_COLS + 1);
+        {
+          offset_cols = ((int) bp->tick + orig_filename_cols + MAX_FILENAME_COLS / 2)
+                        % (orig_filename_cols + MAX_FILENAME_COLS);
+          if (offset_cols > orig_filename_cols)
+            {
+              padding = MAX_FILENAME_COLS - (offset_cols - orig_filename_cols);
+              memset(p, ' ', padding);
+              p += padding;
+              offset_cols = 0;
+            }
+          else
+            padding = 0;
+        }
       else
-        offset_cols = 0;
+        {
+          padding = 0;
+          offset_cols = 0;
+        }
       offset_bytes = cols_to_bytes (bp->f_download, offset_cols, cols_ret);
       bytes_in_filename = cols_to_bytes (bp->f_download + offset_bytes,
-                                         MAX_FILENAME_COLS,
+                                         MAX_FILENAME_COLS - padding,
                                          cols_ret);
       memcpy (p, bp->f_download + offset_bytes, bytes_in_filename);
       p += bytes_in_filename;
-      padding = MAX_FILENAME_COLS - *cols_ret;
-      for (;padding;padding--)
-          *p++ = ' ';
-      *p++ = ' ';
+      padding = MAX_FILENAME_COLS - (padding + *cols_ret);
+      memset (p, ' ', padding + 1);
+      p += padding + 1;
     }
 
   /* "xx% " */
@@ -987,15 +1006,13 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
     {
       int percentage = 100.0 * size / bp->total_length;
       assert (percentage <= 100);
-
-      if (percentage < 100)
-        sprintf (p, "%3d%%", percentage);
-      else
-        strcpy (p, "100%");
-      p += 4;
+      p += sprintf (p, "%3d%%", percentage);
     }
   else
-    APPEND_LITERAL ("    ");
+    {
+      memset (p, ' ', PROGRESS_PERCENT_LEN);
+      p += PROGRESS_PERCENT_LEN;
+    }
 
   /* The progress bar: "[====>      ]" or "[++==>      ]". */
   if (progress_size && bp->total_length > 0)
@@ -1007,7 +1024,6 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
       int dlsz = (double)size / bp->total_length * progress_size;
 
       char *begin;
-      int i;
 
       assert (dlsz <= progress_size);
       assert (insz <= dlsz);
@@ -1017,18 +1033,19 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
 
       /* Print the initial portion of the download with '+' chars, the
          rest with '=' and one '>'.  */
-      for (i = 0; i < insz; i++)
-        *p++ = '+';
+      memset (p, '+', insz);
+      p += insz;
+
       dlsz -= insz;
       if (dlsz > 0)
         {
-          for (i = 0; i < dlsz - 1; i++)
-            *p++ = '=';
+          memset (p, '=', dlsz-1);
+          p += dlsz - 1;
           *p++ = '>';
         }
 
-      while (p - begin < progress_size)
-        *p++ = ' ';
+      memset (p, ' ', (progress_size - (p - begin)));
+      p += (progress_size - (p - begin));
       *p++ = ']';
     }
   else if (progress_size)
@@ -1056,27 +1073,14 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
       *p++ = ']';
 
     }
- ++bp->tick;
+  ++bp->tick;
 
   /* " 234.56M" */
   down_size = human_readable (size, 1000, 2);
-  cols_diff = 7 - count_cols (down_size);
-  while (cols_diff > 0)
-  {
-    *p++=' ';
-    cols_diff--;
-  }
-  sprintf (p, " %s", down_size);
-  move_to_end (p);
-  /* Pad with spaces to 7 chars for the size_grouped field;
-   * couldn't use the field width specifier in sprintf, because
-   * it counts in bytes, not characters. */
-  for (size_grouped_pad = PROGRESS_FILESIZE_LEN - 7;
-       size_grouped_pad > 0;
-       --size_grouped_pad)
-    {
-      *p++ = ' ';
-    }
+  cols_diff = PROGRESS_FILESIZE_LEN - count_cols (down_size);
+  memset (p, ' ', cols_diff);
+  p += cols_diff;
+  p += sprintf (p, "%s", down_size);
 
   /* " 12.52Kb/s or 12.52KB/s" */
   if (hist->total_time > 0 && hist->total_bytes)
@@ -1089,12 +1093,11 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
       wgint dlquant = hist->total_bytes + bp->recent_bytes;
       double dltime = hist->total_time + (dl_total_time - bp->recent_start);
       double dlspeed = calc_rate (dlquant, dltime, &units);
-      sprintf (p, " %4.*f%s", dlspeed >= 99.95 ? 0 : dlspeed >= 9.995 ? 1 : 2,
+      p += sprintf (p, "  %4.*f%s", dlspeed >= 99.95 ? 0 : dlspeed >= 9.995 ? 1 : 2,
                dlspeed,  !opt.report_bps ? short_units[units] : short_units_bits[units]);
-      move_to_end (p);
     }
   else
-    APPEND_LITERAL (" --.-KB/s");
+    APPEND_LITERAL ("  --.-KB/s");
 
   if (!done)
     {
@@ -1129,14 +1132,14 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
               bp->last_eta_time = dl_total_time;
             }
 
-          sprintf (p, get_eta(&bytes_cols_diff),
+          p += sprintf (p, get_eta(&bytes_cols_diff),
                    eta_to_human_short (eta, false));
-          move_to_end (p);
         }
       else if (bp->total_length > 0)
         {
         skip_eta:
-          APPEND_LITERAL ("             ");
+          memset (p, ' ', PROGRESS_ETA_LEN);
+          p += PROGRESS_ETA_LEN;
         }
     }
   else
@@ -1146,21 +1149,25 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
       int ncols;
 
       /* Note to translators: this should not take up more room than
-         available here.  Abbreviate if necessary.  */
-      strcpy (p, _("   in "));
+         available here (6 columns).  Abbreviate if necessary.  */
+      strcpy (p, _("    in "));
       nbytes = strlen (p);
       ncols  = count_cols (p);
       bytes_cols_diff = nbytes - ncols;
-      p += nbytes;
       if (dl_total_time >= 10)
-        strcpy (p, eta_to_human_short ((int) (dl_total_time + 0.5), false));
+        ncols += sprintf (p + nbytes, "%s",  eta_to_human_short ((int) (dl_total_time + 0.5), false));
       else
-        sprintf (p, "%ss", print_decimal (dl_total_time));
-      move_to_end (p);
+        ncols += sprintf (p + nbytes, "%ss", print_decimal (dl_total_time));
+      p += ncols + bytes_cols_diff;
+      memset (p, ' ', PROGRESS_ETA_LEN - ncols);
+      p += PROGRESS_ETA_LEN - ncols;
     }
 
-  while (p - bp->buffer - bytes_cols_diff < bp->width)
-    *p++ = ' ';
+  padding = bp->width - count_cols (bp->buffer);
+  assert (padding >= 0 && "Padding length became non-positive!");
+  padding = padding > 0 ? padding : 0;
+  memset (p, ' ', padding);
+  p += padding;
   *p = '\0';
 
   /* 2014-11-14  Darshit Shah  <darnir@gmail.com>
@@ -1169,8 +1176,11 @@ create_image (struct bar_progress *bp, double dl_total_time, bool done)
    * from the release code since we do not want Wget to crash and burn when the
    * assertion fails. Instead Wget should continue downloading and display a
    * horrible and irritating progress bar that spams the screen with newlines.
+   *
+   * By default, all assertions are disabled in a Wget build and are enabled
+   * only with the --enable-assert configure option.
    */
-  assert (count_cols (bp->buffer) <= bp->width + 1);
+  assert (count_cols (bp->buffer) == bp->width);
 }
 
 /* Print the contents of the buffer as a one-line ASCII "image" so
@@ -1188,8 +1198,6 @@ display_image (char *buf)
 static void
 bar_set_params (char *params)
 {
-  char *term = getenv ("TERM");
-
   if (params)
     {
       char *param = strtok (params, ":");
@@ -1209,12 +1217,6 @@ bar_set_params (char *params)
           dots.  */
        || !isatty (fileno (stderr))
 #endif
-       /* Normally we don't depend on terminal type because the
-          progress bar only uses ^M to move the cursor to the
-          beginning of line, which works even on dumb terminals.  But
-          Jamie Zawinski reports that ^M and ^H tricks don't work in
-          Emacs shell buffers, and only make a mess.  */
-       || (term && 0 == strcmp (term, "emacs"))
        )
       && !current_impl_locked)
     {

src/progress.h

@@ -1,6 +1,6 @@
 /* Download progress.
    Copyright (C) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009,
-   2010, 2011 Free Software Foundation, Inc.
+   2010, 2011, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/ptimer.c

@@ -1,6 +1,6 @@
 /* Portable timers.
-   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software
-   Foundation, Inc.
+   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free
+   Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/ptimer.h

@@ -1,6 +1,6 @@
 /* Declarations for ptimer.c.
-   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software
-   Foundation, Inc.
+   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free
+   Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/recur.c

@@ -1,7 +1,7 @@
 /* Handling of recursive HTTP retrieving.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -50,6 +50,7 @@ as that of the covered work.  */
 #include "html-url.h"
 #include "css-url.h"
 #include "spider.h"
+#include "exits.h"
 
 /* Functions for maintaining the URL queue.  */
 
@@ -181,11 +182,20 @@ static int blacklist_contains (struct hash_table *blacklist, const char *url)
   return ret;
 }
 
-static bool download_child_p (const struct urlpos *, struct url *, int,
-                              struct url *, struct hash_table *, struct iri *);
-static bool descend_redirect_p (const char *, struct url *, int,
-                                struct url *, struct hash_table *, struct iri *);
+typedef enum
+{
+  WG_RR_SUCCESS, WG_RR_BLACKLIST, WG_RR_NOTHTTPS, WG_RR_NONHTTP, WG_RR_ABSOLUTE,
+  WG_RR_DOMAIN, WG_RR_PARENT, WG_RR_LIST, WG_RR_REGEX, WG_RR_RULES,
+  WG_RR_SPANNEDHOST, WG_RR_ROBOTS
+} reject_reason;
 
+static reject_reason download_child (const struct urlpos *, struct url *, int,
+                              struct url *, struct hash_table *, struct iri *);
+static reject_reason descend_redirect (const char *, struct url *, int,
+                              struct url *, struct hash_table *, struct iri *);
+static void write_reject_log_header (FILE *);
+static void write_reject_log_reason (FILE *, reject_reason,
+                              const struct url *, const struct url *);
 
 /* Retrieve a part of the web beginning with START_URL.  This used to
    be called "recursive retrieval", because the old function was
@@ -222,17 +232,21 @@ retrieve_tree (struct url *start_url_parsed, struct iri *pi)
 
   struct iri *i = iri_new ();
 
-#define COPYSTR(x)  (x) ? xstrdup(x) : NULL;
+  FILE *rejectedlog = NULL; /* Don't write a rejected log. */
+
   /* Duplicate pi struct if not NULL */
   if (pi)
     {
+#define COPYSTR(x)  (x) ? xstrdup(x) : NULL;
       i->uri_encoding = COPYSTR (pi->uri_encoding);
       i->content_encoding = COPYSTR (pi->content_encoding);
       i->utf8_encode = pi->utf8_encode;
+#undef COPYSTR
     }
+#ifdef ENABLE_IRI
   else
     set_uri_encoding (i, opt.locale, true);
-#undef COPYSTR
+#endif
 
   queue = url_queue_new ();
   blacklist = make_string_hash_table (0);
@@ -243,6 +257,14 @@ retrieve_tree (struct url *start_url_parsed, struct iri *pi)
                false);
   blacklist_add (blacklist, start_url_parsed->url);
 
+  if (opt.rejected_log)
+    {
+      rejectedlog = fopen (opt.rejected_log, "w");
+      write_reject_log_header (rejectedlog);
+      if (!rejectedlog)
+        logprintf (LOG_NOTQUIET, "%s: %s\n", opt.rejected_log, strerror (errno));
+    }
+
   while (1)
     {
       bool descend = false;
@@ -265,9 +287,9 @@ retrieve_tree (struct url *start_url_parsed, struct iri *pi)
         break;
 
       /* ...and download it.  Note that this download is in most cases
-         unconditional, as download_child_p already makes sure a file
+         unconditional, as download_child already makes sure a file
          doesn't get enqueued twice -- and yet this check is here, and
-         not in download_child_p.  This is so that if you run `wget -r
+         not in download_child.  This is so that if you run `wget -r
          URL1 URL2', and a random URL is encountered once under URL1
          and again under URL2, but at a different (possibly smaller)
          depth, we want the URL's children to be taken into account
@@ -298,52 +320,69 @@ retrieve_tree (struct url *start_url_parsed, struct iri *pi)
           char *redirected = NULL;
           struct url *url_parsed = url_parse (url, &url_err, i, true);
 
-          status = retrieve_url (url_parsed, url, &file, &redirected, referer,
-                                 &dt, false, i, true);
-
-          if (html_allowed && file && status == RETROK
-              && (dt & RETROKF) && (dt & TEXTHTML))
+          if (!url_parsed)
             {
-              descend = true;
-              is_css = false;
-            }
-
-          /* a little different, css_allowed can override content type
-             lots of web servers serve css with an incorrect content type
-          */
-          if (file && status == RETROK
-              && (dt & RETROKF) &&
-              ((dt & TEXTCSS) || css_allowed))
-            {
-              descend = true;
-              is_css = true;
-            }
-
-          if (redirected)
-            {
-              /* We have been redirected, possibly to another host, or
-                 different path, or wherever.  Check whether we really
-                 want to follow it.  */
-              if (descend)
-                {
-                  if (!descend_redirect_p (redirected, url_parsed, depth,
-                                           start_url_parsed, blacklist, i))
-                    descend = false;
-                  else
-                    /* Make sure that the old pre-redirect form gets
-                       blacklisted. */
-                    blacklist_add (blacklist, url);
-                }
-
-              xfree (url);
-              url = redirected;
+              char *error = url_error (url, url_err);
+              logprintf (LOG_NOTQUIET, "%s: %s.\n",url, error);
+              xfree (error);
+              inform_exit_status (URLERROR);
             }
           else
             {
-              xfree (url);
-              url = xstrdup (url_parsed->url);
+
+              status = retrieve_url (url_parsed, url, &file, &redirected, referer,
+                                     &dt, false, i, true);
+
+              if (html_allowed && file && status == RETROK
+                  && (dt & RETROKF) && (dt & TEXTHTML))
+                {
+                  descend = true;
+                  is_css = false;
+                }
+
+              /* a little different, css_allowed can override content type
+                 lots of web servers serve css with an incorrect content type
+              */
+              if (file && status == RETROK
+                  && (dt & RETROKF) &&
+                  ((dt & TEXTCSS) || css_allowed))
+                {
+                  descend = true;
+                  is_css = true;
+                }
+
+              if (redirected)
+                {
+                  /* We have been redirected, possibly to another host, or
+                     different path, or wherever.  Check whether we really
+                     want to follow it.  */
+                  if (descend)
+                    {
+                      reject_reason r = descend_redirect (redirected, url_parsed,
+                                        depth, start_url_parsed, blacklist, i);
+                      if (r == WG_RR_SUCCESS)
+                        {
+                          /* Make sure that the old pre-redirect form gets
+                             blacklisted. */
+                          blacklist_add (blacklist, url);
+                        }
+                      else
+                        {
+                          write_reject_log_reason (rejectedlog, r, url_parsed, start_url_parsed);
+                          descend = false;
+                        }
+                    }
+
+                  xfree (url);
+                  url = redirected;
+                }
+              else
+                {
+                  xfree (url);
+                  url = xstrdup (url_parsed->url);
+                }
+              url_free (url_parsed);
             }
-          url_free (url_parsed);
         }
 
       if (opt.spider)
@@ -409,12 +448,16 @@ retrieve_tree (struct url *start_url_parsed, struct iri *pi)
 
               for (; child; child = child->next)
                 {
+                  reject_reason r;
+
                   if (child->ignore_when_downloading)
                     continue;
                   if (dash_p_leaf_HTML && !child->link_inline_p)
                     continue;
-                  if (download_child_p (child, url_parsed, depth, start_url_parsed,
-                                        blacklist, i))
+
+                  r = download_child (child, url_parsed, depth,
+                                      start_url_parsed, blacklist, i);
+                  if (r == WG_RR_SUCCESS)
                     {
                       ci = iri_new ();
                       set_uri_encoding (ci, i->content_encoding, false);
@@ -427,6 +470,10 @@ retrieve_tree (struct url *start_url_parsed, struct iri *pi)
                          same URL twice.  */
                       blacklist_add (blacklist, child->url->url);
                     }
+                  else
+                    {
+                      write_reject_log_reason (rejectedlog, r, child->url, url_parsed);
+                    }
                 }
 
               if (strip_auth)
@@ -466,6 +513,9 @@ retrieve_tree (struct url *start_url_parsed, struct iri *pi)
       iri_free (i);
     }
 
+  if (rejectedlog)
+    fclose (rejectedlog);
+
   /* If anything is left of the queue due to a premature exit, free it
      now.  */
   {
@@ -501,14 +551,15 @@ retrieve_tree (struct url *start_url_parsed, struct iri *pi)
    by storing these URLs to BLACKLIST.  This may or may not help.  It
    will help if those URLs are encountered many times.  */
 
-static bool
-download_child_p (const struct urlpos *upos, struct url *parent, int depth,
+static reject_reason
+download_child (const struct urlpos *upos, struct url *parent, int depth,
                   struct url *start_url_parsed, struct hash_table *blacklist,
                   struct iri *iri)
 {
   struct url *u = upos->url;
   const char *url = u->url;
   bool u_scheme_like_http;
+  reject_reason reason = WG_RR_SUCCESS;
 
   DEBUGP (("Deciding whether to enqueue \"%s\".\n", url));
 
@@ -517,11 +568,12 @@ download_child_p (const struct urlpos *upos, struct url *parent, int depth,
       if (opt.spider)
         {
           char *referrer = url_string (parent, URL_AUTH_HIDE_PASSWD);
-          DEBUGP (("download_child_p: parent->url is: %s\n", quote (parent->url)));
+          DEBUGP (("download_child: parent->url is: %s\n", quote (parent->url)));
           visited_url (url, referrer);
           xfree (referrer);
         }
       DEBUGP (("Already on the black list.\n"));
+      reason = WG_RR_BLACKLIST;
       goto out;
     }
 
@@ -551,6 +603,7 @@ download_child_p (const struct urlpos *upos, struct url *parent, int depth,
   if (opt.https_only && u->scheme != SCHEME_HTTPS)
     {
       DEBUGP (("Not following non-HTTPS links.\n"));
+      reason = WG_RR_NOTHTTPS;
       goto out;
     }
 #endif
@@ -559,9 +612,14 @@ download_child_p (const struct urlpos *upos, struct url *parent, int depth,
   u_scheme_like_http = schemes_are_similar_p (u->scheme, SCHEME_HTTP);
 
   /* 1. Schemes other than HTTP are normally not recursed into. */
-  if (!u_scheme_like_http && !(u->scheme == SCHEME_FTP && opt.follow_ftp))
+  if (!u_scheme_like_http && !((u->scheme == SCHEME_FTP
+#ifdef HAVE_SSL
+      || u->scheme == SCHEME_FTPS
+#endif
+      ) && opt.follow_ftp))
     {
       DEBUGP (("Not following non-HTTP schemes.\n"));
+      reason = WG_RR_NONHTTP;
       goto out;
     }
 
@@ -571,6 +629,7 @@ download_child_p (const struct urlpos *upos, struct url *parent, int depth,
     if (opt.relative_only && !upos->link_relative_p)
       {
         DEBUGP (("It doesn't really look like a relative link.\n"));
+        reason = WG_RR_ABSOLUTE;
         goto out;
       }
 
@@ -579,6 +638,7 @@ download_child_p (const struct urlpos *upos, struct url *parent, int depth,
   if (!accept_domain (u))
     {
       DEBUGP (("The domain was not accepted.\n"));
+      reason = WG_RR_DOMAIN;
       goto out;
     }
 
@@ -598,6 +658,7 @@ download_child_p (const struct urlpos *upos, struct url *parent, int depth,
         {
           DEBUGP (("Going to \"%s\" would escape \"%s\" with no_parent on.\n",
                    u->dir, start_url_parsed->dir));
+          reason = WG_RR_PARENT;
           goto out;
         }
     }
@@ -610,12 +671,14 @@ download_child_p (const struct urlpos *upos, struct url *parent, int depth,
       if (!accdir (u->dir))
         {
           DEBUGP (("%s (%s) is excluded/not-included.\n", url, u->dir));
+          reason = WG_RR_LIST;
           goto out;
         }
     }
   if (!accept_url (url))
     {
       DEBUGP (("%s is excluded/not-included through regex.\n", url));
+      reason = WG_RR_REGEX;
       goto out;
     }
 
@@ -640,6 +703,7 @@ download_child_p (const struct urlpos *upos, struct url *parent, int depth,
         {
           DEBUGP (("%s (%s) does not match acc/rej rules.\n",
                    url, u->file));
+          reason = WG_RR_RULES;
           goto out;
         }
     }
@@ -650,6 +714,7 @@ download_child_p (const struct urlpos *upos, struct url *parent, int depth,
       {
         DEBUGP (("This is not the same hostname as the parent's (%s and %s).\n",
                  u->host, parent->host));
+        reason = WG_RR_SPANNEDHOST;
         goto out;
       }
 
@@ -692,35 +757,36 @@ download_child_p (const struct urlpos *upos, struct url *parent, int depth,
         {
           DEBUGP (("Not following %s because robots.txt forbids it.\n", url));
           blacklist_add (blacklist, url);
+          reason = WG_RR_ROBOTS;
           goto out;
         }
     }
 
-  /* The URL has passed all the tests.  It can be placed in the
-     download queue. */
-  DEBUGP (("Decided to load it.\n"));
+  out:
 
-  return true;
+  if (reason == WG_RR_SUCCESS)
+    /* The URL has passed all the tests.  It can be placed in the
+       download queue. */
+    DEBUGP (("Decided to load it.\n"));
+  else
+    DEBUGP (("Decided NOT to load it.\n"));
 
- out:
-  DEBUGP (("Decided NOT to load it.\n"));
-
-  return false;
+  return reason;
 }
 
 /* This function determines whether we will consider downloading the
    children of a URL whose download resulted in a redirection,
    possibly to another host, etc.  It is needed very rarely, and thus
-   it is merely a simple-minded wrapper around download_child_p.  */
+   it is merely a simple-minded wrapper around download_child.  */
 
-static bool
-descend_redirect_p (const char *redirected, struct url *orig_parsed, int depth,
+static reject_reason
+descend_redirect (const char *redirected, struct url *orig_parsed, int depth,
                     struct url *start_url_parsed, struct hash_table *blacklist,
                     struct iri *iri)
 {
   struct url *new_parsed;
   struct urlpos *upos;
-  bool success;
+  reject_reason reason;
 
   assert (orig_parsed != NULL);
 
@@ -730,10 +796,10 @@ descend_redirect_p (const char *redirected, struct url *orig_parsed, int depth,
   upos = xnew0 (struct urlpos);
   upos->url = new_parsed;
 
-  success = download_child_p (upos, orig_parsed, depth,
+  reason = download_child (upos, orig_parsed, depth,
                               start_url_parsed, blacklist, iri);
 
-  if (success)
+  if (reason == WG_RR_SUCCESS)
     blacklist_add (blacklist, upos->url->url);
   else
     DEBUGP (("Redirection \"%s\" failed the test.\n", redirected));
@@ -741,7 +807,93 @@ descend_redirect_p (const char *redirected, struct url *orig_parsed, int depth,
   url_free (new_parsed);
   xfree (upos);
 
-  return success;
+  return reason;
+}
+
+
+/* This function writes the rejected log header. */
+static void
+write_reject_log_header (FILE *f)
+{
+  if (!f)
+    return;
+
+  /* Note: Update this header when columns change in any way. */
+  fprintf (f, "REASON\t"
+    "U_URL\tU_SCHEME\tU_HOST\tU_PORT\tU_PATH\tU_PARAMS\tU_QUERY\tU_FRAGMENT\t"
+    "P_URL\tP_SCHEME\tP_HOST\tP_PORT\tP_PATH\tP_PARAMS\tP_QUERY\tP_FRAGMENT\n");
+}
+
+/* This function writes a URL to the reject log. Internal use only. */
+static void
+write_reject_log_url (FILE *fp, const struct url *url)
+{
+  const char *escaped_str;
+  const char *scheme_str;
+
+  if (!fp)
+    return;
+
+  escaped_str = url_escape (url->url);
+
+  switch (url->scheme)
+    {
+      case SCHEME_HTTP:  scheme_str = "SCHEME_HTTP";    break;
+#ifdef HAVE_SSL
+      case SCHEME_HTTPS: scheme_str = "SCHEME_HTTPS";   break;
+      case SCHEME_FTPS:  scheme_str = "SCHEME_FTPS";    break;
+#endif
+      case SCHEME_FTP:   scheme_str = "SCHEME_FTP";     break;
+      default:           scheme_str = "SCHEME_INVALID"; break;
+    }
+
+  fprintf (fp, "%s\t%s\t%s\t%i\t%s\t%s\t%s\t%s",
+    escaped_str,
+    scheme_str,
+    url->host,
+    url->port,
+    url->path,
+    url->params ? url->params : "",
+    url->query ? url->query : "",
+    url->fragment ? url->fragment : "");
+
+  xfree (escaped_str);
+}
+
+/* This function writes out information on why a URL was rejected and its
+   context from download_child such as the URL being rejected and it's
+   parent's URL. The format it uses is comma separated values but with tabs. */
+static void
+write_reject_log_reason (FILE *fp, reject_reason reason,
+                         const struct url *url, const struct url *parent)
+{
+  const char *reason_str;
+
+  if (!fp)
+    return;
+
+  switch (reason)
+    {
+      case WG_RR_SUCCESS:     reason_str = "SUCCESS";     break;
+      case WG_RR_BLACKLIST:   reason_str = "BLACKLIST";   break;
+      case WG_RR_NOTHTTPS:    reason_str = "NOTHTTPS";    break;
+      case WG_RR_NONHTTP:     reason_str = "NONHTTP";     break;
+      case WG_RR_ABSOLUTE:    reason_str = "ABSOLUTE";    break;
+      case WG_RR_DOMAIN:      reason_str = "DOMAIN";      break;
+      case WG_RR_PARENT:      reason_str = "PARENT";      break;
+      case WG_RR_LIST:        reason_str = "LIST";        break;
+      case WG_RR_REGEX:       reason_str = "REGEX";       break;
+      case WG_RR_RULES:       reason_str = "RULES";       break;
+      case WG_RR_SPANNEDHOST: reason_str = "SPANNEDHOST"; break;
+      case WG_RR_ROBOTS:      reason_str = "ROBOTS";      break;
+      default:                reason_str = "UNKNOWN";     break;
+    }
+
+  fprintf (fp, "%s\t", reason_str);
+  write_reject_log_url (fp, url);
+  fprintf (fp, "\t");
+  write_reject_log_url (fp, parent);
+  fprintf (fp, "\n");
 }
 
 /* vim:set sts=2 sw=2 cino+={s: */

src/recur.h

@@ -1,7 +1,7 @@
 /* Declarations for recur.c.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/res.c

@@ -1,6 +1,6 @@
 /* Support for Robot Exclusion Standard (RES).
-   Copyright (C) 2001, 2006, 2007, 2008, 2009, 2010, 2011 Free Software
-   Foundation, Inc.
+   Copyright (C) 2001, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free
+   Software Foundation, Inc.
 
 This file is part of Wget.
 

src/res.h

@@ -1,5 +1,5 @@
 /* Declarations for res.c.
-   Copyright (C) 2001, 2007, 2008, 2009, 2010, 2011 Free Software
+   Copyright (C) 2001, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
    Foundation, Inc.
 
 This file is part of Wget.

src/retr.c

@@ -1,7 +1,7 @@
 /* File retrieval.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2014, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -56,6 +56,7 @@ as that of the covered work.  */
 #include "ptimer.h"
 #include "html-url.h"
 #include "iri.h"
+#include "hsts.h"
 
 /* Total size of downloaded files.  Used to enforce quota.  */
 SUM_SIZE_INT total_downloaded_bytes;
@@ -725,7 +726,7 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file,
   char *mynewloc, *proxy;
   struct url *u = orig_parsed, *proxy_url;
   int up_error_code;            /* url parse error code */
-  char *local_file;
+  char *local_file = NULL;
   int redirection_count = 0;
 
   bool method_suspended = false;
@@ -753,7 +754,7 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file,
 
   result = NOCONERROR;
   mynewloc = NULL;
-  local_file = NULL;
+  xfree(local_file);
   proxy_url = NULL;
 
   proxy = getproxy (u);
@@ -772,6 +773,8 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file,
                      proxy, error);
           xfree (url);
           xfree (error);
+          xfree (proxy);
+          iri_free (pi);
           RESTORE_METHOD;
           result = PROXERR;
           goto bail;
@@ -781,6 +784,8 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file,
           logprintf (LOG_NOTQUIET, _("Error in proxy URL %s: Must be HTTP.\n"), proxy);
           url_free (proxy_url);
           xfree (url);
+          xfree (proxy);
+          iri_free (pi);
           RESTORE_METHOD;
           result = PROXERR;
           goto bail;
@@ -795,10 +800,28 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file,
 #endif
       || (proxy_url && proxy_url->scheme == SCHEME_HTTP))
     {
+#ifdef HAVE_HSTS
+#ifdef TESTING
+      /* we don't link against main.o when we're testing */
+      hsts_store_t hsts_store = NULL;
+#else
+      extern hsts_store_t hsts_store;
+#endif
+
+      if (opt.hsts && hsts_store)
+	{
+	  if (hsts_match (hsts_store, u))
+	    logprintf (LOG_VERBOSE, "URL transformed to HTTPS due to an HSTS policy\n");
+	}
+#endif
       result = http_loop (u, orig_parsed, &mynewloc, &local_file, refurl, dt,
                           proxy_url, iri);
     }
-  else if (u->scheme == SCHEME_FTP)
+  else if (u->scheme == SCHEME_FTP
+#ifdef HAVE_SSL
+      || u->scheme == SCHEME_FTPS
+#endif
+      )
     {
       /* If this is a redirection, temporarily turn off opt.ftp_glob
          and opt.recursive, both being undesirable when following
@@ -807,14 +830,19 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file,
       if (redirection_count)
         oldrec = glob = false;
 
-      result = ftp_loop (u, &local_file, dt, proxy_url, recursive, glob);
+      result = ftp_loop (u, orig_parsed, &local_file, dt, proxy_url,
+                         recursive, glob);
       recursive = oldrec;
 
       /* There is a possibility of having HTTP being redirected to
          FTP.  In these cases we must decide whether the text is HTML
          according to the suffix.  The HTML suffixes are `.html',
          `.htm' and a few others, case-insensitive.  */
-      if (redirection_count && local_file && u->scheme == SCHEME_FTP)
+      if (redirection_count && local_file && (u->scheme == SCHEME_FTP
+#ifdef HAVE_SSL
+          || u->scheme == SCHEME_FTPS
+#endif
+          ))
         {
           if (has_html_suffix_p (local_file))
             *dt |= TEXTHTML;
@@ -845,11 +873,15 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file,
       xfree (mynewloc);
       mynewloc = construced_newloc;
 
-      /* Reset UTF-8 encoding state, keep the URI encoding and reset
+#ifdef ENABLE_IRI
+      /* Reset UTF-8 encoding state, set the URI encoding and reset
          the content encoding. */
       iri->utf8_encode = opt.enable_iri;
+      if (opt.encoding_remote)
+       set_uri_encoding (iri, opt.encoding_remote, true);
       set_content_encoding (iri, NULL);
       xfree (iri->orig_url);
+#endif
 
       /* Now, see if this new location makes sense. */
       newloc_parsed = url_parse (mynewloc, &up_error_code, iri, true);
@@ -941,7 +973,7 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file,
           DEBUGP (("[Couldn't fallback to non-utf8 for %s\n", quote (url)));
     }
 
-  if (local_file && u && *dt & RETROKF)
+  if (local_file && u && (*dt & RETROKF || opt.content_on_error))
     {
       register_download (u->url, local_file);
 
@@ -984,6 +1016,7 @@ retrieve_url (struct url * orig_parsed, const char *origurl, char **file,
 bail:
   if (register_status)
     inform_exit_status (result);
+
   return result;
 }
 
@@ -1019,6 +1052,7 @@ retrieve_from_file (const char *file, bool html, int *count)
           char *error = url_error (url, url_err);
           logprintf (LOG_NOTQUIET, "%s: %s.\n", url, error);
           xfree (error);
+          iri_free (iri);
           return URLERROR;
         }
 
@@ -1035,10 +1069,12 @@ retrieve_from_file (const char *file, bool html, int *count)
       if (dt & TEXTHTML)
         html = true;
 
+#ifdef ENABLE_IRI
       /* If we have a found a content encoding, use it.
        * ( == is okay, because we're checking for identical object) */
       if (iri->content_encoding != opt.locale)
           set_uri_encoding (iri, iri->content_encoding, false);
+#endif
 
       /* Reset UTF-8 encode status */
       iri->utf8_encode = opt.enable_iri;
@@ -1074,12 +1110,20 @@ retrieve_from_file (const char *file, bool html, int *count)
 
       proxy = getproxy (cur_url->url);
       if ((opt.recursive || opt.page_requisites)
-          && (cur_url->url->scheme != SCHEME_FTP || proxy))
+          && ((cur_url->url->scheme != SCHEME_FTP
+#ifdef HAVE_SSL
+          && cur_url->url->scheme != SCHEME_FTPS
+#endif
+          ) || proxy))
         {
           int old_follow_ftp = opt.follow_ftp;
 
           /* Turn opt.follow_ftp on in case of recursive FTP retrieval */
-          if (cur_url->url->scheme == SCHEME_FTP)
+          if (cur_url->url->scheme == SCHEME_FTP
+#ifdef HAVE_SSL
+              || cur_url->url->scheme == SCHEME_FTPS
+#endif
+              )
             opt.follow_ftp = 1;
 
           status = retrieve_tree (parsed_url ? parsed_url : cur_url->url,
@@ -1260,6 +1304,9 @@ getproxy (struct url *u)
     case SCHEME_HTTPS:
       proxy = opt.https_proxy ? opt.https_proxy : getenv ("https_proxy");
       break;
+    case SCHEME_FTPS:
+      proxy = opt.ftp_proxy ? opt.ftp_proxy : getenv ("ftps_proxy");
+      break;
 #endif
     case SCHEME_FTP:
       proxy = opt.ftp_proxy ? opt.ftp_proxy : getenv ("ftp_proxy");

src/retr.h

@@ -1,7 +1,7 @@
 /* Declarations for retr.c.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/spider.c

@@ -1,5 +1,5 @@
 /* Keep track of visited URLs in spider mode.
-   Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 Free Software
+   Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
    Foundation, Inc.
 
 This file is part of GNU Wget.

src/spider.h

@@ -1,5 +1,5 @@
 /* Declarations for spider.c
-   Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011 Free Software
+   Copyright (C) 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
    Foundation, Inc.
 
 This file is part of GNU Wget.

src/ssl.h

@@ -1,6 +1,6 @@
 /* SSL support.
    Copyright (C) 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008,
-   2009, 2010, 2011, 2012 Free Software Foundation, Inc.
+   2009, 2010, 2011, 2012, 2015 Free Software Foundation, Inc.
    Originally contributed by Christian Fraenkel.
 
 This file is part of GNU Wget.
@@ -33,7 +33,7 @@ as that of the covered work.  */
 #define GEN_SSLFUNC_H
 
 bool ssl_init (void);
-bool ssl_connect_wget (int, const char *);
+bool ssl_connect_wget (int, const char *, int *);
 bool ssl_check_certificate (int, const char *);
 
 #endif /* GEN_SSLFUNC_H */

src/sysdep.h

@@ -1,7 +1,7 @@
 /* Dirty system-dependent hacks.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -68,9 +68,6 @@ as that of the covered work.  */
 
 #ifdef NAMESPACE_TWEAKS
 
-/* Request the "Unix 98 compilation environment". */
-#define _XOPEN_SOURCE 500
-
 #endif /* NAMESPACE_TWEAKS */
 
 
@@ -146,16 +143,6 @@ int snprintf (char *str, size_t count, const char *fmt, ...);
 int vsnprintf (char *str, size_t count, const char *fmt, va_list arg);
 #endif
 
-/* Some systems (Linux libc5, "NCR MP-RAS 3.0", and others) don't
-   provide MAP_FAILED, a symbolic constant for the value returned by
-   mmap() when it doesn't work.  Usually, this constant should be -1.
-   This only makes sense for files that use mmap() and include
-   sys/mman.h *before* sysdep.h, but doesn't hurt others.  */
-
-#ifndef MAP_FAILED
-# define MAP_FAILED ((void *) -1)
-#endif
-
 /* Enable system fnmatch only on systems where fnmatch.h is usable.
    If the fnmatch on your system is buggy, undef this symbol and a
    replacement implementation will be used instead.  */

src/test.c

@@ -1,6 +1,6 @@
 /* Unit testing.
-   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software
-   Foundation, Inc.
+   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free
+   Software Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -41,24 +41,20 @@ as that of the covered work.  */
 #error "TESTING not set!!!"
 #endif
 
-const char *test_parse_content_disposition(void);
-const char *test_subdir_p(void);
-const char *test_dir_matches_p(void);
-const char *test_commands_sorted(void);
-const char *test_cmd_spec_restrict_file_names(void);
-const char *test_path_simplify (void);
-const char *test_append_uri_pathel(void);
-const char *test_are_urls_equal(void);
-const char *test_is_robots_txt_url(void);
-
 const char *program_argstring = "TEST";
 
-int tests_run;
+static int tests_run;
 
 static const char *
 all_tests(void)
 {
+#ifdef HAVE_METALINK
+  mu_run_test (test_find_key_value);
+  mu_run_test (test_find_key_values);
+  mu_run_test (test_has_key);
+#endif
   mu_run_test (test_parse_content_disposition);
+  mu_run_test (test_parse_range_header);
   mu_run_test (test_subdir_p);
   mu_run_test (test_dir_matches_p);
   mu_run_test (test_commands_sorted);
@@ -67,6 +63,12 @@ all_tests(void)
   mu_run_test (test_append_uri_pathel);
   mu_run_test (test_are_urls_equal);
   mu_run_test (test_is_robots_txt_url);
+#ifdef HAVE_HSTS
+  mu_run_test (test_hsts_new_entry);
+  mu_run_test (test_hsts_url_rewrite_superdomain);
+  mu_run_test (test_hsts_url_rewrite_congruent);
+  mu_run_test (test_hsts_read_database);
+#endif
 
   return NULL;
 }
@@ -78,6 +80,7 @@ main (int argc _GL_UNUSED, const char *argv[])
 {
   const char *result;
 
+  printf ("[DEBUG] Testing...\n\n");
 #ifdef ENABLE_NLS
   /* Set the current locale.  */
   setlocale (LC_ALL, "");

src/test.h

@@ -1,6 +1,6 @@
 /* Unit testing declarations.
-   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software
-   Foundation, Inc.
+   Copyright (C) 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free
+   Software Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -43,9 +43,12 @@ do { \
   puts("PASSED\n"); \
 } while (0)
 
-extern int tests_run;
 
+const char *test_has_key (void);
+const char *test_find_key_value (void);
+const char *test_find_key_values (void);
 const char *test_parse_content_disposition(void);
+const char *test_parse_range_header(void);
 const char *test_commands_sorted(void);
 const char *test_cmd_spec_restrict_file_names(void);
 const char *test_is_robots_txt_url(void);
@@ -54,6 +57,10 @@ const char *test_append_uri_pathel(void);
 const char *test_are_urls_equal(void);
 const char *test_subdir_p(void);
 const char *test_dir_matches_p(void);
+const char *test_hsts_new_entry(void);
+const char *test_hsts_url_rewrite_superdomain(void);
+const char *test_hsts_url_rewrite_congruent(void);
+const char *test_hsts_read_database(void);
 
 #endif /* TEST_H */
 

src/url.c

@@ -1,7 +1,7 @@
 /* URL handling.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -43,6 +43,11 @@ as that of the covered work.  */
 #include "host.h"  /* for is_valid_ipv6_address */
 #include "c-strcase.h"
 
+#if HAVE_ICONV
+#include <iconv.h>
+#include <langinfo.h>
+#endif
+
 #ifdef __VMS
 #include "vms.h"
 #endif /* def __VMS */
@@ -78,6 +83,13 @@ static struct scheme_data supported_schemes[] =
   { "https",    "https://", DEFAULT_HTTPS_PORT, scm_has_query|scm_has_fragment },
 #endif
   { "ftp",      "ftp://",   DEFAULT_FTP_PORT,   scm_has_params|scm_has_fragment },
+#ifdef HAVE_SSL
+  /*
+   * Explicit FTPS uses the same port as FTP.
+   * Implicit FTPS has its own port (990), but it is disabled by default.
+   */
+  { "ftps",     "ftps://",  DEFAULT_FTP_PORT,  scm_has_params|scm_has_fragment },
+#endif
 
   /* SCHEME_INVALID */
   { NULL,       NULL,       -1,                 0 }
@@ -161,17 +173,8 @@ static const unsigned char urlchr_table[256] =
 #undef U
 #undef RU
 
-/* URL-unescape the string S.
-
-   This is done by transforming the sequences "%HH" to the character
-   represented by the hexadecimal digits HH.  If % is not followed by
-   two hexadecimal digits, it is inserted literally.
-
-   The transformation is done in place.  If you need the original
-   string intact, make a copy before calling this function.  */
-
-void
-url_unescape (char *s)
+static void
+url_unescape_1 (char *s, unsigned char mask)
 {
   char *t = s;                  /* t - tortoise */
   char *h = s;                  /* h - hare     */
@@ -190,6 +193,8 @@ url_unescape (char *s)
           if (!h[1] || !h[2] || !(c_isxdigit (h[1]) && c_isxdigit (h[2])))
             goto copychar;
           c = X2DIGITS_TO_NUM (h[1], h[2]);
+          if (urlchr_test(c, mask))
+            goto copychar;
           /* Don't unescape %00 because there is no way to insert it
              into a C string without effectively truncating it. */
           if (c == '\0')
@@ -201,6 +206,31 @@ url_unescape (char *s)
   *t = '\0';
 }
 
+/* URL-unescape the string S.
+
+   This is done by transforming the sequences "%HH" to the character
+   represented by the hexadecimal digits HH.  If % is not followed by
+   two hexadecimal digits, it is inserted literally.
+
+   The transformation is done in place.  If you need the original
+   string intact, make a copy before calling this function.  */
+void
+url_unescape (char *s)
+{
+  url_unescape_1 (s, 0);
+}
+
+/* URL-unescape the string S.
+
+   This functions behaves identically as url_unescape(), but does not
+   convert characters from "reserved". In other words, it only converts
+   "unsafe" characters.  */
+void
+url_unescape_except_reserved (char *s)
+{
+  url_unescape_1 (s, urlchr_reserved);
+}
+
 /* The core of url_escape_* functions.  Escapes the characters that
    match the provided mask in urlchr_table.
 
@@ -701,7 +731,7 @@ url_parse (const char *url, int *error, struct iri *iri, bool percent_encode)
     {
       char *new_url = NULL;
 
-      iri->utf8_encode = remote_to_utf8 (iri, iri->orig_url ? iri->orig_url : url, (const char **) &new_url);
+      iri->utf8_encode = remote_to_utf8 (iri, iri->orig_url ? iri->orig_url : url, &new_url);
       if (!iri->utf8_encode)
         new_url = NULL;
       else
@@ -1328,8 +1358,9 @@ append_string (const char *str, struct growable *dest)
 
 enum {
   filechr_not_unix    = 1,      /* unusable on Unix, / and \0 */
-  filechr_not_windows = 2,      /* unusable on Windows, one of \|/<>?:*" */
-  filechr_control     = 4       /* a control character, e.g. 0-31 */
+  filechr_not_vms     = 2,      /* unusable on VMS (ODS5), 0x00-0x1F * ? */
+  filechr_not_windows = 4,      /* unusable on Windows, one of \|/<>?:*" */
+  filechr_control     = 8       /* a control character, e.g. 0-31 */
 };
 
 #define FILE_CHAR_TEST(c, mask) \
@@ -1338,11 +1369,14 @@ enum {
 
 /* Shorthands for the table: */
 #define U filechr_not_unix
+#define V filechr_not_vms
 #define W filechr_not_windows
 #define C filechr_control
 
+#define UVWC U|V|W|C
 #define UW U|W
-#define UWC U|W|C
+#define VC V|C
+#define VW V|W
 
 /* Table of characters unsafe under various conditions (see above).
 
@@ -1353,25 +1387,25 @@ enum {
 
 static const unsigned char filechr_table[256] =
 {
-UWC,  C,  C,  C,   C,  C,  C,  C,   /* NUL SOH STX ETX  EOT ENQ ACK BEL */
-  C,  C,  C,  C,   C,  C,  C,  C,   /* BS  HT  LF  VT   FF  CR  SO  SI  */
-  C,  C,  C,  C,   C,  C,  C,  C,   /* DLE DC1 DC2 DC3  DC4 NAK SYN ETB */
-  C,  C,  C,  C,   C,  C,  C,  C,   /* CAN EM  SUB ESC  FS  GS  RS  US  */
-  0,  0,  W,  0,   0,  0,  0,  0,   /* SP  !   "   #    $   %   &   '   */
-  0,  0,  W,  0,   0,  0,  0, UW,   /* (   )   *   +    ,   -   .   /   */
-  0,  0,  0,  0,   0,  0,  0,  0,   /* 0   1   2   3    4   5   6   7   */
-  0,  0,  W,  0,   W,  0,  W,  W,   /* 8   9   :   ;    <   =   >   ?   */
-  0,  0,  0,  0,   0,  0,  0,  0,   /* @   A   B   C    D   E   F   G   */
-  0,  0,  0,  0,   0,  0,  0,  0,   /* H   I   J   K    L   M   N   O   */
-  0,  0,  0,  0,   0,  0,  0,  0,   /* P   Q   R   S    T   U   V   W   */
-  0,  0,  0,  0,   W,  0,  0,  0,   /* X   Y   Z   [    \   ]   ^   _   */
-  0,  0,  0,  0,   0,  0,  0,  0,   /* `   a   b   c    d   e   f   g   */
-  0,  0,  0,  0,   0,  0,  0,  0,   /* h   i   j   k    l   m   n   o   */
-  0,  0,  0,  0,   0,  0,  0,  0,   /* p   q   r   s    t   u   v   w   */
-  0,  0,  0,  0,   W,  0,  0,  C,   /* x   y   z   {    |   }   ~   DEL */
+UVWC, VC, VC, VC,  VC, VC, VC, VC,   /* NUL SOH STX ETX  EOT ENQ ACK BEL */
+  VC, VC, VC, VC,  VC, VC, VC, VC,   /* BS  HT  LF  VT   FF  CR  SO  SI  */
+  VC, VC, VC, VC,  VC, VC, VC, VC,   /* DLE DC1 DC2 DC3  DC4 NAK SYN ETB */
+  VC, VC, VC, VC,  VC, VC, VC, VC,   /* CAN EM  SUB ESC  FS  GS  RS  US  */
+   0,  0,  W,  0,   0,  0,  0,  0,   /* SP  !   "   #    $   %   &   '   */
+   0,  0, VW,  0,   0,  0,  0, UW,   /* (   )   *   +    ,   -   .   /   */
+   0,  0,  0,  0,   0,  0,  0,  0,   /* 0   1   2   3    4   5   6   7   */
+   0,  0,  W,  0,   W,  0,  W, VW,   /* 8   9   :   ;    <   =   >   ?   */
+   0,  0,  0,  0,   0,  0,  0,  0,   /* @   A   B   C    D   E   F   G   */
+   0,  0,  0,  0,   0,  0,  0,  0,   /* H   I   J   K    L   M   N   O   */
+   0,  0,  0,  0,   0,  0,  0,  0,   /* P   Q   R   S    T   U   V   W   */
+   0,  0,  0,  0,   W,  0,  0,  0,   /* X   Y   Z   [    \   ]   ^   _   */
+   0,  0,  0,  0,   0,  0,  0,  0,   /* `   a   b   c    d   e   f   g   */
+   0,  0,  0,  0,   0,  0,  0,  0,   /* h   i   j   k    l   m   n   o   */
+   0,  0,  0,  0,   0,  0,  0,  0,   /* p   q   r   s    t   u   v   w   */
+   0,  0,  0,  0,   W,  0,  0,  C,   /* x   y   z   {    |   }   ~   DEL */
 
-  C, C, C, C,  C, C, C, C,  C, C, C, C,  C, C, C, C, /* 128-143 */
-  C, C, C, C,  C, C, C, C,  C, C, C, C,  C, C, C, C, /* 144-159 */
+  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0, /* 128-143 */
+  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0, /* 144-159 */
   0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,
   0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,
 
@@ -1381,10 +1415,13 @@ UWC,  C,  C,  C,   C,  C,  C,  C,   /* NUL SOH STX ETX  EOT ENQ ACK BEL */
   0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,  0, 0, 0, 0,
 };
 #undef U
+#undef V
 #undef W
 #undef C
 #undef UW
-#undef UWC
+#undef UVWC
+#undef VC
+#undef VW
 
 /* FN_PORT_SEP is the separator between host and port in file names
    for non-standard port numbers.  On Unix this is normally ':', as in
@@ -1393,10 +1430,14 @@ UWC,  C,  C,  C,   C,  C,  C,  C,   /* NUL SOH STX ETX  EOT ENQ ACK BEL */
 #define FN_PORT_SEP  (opt.restrict_files_os != restrict_windows ? ':' : '+')
 
 /* FN_QUERY_SEP is the separator between the file name and the URL
-   query, normally '?'.  Since Windows cannot handle '?' as part of
+   query, normally '?'.  Because VMS and Windows cannot handle '?' in a
    file name, we use '@' instead there.  */
-#define FN_QUERY_SEP (opt.restrict_files_os != restrict_windows ? '?' : '@')
-#define FN_QUERY_SEP_STR (opt.restrict_files_os != restrict_windows ? "?" : "@")
+#define FN_QUERY_SEP \
+ (((opt.restrict_files_os != restrict_vms) && \
+   (opt.restrict_files_os != restrict_windows)) ? '?' : '@')
+#define FN_QUERY_SEP_STR \
+ (((opt.restrict_files_os != restrict_vms) && \
+   (opt.restrict_files_os != restrict_windows)) ? "?" : "@")
 
 /* Quote path element, characters in [b, e), as file name, and append
    the quoted string to DEST.  Each character is quoted as per
@@ -1415,6 +1456,8 @@ append_uri_pathel (const char *b, const char *e, bool escaped,
   int mask;
   if (opt.restrict_files_os == restrict_unix)
     mask = filechr_not_unix;
+  else if (opt.restrict_files_os == restrict_vms)
+    mask = filechr_not_vms;
   else
     mask = filechr_not_windows;
   if (opt.restrict_files_ctrl)
@@ -1493,6 +1536,82 @@ append_uri_pathel (const char *b, const char *e, bool escaped,
   append_null (dest);
 }
 
+static char *
+convert_fname (char *fname)
+{
+  char *converted_fname = fname;
+#if HAVE_ICONV
+  const char *from_encoding = opt.encoding_remote;
+  const char *to_encoding = opt.locale;
+  iconv_t cd;
+  size_t len, done, inlen, outlen;
+  char *s;
+  const char *orig_fname = fname;
+
+  /* Defaults for remote and local encodings.  */
+  if (!from_encoding)
+    from_encoding = "UTF-8";
+  if (!to_encoding)
+    to_encoding = nl_langinfo (CODESET);
+
+  cd = iconv_open (to_encoding, from_encoding);
+  if (cd == (iconv_t)(-1))
+    logprintf (LOG_VERBOSE, _("Conversion from %s to %s isn't supported\n"),
+	       quote (from_encoding), quote (to_encoding));
+  else
+    {
+      inlen = strlen (fname);
+      len = outlen = inlen * 2;
+      converted_fname = s = xmalloc (outlen + 1);
+      done = 0;
+
+      for (;;)
+	{
+	  if (iconv (cd, &fname, &inlen, &s, &outlen) != (size_t)(-1)
+	      && iconv (cd, NULL, NULL, &s, &outlen) != (size_t)(-1))
+	    {
+	      *(converted_fname + len - outlen - done) = '\0';
+	      iconv_close(cd);
+	      DEBUGP (("Converted file name '%s' (%s) -> '%s' (%s)\n",
+		       orig_fname, from_encoding, converted_fname, to_encoding));
+	      xfree (orig_fname);
+	      return converted_fname;
+	    }
+
+	  /* Incomplete or invalid multibyte sequence */
+	  if (errno == EINVAL || errno == EILSEQ)
+	    {
+	      logprintf (LOG_VERBOSE,
+			 _("Incomplete or invalid multibyte sequence encountered\n"));
+	      xfree (converted_fname);
+	      converted_fname = (char *)orig_fname;
+	      break;
+	    }
+	  else if (errno == E2BIG) /* Output buffer full */
+	    {
+	      done = len;
+	      len = outlen = done + inlen * 2;
+	      converted_fname = xrealloc (converted_fname, outlen + 1);
+	      s = converted_fname + done;
+	    }
+	  else /* Weird, we got an unspecified error */
+	    {
+	      logprintf (LOG_VERBOSE, _("Unhandled errno %d\n"), errno);
+	      xfree (converted_fname);
+	      converted_fname = (char *)orig_fname;
+	      break;
+	    }
+	}
+      DEBUGP (("Failed to convert file name '%s' (%s) -> '?' (%s)\n",
+	       orig_fname, from_encoding, to_encoding));
+    }
+
+    iconv_close(cd);
+#endif
+
+  return converted_fname;
+}
+
 /* Append to DEST the directory structure that corresponds the
    directory part of URL's path.  For example, if the URL is
    http://server/dir1/dir2/file, this appends "/dir1/dir2".
@@ -1668,6 +1787,8 @@ url_file_name (const struct url *u, char *replaced_filename)
 
   xfree (temp_fnres.base);
 
+  fname = convert_fname (fname);
+
   /* Check the cases in which the unique extensions are not used:
      1) Clobbering is turned off (-nc).
      2) Retrieval with regetting.
@@ -1741,7 +1862,7 @@ path_simplify (enum url_scheme scheme, char *path)
       else if (h[0] == '.' && h[1] == '.' && (h[2] == '/' || h[2] == '\0'))
         {
           /* Handle "../" by retreating the tortoise by one path
-             element -- but not past beggining.  */
+             element -- but not past beginning.  */
           if (t > beg)
             {
               /* Move backwards until T hits the beginning of the
@@ -1749,7 +1870,11 @@ path_simplify (enum url_scheme scheme, char *path)
               for (--t; t > beg && t[-1] != '/'; t--)
                 ;
             }
-          else if (scheme == SCHEME_FTP)
+          else if (scheme == SCHEME_FTP
+#ifdef HAVE_SSL
+              || scheme == SCHEME_FTPS
+#endif
+              )
             {
               /* If we're at the beginning, copy the "../" literally
                  and move the beginning so a later ".." doesn't remove
@@ -2319,6 +2444,7 @@ test_append_uri_pathel(void)
 
       mu_assert ("test_append_uri_pathel: wrong result",
                  strcmp (dest.base, test_array[i].expected_result) == 0);
+      xfree (dest.base);
     }
 
   return NULL;

src/url.h

@@ -1,7 +1,7 @@
 /* Declarations for url.c.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -36,6 +36,7 @@ as that of the covered work.  */
 #define DEFAULT_HTTP_PORT 80
 #define DEFAULT_FTP_PORT 21
 #define DEFAULT_HTTPS_PORT 443
+#define DEFAULT_FTPS_IMPLICIT_PORT 990
 
 /* This represents how many characters less than the OS max name length a file
  * should be.  More precisely, a file name should be at most
@@ -70,6 +71,9 @@ enum url_scheme {
   SCHEME_HTTPS,
 #endif
   SCHEME_FTP,
+#ifdef HAVE_SSL
+  SCHEME_FTPS,
+#endif
   SCHEME_INVALID
 };
 
@@ -106,6 +110,7 @@ struct url
 char *url_escape (const char *);
 char *url_escape_unsafe_and_reserved (const char *);
 void url_unescape (char *);
+void url_unescape_except_reserved (char *);
 
 struct url *url_parse (const char *, int *, struct iri *iri, bool percent_encode);
 char *url_error (const char *, int);

src/utils.c

@@ -1,7 +1,7 @@
 /* Various utility functions.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -31,14 +31,12 @@ as that of the covered work.  */
 
 #include "wget.h"
 
+#include "sha256.h"
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <time.h>
 #include <unistd.h>
-#ifdef HAVE_MMAP
-# include <sys/mman.h>
-#endif
 #ifdef HAVE_PROCESS_H
 # include <process.h>  /* getpid() */
 #endif
@@ -89,6 +87,18 @@ as that of the covered work.  */
 # define USE_SIGNAL_TIMEOUT
 #endif
 
+/* Some systems (Linux libc5, "NCR MP-RAS 3.0", and others) don't
+   provide MAP_FAILED, a symbolic constant for the value returned by
+   mmap() when it doesn't work.  Usually, this constant should be -1.
+   This only makes sense for files that use mmap() and include
+   sys/mman.h *before* sysdep.h, but doesn't hurt others.  */
+#ifdef HAVE_MMAP
+# include <sys/mman.h>
+# ifndef MAP_FAILED
+#  define MAP_FAILED ((void *) -1)
+# endif
+#endif
+
 #include "utils.h"
 #include "hash.h"
 
@@ -336,7 +346,7 @@ aprintf (const char *fmt, ...)
         {                               /* maybe we have some wrong
                                            format string? */
           logprintf (LOG_ALWAYS,
-                     _("%s: aprintf: text buffer is too big (%ld bytes), "
+                     _("%s: aprintf: text buffer is too big (%d bytes), "
                        "aborting.\n"),
                      exec_name, size);  /* printout a log message */
           abort ();                     /* and abort... */
@@ -1284,7 +1294,10 @@ free_vec (char **vec)
     {
       char **p = vec;
       while (*p)
-        xfree (*p++);
+        {
+          xfree (*p);
+          p++;
+        }
       xfree (vec);
     }
 }
@@ -2503,6 +2516,220 @@ get_max_length (const char *path, int length, int name)
   return ret;
 }
 
+void
+wg_hex_to_string (char *str_buffer, const char *hex_buffer, size_t hex_len)
+{
+  size_t i;
+
+  for (i = 0; i < hex_len; i++)
+    {
+      /* Each byte takes 2 characters.  */
+      sprintf (str_buffer + 2 * i, "%02x", hex_buffer[i] & 0xFF);
+    }
+
+  /* Null-terminate result.  */
+  str_buffer[2 * i] = '\0';
+}
+
+#ifdef HAVE_SSL
+
+/*
+ * Public key pem to der conversion
+ */
+
+static bool
+wg_pubkey_pem_to_der (const char *pem, unsigned char **der, size_t *der_len)
+{
+  char *stripped_pem, *begin_pos, *end_pos;
+  size_t pem_count, stripped_pem_count = 0, pem_len;
+  ssize_t size;
+  unsigned char *base64data;
+
+  *der = NULL;
+  *der_len = 0;
+
+  /* if no pem, exit. */
+  if (!pem)
+    return false;
+
+  begin_pos = strstr (pem, "-----BEGIN PUBLIC KEY-----");
+  if (!begin_pos)
+    return false;
+
+  pem_count = begin_pos - pem;
+  /* Invalid if not at beginning AND not directly following \n */
+  if (0 != pem_count && '\n' != pem[pem_count - 1])
+    return false;
+
+  /* 26 is length of "-----BEGIN PUBLIC KEY-----" */
+  pem_count += 26;
+
+  /* Invalid if not directly following \n */
+  end_pos = strstr (pem + pem_count, "\n-----END PUBLIC KEY-----");
+  if (!end_pos)
+    return false;
+
+  pem_len = end_pos - pem;
+
+  stripped_pem = xmalloc (pem_len - pem_count + 1);
+
+  /*
+   * Here we loop through the pem array one character at a time between the
+   * correct indices, and place each character that is not '\n' or '\r'
+   * into the stripped_pem array, which should represent the raw base64 string
+   */
+  while (pem_count < pem_len) {
+    if ('\n' != pem[pem_count] && '\r' != pem[pem_count])
+      stripped_pem[stripped_pem_count++] = pem[pem_count];
+    ++pem_count;
+  }
+  /* Place the null terminator in the correct place */
+  stripped_pem[stripped_pem_count] = '\0';
+
+  base64data = xmalloc (BASE64_LENGTH(stripped_pem_count));
+
+  size = base64_decode (stripped_pem, base64data);
+
+  if (size < 0) {
+    xfree (base64data);           /* malformed base64 from server */
+  } else {
+    *der = base64data;
+    *der_len = (size_t) size;
+  }
+
+  xfree (stripped_pem);
+
+  return *der_len > 0;
+}
+
+/*
+ * Generic pinned public key check.
+ */
+
+bool
+wg_pin_peer_pubkey (const char *pinnedpubkey, const char *pubkey, size_t pubkeylen)
+{
+  struct file_memory *fm;
+  unsigned char *buf = NULL, *pem_ptr = NULL;
+  size_t size, pem_len;
+  bool pem_read;
+  bool result = false;
+
+  size_t pinkeylen;
+  ssize_t decoded_hash_length;
+  char *pinkeycopy, *begin_pos, *end_pos;
+  unsigned char *sha256sumdigest = NULL, *expectedsha256sumdigest = NULL;
+
+  /* if a path wasn't specified, don't pin */
+  if (!pinnedpubkey)
+    return true;
+  if (!pubkey || !pubkeylen)
+    return result;
+
+  /* only do this if pinnedpubkey starts with "sha256//", length 8 */
+  if (strncmp (pinnedpubkey, "sha256//", 8) == 0) {
+    /* compute sha256sum of public key */
+    sha256sumdigest = xmalloc (SHA256_DIGEST_SIZE);
+    sha256_buffer (pubkey, pubkeylen, sha256sumdigest);
+    expectedsha256sumdigest = xmalloc (SHA256_DIGEST_SIZE + 1);
+
+    /* it starts with sha256//, copy so we can modify it */
+    pinkeylen = strlen (pinnedpubkey) + 1;
+    pinkeycopy = xmalloc (pinkeylen);
+    memcpy (pinkeycopy, pinnedpubkey, pinkeylen);
+
+    /* point begin_pos to the copy, and start extracting keys */
+    begin_pos = pinkeycopy;
+    do
+      {
+        end_pos = strstr (begin_pos, ";sha256//");
+        /*
+         * if there is an end_pos, null terminate,
+         * otherwise it'll go to the end of the original string
+         */
+        if (end_pos)
+          end_pos[0] = '\0';
+
+        /* decode base64 pinnedpubkey, 8 is length of "sha256//" */
+        decoded_hash_length = base64_decode (begin_pos + 8, expectedsha256sumdigest);
+        /* if valid base64, compare sha256 digests directly */
+        if (SHA256_DIGEST_SIZE == decoded_hash_length &&
+           !memcmp (sha256sumdigest, expectedsha256sumdigest, SHA256_DIGEST_SIZE)) {
+          result = true;
+          break;
+        }
+
+        /*
+         * change back the null-terminator we changed earlier,
+         * and look for next begin
+         */
+        if (end_pos) {
+          end_pos[0] = ';';
+          begin_pos = strstr (end_pos, "sha256//");
+        }
+      } while (end_pos && begin_pos);
+
+    xfree (sha256sumdigest);
+    xfree (expectedsha256sumdigest);
+    xfree (pinkeycopy);
+
+    return result;
+  }
+
+  /* fall back to assuming this is a file path */
+  fm = wget_read_file (pinnedpubkey);
+  if (!fm)
+    return result;
+
+  /* Check the file's size */
+  if (fm->length < 0 || fm->length > MAX_PINNED_PUBKEY_SIZE)
+    goto cleanup;
+
+  /*
+   * if the size of our certificate is bigger than the file
+   * size then it can't match
+   */
+  size = (size_t) fm->length;
+  if (pubkeylen > size)
+    goto cleanup;
+
+  /* If the sizes are the same, it can't be base64 encoded, must be der */
+  if (pubkeylen == size) {
+    if (!memcmp (pubkey, fm->content, pubkeylen))
+      result = true;
+    goto cleanup;
+  }
+
+  /*
+   * Otherwise we will assume it's PEM and try to decode it
+   * after placing null terminator
+   */
+  buf = xmalloc (size + 1);
+  memcpy (buf, fm->content, size);
+  buf[size] = '\0';
+
+  pem_read = wg_pubkey_pem_to_der ((const char *) buf, &pem_ptr, &pem_len);
+  /* if it wasn't read successfully, exit */
+  if (!pem_read)
+    goto cleanup;
+
+  /*
+   * if the size of our certificate doesn't match the size of
+   * the decoded file, they can't be the same, otherwise compare
+   */
+  if (pubkeylen == pem_len && !memcmp (pubkey, pem_ptr, pubkeylen))
+    result = true;
+
+ cleanup:
+  xfree (buf);
+  xfree (pem_ptr);
+  wget_read_file_free (fm);
+
+  return result;
+}
+
+#endif /* HAVE_SSL */
+
 #ifdef TESTING
 
 const char *

src/utils.h

@@ -1,7 +1,7 @@
 /* Declarations for utils.c.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -32,9 +32,15 @@ as that of the covered work.  */
 #ifndef UTILS_H
 #define UTILS_H
 
+# include <stdlib.h>
+
 /* Constant is using when we don`t know attempted size exactly */
 #define UNKNOWN_ATTEMPTED_SIZE -3
 
+#ifndef MAX_PINNED_PUBKEY_SIZE
+#define MAX_PINNED_PUBKEY_SIZE 1048576 /* 1MB */
+#endif
+
 /* Macros that interface to malloc, but know about type sizes, and
    cast the result to the appropriate type.  The casts are not
    necessary in standard C, but Wget performs them anyway for the sake
@@ -155,6 +161,13 @@ long get_max_length (const char *path, int length, int name);
 size_t strlcpy (char *dst, const char *src, size_t size);
 #endif
 
+void wg_hex_to_string (char *str_buffer, const char *hex_buffer, size_t hex_len);
+
 extern unsigned char char_prop[];
 
+#ifdef HAVE_SSL
+/* Check pinned public key. */
+bool wg_pin_peer_pubkey (const char *pinnedpubkey, const char *pubkey, size_t pubkeylen);
+#endif
+
 #endif /* UTILS_H */

src/version.h

@@ -1,5 +1,5 @@
 /* Extern declarations for printing version information
-   Copyright (C) 2013 Free Software Foundation, Inc.
+   Copyright (C) 2013, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 

src/warc.c

@@ -1,5 +1,5 @@
 /* Utility functions for writing WARC files.
-   Copyright (C) 2011, 2012 Free Software Foundation, Inc.
+   Copyright (C) 2011, 2012, 2015 Free Software Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -27,10 +27,6 @@ Corresponding Source for a non-source form of such a combination
 shall include the source code for the parts of OpenSSL used as well
 as that of the covered work.  */
 
-#ifndef _GNU_SOURCE
-#define _GNU_SOURCE
-#endif
-
 #include "wget.h"
 #include "hash.h"
 #include "utils.h"
@@ -102,7 +98,7 @@ static bool warc_write_ok;
 static FILE *warc_current_cdx_file;
 
 /* The record id of the warcinfo record of the current WARC file.  */
-static char *warc_current_warcinfo_uuid_str;
+static char warc_current_warcinfo_uuid_str[48];
 
 /* The file name of the current WARC file. */
 static char *warc_current_filename;
@@ -250,6 +246,22 @@ warc_write_header (const char *name, const char *value)
   return warc_write_ok;
 }
 
+/* Writes a WARC header with a URI as value to the current WARC record.
+   This method may be run after warc_write_start_record and
+   before warc_write_block_from_file.  */
+static bool
+warc_write_header_uri (const char *name, const char *value)
+{
+  if (value)
+    {
+      warc_write_string (name);
+      warc_write_string (": <");
+      warc_write_string (value);
+      warc_write_string (">\r\n");
+    }
+  return warc_write_ok;
+}
+
 /* Copies the contents of DATA_IN to the WARC record.
    Adds a Content-Length header to the WARC record.
    Run this method after warc_write_header,
@@ -404,7 +416,7 @@ warc_write_date_header (const char *timestamp)
    the current WARC record.  If IP is NULL, no header will
    be written.  */
 static bool
-warc_write_ip_header (ip_address *ip)
+warc_write_ip_header (const ip_address *ip)
 {
   if (ip != NULL)
     return warc_write_header ("WARC-IP-Address", print_address (ip));
@@ -435,9 +447,7 @@ warc_sha1_stream_with_payload (FILE *stream, void *res_block, void *res_payload,
   off_t pos;
   off_t sum;
 
-  char *buffer = malloc (BLOCKSIZE + 72);
-  if (!buffer)
-    return 1;
+  char *buffer = xmalloc (BLOCKSIZE + 72);
 
   /* Initialize the computation context.  */
   sha1_init_ctx (&ctx_block);
@@ -542,14 +552,17 @@ warc_sha1_stream_with_payload (FILE *stream, void *res_block, void *res_payload,
 /* Converts the SHA1 digest to a base32-encoded string.
    "sha1:DIGEST\0"  (Allocates a new string for the response.)  */
 static char *
-warc_base32_sha1_digest (char *sha1_digest)
+warc_base32_sha1_digest (const char *sha1_digest, char *sha1_base32, size_t sha1_base32_size)
 {
-  /* length: "sha1:" + digest + "\0" */
-  char *sha1_base32 = malloc (BASE32_LENGTH(SHA1_DIGEST_SIZE) + 1 + 5 );
-  base32_encode (sha1_digest, SHA1_DIGEST_SIZE, sha1_base32 + 5,
-                 BASE32_LENGTH(SHA1_DIGEST_SIZE) + 1);
-  memcpy (sha1_base32, "sha1:", 5);
-  sha1_base32[BASE32_LENGTH(SHA1_DIGEST_SIZE) + 5] = '\0';
+  if (sha1_base32_size >= BASE32_LENGTH(SHA1_DIGEST_SIZE) + 5 + 1)
+    {
+      memcpy (sha1_base32, "sha1:", 5);
+      base32_encode (sha1_digest, SHA1_DIGEST_SIZE, sha1_base32 + 5,
+                     sha1_base32_size - 5);
+    }
+  else
+    *sha1_base32 = 0;
+
   return sha1_base32;
 }
 
@@ -571,18 +584,14 @@ warc_write_digest_headers (FILE *file, long payload_offset)
       if (warc_sha1_stream_with_payload (file, sha1_res_block,
           sha1_res_payload, payload_offset) == 0)
         {
-          char *digest;
+          char digest[BASE32_LENGTH(SHA1_DIGEST_SIZE) + 1 + 5];
 
-          digest = warc_base32_sha1_digest (sha1_res_block);
-          warc_write_header ("WARC-Block-Digest", digest);
-          xfree (digest);
+          warc_write_header ("WARC-Block-Digest",
+              warc_base32_sha1_digest (sha1_res_block, digest, sizeof(digest)));
 
           if (payload_offset >= 0)
-            {
-              digest = warc_base32_sha1_digest (sha1_res_payload);
-              warc_write_header ("WARC-Payload-Digest", digest);
-              xfree (digest);
-            }
+              warc_write_header ("WARC-Payload-Digest",
+                  warc_base32_sha1_digest (sha1_res_payload, digest, sizeof(digest)));
         }
     }
 }
@@ -734,7 +743,7 @@ warc_uuid_str (char *urn_str)
 /* Write a warcinfo record to the current file.
    Updates warc_current_warcinfo_uuid_str. */
 static bool
-warc_write_warcinfo_record (char *filename)
+warc_write_warcinfo_record (const char *filename)
 {
   FILE *warc_tmp;
   char timestamp[22];
@@ -743,7 +752,6 @@ warc_write_warcinfo_record (char *filename)
   /* Write warc-info record as the first record of the file. */
   /* We add the record id of this info record to the other records in the
      file. */
-  warc_current_warcinfo_uuid_str = (char *) malloc (48);
   warc_uuid_str (warc_current_warcinfo_uuid_str);
 
   warc_timestamp (timestamp, sizeof(timestamp));
@@ -827,14 +835,15 @@ warc_start_new_file (bool meta)
   if (warc_current_file != NULL)
     fclose (warc_current_file);
 
-  xfree (warc_current_warcinfo_uuid_str);
+  *warc_current_warcinfo_uuid_str = 0;
   xfree (warc_current_filename);
 
   warc_current_file_number++;
 
   base_filename_length = strlen (opt.warc_filename);
   /* filename format:  base + "-" + 5 digit serial number + ".warc.gz" */
-  new_filename = malloc (base_filename_length + 1 + 5 + 8 + 1);
+  new_filename = xmalloc (base_filename_length + 1 + 5 + 8 + 1);
+
   warc_current_filename = new_filename;
 
   /* If max size is enabled, we add a serial number to the file names. */
@@ -995,7 +1004,7 @@ warc_process_cdx_line (char *lineptr, int field_num_original_url,
         {
           /* This is a valid line with a valid checksum. */
           struct warc_cdx_record *rec;
-          rec = malloc (sizeof (struct warc_cdx_record));
+          rec = xmalloc (sizeof (struct warc_cdx_record));
           rec->url = original_url;
           rec->uuid = record_id;
           memcpy (rec->digest, checksum_v, SHA1_DIGEST_SIZE);
@@ -1098,7 +1107,7 @@ _("CDX file does not list record ids. (Missing column 'u'.)\n"));
    digest.  Returns NULL if the url is not found or if the payload digest
    does not match, or if CDX deduplication is disabled. */
 static struct warc_cdx_record *
-warc_find_duplicate_cdx_record (char *url, char *sha1_digest_payload)
+warc_find_duplicate_cdx_record (const char *url, char *sha1_digest_payload)
 {
   struct warc_cdx_record *rec_existing;
 
@@ -1228,7 +1237,7 @@ warc_close (void)
   if (warc_current_file != NULL)
     {
       warc_write_metadata ();
-      xfree (warc_current_warcinfo_uuid_str);
+      *warc_current_warcinfo_uuid_str = 0;
       fclose (warc_current_file);
     }
   if (warc_current_cdx_file != NULL)
@@ -1293,12 +1302,13 @@ warc_tempfile (void)
    Calling this function will close body.
    Returns true on success, false on error. */
 bool
-warc_write_request_record (char *url, char *timestamp_str, char *record_uuid,
-                           ip_address *ip, FILE *body, off_t payload_offset)
+warc_write_request_record (const char *url, const char *timestamp_str,
+                           const char *record_uuid, const ip_address *ip,
+                           FILE *body, off_t payload_offset)
 {
   warc_write_start_record ();
   warc_write_header ("WARC-Type", "request");
-  warc_write_header ("WARC-Target-URI", url);
+  warc_write_header_uri ("WARC-Target-URI", url);
   warc_write_header ("Content-Type", "application/http;msgtype=request");
   warc_write_date_header (timestamp_str);
   warc_write_header ("WARC-Record-ID", record_uuid);
@@ -1382,18 +1392,18 @@ warc_write_cdx_record (const char *url, const char *timestamp_str,
    Calling this function will close body.
    Returns true on success, false on error. */
 static bool
-warc_write_revisit_record (char *url, char *timestamp_str,
-                           char *concurrent_to_uuid, char *payload_digest,
-                           char *refers_to, ip_address *ip, FILE *body)
+warc_write_revisit_record (const char *url, const char *timestamp_str,
+                           const char *concurrent_to_uuid, const char *payload_digest,
+                           const char *refers_to, const ip_address *ip, FILE *body)
 {
   char revisit_uuid [48];
-  char *block_digest = NULL;
+  char block_digest[BASE32_LENGTH(SHA1_DIGEST_SIZE) + 1 + 5];
   char sha1_res_block[SHA1_DIGEST_SIZE];
 
   warc_uuid_str (revisit_uuid);
 
   sha1_stream (body, sha1_res_block);
-  block_digest = warc_base32_sha1_digest (sha1_res_block);
+  warc_base32_sha1_digest (sha1_res_block, block_digest, sizeof(block_digest));
 
   warc_write_start_record ();
   warc_write_header ("WARC-Type", "revisit");
@@ -1413,7 +1423,6 @@ warc_write_revisit_record (char *url, char *timestamp_str,
   warc_write_end_record ();
 
   fclose (body);
-  xfree (block_digest);
 
   return warc_write_ok;
 }
@@ -1432,13 +1441,13 @@ warc_write_revisit_record (char *url, char *timestamp_str,
    Calling this function will close body.
    Returns true on success, false on error. */
 bool
-warc_write_response_record (char *url, char *timestamp_str,
-                            char *concurrent_to_uuid, ip_address *ip,
-                            FILE *body, off_t payload_offset, char *mime_type,
-                            int response_code, char *redirect_location)
+warc_write_response_record (const char *url, const char *timestamp_str,
+                            const char *concurrent_to_uuid, const ip_address *ip,
+                            FILE *body, off_t payload_offset, const char *mime_type,
+                            int response_code, const char *redirect_location)
 {
-  char *block_digest = NULL;
-  char *payload_digest = NULL;
+  char block_digest[BASE32_LENGTH(SHA1_DIGEST_SIZE) + 1 + 5];
+  char payload_digest[BASE32_LENGTH(SHA1_DIGEST_SIZE) + 1 + 5];
   char sha1_res_block[SHA1_DIGEST_SIZE];
   char sha1_res_payload[SHA1_DIGEST_SIZE];
   char response_uuid [48];
@@ -1471,17 +1480,16 @@ warc_write_response_record (char *url, char *timestamp_str,
                 }
 
               /* Send the original payload digest. */
-              payload_digest = warc_base32_sha1_digest (sha1_res_payload);
+              warc_base32_sha1_digest (sha1_res_payload, payload_digest, sizeof(payload_digest));
               result = warc_write_revisit_record (url, timestamp_str,
                          concurrent_to_uuid, payload_digest, rec_existing->uuid,
                          ip, body);
-              xfree (payload_digest);
 
               return result;
             }
 
-          block_digest = warc_base32_sha1_digest (sha1_res_block);
-          payload_digest = warc_base32_sha1_digest (sha1_res_payload);
+          warc_base32_sha1_digest (sha1_res_block, block_digest, sizeof(block_digest));
+          warc_base32_sha1_digest (sha1_res_payload, payload_digest, sizeof(payload_digest));
         }
     }
 
@@ -1516,9 +1524,6 @@ warc_write_response_record (char *url, char *timestamp_str,
       response_uuid);
     }
 
-  xfree (block_digest);
-  xfree (payload_digest);
-
   return warc_write_ok;
 }
 
@@ -1535,16 +1540,18 @@ warc_write_response_record (char *url, char *timestamp_str,
    Calling this function will close body.
    Returns true on success, false on error. */
 static bool
-warc_write_record (const char *record_type, char *resource_uuid,
+warc_write_record (const char *record_type, const char *resource_uuid,
                  const char *url, const char *timestamp_str,
                  const char *concurrent_to_uuid,
-                 ip_address *ip, const char *content_type, FILE *body,
+                 const ip_address *ip, const char *content_type, FILE *body,
                  off_t payload_offset)
 {
   if (resource_uuid == NULL)
     {
-      resource_uuid = alloca (48);
-      warc_uuid_str (resource_uuid);
+      /* using uuid_buf allows const for resource_uuid in function declaration */
+      char *uuid_buf = alloca (48);
+      warc_uuid_str (uuid_buf);
+      resource_uuid = uuid_buf;
     }
 
   if (content_type == NULL)
@@ -1580,9 +1587,9 @@ warc_write_record (const char *record_type, char *resource_uuid,
    Calling this function will close body.
    Returns true on success, false on error. */
 bool
-warc_write_resource_record (char *resource_uuid, const char *url,
+warc_write_resource_record (const char *resource_uuid, const char *url,
                  const char *timestamp_str, const char *concurrent_to_uuid,
-                 ip_address *ip, const char *content_type, FILE *body,
+                 const ip_address *ip, const char *content_type, FILE *body,
                  off_t payload_offset)
 {
   return warc_write_record ("resource",
@@ -1602,7 +1609,7 @@ warc_write_resource_record (char *resource_uuid, const char *url,
    Calling this function will close body.
    Returns true on success, false on error. */
 bool
-warc_write_metadata_record (char *record_uuid, const char *url,
+warc_write_metadata_record (const char *record_uuid, const char *url,
                  const char *timestamp_str, const char *concurrent_to_uuid,
                  ip_address *ip, const char *content_type, FILE *body,
                  off_t payload_offset)

src/warc.h

@@ -12,15 +12,15 @@ char * warc_timestamp (char *timestamp, size_t timestamp_size);
 
 FILE * warc_tempfile (void);
 
-bool warc_write_request_record (char *url, char *timestamp_str,
-  char *concurrent_to_uuid, ip_address *ip, FILE *body, off_t payload_offset);
-bool warc_write_response_record (char *url, char *timestamp_str,
-  char *concurrent_to_uuid, ip_address *ip, FILE *body, off_t payload_offset,
-  char *mime_type, int response_code, char *redirect_location);
-bool warc_write_resource_record (char *resource_uuid, const char *url,
-  const char *timestamp_str, const char *concurrent_to_uuid, ip_address *ip,
+bool warc_write_request_record (const char *url, const char *timestamp_str,
+  const char *concurrent_to_uuid, const ip_address *ip, FILE *body, off_t payload_offset);
+bool warc_write_response_record (const char *url, const char *timestamp_str,
+  const char *concurrent_to_uuid, const ip_address *ip, FILE *body, off_t payload_offset,
+  const char *mime_type, int response_code, const char *redirect_location);
+bool warc_write_resource_record (const char *resource_uuid, const char *url,
+  const char *timestamp_str, const char *concurrent_to_uuid, const ip_address *ip,
   const char *content_type, FILE *body, off_t payload_offset);
-bool warc_write_metadata_record (char *record_uuid, const char *url,
+bool warc_write_metadata_record (const char *record_uuid, const char *url,
   const char *timestamp_str, const char *concurrent_to_uuid, ip_address *ip,
   const char *content_type, FILE *body, off_t payload_offset);
 

src/wget.h

@@ -1,7 +1,7 @@
 /* Miscellaneous declarations.
    Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
-   2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
-   Inc.
+   2005, 2006, 2007, 2008, 2009, 2010, 2011, 2015 Free Software
+   Foundation, Inc.
 
 This file is part of GNU Wget.
 
@@ -48,12 +48,15 @@ as that of the covered work.  */
 
 /* Disable assertions when debug support is not compiled in. */
 #ifndef ENABLE_DEBUG
+#ifndef NDEBUG
 # define NDEBUG
 #endif
+#endif
 
 /* Is OpenSSL or GNUTLS available? */
 #if defined HAVE_LIBSSL || defined HAVE_LIBSSL32 || defined HAVE_LIBGNUTLS
 # define HAVE_SSL
+# define HAVE_HSTS /* There's no sense in enabling HSTS without SSL */
 #endif
 
 /* `gettext (FOO)' is long to write, so we use `_(FOO)'.  If NLS is
@@ -331,7 +334,9 @@ enum
   SEND_NOCACHE         = 0x0008,        /* send Pragma: no-cache directive */
   ACCEPTRANGES         = 0x0010,        /* Accept-ranges header was found */
   ADDED_HTML_EXTENSION = 0x0020,        /* added ".html" extension due to -E */
-  TEXTCSS              = 0x0040         /* document is of type text/css */
+  TEXTCSS              = 0x0040,        /* document is of type text/css */
+  IF_MODIFIED_SINCE    = 0x0080,        /* use if-modified-since header */
+  METALINK_METADATA    = 0x0100         /* use HTTP response for Metalink metadata */
 };
 
 /* Universal error type -- used almost everywhere.  Error reporting of
@@ -346,12 +351,17 @@ typedef enum
   FTPSRVERR, FTPRETRINT, FTPRESTFAIL, URLERROR, FOPENERR,
   FOPEN_EXCL_ERR, FWRITEERR, HEOF, GATEWAYTIMEOUT,
   HERR, RETROK, RECLEVELEXC, WRONGCODE,
-  FTPINVPASV, FTPNOPASV, CONTNOTSUPPORTED, RETRUNNEEDED, RETRFINISHED,
+  FTPINVPASV, FTPNOPASV, FTPNOPBSZ, FTPNOPROT, FTPNOAUTH,
+  CONTNOTSUPPORTED, RETRUNNEEDED, RETRFINISHED,
   READERR, TRYLIMEXC, FILEBADFILE, RANGEERR,
   RETRBADPATTERN, PROXERR,
   AUTHFAILED, QUOTEXC, WRITEFAILED, SSLINITFAILED, VERIFCERTERR,
   UNLINKERR, NEWLOCATION_KEEP_POST, CLOSEFAILED, ATTRMISSING, UNKNOWNATTR,
-  WARC_ERR, WARC_TMP_FOPENERR, WARC_TMP_FWRITEERR
+  WARC_ERR, WARC_TMP_FOPENERR, WARC_TMP_FWRITEERR,
+  TIMECONV_ERR,
+  METALINK_PARSE_ERROR, METALINK_RETR_ERROR,
+  METALINK_CHKSUM_ERROR, METALINK_SIG_ERROR, METALINK_MISSING_RESOURCE,
+  RETR_WITH_METALINK
 } uerr_t;
 
 /* 2005-02-19 SMS.

testenv/Makefile.am

@@ -1,5 +1,5 @@
 # Makefile for `wget' utility
-# Copyright (C) 2013 Free Software Foundation, Inc.
+# Copyright (C) 2013, 2015 Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -26,39 +26,62 @@
 # as that of the covered work.
 
 
+if METALINK_IS_ENABLED
+  METALINK_TESTS = Test-metalink-xml.py             \
+    Test-metalink-http.py
+else
+  METALINK_TESTS =
+endif
+
 AUTOMAKE_OPTIONS = parallel-tests
 AM_TESTS_ENVIRONMENT = export WGETRC=/dev/null; MAKE_CHECK=True; export MAKE_CHECK;\
  export PYTHONPATH=$$PYTHONPATH:$(srcdir); export VALGRIND_TESTS="@VALGRIND_TESTS@";
-if HAVE_PYTHON3
-  TESTS = Test-auth-basic-fail.py           \
-    Test-auth-basic.py                      \
-    Test-auth-both.py                       \
-    Test-auth-digest.py                     \
-    Test-auth-no-challenge.py               \
-    Test-auth-no-challenge-url.py           \
-    Test-auth-retcode.py                    \
-    Test-auth-with-content-disposition.py   \
-    Test-c-full.py                          \
-    Test-Content-disposition-2.py           \
-    Test-Content-disposition.py             \
-    Test-cookie-401.py                      \
-    Test-cookie-domain-mismatch.py          \
-    Test-cookie-expires.py                  \
-    Test-cookie.py                          \
-    Test-Head.py                            \
-    Test--https.py                          \
-    Test--https-crl.py                      \
-    Test-O.py                               \
-    Test-Post.py                            \
-    Test-504.py                             \
-    Test--spider-r.py                       \
-    Test-redirect-crash.py
 
-  # added test cases expected to fail here and under TESTS
-  XFAIL_TESTS =
+if WITH_SSL
+  AM_TESTS_ENVIRONMENT += export SSL_TESTS=1;
 endif
 
-EXTRA_DIST = certs conf exc misc server test README $(TESTS) $(XFAIL_TESTS)
+if HAVE_PYTHON3
+  TESTS = Test-auth-basic-fail.py                   \
+    Test-auth-basic.py                              \
+    Test-auth-both.py                               \
+    Test-auth-digest.py                             \
+    Test-auth-no-challenge.py                       \
+    Test-auth-no-challenge-url.py                   \
+    Test-auth-retcode.py                            \
+    Test-auth-with-content-disposition.py           \
+    Test-c-full.py                                  \
+    Test-Content-disposition-2.py                   \
+    Test-Content-disposition.py                     \
+    Test--convert-links--content-on-error.py        \
+    Test-cookie-401.py                              \
+    Test-cookie-domain-mismatch.py                  \
+    Test-cookie-expires.py                          \
+    Test-cookie.py                                  \
+    Test-Head.py                                    \
+    Test--https.py                                  \
+    Test--https-crl.py                              \
+    Test-missing-scheme-retval.py                   \
+    Test-pinnedpubkey-der-https.py                  \
+    Test-pinnedpubkey-der-no-check-https.py         \
+    Test-pinnedpubkey-hash-https.py                 \
+    Test-pinnedpubkey-hash-no-check-fail-https.py   \
+    Test-pinnedpubkey-pem-fail-https.py             \
+    Test-pinnedpubkey-pem-https.py                  \
+    Test-hsts.py                                    \
+    Test-O.py                                       \
+    Test-Post.py                                    \
+    Test-504.py                                     \
+    Test--spider-r.py                               \
+    Test--rejected-log.py                           \
+    Test-redirect-crash.py                          \
+    Test-reserved-chars.py                          \
+    Test-condget.py                                 \
+    $(METALINK_TESTS)
+
+endif
+
+EXTRA_DIST = certs conf exc misc server test README $(TESTS)
 
 TEST_EXTENSIONS = .py
 PY_LOG_COMPILER = python3

testenv/README

@@ -97,6 +97,7 @@ Environment Variables:
   the test suite will execute all the tests via this command.
   If it is set to "1", valgrind memcheck is enabled with hard coded options.
   This variable is set by ./configure --enable-valgrind-tests.
+* SSL_TESTS: This must be set to run any https tests.
 
 
 File Structure:
@@ -124,7 +125,7 @@ WgetFile (str name, str contents, str timestamp, dict rules)
 None except name is a mandatory paramter, one may pass only those parameters
 that are required by the File object.
 
-The timestamp string should be a valid Unix Timestamp as defined in RFC xxxx.
+The timestamp string should be in a format: "YYYY-MM-DD HH:MM:SS" in UTC zone.
 The rules object is a dictionary element, with the key as the Rule Name and
 value as the Rule Data. In most cases, the Rule Data is another dictionary.
 
@@ -184,7 +185,8 @@ This section lists the currently supported File Rules and their structure.
 
     * SendHeader    : This list of Headers will be sent in EVERY response to a
     request for the respective file. It follows the same value format as
-    ExpectHeader.
+    ExpectHeader. Additionally you can specify a list of strings as <Header Data>
+    if you want the header repeated with multiple values.
 
     * Response      : The HTTP Response Code to send to a request for this File.
     The value is an Integer that represents a valid HTTP Response Code.

testenv/Test--convert-links--content-on-error.py

@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from misc.wget_file import WgetFile
+
+"""
+    This test ensures that Wget link conversion works also on HTTP error pages.
+"""
+############# File Definitions ###############################################
+a_x_FileContent = """
+<!DOCTYPE html>
+<html lang="en">
+<head>
+	<meta charset="UTF-8">
+	<title></title>
+</head>
+<body>
+	<a href="/b/y.html"></a>
+</body>
+</html>
+"""
+a_x_LocalFileContent = """
+<!DOCTYPE html>
+<html lang="en">
+<head>
+	<meta charset="UTF-8">
+	<title></title>
+</head>
+<body>
+	<a href="../b/y.html"></a>
+</body>
+</html>
+"""
+
+error_FileContent = '404 page content'
+error_FileRules = {
+    'Response': 404 ,
+    'SendHeader': {
+        'Content-Length': len(error_FileContent),
+        'Content-Type': 'text/plain',
+    }
+}
+
+a_x_File = WgetFile ("a/x.html", a_x_FileContent)
+robots_File = WgetFile ("robots.txt", '')
+error_File = WgetFile ("b/y.html", error_FileContent, rules=error_FileRules)
+
+B_File = WgetFile ("a/x.html", a_x_LocalFileContent)
+
+WGET_OPTIONS = "--no-host-directories -r -l2 --convert-links --content-on-error"
+WGET_URLS = [["a/x.html"]]
+
+Files = [[a_x_File, robots_File, error_File]]
+
+ExpectedReturnCode = 8
+ExpectedDownloadedFiles = [B_File, robots_File, error_File]
+
+################ Pre and Post Test Hooks #####################################
+pre_test = {
+    "ServerFiles"       : Files
+}
+test_options = {
+    "WgetCommands"      : WGET_OPTIONS,
+    "Urls"              : WGET_URLS
+}
+post_test = {
+    "ExpectedFiles"     : ExpectedDownloadedFiles,
+    "ExpectedRetcode"   : ExpectedReturnCode
+}
+
+err = HTTPTest (
+                pre_hook=pre_test,
+                test_params=test_options,
+                post_hook=post_test
+).begin ()
+
+exit (err)

testenv/Test--https-crl.py

@@ -8,7 +8,9 @@ import os
 """
     This test ensures that Wget can download files from HTTPS Servers
 """
-TEST_NAME = "HTTPS CRL"
+if os.getenv('SSL_TESTS') is None:
+    exit (77)
+
 ############# File Definitions ###############################################
 File1 = "Would you like some Tea?"
 File2 = "With lemon or cream?"
@@ -40,7 +42,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test,

testenv/Test--https.py

@@ -8,7 +8,9 @@ import os
 """
     This test ensures that Wget can download files from HTTPS Servers
 """
-TEST_NAME = "HTTPS Downloads"
+if os.getenv('SSL_TESTS') is None:
+    exit (77)
+
 ############# File Definitions ###############################################
 File1 = "Would you like some Tea?"
 File2 = "With lemon or cream?"
@@ -45,7 +47,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test,

testenv/Test--rejected-log.py

@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+from sys import exit
+from test.http_test import HTTPTest
+from misc.wget_file import WgetFile
+
+"""
+    This test executed Wget in recursive mode with a rejected log outputted.
+"""
+############# File Definitions ###############################################
+mainpage = """
+<html>
+<head>
+  <title>Main Page</title>
+</head>
+<body>
+  <p>
+    Recurse to a <a href="http://127.0.0.1:{{port}}/secondpage.html">second page</a>.
+  </p>
+</body>
+</html>
+"""
+
+secondpage = """
+<html>
+<head>
+  <title>Second Page</title>
+</head>
+<body>
+  <p>
+    Recurse to a <a href="http://127.0.0.1:{{port}}/thirdpage.html">third page</a>.
+    Try the blacklisted <a href="http://127.0.0.1:{{port}}/index.html">main page</a>.
+  </p>
+</body>
+</html>
+"""
+
+thirdpage = """
+<html>
+<head>
+  <title>Third Page</title>
+</head>
+<body>
+  <p>
+    Try a hidden <a href="http://127.0.0.1:{{port}}/dummy.txt">dummy file</a>.
+    Try to leave to <a href="http://no.such.domain/">another domain</a>.
+  </p>
+</body>
+</html>
+"""
+
+robots = """
+User-agent: *
+Disallow: /dummy.txt
+"""
+
+log = """\
+REASON\tU_URL\tU_SCHEME\tU_HOST\tU_PORT\tU_PATH\tU_PARAMS\tU_QUERY\tU_FRAGMENT\tP_URL\tP_SCHEME\tP_HOST\tP_PORT\tP_PATH\tP_PARAMS\tP_QUERY\tP_FRAGMENT
+BLACKLIST\thttp%3A//127.0.0.1%3A{{port}}/index.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tindex.html\t\t\t\thttp%3A//127.0.0.1%3A{{port}}/secondpage.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tsecondpage.html\t\t\t
+ROBOTS\thttp%3A//127.0.0.1%3A{{port}}/dummy.txt\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tdummy.txt\t\t\t\thttp%3A//127.0.0.1%3A{{port}}/thirdpage.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tthirdpage.html\t\t\t
+SPANNEDHOST\thttp%3A//no.such.domain/\tSCHEME_HTTP\tno.such.domain\t80\t\t\t\t\thttp%3A//127.0.0.1%3A{{port}}/thirdpage.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tthirdpage.html\t\t\t
+"""
+
+dummyfile = "Don't care."
+
+
+index_html = WgetFile ("index.html", mainpage)
+secondpage_html = WgetFile ("secondpage.html", secondpage)
+thirdpage_html = WgetFile ("thirdpage.html", thirdpage)
+robots_txt = WgetFile ("robots.txt", robots)
+dummy_txt = WgetFile ("dummy.txt", dummyfile)
+log_csv = WgetFile ("log.csv", log)
+
+WGET_OPTIONS = "-nd -r --rejected-log log.csv"
+WGET_URLS = [["index.html"]]
+
+Files = [[index_html, secondpage_html, thirdpage_html, robots_txt, dummy_txt]]
+
+ExpectedReturnCode = 0
+ExpectedDownloadedFiles = [index_html, secondpage_html, thirdpage_html, robots_txt, log_csv]
+
+################ Pre and Post Test Hooks #####################################
+pre_test = {
+    "ServerFiles"       : Files
+}
+test_options = {
+    "WgetCommands"      : WGET_OPTIONS,
+    "Urls"              : WGET_URLS
+}
+post_test = {
+    "ExpectedFiles"     : ExpectedDownloadedFiles,
+    "ExpectedRetcode"   : ExpectedReturnCode
+}
+
+err = HTTPTest (
+                pre_hook=pre_test,
+                test_params=test_options,
+                post_hook=post_test
+).begin ()
+
+exit (err)

testenv/Test--spider-r.py

@@ -6,7 +6,6 @@ from misc.wget_file import WgetFile
 """
     This test executed Wget in Spider mode with recursive retrieval.
 """
-TEST_NAME = "Recursive Spider"
 ############# File Definitions ###############################################
 mainpage = """
 <html>
@@ -97,7 +96,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-504.py

@@ -18,7 +18,6 @@ from misc.wget_file import WgetFile
     considered 504 as a general Server Error, it would be a fatal failure and
     Wget would request File1 only once.
 """
-TEST_NAME = "504 Gateway Timeouts"
 ############# File Definitions ###############################################
 File1 = """All happy families are alike;
 Each unhappy family is unhappy in its own way"""
@@ -63,7 +62,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test

testenv/Test-Content-disposition-2.py

@@ -7,7 +7,6 @@ from misc.wget_file import WgetFile
     This test ensures that Wget parses the Content-Disposition header
     correctly and creates the appropriate file when the said filename exists.
 """
-TEST_NAME = "Content Disposition Clobber"
 ############# File Definitions ###############################################
 File1 = "Teapot"
 File2 = "The Teapot Protocol"
@@ -45,7 +44,6 @@ post_test = {
 }
 
 err = HTTPTest (
-                name=TEST_NAME,
                 pre_hook=pre_test,
                 test_params=test_options,
                 post_hook=post_test