* src/url.c (url_skip_credentials): Properly re-implement userinfo parsing (rfc2396)
The reason why the implementation is based on RFC 2396, an outdated standard,
is that the whole file is based on that RFC, and mixing standard here might be
dangerous.
Rather than reading from stdin only once, leave the pipe open until the
other end closes it and keep reading from the file after each set of
URLs is read
* src/html-url.h(get_urls_file): Update prototype to add additional
param
* src/html-url.c(get_urls_file): Pass through read_again to
wget_read_from_file.
* src/retr.c(retrieve_from_file): Split the function into two. Introduce
`retrieve_from_url_list` that actually performs the retrieval.
Also, if `url_list` returns that the fd has been left open, then
continue reading from it until the fd is closed.
(retrieve_from_url_list): New function that does the retrieval from
a list of URLs that was read from a file.
* src/utils.c(wget_read_from_file): Rename old function `wget_read_file`
to this.
Accept an additional output parameter that states whether the fd was
left open and if we should continue reading from it after the current
set of URLs have been processed
(wget_read_file): Write it as a new wrapper function around
`wget_read_from_file` to maintain API comptability across other users
The definition of debug_logprintf in src/log.c is guarded by ENABLE_DEBUG
(although its prototype is unconditionally available in src/log.h).
The uses of debug_logprintf in src/retr.c aren't guarded by ENABLE_DEBUG.
Use the DEBUGP macro which is designed for this purpose.
* src/retr.c (getproxy): Use DEBUGP macro.
Fixes: https://gitlab.com/gnuwget/wget/-/issues/19
Copyright-paperwork-exempt: Yes
* testenv/Test-recursive-pathmax.py: Add a new testcase. This test tries
to check that Wget allows downloading long filenames as far as allowed
by the OS and filesystem.
* tests/Makefile.am: Remove some tests that are redundant with the
Python testenv
* tests/Test-auth-basic.px: Delete file
* tests/Test-auth-no-challenge.px: Same
* tests/Test-auth-no-challenge-url.px: Same
* tests/Test-auth-retcode.px: Same
* tests/Test-auth-with-content-disposition.px: Same
* tests/Test-k.px: Same
* testenv/Makefile.am: Add two new tests, Test-k.py and Test-https-k.py
* testenv/Test-k.py: New file. Add a test based on tests/Test-k.px
* testenv/Test-https-k.py: New file. Add a new test to ensure that the
protocol of the original host URL is retained when creatign absolute
links.
This test is added as a result of an issue reported on StackExchange:
https://superuser.com/questions/1348940/making-wgets-convert-links-respect-http-vs-https
Add support for libproxy, which is capable to extract desktop
environment proxy configurations from dozens of systems and platforms.
This also enables wget to handle pac/wpad proxy server.
* configure.ac: Add check for libbproxy.
* src/retr.c (getproxy): Retrieve proxy via libproxy.
Copyright-paperwork-exempt: Yes
If the download rate is TB/s, a read buffer overflow happended
that either caused a crash or printed whatever string was pointed to.
* src/retr.c (retr_rate): Add missing array entrie for TB/s and Tb/s,
(test_retr_rate): New test function.
* tests/unit-tests.c (all_tests): Run test 'test_retr_rate'.
* tests/unit-tests.h: Add prototype for test_retr_rate.
Reported-by: Wiebe Cazemier <wiebe@halfgaar.net>
* testenv/server/http/http_server.py (HTTPSServer): Update for
ssl.SSLContext APIs instead of deprecated ssl.wrap_socket().
ssl.wrap_socket() was deprecated in 3.7 and removed in 3.12.
This should be compatible back to 3.6 (RHEL 8 and newer).
Copyright-paperwork-exempt: Yes
* src/convert.c(convert_links): Print the actual quoted newname when printing DEBUG output
(local_quote_string): Also quote the ' ' charcter as %20. While it is okay
to leave the characted as-is, quoting it covers more edge cases.
And it should resolve a >10 year old bug with CSS url() parameters not being quoted
Bug-Id: 64082
Reported-By: Ethan Gibbs <ethan@snowsign.net>
Discussed-At: https://stackoverflow.com/q/13300017
url() parameters in CSS cannot have spaces in them. Ensure that Wget does not do that
when using --convert-links
* testenv/test_css_url.py: New file
* testenv/Makefile: Add test_css_url.py to tests
Bug-Id: 64082
* doc/wget.texi: Update option '--html-extension' to '--adjust-extension'.
Renamed in Wget 1.12 to better reflect its behavior.
Copyright-paperwork-exempt: Yes
* doc/wget.texi: s/time stamp/timestamp/
* src/ftp-ls.c(clean_line): Same
(ftp_parse_vms_ls): Don't cast return value of x*alloc
* src/ftp.c: Don't cast return value of alloca
* src/css-url.c: wget.h should be the first include
* testenv/Makefile.am: Use $(PATH_SEPARATOR) instead of : when declaring
a path
* testenv/Test-no_proxy-env.py: s/can not/cannot/
* configure.ac: Replace IRI_IS_ENABLED with WITH_IRI and
METALINK_IS_ENABLED with WITH_METALINK
* src/Makefile.am: Add the .c and .h files for IRI, Metalink and
Xattr support to wget_SOURCES directly when needed instead of
adding them as object files.
* testenv/Makefile.am: Rename METALINK_IS_ENABLED to WITH_METALINK
AC_LIBOBJ is to be used for providing replacement functions for
compatibility reasons. Not for conditional compilation of entire source
files. Autotools provides other methods for achieving that. However, our
build system has (ab-)used the AC_LIBOBJ macros for doing this for over
15 years. That stops today.
* configure.ac: Replace uses of AC_LIBOBJ with automake conditionals
that can be used in Makefiles
* src/Makefile.am: Use the defined conditionals to select which
files get built.
* configure.ac: Use AM_GNU_GETTEXT_REQUIRE_VERSION to specify a minimum
version of gettext needed. This should allow newer versions of gettext
to be copied for autipoint to use when available.
This commit hopefully fixes the annoying issue when building from
source on a system that has a newer version of gettext.
When continuing a FTP download, or not starting one because the file is
already fully retrieved, don't include the size of the file in the
total_downloaded_bytes. Only the actual amount of data retrieved over
the network should be considered there.
Fixes: #61277
Reported-By: Michal Ruprich <formaiko>
This reverts commit affad27664.
Manual tests with very long path names did not work with this patch.
We have to wait for a patch including automated tests.
This solves an issue where the server expects the Host: header
as first one. This seems plausible (ahem) as the Host: header is the
only one that is required.
* fuzz/fuzzer.h: Ignore -Wunused-parameter.
* fuzz/wget_options_fuzzer.c: Let getaddrinfo() fail in while fuzzing.
* fuzz/wget_options_fuzzer.in/*: Update corpora from OSS-Fuzz.
* src/html-url.c (get_urls_html_fm): Remove misleading debug message.
* src/recur.c (retrieve_tree): Fix no-follow -> follow in DEBUGP.
The attribute in html is "nofollow" so it is more consistent to call it
so than to hyphenate it.
Copyright-paperwork-exempt: Yes
* src/ftp.c (ftp_retrieve_list): change quote to quote_n
* src/iri.c (do_conversion): change quote to quote_n
* src/url.c (convert_fname): change quote to quote_n
The implementation quote() reuses the buffer it returns which
leads to printing the same string for each quote() call in one
output line. Instead, quote_n() should be used as highlighted in
the doc:
https://www.gnu.org/software/gnulib/manual/html_node/Quoting.html
Copyright-paperwork-exempt: Yes
This prevents needing a special case in the fuzz tests to detect
valgrind usage and a new exec. Instead, we simply detect the environment
in a shell script and start the test with valgrind in the first place.
* fuzz/test-runner.sh: New script for running the fuzz tests
* fuzz/main.c (main): Remove code for re-running under valgrind
* fuzz/Makefile.am: Set the LOG_COMPILER to test-runner.sh and remember
to add it to the list of extra files for packaging
* src/ftp.c(getftp): Don't free `target`. If it is not pointing to
targetbuf, then it still pointing to its original location of u->dir.
This location will be free'd later. Doing so now causes a double free
and hence crashes Wget
* tests/Test-ftp-dir.px: New test to show double free error
* tests/Makefile.am: Add new test
* src/init.c (cmd_bytes_sum): Use WGINT_MIN and WGINT_MAX in check.
* src/options.h (struct options): Make 'quota' of type wgint.
* src/retr.c: Make 'total_downloaded_bytes' of type wgint.
* src/utils.h: Fix comment.
* src/wget.h: Add WGINT_MIN, remove SUM_SIZE_INT.
The maintainer makefile provides targets for making a release. They need
these things:
* cfg.mk: Set the hash of the NEWS file as it was at the beginning of
this release
* .prev-version: Create the .prev-version file
* src/utils.c: Use PRId64 to correctly identify the format specifier for
wgint values. This fixes a warning on 32-bit systems where wgint is a
long long int instead of the long int that the format specifier
indicated.
Reported-by: Jeffrey Walton
time_t on VMS is typically unsigned. (Lazy man's solution to 2038?)
I added "(time_t)" type casts to negative values ("-1"), and changed
tests to avoid complaints.
* src/hsts.c (hsts_add_entry): Explicitly cast potentially negative time
values to time_t to handle VMS quirks.
(hsts_store_entry): Same
(get_hsts_store_filename): Use new ajoin_dir_file function to join
filenames
(test_hsts_read_database): Same
* src/init.c (struct options): use-askpass is not implemented on VMS
(ajoin_dir_file): New Function to join filenames in a platform
agnostic manner
(wgetrc_user_file_name): Use ajoin_dir_file to join paths. Doing this
correctly, eliminates the need for a special case on VMS
* src/init.h: Add prototype for ajoin_dir_file
* src/log.c (check_redirect_output): Ignore on VMS
* src/main.c(option_data): Disable use-askpass on VMS
(print_help): Same
(get_hsts_database): Use ajoin_dir_file to join paths
(print_version): Add VMS specific information to Version output
* src/utils.c (fork_to_background): Fix signature on VMS
Co-authored-by: Darshit Shah <darnir@gnu.org>
* src/http.c (test_parse_range_header): Remove use of SIZEOF_WGINT.
* src/utils.c (human_readable): Remove superfluous HR_NUMTYPE,
* (number_to_string): Remove use of SIZEOF_WGINT.
* src/utils.h: Remove use of SIZEOF_WGINT and HR_NUMTYPE.
* src/wget.h: Remove #define SIZEOF_WGINT.
Gnulib's stdint.h module promises a C99 compliant stdint.h file on all
platforms. Thus allowing us to directly use the fixed wodth integer
type, int64_t wihout needing to resort to all the checks being
performed.
* src/wget.h: Assume that int64_t is always available and use it
* src/mswindows.h: Remove portability code since gnulib handles it
* configure.ac: Remove sizeof checks for integer types that are no
longer used
testenv/test/base_test.py: Add new variable SKIP_TEST
testenv/Test--https-crl.py: Use SKIP_TEST instead of magic number
testenv/Test--https.py: Same
testenv/Test-hsts.py: Same
testenv/Test-no_proxy-env.py: Same
testenv/Test-pinnedpubkey-der-https.py: Same
testenv/Test-pinnedpubkey-der-no-check-https.py: Same
testenv/Test-pinnedpubkey-hash-https.py: Same
testenv/Test-pinnedpubkey-hash-no-check-fail-https.py: Same
testenv/Test-pinnedpubkey-pem-fail-https.py: Same
testenv/Test-pinnedpubkey-pem-https.py: Same
m4/exitfail.m4: Remove unneeded file
m4/wchar.m4: Same
m4/wctype.m4: Same
The latest versions of these files will be copied by gnulib-tool upon
its invokation. No need to track them around here
Use the --po-domain option to gnulib-tool to create a new textdomain
that can be used by gnulib files for translations. This way, we don't
have to maintain the list of all files that require translations in
gnulib.
* bootstrap.conf: Use --po-domain and --po-base options to create a
separate base for gnulib translations
* src/main.c(i18n_initialize): Call bindtextdomain on wget-gnulib to
include those translations as well
* Makefile.am: Add new directory gnulib_po to SUBDIRS
* configure.ac: Generate gnulib_po/Makefile.in
* lib/Makefile.am: Set AM_CPPFLAGS to empty since gnulib.mk expects it
to be set
This silences the wget_options_fuzzer which triggered #28610 on
OSS-Fuzz. This issue can not happen with the Wget utility.
The fuzzer runs main(),...,cleanup() in a loop which the Wget utility
never does.
* testenv/Test-no_proxy-env.py: Added new test for no_proxy env
* testenv/Makefile.am: Added the new test to Makefile
Added new test with 5 cases, which are testing various combinations
of no_proxy environment variable definition and requested URLs.
The test is skipped if the system does not support resolution of
localhost subdomains to lcalhost address.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
* testenv/README: Added description for new EnvironmentVariable hook
* testenv/conf/environment_variable.py: Added implementation of
EnvironmentVariable hook
* testenv/test/base_test.py: Modified exec_wget() to enable use of
EnvironmentVariable hook
Added new test hook called EnvironmentVariables, for defining environment
variables when wget is executed in tests. This is handy for testing
environment variables, which are accepted by wget.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
* testenv/test/http_test.py: Ensure that HTTPTest.begin() always returns
a value
Previously the HTTPTest.begin() method always returned None. However this
is not consistent with the begin() implementation of the parent class
(BaseTest). This change ensures that HTTPTest.begin() returns a value.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
There is a bug that causes wget to exit with SIGABRT when trying to
receive files through FTP from a server with a certificate that failed
the verification.
The bug is filed in RedHat Bugzilla for Fedora:
https://bugzilla.redhat.com/show_bug.cgi?id=1475861
Reported-by: Artem Egorenkov <aegorenk@redhat.com>
* connect.c (fd_read, fd_peek): Let implementation take care about timeout.
* gnutls.c (_do_handshake, _do_reauth, wgnutls_read_timeout): Fix support for interactive timeout.
* gnutls.c (wgnutls_peek): Let wgnutls_read_timeout() take care about timeout.
* openssl.c (openssl_read_peek): Fix 0 (-1) timeout.
* retr.c (fd_read_body): Avoid wrong 'interactive timeout'.
Add new implementation for openssl_read, openssl_peek
ssl_connect_with_timeout (openssl). Thats allow continue
read after ETIMEDOUT (if timeout < opt.read_timeout)
without 'Retrying', and do not create thread (under MSWin)
for every read. Old implementation, (with fix for 'timeout')
avaible for build with: -DOPENSSL_RUN_WITHTIMEOUT
Add timeout for transport_implementation {reader,peeker};
* src/init.c (cmd_time): Add check for negative value.
* src/connect.h (transport_implementation {reader,peeker}): Change function prototype.
* src/connect.c (select_fd_nb): New function, aviod conversion to blocked under MSWin.
* src/gnutls.c
(gnutls_read): Fix: using timeout, seting ETIMEDOUT, conversion to blocked.
(gnutls_peek): Likewise.
(wgnutls_errstr): Add errmsg for ETIMEDOUT.
(_do_handshake) (_do_reauth): Fix conversion to blocked.
* src/openssl.c:
Add new implementation for openssl_read, openssl_peek, ssl_connect_with_timeout.
(init_prng): when option --random-file given warn user if RAND_load_file() fail.
* src/gnutls.c (ssl_init): Use LOG_VERBOSE verbosity for informative
message related to loading CRL or CA certificate file.
Before change [1], wget didn't produce any output related to loading CA
certificates when --no-verbose option has been used. When --no-verbose
option is used, only error messages and basic information should get
printed. Information about loading CRL or CA certificate is probably not
a basic information. Any error when loading the CRL or CA certificate
will be still printed with --no-verbose.
Some users rely on wget not printing such information and they consider
it a regression.
Reported as https://bugzilla.redhat.com/show_bug.cgi?id=1807267
[1] http://git.savannah.gnu.org/cgit/wget.git/commit/?id=e4a8fe84e2b813b65d91aec29298eecabe4850a5
Signed-off-by: Tomas Hozza <thozza@redhat.com>
* configure.ac: Add AX_CODE_COVERAGE.
* Makefile.am: Add rules 'check-coverage' and 'fuzz-coverage'.
* fuzz/Makefile.am: Amend LDADD, AM_CFLAGS and AM_CPPFLAGS.
* src/Makefile.am: Amend LDADD, AM_CFLAGS and AM_CPPFLAGS.
This add a new configure option, --enable-code-coverage.
With this option enabled, generate test code coverage with
make code-coverage
and generate coverage for the fuzz test corpora with
make fuzz-coverage
Following the "privacy by design" principle, which is now European law by
virtue of the GDPR since 2018, the client should not transmit the operating
system.
Fixes#57884
Reported-by: Bruno Haible
* src/http.c (http_loop): overwrite 'restval' only if 'len' is
greater than it. Else substract 'len' to 'restval'.
When retrying a request, only send a 'Range' header if the previous
request returned data.
* src/progress.c (create_image): Likewise
The note was not even picked by xgettext to be included in the .pot,
so barely any translators even saw the note.
Also, this is one of the most cryptic messages to translate in wget,
unless the translator happens to guess where the string is used,
or looks at the source.
Copyright-paperwork-exempt: Yes
* src/wget.h: Replace XDIGIT_TO_NUM by _unhex()
* src/html-parse.c (decode_entity): Use _unhex()
* src/host.c (is_valid_ipv6_address): Use _unhex()
* src/url.c (url_unescape_1): Use unsigned char to avoid UB
* src/res.c (free_specs): Likewise
Code taken from Wget2. XDIGIT_TO_NUM had a signed issue, detected by
scan-build static anylyzer.
* testenv/README: Document $WGET_PATH
* testenv/test/base_test.py: Use $WGET_PATH instead ../src/wget if set
Previously tests in testenv/ directory were run only on wget binary
which was built from sources in src/ directory. However as a
wget maintainer in a Linux distribution, I would like to be able to run
upstream tests on the wget binary distributed with the distribution.
This change enables one to define WGET_PATH environment variable
to a path to wget binary which should be used by tests.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
of the header field while RejectHeader rejects ONLY the specified full header.
Since we wanted to be sure a header field is not sent to the server we wrote this rule.
* doc/wget.texi: Added --disable-header documentation.
* fuzz/wget_options_fuzzer.dict: Update with --disable-header inputs.
* src/http.c (disabled_header): Checks for disabled headers
(request_set_header): Doesn't let header to be set if disabled
(gethttp): frees disabled header to let overriding
* src/init.c (cmd_dis_header), (check_user_disabled_header) added new option disabled_headers.
* src/main.c: added new option --disable-header, added help description
* src/options.h: added new option --disable-header
* src/utils.h (vec_remove_header)
* src/utils.c (vec_remove_header) removes all header instances from vector
* testenv/Makefile.am: Added new test files
* testenv/server/http/http_server.py: Added new rule RejectHeaderField
* testenv/conf/reject_header_field.py: Added new rule RejectHeaderField
* testenv/README: Added help description for new rule
* testenv/Test-disable-default-headers.py: Test without using --header
* testenv/Test-disable-headers-after.py: Test using --header before --disable-header
* testenv/Test-disable-headers-before.py: Test using --header after --disable-header
Signed-off-by: sulfastor <torresmoisesa@gmail.com>, adham elkarn <adhamelkarn@hotmail.com>
* .gitignore: Exclude /lib/Makefile.am
* bootstrap.conf: Use --makefile-name=gnulib.mk with gnulib-tool
* configure.ac: Use wget_MANYWARNINGS()
* lib/Makefile.am: Define empty noinst_LIBRARIES and MAINTAINERCLEANFILES
* m4/wget_manywarnings.m4: New file from GNU Wget2
* src/Makefile.am: Use WARN_CFLAGS for AM_CFLAGS
ssl_init fails with openssl 1.1.1 when openssl.cnf is not found.
Redundant calls to intialization functions were removed as
OPENSSL_config takes care of them for openssl versions < 1.1.0.
For versions > 1.1.0, OPENSSL_init_ssl is preferred.
Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
Copyright-paperwork-exempt: Yes
* configure.ac: Search for dlopen and add library to $FUZZ_LIBS
* fuzz/Makefile.am: Link with $FUZZ_LIBS instead of -ldl
This fixes linking on BSD systems.
Reported-by: Nam Nguyen
Also the Referer info is reduced to scheme+host+port.
* src/ftp.c (getftp): Change params of set_file_metadata()
* src/http.c (gethttp): Change params of set_file_metadata()
* src/xattr.c (set_file_metadata): Remove user/password from origin URL,
reduce Referer value to scheme/host/port.
* src/xattr.h: Change prototype of set_file_metadata()
* src/init.c (defaults): Set enable_xattr to false by default
* src/main.c (print_help): Reverse option logic of --xattr
* doc/wget.texi: Add description for --xattr
Users may not be aware that the origin URL and Referer are saved
including credentials, and possibly access tokens within
the urls.
This reverts commit 6f3b995993.
The code is obviously wrong, see https://savannah.gnu.org/bugs/?54963
Also, the example from the original post doesn't work any more.
With other words, the broken server behavior has been fixed meanwhile.
* src/openssl.c: Check for OPENSSL_NO_ENGINE before
including openssl/engine.h and before calling ENGINE_load_builtin_engines()
Fixes compilation with no engines compiled.
Copyright-paperwork-exempt: Yes
Signed-off-by: Rosen Penev <rosenp@gmail.com>
* tests/SSLTest.pm: Rename server cert and key file
* tests/Test-https*.px: Fix and remove OpenSSL hard-coding
* tests/certs/create-certs.sh: Script to generate test files
* tests/certs/*-template.txt: GnuTLS template files for certs and crl
* tests/certs/*.pem: Keys, certs, crls
* tests/certs/README: Removed commands, link to create-certs.sh
Current sufmatch does not match when domain is dot-prefixed.
The example of no_proxy in man (.mit.edu) does use a dot-prefixed
domain.
Signed-off-by: Luiz Angelo Daros de Luca <luizluca@gmail.com>
Copyright-paperwork-exempt: Yes
* configure.ac: Check for libpcre2-8
* src/init.c (choices): Test for HAVE_LIBPCRE2
* src/main.c (main): Set regex compile and match functions
* src/options.h: Test for HAVE_LIBPCRE2
* src/utils.c: Include pcre2.h, add functions
compile_pcre2_regex() and match_pcre2_regex()
* src/utils.h: Declare compile_pcre2_regex() and match_pcre2_regex()
Fixes#54677
Reported-by: Noël Köthe
* doc/wget.texi: Add "TLSv1_3" to --secure-protocol
* src/gnutls.c (set_prio_default): Use GNUTLS_TLS1_3 where needed
Wget currently allows specifying "TLSv1_3" as the parameter for
--secure-protocol option. However it is only implemented for OpenSSL
and in case wget is compiled with GnuTLS, it causes wget to abort with:
GnuTLS: unimplemented 'secure-protocol' option value 6
GnuTLS contains TLS 1.3 implementation since version 3.6.3 [1]. However
currently it must be enabled explicitly in the application of it to be
used. This will change after the draft is finalized. [2] However for
the time being, I enabled it explicitly in case "TLSv1_3" is used with
--secure-protocol.
I also fixed man page to contain "TLSv1_3" in all listings of available
parameters for --secure-protocol
[1] https://lists.gnupg.org/pipermail/gnutls-devel/2018-July/008584.html
[2] https://nikmav.blogspot.com/2018/05/gnutls-and-tls-13.html
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Error: RESOURCE_LEAK (CWE-772): - REAL ERROR
wget-1.19.5/src/warc.c:1376: alloc_fn: Storage is returned from allocation function "url_escape".
wget-1.19.5/src/url.c:284:3: alloc_fn: Storage is returned from allocation function "url_escape_1".
wget-1.19.5/src/url.c:255:3: alloc_fn: Storage is returned from allocation function "xmalloc".
wget-1.19.5/lib/xmalloc.c:41:11: alloc_fn: Storage is returned from allocation function "malloc".
wget-1.19.5/lib/xmalloc.c:41:11: var_assign: Assigning: "p" = "malloc(n)".
wget-1.19.5/lib/xmalloc.c:44:3: return_alloc: Returning allocated memory "p".
wget-1.19.5/src/url.c:255:3: var_assign: Assigning: "newstr" = "xmalloc(newlen + 1)".
wget-1.19.5/src/url.c:258:3: var_assign: Assigning: "p2" = "newstr".
wget-1.19.5/src/url.c:275:3: return_alloc: Returning allocated memory "newstr".
wget-1.19.5/src/url.c:284:3: return_alloc_fn: Directly returning storage allocated by "url_escape_1".
wget-1.19.5/src/warc.c:1376: var_assign: Assigning: "redirect_location" = storage returned from "url_escape(redirect_location)".
wget-1.19.5/src/warc.c:1381: noescape: Resource "redirect_location" is not freed or pointed-to in "fprintf".
wget-1.19.5/src/warc.c:1387: leaked_storage: Returning without freeing "redirect_location" leaks the storage that it points to.
\# 1385| fflush (warc_current_cdx_file);
\# 1386|
\# 1387|-> return true;
\# 1388| }
\# 1389|
url_escape() really returns a newly allocated memory and it leaks when the warc_write_cdx_record() returns. The memory returned from url_escape() is usually stored in a temporary variable in other parts of the project and then freed. I took the same approach.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
In warc_write_start_record() function, the reutrn value of dup() is
directly used in gzdopen() call and not stored anywhere. However the
zlib documentation says that "The duplicated descriptor should be saved
to avoid a leak, since gzdopen does not close fd if it fails." [1].
This change stores the FD in a variable and closes it in case gzopen()
fails.
[1] https://www.zlib.net/manual.html
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/warc.c:217: open_fn: Returning handle opened by "dup".
wget-1.19.5/src/warc.c:217: leaked_handle: Failing to save or close handle opened by "dup(fileno(warc_current_file))" leaks it.
\# 215|
\# 216| /* Start a new GZIP stream. */
\# 217|-> warc_current_gzfile = gzdopen (dup (fileno (warc_current_file)), "wb9");
\# 218| warc_current_gzfile_uncompressed_size = 0;
\# 219|
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/utils.c:914: open_fn: Returning handle opened by "open". [Note: The source code implementation of the function has been overridden by a user model.]
wget-1.19.5/src/utils.c:914: var_assign: Assigning: "fd" = handle returned from "open(fname, flags, mode)".
wget-1.19.5/src/utils.c:921: noescape: Resource "fd" is not freed or pointed-to in "fstat". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/utils.c:924: leaked_handle: Handle variable "fd" going out of scope leaks the handle.
\# 922| {
\# 923| logprintf (LOG_NOTQUIET, _("Failed to stat file %s, error: %s\n"), fname, strerror(errno));
\# 924|-> return -1;
\# 925| }
\# 926| #if !(defined(WINDOWS) || defined(__VMS))
This seems to be a real issue, since the opened file descriptor in "fd"
would leak. There is also additional check below the "fstat" call, which
closes the opened "fd".
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/http.c:4486: alloc_fn: Storage is returned from allocation function "url_string".
wget-1.19.5/src/url.c:2248:3: alloc_fn: Storage is returned from allocation function "xmalloc".
wget-1.19.5/lib/xmalloc.c:41:11: alloc_fn: Storage is returned from allocation function "malloc".
wget-1.19.5/lib/xmalloc.c:41:11: var_assign: Assigning: "p" = "malloc(n)".
wget-1.19.5/lib/xmalloc.c:44:3: return_alloc: Returning allocated memory "p".
wget-1.19.5/src/url.c:2248:3: var_assign: Assigning: "result" = "xmalloc(size)".
wget-1.19.5/src/url.c:2248:3: var_assign: Assigning: "p" = "result".
wget-1.19.5/src/url.c:2250:3: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2253:7: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2257:11: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2264:3: noescape: Resource "p" is not freed or pointed-to in function "memcpy". [Note: The source code implementation of the function has been overridden by a builtin model.]
wget-1.19.5/src/url.c:2270:7: identity_transfer: Passing "p" as argument 1 to function "number_to_string", which returns an offset off that argument.
wget-1.19.5/src/utils.c:1776:11: var_assign_parm: Assigning: "p" = "buffer".
wget-1.19.5/src/utils.c:1847:3: return_var: Returning "p", which is a copy of a parameter.
wget-1.19.5/src/url.c:2270:7: noescape: Resource "p" is not freed or pointed-to in function "number_to_string".
wget-1.19.5/src/utils.c:1774:25: noescape: "number_to_string(char *, wgint)" does not free or save its parameter "buffer".
wget-1.19.5/src/url.c:2270:7: var_assign: Assigning: "p" = "number_to_string(p, url->port)".
wget-1.19.5/src/url.c:2273:3: noescape: Resource "p" is not freed or pointed-to in function "full_path_write".
wget-1.19.5/src/url.c:1078:47: noescape: "full_path_write(struct url const *, char *)" does not free or save its parameter "where".
wget-1.19.5/src/url.c:2287:3: return_alloc: Returning allocated memory "result".
wget-1.19.5/src/http.c:4486: var_assign: Assigning: "hurl" = storage returned from "url_string(u, URL_AUTH_HIDE_PASSWD)".
wget-1.19.5/src/http.c:4487: noescape: Resource "hurl" is not freed or pointed-to in "logprintf".
wget-1.19.5/src/http.c:4513: leaked_storage: Variable "hurl" going out of scope leaks the storage it points to.
\# 4511| {
\# 4512| printwhat (count, opt.ntry);
\# 4513|-> continue;
\# 4514| }
\# 4515| else
There are two conditional branches, which call continue, without freeing memory potentially allocated and pointed to by"hurl" pointer. In fase "!opt.verbose" is True and some of the appropriate conditions in the following if/else if construction, in which "continue" is called, are also true, then the memory allocated to "hurl" will leak.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/http.c:2434: alloc_fn: Storage is returned from allocation function "xmalloc".
wget-1.19.5/lib/xmalloc.c:41:11: alloc_fn: Storage is returned from allocation function "malloc".
wget-1.19.5/lib/xmalloc.c:41:11: var_assign: Assigning: "p" = "malloc(n)".
wget-1.19.5/lib/xmalloc.c:44:3: return_alloc: Returning allocated memory "p".
wget-1.19.5/src/http.c:2434: var_assign: Assigning: "auth_stat" = storage returned from "xmalloc(4UL)".
wget-1.19.5/src/http.c:2446: noescape: Resource "auth_stat" is not freed or pointed-to in "create_authorization_line".
wget-1.19.5/src/http.c:5203:70: noescape: "create_authorization_line(char const *, char const *, char const *, char const *, char const *, _Bool *, uerr_t *)" does not free or save its parameter "auth_err".
wget-1.19.5/src/http.c:2476: leaked_storage: Variable "auth_stat" going out of scope leaks the storage it points to.
\# 2474| /* Creating the Authorization header went wrong */
\# 2475| }
\# 2476|-> }
\# 2477| else
\# 2478| {
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/http.c:2431: alloc_fn: Storage is returned from allocation function "url_full_path".
wget-1.19.5/src/url.c:1105:19: alloc_fn: Storage is returned from allocation function "xmalloc".
wget-1.19.5/lib/xmalloc.c:41:11: alloc_fn: Storage is returned from allocation function "malloc".
wget-1.19.5/lib/xmalloc.c:41:11: var_assign: Assigning: "p" = "malloc(n)".
wget-1.19.5/lib/xmalloc.c:44:3: return_alloc: Returning allocated memory "p".
wget-1.19.5/src/url.c:1105:19: var_assign: Assigning: "full_path" = "xmalloc(length + 1)".
wget-1.19.5/src/url.c:1107:3: noescape: Resource "full_path" is not freed or pointed-to in function "full_path_write".
wget-1.19.5/src/url.c:1078:47: noescape: "full_path_write(struct url const *, char *)" does not free or save its parameter "where".
wget-1.19.5/src/url.c:1110:3: return_alloc: Returning allocated memory "full_path".
wget-1.19.5/src/http.c:2431: var_assign: Assigning: "pth" = storage returned from "url_full_path(u)".
wget-1.19.5/src/http.c:2446: noescape: Resource "pth" is not freed or pointed-to in "create_authorization_line".
wget-1.19.5/src/http.c:5203:40: noescape: "create_authorization_line(char const *, char const *, char const *, char const *, char const *, _Bool *, uerr_t *)" does not free or save its parameter "path".
wget-1.19.5/src/http.c:2476: leaked_storage: Variable "pth" going out of scope leaks the storage it points to.
\# 2474| /* Creating the Authorization header went wrong */
\# 2475| }
\# 2476|-> }
\# 2477| else
\# 2478| {
Both "pth" and "auth_stat" are allocated in "check_auth()" function. These are used for creating the HTTP Authorization Request header via "create_authorization_line()" function. In case the creation went OK (auth_err == RETROK), then the memory previously allocated to "pth" and "auth_stat" is freed. However if the creation failed, then the memory is never freed and it leaks.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Error: RESOURCE_LEAK (CWE-772):
wget-1.19.5/src/ftp.c:1493: alloc_fn: Storage is returned from allocation function "fopen".
wget-1.19.5/src/ftp.c:1493: var_assign: Assigning: "fp" = storage returned from "fopen(con->target, "wb")".
wget-1.19.5/src/ftp.c:1811: leaked_storage: Variable "fp" going out of scope leaks the storage it points to.
\# 1809| if (fp && !output_stream)
\# 1810| fclose (fp);
\# 1811|-> return err;
\# 1812| }
\# 1813|
It can happen, that "if (!output_stream || con->cmd & DO_LIST)" on line #1398 can be true, even though "output_stream != NULL". In this case a new file is opened to "fp". Later it may happen in the FTPS branch, that some error will occure and code will jump to label "exit_error". In "exit_error", the "fp" is closed only if "output_stream == NULL". However this may not be true as described earlier and "fp" leaks.
On line #1588, there is the following conditional free of "fp":
/* Close the local file. */
if (!output_stream || con->cmd & DO_LIST)
fclose (fp);
Therefore the conditional at the end of the function after "exit_error" label should be modified to:
if (fp && (!output_stream || con->cmd & DO_LIST))
fclose (fp);
This will ensure that "fp" does not leak in any case it sould be opened.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
In Fedora, we are implementing crypto policies, in order to enhance the
security of user systems. This is done on the system level by global
configuration. It may happen that due to the active policy, only
TLSv1.2 or higher will be available in crypto libraries. While wget as
a client will by default determine the minimal TLS version supported by
both client and server, the HTTPS server implementation in testenv/
hardcodes use of TLSv1. As a result all HTTPS related tests fail in
case a more hardened crypto policy is set on the Fedora system.
This change removes the explicit TLS version setting and leaves the
determination of the minimal supported TLS version on the server and
client.
More information about Fedora change can be found here:
https://fedoraproject.org/wiki/Changes/StrongCryptoSettings
Signed-off-by: Tomas Hozza <thozza@redhat.com>
* src/http.c (resp_new): Replace \r\n by space in continuation lines
Fixes#53763
"Malicious website can write arbitrary cookie entries to cookie jar"
HTTP header parsing left the \r\n from continuation line intact.
The Set-Cookie code didn't check and could be tricked to write
\r\n into the cookie jar, allowing a server to generate cookies at will.
* tests/Test-https-badcerts.px: Fix test return value
* tests/Test-https-crl.px: Likewise
* README: How to create certs with GnuTLS's certtool
* tests/certs/revokedcrl.pem: Recreated revocation
* tests/certs/server.crt: Recreated server cert with no expiry
* tests/certs/test-ca-cert.pem: Recreated CA cert with no expiry
* src/openssl.c (init_prng): keep gathering entropy even though we
already have enough
(ssl_connect_with_timeout_callback): reseed PRNG again just before
the handshake
Reported-by: Jeffrey Walton <noloader@gmail.com>
This commit hardens SSL/TLS a bit more in the following ways:
* Explicitly exclude NULL authentication and the 'MEDIUM' cipher list
category. Ciphers in the 'HIGH' level are only considered - this
includes all symmetric ciphers with key lengths larger than 128 bits,
and some ('modern') 128-bit ciphers, such as AES in GCM mode.
* Allow RSA key exchange by default, but exclude it when
Perfect Forward Secrecy is desired (with --secure-protocol=PFS).
* Introduce new option --ciphers to set the cipher list that the SSL/TLS
engine will favor. This string is fed directly to the underlying TLS
library (GnuTLS or OpenSSL) without further processing, and hence its
format and syntax are directly dependent on the specific library.
Reported-by: Jeffrey Walton <noloader@gmail.com>
* src/css-url.c (get_uri_string): Check input length
* fuzz/wget_css_fuzzer.repro/buffer-overflow-6600180399865856:
Add reproducer corpus
Fixes OSS-Fuzz issue #8033.
This is a long standing bug affecting all versions <= 1.19.4.
* src/css-url.c (get_urls_css): Check input string length
* fuzz/wget_css_fuzzer.repro/negative-size-param-5724866467594240:
Add reproducer corpus
Fixes OSS-Fuzz issue #8032.
This is a long standing bug affecting all versions <= 1.19.4.
* fuzz/Makefile.am: Do not include corpora in tarball
* fuzz/main.c: SKIP if corpora directory isn't found (make check)
The fuzz corpora are thousands of files, not needed for a standard build
from a distribution tarball. The reproducers of former issues are being
included for regression testing.
* src/css-tokens.h: Add enums and fixate values
* src/css.l: Include config.h,
ignore several compiler warnings,
update the grammar to CSS 2.2
Fixes OSS-Fuzz issue #8010 (slowness issue).
This is a long standing bug affecting all versions <= 1.19.4.
Some crafted CSS input was extremely slow / CPU wasting, so it could
be used as a DOS attack against website scanning.
The code/grammar changes were backported from Wget2.x.
* fuzz/Makefile.am: Add wget_ftpls_fuzzer
* fuzz/wget_ftpls_fuzzer.c: New fuzzer
* fuzz/wget_ftpls_fuzzer.dict: Fuzzer dictionary
* fuzz/wget_ftpls_fuzzer.in/starter: Starting corpus
* src/ftp-ls.c: Parsing function take FILE * as argument,
new function ftp_parse_ls_fp()
* src/ftp.c: Remove static from freefileinfo()
* src/ftp.h: Add ftp_parse_ls_fp() and freefileinfo()
* fuzz/Makefile.am: Add wget_html_fuzzer
* fuzz/wget_html_fuzzer.c: New fuzzer
* fuzz/wget_html_fuzzer.dict: HTML dictionary for fuzzing
* fuzz/wget_html_fuzzer.in: Initial corpora
* src/html-url.c: Add new function get_urls_html_fm()
* src/html-url.h: Add ne function get_urls_html_fm()
* src/wget.h: Fix define for fopen_wgetrc()
* fuzz/wget_options_fuzzer.c: Add fopen_wget() and fopen_wgetrc()
* src/utils.c: Use fopen_wgetrc() for config files,
don't read from stdin when fuzzing
* src/wget.h: Define fopen as fopen_wget when fuzzing,
define fopen_wgetrc as fopen when not fuzzing
* src/ftp.c (ftp_loop_internal): Set warc_tmp to NULL after ffclose()
* src/init.c (cleanup): Set output_stream to NULL after fclose()
* src/log.c (log_close): Set global stream vars to NULL after closing
* src/recur.c (retrieve_tree): Set rejectedlog to NULL after closing
* src/warc.c (warc_close): Set stream vars to NULL after closing
* Makefile.am: Add fuzz/ to SUBDIRS
* cfg.mk: Fix 'make syntax-check'
* configure.ac: Add --enable-fuzzing
* fuzz/Makefile.am: New file
* fuzz/README.md: New file
* fuzz/fuzzer.h: New file
* fuzz/get_all_corpora: New file
* fuzz/get_ossfuzz_corpora: New file
* fuzz/glob_crash.c: New file
* fuzz/main.c: New file
* fuzz/run-afl.sh: New file
* fuzz/run-clang.sh: New file
* fuzz/view-coverage.sh: New file
* fuzz/wget_options_fuzzer.c: New file
* fuzz/wget_options_fuzzer.dict: New file
* src/init.c (cleanup): Free more resources
* src/main.c (init_switches): Initialize only once,
(print_usage): Don't print if TESTING is defined
* src/utils.h: Include wget.h
Gzip compression has a number of bugs which need to be ironed out before
we can support it by default. Some of these stem from a misunderstanding
of the HTTP spec, but a lot of them are also due to many web servers not
being compliant with RFC 7231.
With this commit, I am marking GZip compression support as experimental
in GNU Wget pending further investigation and the addition of tests.
* src/init.c (defaults): Switch of compression support by default
* docs/wget.texi: State that compression is experimental
* src/log.c (check_redirect_output): tcgetpgrp can return -1 (ENOTTY),
be sure to check whether a valid controlling terminal exists before
redirecting.
Fixes: #51181
* testenv/server/http/http_server.py: If there are multiple requests in
which the requested range is unsatisfiable, then send a body in the in
the 2nd response onwards
* testenv/Test-416.py: New test to check how Wget handles 416 responses
* http.c(gethttp): In case of a 416 response, try to drain the socket of
any bytes before reusing the connection
Reported-By: Iru Cai <mytbk920423@gmail.com>
* src/http.c(gethttp): When Encoding is gzip, ensure that the
Content-Type Header was actually seen. Without this, the "type" variable
is null causing a Segfault.
Reported-By: Noël Köthe <noel@debian.org>
* testenv/conf/expected_files.py: Ignore dirmngr.conf, gpg.conf
Libgpgme creates certain files if $HOME doesn't contain the .gnupg directory.
These files disturb some metalink tests if we don't ignore them.
Reported-by: Arkadiusz Miśkiewicz <arekm@maven.pl>
* src/retr.c (fd_read_body): Stop processing on negative chunk size
Reported-by: Antti Levomäki, Christian Jalio, Joonas Pihlaja from Forcepoint
Reported-by: Juhani Eronen from Finnish National Cyber Security Centre
* src/http.c (skip_short_body): Return error on negative chunk size
Reported-by: Antti Levomäki, Christian Jalio, Joonas Pihlaja from Forcepoint
Reported-by: Juhani Eronen from Finnish National Cyber Security Centre
* src/http.c (gethttp): Move 304 code before --adjust-extension code
This fixes applying --adjust-extension in combination with 304
HTTP responses. It could lead to .html extensions to arbitrary
files.
Reported-by: anfractuosity
Although internally code uses option for (not) reading .netrc for
credentials, it was not possible to turn this behavior off on command
line. Note that it was possible to turn it off using wgetrc.
Idea for this change came from Bruce Jerrick (bmj001@gmail.com).
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1425097
Signed-off-by: Tomas Hozza <thozza@redhat.com>
There seemed to be a copy&paste error in http.c code, which decides
whether to get credentials from .netrc. In ftp.c "user" and "pass"
variables are char*, while in http.c, these are char**. For this reason
they should be dereferenced when determining if password and user login
is set to some value.
Also since both variables are dereferenced on lines above the changed
code, it does not really make sense to check if they are NULL.
This patch is based on fix from Bruce Jerrick <bmj001@gmail.com>.
Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1425097
Signed-off-by: Tomas Hozza <thozza@redhat.com>
Getting credentials from .netrc has been broken from time to time, thus
adding a test coverage to prevent regressions.
Also added setting of "HOME" environment variable when executing wget,
to make sure LocalFiles like .netrc, which are created just for the
test, are actually used.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
* testenv/certs/README: Amend cert creation extensions
* testenv/certs/ca-cert.pem: Created without OCSP signing purpose
Having the OCSP signing purpose set made newer versions of OpenSSL
fail due to stricter checking. Test version of OpenSSL was 1.1.0e.
* tests/certs/wotca.pem: New file
* tests/Test-https-weboftrust.px: Remove creation of wotca.pem
This change let Test-https-weboftrust.px survive VPATH builds.
* tests/Test-https-badcerts.px : New file
* tests/Test-https-clientcert.px : New file
* tests/Test-https-crl.px : New file
* tests/Test-https-weboftrust.px : New file
* tests/certs/interca.conf : New file
* tests/certs/rootca.conf : New file
* tests/certs/test-ca-key.pem : New file
Added all new SSL / HTTPS tests to make check
Added Test for SSL Web of Trust, accept only if CA chain of trust is intact.
Added a test script for client certificate
Added Test for crlfile option of wget
Added test to make sure that wget doesn't accept expired or invalid certs
Some clean up : 1, Removed cause of warnings from perl & other cosmetic changes
2, Fix make -j 4 check such that it passes all tests
* src/main.c: The --secure-protocol option accepts also values TLSv1_1
and TLSv1_2, as mentioned in the man page. However the help message
doesn't mention these two values. This patch adds TLSv1_1 and TLSv1_2 as
possible values to the help message.
Signed-off-by: Tomas Hozza <thozza@redhat.com>
* tests/Test-https-badcerts.px : New file
* tests/Test-https-clientcert.px : New file
* tests/Test-https-crl.px : New file
* tests/Test-https-weboftrust.px : New file
* tests/certs/interca.conf : New file
* tests/certs/rootca.conf : New file
* tests/certs/test-ca-key.pem : New file
Added all new SSL / HTTPS tests to make check
Added Test for SSL Web of Trust, accept only if CA chain of trust is intact.
Added a test script for client certificate
Added Test for crlfile option of wget
Added test to make sure that wget doesn't accept expired or invalid certs
Some clean up : Removed cause of warnings from perl & other cosmetic changes
* tests/SSLServer.pm: New file
* tests/SSLTest.pm: New file
* tests/Test-https-pfs.px: New file
* tests/Test-https-selfsigned.px: New file
* tests/Test-https-tlsv1.px: New file
* tests/Test-https-tlsv1x.px: New file
* tests/certs/server.crt: New file
* tests/certs/server.key: New file
* tests/certs/test-ca-cert.pem: New file
Added 4 new test scripts all for SSL.
Added base pm for SSL testing.
Added SSL tests for TLSv1, TLSv1_1 and PFS.
Added test for self signed cert : check that it fails without
--no-check-certificate and passes with that flag.
* src/url.c: Check iconv() against 0, not -1
On some libiconv implementations, unknown codepoints become
encoded as ?, e.g. when converting a non-ascii codepoint to ASCII.
This results in ambigious file names which also fails our tests.
* src/connect.c (connect_to_ip): Use xfree() instead of idn2_free()
* src/host.c (lookup_host): Use xfree() instead of idn2_free()
* src/iri.h: Do not include idn2.h
* src/url.c (url_free): Use xfree() instead of idn2_free()
* src/url.h (struct url): Remove 'idn_allocated' from struct
Reported-by: Gisle Vanem
* tests/FTPServer.pm: Escape '{' in RE to fix warnings
* tests/FTPTest.pm: Likewise
* tests/HTTPServer.pm: Likewise
* tests/HTTPTest.pm: Likewise
* tests/Test-proxied-https-auth-keepalive.px: Likewise
* tests/Test-proxied-https-auth.px: Likewise
Escape '{' in RE to fix warnings:
Unescaped left brace in regex is deprecated, passed through in regex;
marked by <-- HERE in m/{{ <-- HERE port}}/
* src/utils.h: Add struct file_stat_s declaration,
change prototypes of file_exists_p(),
add prototypes for fopen_stat() and open_stat().
* src/utils.c: Extend file_exists_p(),
new function fopen_stat() and open_stat(),
add new param for file_exists_p().
* src/init.h: Add param file_stats_t to run_wgetrc().
* src/ftp.c: Amend calls to extended functions.
* src/hsts.c: Likewise.
* src/http.c: Likewise.
* src/init.c: Likewise.
* src/main.c: Likewise.
* src/metalink.c: Likewise.
* src/retr.c: Likewise.
* src/url.c: Likewise.
Added fopen_stat() and open_stat() that checks to makes sure the file didn't
change underneath us.
Return error from file_exists_p().
Added a way to return error from this file without major surgery to the
callers.
Fixes: #20369
* src/iri.c: Check for libidn2 < 0.14 to include libunistring headers
The unistring functions are used only when an older version of libidn2
is used, so don't include its headers either w/newer libdin2 versions.
The WARC spec requires that all URIs be enclosed in angle brackets. This
was being done in most cases, but not for "WARC-Target-URI" fields in
WARC blocks of type "response", "resource", "revisit", and "metadata".
* src/http.c (gethttp): Move 504 handling to correct place.
(http_loop): Fix memeory leak.
* testenv/server/http/http_server.py: Add Content-Length header on non-2xx
status codes with a body
Reported-by: Adam Sampson
In a non-ASCII environment, the local path may contain non-ASCII
characters. The server responded file name must be converted before
it is concatenated to the local path. Conversion after concatenation
may result in 'iconv' errors.
* doc/wget.texi: Add description for --retry-on-http-error
* src/http.c (gethttp):
Consider given HTTP response codes as non-fatal, transient errors.
Supply a comma-separated list of 3-digit HTTP response codes as
argument. Useful to work around special circumstances where retries
are required, but the server responds with an error code normally not
retried by Wget. Such errors might be 503 (Service Unavailable) and
429 (Too Many Requests). Retries enabled by this option are performed
subject to the normal retry timing and retry count limitations of
Wget.
Using this option is intended to support special use cases only and is
generally not recommended, as it can force retries even in cases where
the server is actually trying to decrease its load. Please use it
wisely and only if you know what you are doing.
Example use and a starting point for manual testing:
wget --retry-on-http-error=429,503 http://httpbin.org/status/503
* configure.ac: Remove checking for IRI=no
On certain systems, gnulib falls back to libunistring
for mbtowc() and wcwidth() (used in src/progress.c).
Reported-by: Zhiming Wang
* src/ftp.c (getftp): Fix password/user selection
* src/http.c (initialize_request): Likewise
Before, netrc password won over interactive
--ask-password but now --ask-password wins
after change of program logic
Fixes Issue #48811
* src/connect.c: check that the fd is not bigger than FD_SETSIZE
before using FD_SET. An fd_set cannot hold fds bigger than
FD_SETSIZE, causing out-of-bounds write to a buffer on the stack.
Reported by: Jann Horn <jannh@google.com>
* src/gnutls.c (ssl_connect_wget): Call gnutls_set_default_priority()
for --secure-protocol=auto (default).
The patch fixes a behavior that may have unintended side-effects in
certain gnutls versions. Instead use the default priorities when no
options are given.
Signed-off-by: Nikos Mavrogiannopoulos <nmav@gnutls.org>
* src/http-ntlm.c: Rename base64_{encode,decode}
* src/http.c: Likewise
* src/utils.c: Likewise
* src/utils.h: Likewise
When statically linking with gnutls, we get definition clash error for
base64_encode which is also defined by gnutls.
To prevent definition clash, rename base64_{encode,decode}
Signed-off-by: Rahul Bedarkar <rahul.bedarkar@imgtec.com>
* .travis.yml: Install libidn2-dev instead libidn11-dev.
* bootstrap.conf: Add modules libunistring-optional, unistr/base,
unicase/tolower.
* configure.ac: Check for libidn2.
* src/Makefile.am: Add $(LTLIBUNISTRING) to LDADD.
* tests/Makefile.am: Set LDADD similar to LDADD in src/Makefile.am
* src/connect.c: Use libidn2 code instead of libidn.
* src/host.c: Likewise.
* src/iri.c: Likewise.
* src/iri.h: Likewise.
* src/options.h: Likewise.
* src/url.c: Likewise.
* src/url.h: Likewise.
* src/log.c: Fix C99 comment.
IDN2003 should not be used any more due to security concerns.
We use libunistring (resp. the unicode code from gnulib) for
lowercasing UTF-8 before we give data to libidn2.
TR#46 is missing, no support in libidn2 nor in libunistring.
* src/log.c: Use tcgetpgrp(STDIN_FILENO) != getpgrp() to determine when to print to STD* or logfile.
Deprecate log_request_redirect_output function.
Use different file handles for STD* and logfile, to easily switch between them when changing fg/bg.
* src/log.h: Make redirect_output function externally linked.
* src/main.c: Don't use deprecated log_request_redirect_output function. Use redirect_output instead.
* src/mswindows.c: Don't use deprecated log_request_redirect_output function. Use redirect_output instead.
* src/recur.c (descend_redirect): Ignore WG_RR_LIST and WG_RR_REGEX
for redirections.
* testenv/Makefile.am: Add Test-recursive-redirect.py
* testenv/Test-recursive-redirect.py: New test
Test-recursive-redirect.py written by Dale R. Worley.
Reported-by: "Dale R. Worley" <worley@ariadne.com>
* testenv/misc/metalinkv3_xml.py: New Metalink/XML v3 python class
* testenv/Test-metalink-xml-abspath-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-abspath.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-absprefix-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-absprefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-continue.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-emptyprefix-trust.py: Update test to use
the new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homepath-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homepath.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-homeprefix-trust.py: Update test to use
the new Metalink/XML v3 python class
* testenv/Test-metalink-xml-homeprefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-nohash.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-nourls.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-prefix-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-prefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-relpath-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-relpath.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-relprefix-trust.py: Update test to use the
new Metalink/XML v3 python class
* testenv/Test-metalink-xml-relprefix.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-size.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml-trust.py: Update test to use the new
Metalink/XML v3 python class
* testenv/Test-metalink-xml.py: Update test to use the new
Metalink/XML v3 python class
* src/http.c (metalink_from_http): Process the Content-Type header.
Add an application/metalink4+xml URL as metalink metaurl. If the
option opt.content_disposition is true, the Content-Disposition's
filename is the metaurl's name
* doc/wget.texi: Update --content-disposition and --metalink-over-http
* doc/metalink-standard.txt: Update doc. Content-Type/Disposition
processing through --metalink-over-http. Update download naming
system about --trust-server-names and --content-disposition
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-http-xml-type.py: New file. Metalink/HTTP
Content-Type/Disposition header automated Metalink/XML tests
* testenv/Test-metalink-http-xml-type-trust.py: New file. Metalink/HTTP
Content-Type/Disposition header with --trust-server-names automated
Metalink/XML tests
* testenv/Test-metalink-http-xml-type-content.py: New file. Metalink/HTTP
Content-Type/Disposition header with --content-disposition automated
Metalink/XML tests
* testenv/Test-metalink-http-xml-type-trust-content.py: New file.
Metalink/HTTP Content-Type/Disposition header with --trust-server-names
and --content-disposition automated Metalink/XML tests
Process the Content-Type header, identify an application/metalink4+xml
file. The Content-Disposition could provide an alternate name through
the "filename" field for the metalink xml file. Respectively, the cli
options --metalink-over-http and --content-disposition are required.
When Metalink/XML auto-processing, to use the Content-Disposition's
filename, the cli option --trust-server-names is also required.
* src/http.c (http_loop): Prevent SIGSEGV when hstat.local_file is
NULL, opt.content_disposition has a role in leaving the value unset
* src/http.c (gethttp): If hs->local_file is NULL (aka http_loop()'s
hstat.local_file), set it to the value of hs->metalink->origin
* src/metalink.c (retrieve_from_metalink): If opt.trustservernames is
true, use the basename of the metaurl's name to save the xml file
* doc/metalink-standard.txt: Update doc. With --trust-server-names any
Metalink/HTTP Link application/metalink4+xml file is saved using the
basename of the "name" field, if any. Update Metalink/HTTP examples
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-http-xml-trust-name.py: New file. Metalink/HTTP
automated Metalink/XML, save xml files using the "name" field tests
* src/metalink.c (retrieve_from_metalink): Reject any metalink:file
without hashes. Prompt the error and switch to the next file
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-nohash.py: New file. Metalink/XML with no
hashes tests
Prevent SIGSEGV.
* src/http.c (metalink_from_http): Fix hash_bin_len type. Use ssize_t
instead than size_t. Reject -1 as base64_decode() return value
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-http-baddigest.py: New file. Metalink/HTTP
malformed base64 Digest header tests
On malformed base64 input, ssize_t base64_decode() returns -1. Such
value is too big for a size_t variable, and used as xmalloc() value
will exaust all the memory.
* NEWS: Mention the effect of --metalink-index over Metalink
* src/init.c: Add new option metalinkindex (opt.metalink_index),
initialize to -1
* src/main.c: Add new option metalink-index (--metalink-index=NUMBER)
* src/options.h: Add new option metalink_index (int)
* src/metalink.h: Add declaration of functions fetch_metalink_file(),
replace_metalink_basename()
* src/metalink.c: Add functions fetch_metalink_file() simple file
fetch, replace_metalink_basename() replace file basename
* src/metalink.c (retrieve_from_metalink): New. Process Metalink
application/metalink4+xml of opt.metalink_index ordinal number
* doc/wget.texi: Add new option metalink-index (--metalink-index)
documentation
* doc/metalink-standard.txt: Updated doc. Add documentation about
Metalink application/metalink4+xml metaurls download naming system
* doc/metalink-standard.txt: Update Metalink/XML and HTTP examples
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-http-xml.py: New file. Metalink/HTTP automated
Metalink/XML "application/metalink4+xml" --metalink-index tests
* testenv/Test-metalink-http-xml-trust.py: New file. Metalink/HTTP
automated Metalink/XML "application/metalink4+xml" --metalink-index
retrieval with --trust-server-names tests
WARNING: Do not use lib/dirname.c (dir_name) to get the directory
name, it may append a dot '.' character to the directory name.
* src/http.c (metalink_from_http): Parse Metalink/HTTP header for
metaurls application/metalink4+xml media types
* src/metalink.h: Add function declaration metalink_meta_cmp()
* src/metalink.c: Add function metalink_meta_cmp() compare metalink
metaurls priorities
Add Metalink/HTTP application/metalink4+xml media types as metaurls to
the metalink variable that will be used to download the files.
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-emptyprefix-trust.py: New file.
Metalink/XML empty directory prefix (--directory-prefix '') tests
Detect a '/' character wrongfully concatenated to an empty directory
prefix '' (not NULL), resulting in an absolute path as '/dir/file',
instead than 'dir/file'.
* src/metalink.h: Add declaration of function dequote_metalink_string()
* src/metalink.c: Add function dequote_metalink_string() remove
surrounding quotes from string, \' or \"
* src/metalink.c (find_key_value, find_key_values): Call dequote_metalink_string()
to remove the surrounding quotes from the parsed value
* src/metalink.c (test_find_key_value, test_find_key_values): Add
quoted key's values for unit-tests
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-http-quoted.py: New file. Metalink/HTTP quoted
values tests
Some Metalink/HTTP keys, like "type" [2], may have a quoted value [1]:
Link: <http://example.com/example.ext.meta4>; rel=describedby;
type="application/metalink4+xml"
Wget was expecting a dequoted value from the Metalink module. This
patch addresses this problem.
References:
[1] Metalink/HTTP: Mirrors and Hashes
1.1. Example Metalink Server Response
https://tools.ietf.org/html/rfc6249#section-1.1
[2] Additional Link Relations
6. "type"
https://tools.ietf.org/html/rfc6903#section-6
* src/metalink.h: Add declaration of function clean_metalink_string()
* src/metalink.c: Add directive #include "xmemdup0.h"
* src/metalink.c: Add function clean_metalink_string() remove leading
and trailing white spaces and CRLF from string
* src/metalink.c (retrieve_from_metalink): Remove leading and trailing
white spaces and CRLF from url resource mres->url
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-urlbreak.py: New test. Metalink/XML white
spaces and CRLF in url resources tests
White spaces and CRLF are not automatically removed by libmetalink
from url strings. The Wget's Metalink module was unable to process
such url strings. This patch implements the processing of such url
strings cleaning off leading and trailing white spaces and CRLF.
If a parsed Metalink/XML url string contains strings separated by
CRLF, only the first of the series is accepted.
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-nourls.py: New file. Metalink/XML unknown
urls tests
Test if when there are no good Metalink url resources there is any
segmentation fault.
* src/wget.h (uerr_t): Add error code METALINK_SIZE_ERROR to enum
* src/metalink.c (retrieve_from_metalink): Use boolean variable
size_ok, when false set retr_err to METALINK_SIZE_ERROR
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-size.py: New file. Metalink/XML file size
tests (<size></size>)
Before this patch, no appropriate error code was returned to inform a
file size mismatch.
This patch introduces the error code METALINK_SIZE_ERROR to inform a
file size mismatch.
* NEWS: Mention the effect of --trust-server-names over Metalink
* src/metalink.h: Add declaration of function append_suffix_number()
* src/metalink.c: Add function append_suffix_number() append number to
string
* src/metalink.c (retrieve_from_metalink): Safer Metalink/XML and
Metalink/HTTP download naming system, opt.trustservernames based
* doc/metalink-standard.txt: Update doc. Explain new Metalink/XML and
Metalin/HTTP download naming system and --trust-server-names role
* testenv/Makefile.am: Add new files
* testenv/Test-metalink-xml-continue.py: Update test. Metalink/XML
continue/keep existing files (HTTP 416) with --continue tests
* testenv/Test-metalink-xml.py: Update test. Metalink/XML naming tests
* testenv/Test-metalink-xml-trust.py: New file. Metalink/XML naming
tests with --trust-server-names
* testenv/Test-metalink-xml-abspath.py: Update test. Metalink/XML
absolute path tests
* testenv/Test-metalink-xml-abspath-trust.py: New file. Metalink/XML
absolute path tests with --trust-server-names
* testenv/Test-metalink-xml-relpath.py: Update test. Metalink/XML
relative path tests
* testenv/Test-metalink-xml-relpath-trust.py: New file. Metalink/XML
relative path tests with --trust-server-names
* testenv/Test-metalink-xml-homepath.py: Update test. Metalink/XML
home path and ~ (tilde) tests
* testenv/Test-metalink-xml-homepath-trust.py: New file. Metalink/XML
home path and ~ (tilde) tests with --trust-server-names
* testenv/Test-metalink-xml-prefix.py: New file. Metalink/XML naming
tests with --directory-prefix
* testenv/Test-metalink-xml-prefix-trust.py: New file. Metalink/XML
naming tests with --directory-prefix and --trust-server-names
* testenv/Test-metalink-xml-absprefix.py: New file. Metalink/XML
absolute --directory-prefix tests
* testenv/Test-metalink-xml-absprefix-trust.py: New file. Metalink/XML
absolute --directory-prefix tests with --trust-server-names
* testenv/Test-metalink-xml-relprefix.py: New file. Metalink/XML
relative --directory-prefix tests
* testenv/Test-metalink-xml-relprefix-trust.py: New file. Metalink/XML
relative --directory-prefix tests with --trust-server-names
* testenv/Test-metalink-xml-homeprefix.py: New file. Metalink/XML home
--directory-prefix tests
* testenv/Test-metalink-xml-homeprefix-trust.py: New file. Metalink/XML
home --directory-prefix tests with --trust-server-names
The option --trust-server-names allows to use the file names parsed
from a Metalink/XML file. Without --trust-server-names, the safety
mechanism provides secure and predictable file names.
* NEWS: Mention the use of a safe Metalink destination path
* src/metalink.h: Add declaration of functions get_metalink_basename(),
last_component(), metalink_check_safe_path()
* src/metalink.c: Add directive #include "dosname.h"
* src/metalink.c: Add function get_metalink_basename() to return the
basename of a file name, strip w32's drive letter prefixes
* src/metalink.c (retrieve_from_metalink): Enforce Metalink file name
verification, if the file name is unsafe try its basename
* doc/metalink.txt: Update document. Explain --directory-prefix
The function get_metalink_basename() uses FILE_SYSTEM_PREFIX_LEN to
catch any 'C:D:file' (w32 environment), then it removes each drive
letter prefix, i.e. 'C:' and 'D:'.
Unsafe file names contain an absolute, relative, or home path. Safe
paths can be verified by libmetalink's metalink_check_safe_path().
* NEWS: Mention the effect of --directory-prefix over Metalink
* src/metalink.c (retrieve_from_metalink): Add opt.dir_prefix as
prefix to the metalink:file name mfile->name
* doc/metalink.txt: Update document. Explain --directory-prefix
When --directory-prefix=<prefix> is used, set the top of the retrieval
tree to prefix. The default is . (the current directory). Metalink/XML
and Metalink/HTTP files will be downloaded under prefix.
* src/metalink.c (retrieve_from_metalink): Change mfile->name to
filename when referring to the downloaded file
The file name could have been changed by unique_create() (or by any
other mean) before downloading. Use the name of the downloaded file
(filename) when printing output which refer to it.
* NEWS: Mention Metalink's file size verification
* src/metalink.c (retrieve_from_metalink): Add file size computation
* doc/metalink.txt: Update document. Remove resolved bugs
Reject downloaded files when they do not agree with their Metalink/XML
metalink:size: https://tools.ietf.org/html/rfc5854#section-4.2.14
At the moment of writing, Metalink/HTTP headers do not provide a file
size field. This information could be obtained from the Content-Length
header field: https://tools.ietf.org/html/rfc6249#section-7
* testenv/Test-metalink-xml-relpath.py: Update test
* testenv/Test-metalink-xml-homepath.py: New file. Reject home paths
* testenv/Makefile.am: Add new file
When --input-metalink=<file> is used, each metalink:file name is
verified by libmetalink's metalink_check_safe_path(). By design,
absolute, relative and home paths are rejected.
At the moment of writing, when --metalink-over-http is used, absolute,
relative, and home paths aren't a concern. The destination file name
is a combination of URL's file name and cli's "Directory Options"
handled by src/url.c (url_file_name).
* NEWS: Mention the effects of --continue over Metalink
* src/metalink.c (retrieve_from_metalink): On download error, resume
output_stream with the next mres->url. Keep fully downloaded files
started with --continue, otherwise rename/remove the file
* testenv/Makefile.am: Add new file
* testenv/Test-metalink-xml-continue.py: New file. Metalink/XML
continue/keep existing files (HTTP 416) with --continue tests
Before this patch, with --continue, existing and/or fully retrieved
files which fail the sanity tests were renamed (--keep-badhash), or
removed.
This patch ensures that --continue doesn't rename/remove existing
and/or fully retrieved files (HTTP 416) which fail the sanity tests.
* NEWS: Mention the Metalink "path/file" name format handling
* src/metalink.c (retrieve_from_metalink): Fix NULL filename, set
filename to the right "path/file" value
* src/metalink.c (retrieve_from_metalink): Fix NULL output_stream, set
output_stream to filename when it is created by retrieve_url()
* src/metalink.c (retrieve_from_metalink): Add RFC5854 comments about
proper metalink:file "path/file" name format handling
* doc/metalink.txt: Update document. Remove resolved bugs
If unique_create() cannot create/open the destination file, filename
and output_stream remain NULL. If fopen() is used instead, filename
always remains NULL. Both functions cannot create "path/file" trees.
Setting filename to the right value is sufficient to prevent SIGSEGV
generating from testing a NULL value. This also allows retrieve_url()
to create a "path/file" tree through opt.output_document.
Reading NULL as output_stream, when it shall not be, leads to wrong
results. For instance, a non-NULL output_stream tells when a stream
was interrupted, reading NULL instead means to assume the contrary.
This patch conforms to the RFC5854 specification:
The Metalink Download Description Format
4.1.2.1. The "name" Attribute
https://tools.ietf.org/html/rfc5854#section-4.1.2.1
* doc/metalink.txt
Evaluation of "Directory Options" on the command line interacting with
the option '--input-metalink=file':
$ wget --input-metalink=file <directory options>
* testenv/Test-metalink-http.py: Use python .replace
* testenv/Test-metalink-xml.py: Use python .replace
* testenv/Test-metalink-xml-abspath.py: Use python .replace
* testenv/Test-metalink-xml-relpath.py: Use python .replace
Use python .replace instead than re.sub, remove 'import re'.
* testenv/Test-metalink-xml-abspath.py: Change Metalink/XML v3 file
name from test.meta4 into test.metalink
* testenv/Test-metalink-xml-relpath.py: Change Metalink/XML v3 file
name from test.meta4 into test.metalink
* testenv/Test-metalink-xml.py: Change Metalink/XML v3 file name from
test.meta4 into test.metalink
* src/html-url.c (tag_handle_img): Check append_url() for NULL
return value before dereference.
Crashed reproducable with parsing srcset="data:..." inline data.
Reported-by: Coverity
* src/http.c: Add const to first param of initialize_request(),
initialize_proxy_configuration(), establish_connection(),
check_file_output(), check_auth(), gethttp(), http_loop().
* src/http.h: Add const to first param of http_loop().
* src/connect.c (connect_to_ip): Check return value of setsockopt.
* src/ftp.c (ftp_retrieve_list): Check return value of chmod.
* src/http.c (digest_authentication_encode): Cleanup code.
* src/init.c (setval_internal): Explicitely check comind range.
* src/main.c (main): Explicitely check optarg.
* src/retr.c (retr_rate): Use snprintf instead sprintf,
(retrieve_from_file): More verbose error message,
(rotate_backups): Use snprintf instead sprintf, check return
value of rename().
* src/url.c (mkalldirs): Check return value of unlink().
* src/utils.c (strdupdelim): Explicitely check beg and end for NULL,
(merge_vecs): Fix sizeof argument to char *,
(stable_sort): Use malloc instead of alloca.
* bootstrap.conf: Add xmemdup0 and strpbrk.
* src/init.c (cmd_use_askpass): Add 'const' to char *,
remove check for file existence.
* src/main.c (run_use_askpass): C89 compat init of argv,
added \n to error messages,
fixed stripping of \n and \r from input,
make run_use_askpass and use_askpass static.
* doc/wget.texi: Add --use-askpass to documentation.
* src/init.c: Add cmd_use_askpasss to set opt.use_askpass based on
argument, WGET_ASKPASS, and SSH_ASKPASS environment variables.
opt.wget-askpass is freed in cleanup ()
* src/main.c: Update options & add spawn process of opt.use_askpass
command.
* src/options.h: Addition of string use_askpass.
* src/url.c: Function scheme_leading_string to access the leading
string of a parsed url.
* src/url.h: Prototype for scheme_leading_string for returning the
leading string.
* bootstrap.conf: Add posix_spawn to gnulib_modules
This adds the --use-askpass option which is disabled by default.
--use-askpass=COMMAND will request the username and password for a given
URL by executing the external program COMMAND. If COMMAND is left
blank, then the external program in the environment variable
WGET_ASKPASS will be used. If WGET_ASKPASS is not set then the
environment variable SSH_ASKPASS is used. If there is no value set, an
error is returned. If an error occurs requesting the username or
password, wget will exit.
Signed-off-by: Liam R. Howlett <Liam.Howlett@WindRiver.com>
* testenv/Test-recursive-basic.py: New file. Test basic recursion
* testenv/Test-recursive-include.py: New File. Recursion test with
include directories
* testenv/Test-redirect.py: New File. Basic redirection tests
* testenv/Makefile.am: Add new tests to makefile
* configure.ac: Add --with-psl-file to set a PSL file
* src/cookies.c (check_domain_match): Load PSL_FILE with
fallback to built-in data.
This change allows package maintainers to make Wget use the latest
PSL (DAFSA or plain text), without updating libpsl itself.
E.g. Debian now comes with a DAFSA binary within the 'publicsuffix'
package which allows very fast loading (no parsing or processing needed).
* src/cookies.c (cookie_header): Use heap instead of stack.
* src/http.c (request_send): Likewise.
If wget has to handle an insanely large amount of cookies (~700,000 on
32 bit systems or ~530,000 on 64 bit systems), the stack is not large
enough to hold these pointers, leading to undefined behaviour according
to POSIX; expect a segmentation fault in real life. ;)
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
The signal handler for SIGALRM calls longjmp, but the handler is
installed before the jump target has been initialized. If another
process sends SIGALRM right between handler installation and target
initialization, the jump leads to undefined behavior.
This can easily be fixed by moving the signal handler installation
into the "SETJMP == 0" conditional block, which means that the target
has just been initialized.
* src/utils.c: call signal after SETJMP.
Signed-off-by: Tobias Stoeckmann <tobias@stoeckmann.org>
* src/init.c: Remove hyphens from command names
* src/main.c: Likewise
Options with hyphens (or underscores) in their command name cannot be
set in a wgetrc file.
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
* src/metalink.c (retrieve_from_metalink): Continue file download if
opt.always_rest is true
Without --continue, download as a new file with an unique name (this
conforms to the old behaviour).
* bootstrap.conf: Add crypto/md2, and crypto/md4
* src/metalink.c (retrieve_from_metalink): Add md2, and md4 support
This patch adds support for the deprecated (insecure) md2, and md4
Message-Digest algorithms to the Metalink module.
* bootstrap.conf: Add crypto/sha512
* src/metalink.c (retrieve_from_metalink): Add md5, sha1, sha224,
sha384, and sha512 support
Metalink's checksum verification was limited to sha256. This patch
adds support for md5, sha1, sha224, sha384, and sha512.
* configure.ac: Check for xattr availability
* src/Makefile.am: Add xattr.c
* src/ftp.c: Include xattr.h.
(getftp): Set attributes if enabled.
* src/http.c: Include xattr.h.
(gethttp): Add parameter 'original_url',
set attributes if enabled.
(http_loop): Add 'original_url' to call of gethttp().
* src/init.c: Add new option --xattr.
* src/main.c: Add new option --xattr, add description to help text.
* src/options.h: Add new config member 'enable_xattr'.
* src/xatrr.c: New file.
* src/xattr.h: New file.
These attributes provide a lightweight method of later determining
where a file was downloaded from.
This patch changes:
* autoconf detects whether extended attributes are available and
enables the code if they are.
* The new flags --xattr and --no-xattr control whether xattr is enabled.
* The new command "xattr = (on|off)" can be used in ~/.wgetrc or /etc/wgetrc
* The original and redirected URLs are recorded as shown below.
* This works for both single fetches and recursive mode.
The attributes that are set are:
user.xdg.origin.url: The URL that the content was fetched from.
user.xdg.referrer.url: The URL that was originally requested.
Here is an example, where http://archive.org redirects to https://archive.org:
$ wget --xattr http://archive.org
...
$ getfattr -d index.html
user.xdg.origin.url="https://archive.org/"
user.xdg.referrer.url="http://archive.org/"
These attributes were chosen based on those stored by Google Chrome
https://bugs.chromium.org/p/chromium/issues/detail?id=45903
and curl https://github.com/curl/curl/blob/master/src/tool_xattr.c
* src/openssl.c (ssl_init): Use SSL_is_init_finished() instead of
SSL_state(), conditionally skip SSLeay function calls
The python test suite makes SSL_peek() hang, consuming 100% CPU time.
This does not happen on real world TLS connections, though, but needs
investigations.
* Test-pinnedpubkey-hash-https.py: Read hashed pubkey from file
* Test-pinnedpubkey-hash-no-check-fail-https.py: Use invalid hash
* certs/make_ca.sh: New script to generate test certs non-interactive
* certs/ca-template.cfg: New file (template for CA cert)
* certs/server-template.cfg: New file (template for server cert)
* certs/server-pubkey-sha256.base64: New file (pubkey sha256 hash)
* src/hsts.c (hsts_file_access_valid): we should check for "world-writable"
files only on Unix-based systems. It's difficult to mimic the same behavior
on Windows, so it's better to just not do it.
Reported-by: Gisle Vanem <gvanem@yahoo.no>
Reported-by: Eli Zaretskii <eliz@gnu.org>
If not --trust-server-names is used, FTP will also get the destination
file name from the original url specified by the user instead of the
redirected url. Closes CVE-2016-4971.
* src/ftp.c (ftp_get_listing): Add argument original_url.
(getftp): Likewise.
(ftp_loop_internal): Likewise. Use original_url to generate the
file name if --trust-server-names is not provided.
(ftp_retrieve_glob): Likewise.
(ftp_loop): Likewise.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
* testenv/Makefile.am: Add Test-missing-scheme-retval.py
* testenv: New file Test-missing-scheme-retval.py
Idea and Perl implementation by Zdenek Dohnal <zdohnal@redhat.com>
* src/main.c (save_hsts): save the in-memory HSTS database to a file
only if something changed.
* src/hsts.c (struct hsts_store): new field 'changed'.
(hsts_match): update field 'changed' accordingly.
(hsts_store_entry): update field 'changed' accordingly.
(hsts_store_has_changed): new function.
* src/hsts.h (hsts_store_has_changed): new function.
* hsts.c (hsts_file_access_valid): check that the file is a regular
file, and that it's not world-writable.
(hsts_store_open): if the HSTS database file does not meet the
above requirements, disable HSTS at all.
* src/hsts.c (hsts_store_entry): strictly comply with RFC 6797.
RFC 6797 states in section 8.1 that the UA's cached information should
only be updated if:
"either or both of the max-age and includeSubDomains header field
value tokens are conveying information different than that already
maintained by the UA."
* testenv/Makefile.am: Add new tests
* testenv/Test-pinnedpubkey-der-https.py: New test
* testenv/Test-pinnedpubkey-der-no-check-https.py: New Test
* testenv/Test-pinnedpubkey-hash-https.py: New test
* testenv/Test-pinnedpubkey-hash-no-check-fail-https.py: New test
* testenv/Test-pinnedpubkey-pem-fail-https.py: New test
* testenv/Test-pinnedpubkey-pem-https.py: New test
* testenv/certs/README: How to generate public keys with openssl tool
* testenv/certs/server-pubkey.der: New key file (DER format)
* testenv/certs/server-pubkey.pem: New key file (PEM format)
* testenv/test/base_test.py (__init__): Use test file name for name,
remove 'name' parameter
* testenv/test/http_test.py (__init__): Remove 'name' parameter
* testenv/*.py: Remove TEST_NAME
Using a fixed string (TEST_NAME) to build the working directory
for testing caused random failures (or successes) when tests
share the same TEST_NAME value. Not easy to find without digging
into the python test suite code.
We now use the test file name, which is unique within the test
environment.
* src/metalink.c (retrieve_from_metalink): Fix debug message to print the
fingerprint instead of a pointer.
* Do not delete the ChangeLog file since it is required by the Makefile
and breaks compilation
* wget.texi: Replace server.com by example.com,
replace ftp://wuarchive.wustl.edu by https://example.com,
use HTTPS instead of HTTP where possible,
fix list archive reference,
remove reference to wget-notify@addictivecode.org,
change bugtracker URL to bugtracker on Savannah,
replace yoyodyne.com by example.com,
fix URL to VMS port
* README.checkout: Add description for libares
* configure.ac: Add check for libares
* doc/wget.texi: Add docs for the new options
* src/build_info.c.in: Add +/-cares for --version output
* src/host.c:
(merge_address_lists): New static function
(address_list_from_hostent): New static function
(wait_ares): New static function
(callback): New static function
(lookup_host): Add libares resolver code
* src/init.c: Add new options,
(cleanup): Add cleanup code
* src/main.c: Add global libares channel variable
(cmdline_option option_data): Add new options
(print_help): Add short descriptions
(main): Add libares init code
* src/options.h (struct options): Add option members
The new options allow to specify alternative DNS servers and
an alternate packet route for the resolver packets.
Wget has to built with libares, enabled at configure time by
./configure --with-cares.
* src/gnutls.c (ssl_connect_wget, ssl_check_certificate): Fix SNI server name
* src/openssl.c (ssl_connect_wget, ssl_check_certificate): Fix SNI server name
Fixes#47408
* src/url.c [HAVE_ICONV]: Include iconv.h and langinfo.h.
(convert_fname): New function.
[HAVE_ICONV]: Convert file name from remote encoding to local
encoding.
(url_file_name): Call convert_fname.
(filechr_table): Don't consider bytes in 128..159 as control
characters.
* tests/Test-ftp-iri.px: Fix the expected file name to match the
new file-name recoding. State the remote encoding explicitly on
the Wget command line.
* NEWS: Mention the URI recoding when built with libiconv.
* src/iri.c: Kick out the last converted character from iconv()
Thanks to Eli Zaretskii <eliz@gnu.org> for suggesting the fix.
Reported-by: "Andries E. Brouwer" <Andries.Brouwer@cwi.nl>
[AS_HELP_STRING([--enable-fsanitize-asan], [Turn on Address Sanitizer (for developers) (mutually exclusive with Memory/Thread sanitizer or Valgrind tests)])],
[AS_HELP_STRING([--enable-fsanitize-msan], [Turn on Memory Sanitizer (for developers) (mutually exclusive with Address/Thread sanitizer or Valgrind tests)])],
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
